1# Go support for Protocol Buffers 2 3[![GoDev](https://img.shields.io/static/v1?label=godev&message=reference&color=00add8)](https://pkg.go.dev/mod/google.golang.org/protobuf) 4[![Build Status](https://travis-ci.org/protocolbuffers/protobuf-go.svg?branch=master)](https://travis-ci.org/protocolbuffers/protobuf-go) 5 6This project hosts the Go implementation for 7[protocol buffers](https://developers.google.com/protocol-buffers), which is a 8language-neutral, platform-neutral, extensible mechanism for serializing 9structured data. The protocol buffer language is a language for specifying the 10schema for structured data. This schema is compiled into language specific 11bindings. This project provides both a tool to generate Go code for the 12protocol buffer language, and also the runtime implementation to handle 13serialization of messages in Go. See the 14[protocol buffer developer guide](https://developers.google.com/protocol-buffers/docs/overview) 15for more information about protocol buffers themselves. 16 17This project is comprised of two components: 18 19* Code generator: The 20 [`protoc-gen-go`](https://pkg.go.dev/google.golang.org/protobuf/cmd/protoc-gen-go) 21 tool is a compiler plugin to `protoc`, the protocol buffer compiler. It 22 augments the `protoc` compiler so that it knows how to 23 [generate Go specific code for a given `.proto` file](https://developers.google.com/protocol-buffers/docs/reference/go-generated). 24 25* Runtime library: The 26 [`protobuf`](https://pkg.go.dev/mod/google.golang.org/protobuf) module 27 contains a set of Go packages that form the runtime implementation of 28 protobufs in Go. This provides the set of interfaces that 29 [define what a message is](https://pkg.go.dev/google.golang.org/protobuf/reflect/protoreflect) 30 and functionality to serialize message in various formats (e.g., 31 [wire](https://pkg.go.dev/google.golang.org/protobuf/proto), 32 [JSON](https://pkg.go.dev/google.golang.org/protobuf/encoding/protojson), 33 and 34 [text](https://pkg.go.dev/google.golang.org/protobuf/encoding/prototext)). 35 36See the 37[developer guide for protocol buffers in Go](https://developers.google.com/protocol-buffers/docs/gotutorial) 38for a general guide for how to get started using protobufs in Go. 39 40This project is the second major revision of the Go protocol buffer API 41implemented by the 42[`google.golang.org/protobuf`](https://pkg.go.dev/mod/google.golang.org/protobuf) 43module. The first major version is implemented by the 44[`github.com/golang/protobuf`](https://pkg.go.dev/mod/github.com/golang/protobuf) 45module. 46 47## Package index 48 49Summary of the packages provided by this module: 50 51* [`proto`](https://pkg.go.dev/google.golang.org/protobuf/proto): Package 52 `proto` provides functions operating on protobuf messages such as cloning, 53 merging, and checking equality, as well as binary serialization. 54* [`encoding/protojson`](https://pkg.go.dev/google.golang.org/protobuf/encoding/protojson): 55 Package `protojson` serializes protobuf messages as JSON. 56* [`encoding/prototext`](https://pkg.go.dev/google.golang.org/protobuf/encoding/prototext): 57 Package `prototext` serializes protobuf messages as the text format. 58* [`encoding/protowire`](https://pkg.go.dev/google.golang.org/protobuf/encoding/protowire): 59 Package `protowire` parses and formats the low-level raw wire encoding. Most 60 users should use package `proto` to serialize messages in the wire format. 61* [`reflect/protoreflect`](https://pkg.go.dev/google.golang.org/protobuf/reflect/protoreflect): 62 Package `protoreflect` provides interfaces to dynamically manipulate 63 protobuf messages. 64* [`reflect/protoregistry`](https://pkg.go.dev/google.golang.org/protobuf/reflect/protoregistry): 65 Package `protoregistry` provides data structures to register and lookup 66 protobuf descriptor types. 67* [`reflect/protodesc`](https://pkg.go.dev/google.golang.org/protobuf/reflect/protodesc): 68 Package `protodesc` provides functionality for converting 69 `descriptorpb.FileDescriptorProto` messages to/from the reflective 70 `protoreflect.FileDescriptor`. 71* [`testing/protocmp`](https://pkg.go.dev/google.golang.org/protobuf/testing/protocmp): 72 Package `protocmp` provides protobuf specific options for the `cmp` package. 73* [`testing/protopack`](https://pkg.go.dev/google.golang.org/protobuf/testing/protopack): 74 Package `protopack` aids manual encoding and decoding of the wire format. 75* [`testing/prototest`](https://pkg.go.dev/google.golang.org/protobuf/testing/prototest): 76 Package `prototest` exercises the protobuf reflection implementation for 77 concrete message types. 78* [`types/dynamicpb`](https://pkg.go.dev/google.golang.org/protobuf/types/dynamicpb): 79 Package `dynamicpb` creates protobuf messages at runtime from protobuf 80 descriptors. 81* [`types/known/anypb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/anypb): 82 Package `anypb` is the generated package for `google/protobuf/any.proto`. 83* [`types/known/timestamppb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/timestamppb): 84 Package `timestamppb` is the generated package for 85 `google/protobuf/timestamp.proto`. 86* [`types/known/durationpb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/durationpb): 87 Package `durationpb` is the generated package for 88 `google/protobuf/duration.proto`. 89* [`types/known/wrapperspb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/wrapperspb): 90 Package `wrapperspb` is the generated package for 91 `google/protobuf/wrappers.proto`. 92* [`types/known/structpb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/structpb): 93 Package `structpb` is the generated package for 94 `google/protobuf/struct.proto`. 95* [`types/known/fieldmaskpb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/fieldmaskpb): 96 Package `fieldmaskpb` is the generated package for 97 `google/protobuf/field_mask.proto`. 98* [`types/known/apipb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/apipb): 99 Package `apipb` is the generated package for 100 `google/protobuf/api.proto`. 101* [`types/known/typepb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/typepb): 102 Package `typepb` is the generated package for 103 `google/protobuf/type.proto`. 104* [`types/known/sourcecontextpb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/sourcecontextpb): 105 Package `sourcecontextpb` is the generated package for 106 `google/protobuf/source_context.proto`. 107* [`types/known/emptypb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/emptypb): 108 Package `emptypb` is the generated package for 109 `google/protobuf/empty.proto`. 110* [`types/descriptorpb`](https://pkg.go.dev/google.golang.org/protobuf/types/descriptorpb): 111 Package `descriptorpb` is the generated package for 112 `google/protobuf/descriptor.proto`. 113* [`types/pluginpb`](https://pkg.go.dev/google.golang.org/protobuf/types/pluginpb): 114 Package `pluginpb` is the generated package for 115 `google/protobuf/compiler/plugin.proto`. 116* [`compiler/protogen`](https://pkg.go.dev/google.golang.org/protobuf/compiler/protogen): 117 Package `protogen` provides support for writing protoc plugins. 118* [`cmd/protoc-gen-go`](https://pkg.go.dev/google.golang.org/protobuf/cmd/protoc-gen-go): 119 The `protoc-gen-go` binary is a protoc plugin to generate a Go protocol 120 buffer package. 121 122## Reporting issues 123 124The issue tracker for this project is currently located at 125[golang/protobuf](https://github.com/golang/protobuf/issues). 126 127Please report any issues there with a sufficient description of the bug or 128feature request. Bug reports should ideally be accompanied by a minimal 129reproduction of the issue. Irreproducible bugs are difficult to diagnose and fix 130(and likely to be closed after some period of time). Bug reports must specify 131the version of the 132[Go protocol buffer module](https://github.com/protocolbuffers/protobuf-go/releases) 133and also the version of the 134[protocol buffer toolchain](https://github.com/protocolbuffers/protobuf/releases) 135being used. 136 137## Contributing 138 139This project is open-source and accepts contributions. See the 140[contribution guide](https://github.com/protocolbuffers/protobuf-go/blob/master/CONTRIBUTING.md) 141for more information. 142 143## Compatibility 144 145This module and the generated code are expected to be stable over time. However, 146we reserve the right to make breaking changes without notice for the following 147reasons: 148 149* **Security:** A security issue in the specification or implementation may 150 come to light whose resolution requires breaking compatibility. We reserve 151 the right to address such issues. 152* **Unspecified behavior:** There are some aspects of the protocol buffer 153 specification that are undefined. Programs that depend on unspecified 154 behavior may break in future releases. 155* **Specification changes:** It may become necessary to address an 156 inconsistency, incompleteness, or change in the protocol buffer 157 specification, which may affect the behavior of existing programs. We 158 reserve the right to address such changes. 159* **Bugs:** If a package has a bug that violates correctness, a program 160 depending on the buggy behavior may break if the bug is fixed. We reserve 161 the right to fix such bugs. 162* **Generated additions**: We reserve the right to add new declarations to 163 generated Go packages of `.proto` files. This includes declared constants, 164 variables, functions, types, fields in structs, and methods on types. This 165 may break attempts at injecting additional code on top of what is generated 166 by `protoc-gen-go`. Such practice is not supported by this project. 167* **Internal changes**: We reserve the right to add, modify, and remove 168 internal code, which includes all unexported declarations, the 169 [`protoc-gen-go/internal_gengo`](https://pkg.go.dev/google.golang.org/protobuf/cmd/protoc-gen-go/internal_gengo) 170 package, the 171 [`runtime/protoimpl`](https://pkg.go.dev/google.golang.org/protobuf/runtime/protoimpl?tab=doc) 172 package, and all packages under 173 [`internal`](https://pkg.go.dev/google.golang.org/protobuf/internal). 174 175Any breaking changes outside of these will be announced 6 months in advance to 176[protobuf@googlegroups.com](https://groups.google.com/forum/#!forum/protobuf). 177 178Users should use generated code produced by a version of 179[`protoc-gen-go`](https://pkg.go.dev/google.golang.org/protobuf/cmd/protoc-gen-go) 180that is identical to the runtime version provided by the 181[protobuf module](https://pkg.go.dev/mod/google.golang.org/protobuf). This 182project promises that the runtime remains compatible with code produced by a 183version of the generator that is no older than 1 year from the version of the 184runtime used, according to the release dates of the minor version. Generated 185code is expected to use a runtime version that is at least as new as the 186generator used to produce it. Generated code contains references to 187[`protoimpl.EnforceVersion`](https://pkg.go.dev/google.golang.org/protobuf/runtime/protoimpl?tab=doc#EnforceVersion) 188to statically ensure that the generated code and runtime do not drift 189sufficiently far apart. 190 191## Historical legacy 192 193This project is the second major revision 194([released in 2020](https://blog.golang.org/a-new-go-api-for-protocol-buffers)) 195of the Go protocol buffer API implemented by the 196[`google.golang.org/protobuf`](https://pkg.go.dev/mod/google.golang.org/protobuf) 197module. The first major version 198([released publicly in 2010](https://blog.golang.org/third-party-libraries-goprotobuf-and)) 199is implemented by the 200[`github.com/golang/protobuf`](https://pkg.go.dev/mod/github.com/golang/protobuf) 201module. 202 203The first version predates the release of Go 1 by several years. It has a long 204history as one of the first core pieces of infrastructure software ever written 205in Go. As such, the Go protobuf project was one of many pioneers for determining 206what the Go language should even look like and what would eventually be 207considered good design patterns and “idiomatic” Go (by simultaneously being 208both positive and negative examples of it). 209 210Consider the changing signature of the `proto.Unmarshal` function as an example 211of Go language and library evolution throughout the life of this project: 212 213```go 214// 2007/09/25 - Conception of Go 215 216// 2008/11/12 217export func UnMarshal(r io.Read, pb_e reflect.Empty) *os.Error 218 219// 2008/11/13 220export func UnMarshal(buf *[]byte, pb_e reflect.Empty) *os.Error 221 222// 2008/11/24 223export func UnMarshal(buf *[]byte, pb_e interface{}) *os.Error 224 225// 2008/12/18 226export func UnMarshal(buf []byte, pb_e interface{}) *os.Error 227 228// 2009/01/20 229func UnMarshal(buf []byte, pb_e interface{}) *os.Error 230 231// 2009/04/17 232func UnMarshal(buf []byte, pb_e interface{}) os.Error 233 234// 2009/05/22 235func Unmarshal(buf []byte, pb_e interface{}) os.Error 236 237// 2011/11/03 238func Unmarshal(buf []byte, pb_e interface{}) error 239 240// 2012/03/28 - Release of Go 1 241 242// 2012/06/12 243func Unmarshal(buf []byte, pb Message) error 244``` 245 246These changes demonstrate the difficulty of determining what the right API is 247for any new technology. It takes time multiplied by many users to determine what 248is best; even then, “best” is often still somewhere over the horizon. 249 250The change on June 6th, 2012 added a degree of type-safety to Go protobufs by 251declaring a new interface that all protobuf messages were required to implement: 252 253```go 254type Message interface { 255 Reset() 256 String() string 257 ProtoMessage() 258} 259``` 260 261This interface reduced the set of types that can be passed to `proto.Unmarshal` 262from the universal set of all possible Go types to those with a special 263`ProtoMessage` marker method. The intention of this change is to limit the 264protobuf API to only operate on protobuf data types (i.e., protobuf messages). 265For example, there is no sensible operation if a Go channel were passed to the 266protobuf API as a channel cannot be serialized. The restricted interface would 267prevent that. 268 269This interface does not behaviorally describe what a protobuf message is, but 270acts as a marker with an undocumented expectation that protobuf messages must be 271a Go struct with a specific layout of fields with formatted tags. This 272expectation is not statically enforced by the Go language, for it is an 273implementation detail checked dynamically at runtime using Go reflection. Back 274in 2012, the only types with this marker were those generated by 275`protoc-gen-go`. Since `protoc-gen-go` would always generate messages with the 276proper layout of fields, this was deemed an acceptable and dramatic improvement 277over `interface{}`. 278 279Over the next 10 years, 280[use of Go would skyrocket](https://blog.golang.org/10years) and use of 281protobufs in Go would skyrocket as well. With increased popularity also came 282more diverse usages and requirements for Go protobufs and an increased number of 283custom `proto.Message` implementations that were not generated by 284`protoc-gen-go`. 285 286The increasingly diverse ecosystem of Go types implementing the `proto.Message` 287interface led to incompatibilities, which often occurred when: 288 289* **Passing custom `proto.Message` types to the protobuf APIs**: A concrete 290 message implementation might work with some top-level functions (e.g., 291 `proto.Marshal`), but cause others (e.g., `proto.Equal`) to choke and panic. 292 This occurs because the type only had partial support for being an actual 293 message by only implementing the `proto.Marshaler` interface or having 294 malformed struct field tags that happened to work with one function, but not 295 another. 296 297* **Using Go reflection on any `proto.Message` types**: A common desire is to 298 write general-purpose code that operates on any protobuf message. For 299 example, a microservice might want to populate a `trace_id` field if it is 300 present in a message. To accomplish this, one would use Go reflection to 301 introspect the message type, and assume it were a pointer to a Go struct 302 with a field named `TraceId` (as would be commonly produced by 303 `protoc-gen-go`). If the concrete message type did not match this 304 expectation, it either failed to work or even resulted in a panic. Such was 305 the case for concrete message types that might be backed by a Go map instead 306 of a Go struct. 307 308Both of these issues are solved by following the idiom that _interfaces should 309describe behavior, not data_. This means that the interface itself should 310provide sufficient functionality through its methods that users can introspect 311and interact with all aspects of a protobuf message through a principled API. 312This feature is called _protobuf reflection_. Just as how Go reflection provides 313an API for programmatically interacting with any arbitrary Go value, protobuf 314reflection provides an API for programmatically interacting with any arbitrary 315protobuf message. 316 317Since an interface cannot be extended in a backwards compatible way, this 318suggested the need for a new major version that defines a new `proto.Message` 319interface: 320 321```go 322type Message interface { 323 ProtoReflect() protoreflect.Message 324} 325``` 326 327The new 328[`proto.Message`](https://pkg.go.dev/google.golang.org/protobuf/proto?tab=doc#Message) 329interface contains a single `ProtoReflect` method that returns a 330[`protoreflect.Message`](https://pkg.go.dev/google.golang.org/protobuf/reflect/protoreflect?tab=doc#Message), 331which is a reflective view over a protobuf message. In addition to making a 332breaking change to the `proto.Message` interface, we took this opportunity to 333cleanup the supporting functionality that operate on a `proto.Message`, split up 334complicated functionality apart into manageable packages, and to hide 335implementation details away from the public API. 336 337The goal for this major revision is to improve upon all the benefits of, while 338addressing all the shortcomings of the old API. We hope that it will serve the 339Go ecosystem well for the next 10 years and beyond. 340