• Home
  • History
  • Annotate
Name Date Size #Lines LOC

..03-May-2022-

.github/workflows/H18-Mar-2021-

cmd/protoc-gen-go/H18-Mar-2021-

compiler/protogen/H18-Mar-2021-

encoding/H18-Mar-2021-

internal/H18-Mar-2021-

proto/H18-Mar-2021-

reflect/H18-Mar-2021-

runtime/H18-Mar-2021-

testing/H18-Mar-2021-

types/H18-Mar-2021-

.gitignoreH A D18-Mar-2021306

AUTHORSH A D18-Mar-2021174

CONTRIBUTING.mdH A D18-Mar-20212.6 KiB

CONTRIBUTORSH A D18-Mar-2021171

LICENSEH A D18-Mar-20211.4 KiB

PATENTSH A D18-Mar-20211.3 KiB

README.mdH A D18-Mar-202117.5 KiB

go.modH A D18-Mar-2021123

go.sumH A D18-Mar-2021648

integration_test.goH A D18-Mar-202115.2 KiB

regenerate.bashH A D18-Mar-2021297

release.bashH A D18-Mar-20213.7 KiB

test.bashH A D18-Mar-2021295

README.md

1# Go support for Protocol Buffers
2
3[![Go Reference](https://pkg.go.dev/badge/google.golang.org/protobuf.svg)](https://pkg.go.dev/google.golang.org/protobuf)
4[![Build Status](https://travis-ci.org/protocolbuffers/protobuf-go.svg?branch=master)](https://travis-ci.org/protocolbuffers/protobuf-go)
5
6This project hosts the Go implementation for
7[protocol buffers](https://developers.google.com/protocol-buffers), which is a
8language-neutral, platform-neutral, extensible mechanism for serializing
9structured data. The protocol buffer language is a language for specifying the
10schema for structured data. This schema is compiled into language specific
11bindings. This project provides both a tool to generate Go code for the
12protocol buffer language, and also the runtime implementation to handle
13serialization of messages in Go. See the
14[protocol buffer developer guide](https://developers.google.com/protocol-buffers/docs/overview)
15for more information about protocol buffers themselves.
16
17This project is comprised of two components:
18
19*   Code generator: The
20    [`protoc-gen-go`](https://pkg.go.dev/google.golang.org/protobuf/cmd/protoc-gen-go)
21    tool is a compiler plugin to `protoc`, the protocol buffer compiler. It
22    augments the `protoc` compiler so that it knows how to
23    [generate Go specific code for a given `.proto` file](https://developers.google.com/protocol-buffers/docs/reference/go-generated).
24
25*   Runtime library: The
26    [`protobuf`](https://pkg.go.dev/mod/google.golang.org/protobuf) module
27    contains a set of Go packages that form the runtime implementation of
28    protobufs in Go. This provides the set of interfaces that
29    [define what a message is](https://pkg.go.dev/google.golang.org/protobuf/reflect/protoreflect)
30    and functionality to serialize message in various formats (e.g.,
31    [wire](https://pkg.go.dev/google.golang.org/protobuf/proto),
32    [JSON](https://pkg.go.dev/google.golang.org/protobuf/encoding/protojson),
33    and
34    [text](https://pkg.go.dev/google.golang.org/protobuf/encoding/prototext)).
35
36See the
37[developer guide for protocol buffers in Go](https://developers.google.com/protocol-buffers/docs/gotutorial)
38for a general guide for how to get started using protobufs in Go.
39
40This project is the second major revision of the Go protocol buffer API
41implemented by the
42[`google.golang.org/protobuf`](https://pkg.go.dev/mod/google.golang.org/protobuf)
43module. The first major version is implemented by the
44[`github.com/golang/protobuf`](https://pkg.go.dev/mod/github.com/golang/protobuf)
45module.
46
47## Package index
48
49Summary of the packages provided by this module:
50
51*   [`proto`](https://pkg.go.dev/google.golang.org/protobuf/proto): Package
52    `proto` provides functions operating on protobuf messages such as cloning,
53    merging, and checking equality, as well as binary serialization.
54*   [`encoding/protojson`](https://pkg.go.dev/google.golang.org/protobuf/encoding/protojson):
55    Package `protojson` serializes protobuf messages as JSON.
56*   [`encoding/prototext`](https://pkg.go.dev/google.golang.org/protobuf/encoding/prototext):
57    Package `prototext` serializes protobuf messages as the text format.
58*   [`encoding/protowire`](https://pkg.go.dev/google.golang.org/protobuf/encoding/protowire):
59    Package `protowire` parses and formats the low-level raw wire encoding. Most
60    users should use package `proto` to serialize messages in the wire format.
61*   [`reflect/protoreflect`](https://pkg.go.dev/google.golang.org/protobuf/reflect/protoreflect):
62    Package `protoreflect` provides interfaces to dynamically manipulate
63    protobuf messages.
64*   [`reflect/protoregistry`](https://pkg.go.dev/google.golang.org/protobuf/reflect/protoregistry):
65    Package `protoregistry` provides data structures to register and lookup
66    protobuf descriptor types.
67*   [`reflect/protodesc`](https://pkg.go.dev/google.golang.org/protobuf/reflect/protodesc):
68    Package `protodesc` provides functionality for converting
69    `descriptorpb.FileDescriptorProto` messages to/from the reflective
70    `protoreflect.FileDescriptor`.
71*   [`testing/protocmp`](https://pkg.go.dev/google.golang.org/protobuf/testing/protocmp):
72    Package `protocmp` provides protobuf specific options for the `cmp` package.
73*   [`testing/protopack`](https://pkg.go.dev/google.golang.org/protobuf/testing/protopack):
74    Package `protopack` aids manual encoding and decoding of the wire format.
75*   [`testing/prototest`](https://pkg.go.dev/google.golang.org/protobuf/testing/prototest):
76    Package `prototest` exercises the protobuf reflection implementation for
77    concrete message types.
78*   [`types/dynamicpb`](https://pkg.go.dev/google.golang.org/protobuf/types/dynamicpb):
79    Package `dynamicpb` creates protobuf messages at runtime from protobuf
80    descriptors.
81*   [`types/known/anypb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/anypb):
82    Package `anypb` is the generated package for `google/protobuf/any.proto`.
83*   [`types/known/timestamppb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/timestamppb):
84    Package `timestamppb` is the generated package for
85    `google/protobuf/timestamp.proto`.
86*   [`types/known/durationpb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/durationpb):
87    Package `durationpb` is the generated package for
88    `google/protobuf/duration.proto`.
89*   [`types/known/wrapperspb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/wrapperspb):
90    Package `wrapperspb` is the generated package for
91    `google/protobuf/wrappers.proto`.
92*   [`types/known/structpb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/structpb):
93    Package `structpb` is the generated package for
94    `google/protobuf/struct.proto`.
95*   [`types/known/fieldmaskpb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/fieldmaskpb):
96    Package `fieldmaskpb` is the generated package for
97    `google/protobuf/field_mask.proto`.
98*   [`types/known/apipb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/apipb):
99    Package `apipb` is the generated package for
100    `google/protobuf/api.proto`.
101*   [`types/known/typepb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/typepb):
102    Package `typepb` is the generated package for
103    `google/protobuf/type.proto`.
104*   [`types/known/sourcecontextpb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/sourcecontextpb):
105    Package `sourcecontextpb` is the generated package for
106    `google/protobuf/source_context.proto`.
107*   [`types/known/emptypb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/emptypb):
108    Package `emptypb` is the generated package for
109    `google/protobuf/empty.proto`.
110*   [`types/descriptorpb`](https://pkg.go.dev/google.golang.org/protobuf/types/descriptorpb):
111    Package `descriptorpb` is the generated package for
112    `google/protobuf/descriptor.proto`.
113*   [`types/pluginpb`](https://pkg.go.dev/google.golang.org/protobuf/types/pluginpb):
114    Package `pluginpb` is the generated package for
115    `google/protobuf/compiler/plugin.proto`.
116*   [`compiler/protogen`](https://pkg.go.dev/google.golang.org/protobuf/compiler/protogen):
117    Package `protogen` provides support for writing protoc plugins.
118*   [`cmd/protoc-gen-go`](https://pkg.go.dev/google.golang.org/protobuf/cmd/protoc-gen-go):
119    The `protoc-gen-go` binary is a protoc plugin to generate a Go protocol
120    buffer package.
121
122## Reporting issues
123
124The issue tracker for this project is currently located at
125[golang/protobuf](https://github.com/golang/protobuf/issues).
126
127Please report any issues there with a sufficient description of the bug or
128feature request. Bug reports should ideally be accompanied by a minimal
129reproduction of the issue. Irreproducible bugs are difficult to diagnose and fix
130(and likely to be closed after some period of time). Bug reports must specify
131the version of the
132[Go protocol buffer module](https://github.com/protocolbuffers/protobuf-go/releases)
133and also the version of the
134[protocol buffer toolchain](https://github.com/protocolbuffers/protobuf/releases)
135being used.
136
137## Contributing
138
139This project is open-source and accepts contributions. See the
140[contribution guide](https://github.com/protocolbuffers/protobuf-go/blob/master/CONTRIBUTING.md)
141for more information.
142
143## Compatibility
144
145This module and the generated code are expected to be stable over time. However,
146we reserve the right to make breaking changes without notice for the following
147reasons:
148
149*   **Security:** A security issue in the specification or implementation may
150    come to light whose resolution requires breaking compatibility. We reserve
151    the right to address such issues.
152*   **Unspecified behavior:** There are some aspects of the protocol buffer
153    specification that are undefined. Programs that depend on unspecified
154    behavior may break in future releases.
155*   **Specification changes:** It may become necessary to address an
156    inconsistency, incompleteness, or change in the protocol buffer
157    specification, which may affect the behavior of existing programs. We
158    reserve the right to address such changes.
159*   **Bugs:** If a package has a bug that violates correctness, a program
160    depending on the buggy behavior may break if the bug is fixed. We reserve
161    the right to fix such bugs.
162*   **Generated additions**: We reserve the right to add new declarations to
163    generated Go packages of `.proto` files. This includes declared constants,
164    variables, functions, types, fields in structs, and methods on types. This
165    may break attempts at injecting additional code on top of what is generated
166    by `protoc-gen-go`. Such practice is not supported by this project.
167*   **Internal changes**: We reserve the right to add, modify, and remove
168    internal code, which includes all unexported declarations, the
169    [`protoc-gen-go/internal_gengo`](https://pkg.go.dev/google.golang.org/protobuf/cmd/protoc-gen-go/internal_gengo)
170    package, the
171    [`runtime/protoimpl`](https://pkg.go.dev/google.golang.org/protobuf/runtime/protoimpl?tab=doc)
172    package, and all packages under
173    [`internal`](https://pkg.go.dev/google.golang.org/protobuf/internal).
174
175Any breaking changes outside of these will be announced 6 months in advance to
176[protobuf@googlegroups.com](https://groups.google.com/forum/#!forum/protobuf).
177
178Users should use generated code produced by a version of
179[`protoc-gen-go`](https://pkg.go.dev/google.golang.org/protobuf/cmd/protoc-gen-go)
180that is identical to the runtime version provided by the
181[protobuf module](https://pkg.go.dev/mod/google.golang.org/protobuf). This
182project promises that the runtime remains compatible with code produced by a
183version of the generator that is no older than 1 year from the version of the
184runtime used, according to the release dates of the minor version. Generated
185code is expected to use a runtime version that is at least as new as the
186generator used to produce it. Generated code contains references to
187[`protoimpl.EnforceVersion`](https://pkg.go.dev/google.golang.org/protobuf/runtime/protoimpl?tab=doc#EnforceVersion)
188to statically ensure that the generated code and runtime do not drift
189sufficiently far apart.
190
191## Historical legacy
192
193This project is the second major revision
194([released in 2020](https://blog.golang.org/a-new-go-api-for-protocol-buffers))
195of the Go protocol buffer API implemented by the
196[`google.golang.org/protobuf`](https://pkg.go.dev/mod/google.golang.org/protobuf)
197module. The first major version
198([released publicly in 2010](https://blog.golang.org/third-party-libraries-goprotobuf-and))
199is implemented by the
200[`github.com/golang/protobuf`](https://pkg.go.dev/mod/github.com/golang/protobuf)
201module.
202
203The first version predates the release of Go 1 by several years. It has a long
204history as one of the first core pieces of infrastructure software ever written
205in Go. As such, the Go protobuf project was one of many pioneers for determining
206what the Go language should even look like and what would eventually be
207considered good design patterns and “idiomatic” Go (by simultaneously being
208both positive and negative examples of it).
209
210Consider the changing signature of the `proto.Unmarshal` function as an example
211of Go language and library evolution throughout the life of this project:
212
213```go
214// 2007/09/25 - Conception of Go
215
216// 2008/11/12
217export func UnMarshal(r io.Read, pb_e reflect.Empty) *os.Error
218
219// 2008/11/13
220export func UnMarshal(buf *[]byte, pb_e reflect.Empty) *os.Error
221
222// 2008/11/24
223export func UnMarshal(buf *[]byte, pb_e interface{}) *os.Error
224
225// 2008/12/18
226export func UnMarshal(buf []byte, pb_e interface{}) *os.Error
227
228// 2009/01/20
229func UnMarshal(buf []byte, pb_e interface{}) *os.Error
230
231// 2009/04/17
232func UnMarshal(buf []byte, pb_e interface{}) os.Error
233
234// 2009/05/22
235func Unmarshal(buf []byte, pb_e interface{}) os.Error
236
237// 2011/11/03
238func Unmarshal(buf []byte, pb_e interface{}) error
239
240// 2012/03/28 - Release of Go 1
241
242// 2012/06/12
243func Unmarshal(buf []byte, pb Message) error
244```
245
246These changes demonstrate the difficulty of determining what the right API is
247for any new technology. It takes time multiplied by many users to determine what
248is best; even then, “best” is often still somewhere over the horizon.
249
250The change on June 6th, 2012 added a degree of type-safety to Go protobufs by
251declaring a new interface that all protobuf messages were required to implement:
252
253```go
254type Message interface {
255   Reset()
256   String() string
257   ProtoMessage()
258}
259```
260
261This interface reduced the set of types that can be passed to `proto.Unmarshal`
262from the universal set of all possible Go types to those with a special
263`ProtoMessage` marker method. The intention of this change is to limit the
264protobuf API to only operate on protobuf data types (i.e., protobuf messages).
265For example, there is no sensible operation if a Go channel were passed to the
266protobuf API as a channel cannot be serialized. The restricted interface would
267prevent that.
268
269This interface does not behaviorally describe what a protobuf message is, but
270acts as a marker with an undocumented expectation that protobuf messages must be
271a Go struct with a specific layout of fields with formatted tags. This
272expectation is not statically enforced by the Go language, for it is an
273implementation detail checked dynamically at runtime using Go reflection. Back
274in 2012, the only types with this marker were those generated by
275`protoc-gen-go`. Since `protoc-gen-go` would always generate messages with the
276proper layout of fields, this was deemed an acceptable and dramatic improvement
277over `interface{}`.
278
279Over the next 10 years,
280[use of Go would skyrocket](https://blog.golang.org/10years) and use of
281protobufs in Go would skyrocket as well. With increased popularity also came
282more diverse usages and requirements for Go protobufs and an increased number of
283custom `proto.Message` implementations that were not generated by
284`protoc-gen-go`.
285
286The increasingly diverse ecosystem of Go types implementing the `proto.Message`
287interface led to incompatibilities, which often occurred when:
288
289*   **Passing custom `proto.Message` types to the protobuf APIs**: A concrete
290    message implementation might work with some top-level functions (e.g.,
291    `proto.Marshal`), but cause others (e.g., `proto.Equal`) to choke and panic.
292    This occurs because the type only had partial support for being an actual
293    message by only implementing the `proto.Marshaler` interface or having
294    malformed struct field tags that happened to work with one function, but not
295    another.
296
297*   **Using Go reflection on any `proto.Message` types**: A common desire is to
298    write general-purpose code that operates on any protobuf message. For
299    example, a microservice might want to populate a `trace_id` field if it is
300    present in a message. To accomplish this, one would use Go reflection to
301    introspect the message type, and assume it were a pointer to a Go struct
302    with a field named `TraceId` (as would be commonly produced by
303    `protoc-gen-go`). If the concrete message type did not match this
304    expectation, it either failed to work or even resulted in a panic. Such was
305    the case for concrete message types that might be backed by a Go map instead
306    of a Go struct.
307
308Both of these issues are solved by following the idiom that _interfaces should
309describe behavior, not data_. This means that the interface itself should
310provide sufficient functionality through its methods that users can introspect
311and interact with all aspects of a protobuf message through a principled API.
312This feature is called _protobuf reflection_. Just as how Go reflection provides
313an API for programmatically interacting with any arbitrary Go value, protobuf
314reflection provides an API for programmatically interacting with any arbitrary
315protobuf message.
316
317Since an interface cannot be extended in a backwards compatible way, this
318suggested the need for a new major version that defines a new `proto.Message`
319interface:
320
321```go
322type Message interface {
323    ProtoReflect() protoreflect.Message
324}
325```
326
327The new
328[`proto.Message`](https://pkg.go.dev/google.golang.org/protobuf/proto?tab=doc#Message)
329interface contains a single `ProtoReflect` method that returns a
330[`protoreflect.Message`](https://pkg.go.dev/google.golang.org/protobuf/reflect/protoreflect?tab=doc#Message),
331which is a reflective view over a protobuf message. In addition to making a
332breaking change to the `proto.Message` interface, we took this opportunity to
333cleanup the supporting functionality that operate on a `proto.Message`, split up
334complicated functionality apart into manageable packages, and to hide
335implementation details away from the public API.
336
337The goal for this major revision is to improve upon all the benefits of, while
338addressing all the shortcomings of the old API. We hope that it will serve the
339Go ecosystem well for the next 10 years and beyond.
340