1Protocol Buffers for Go with Gadgets
2
3GoGoProtobuf http://github.com/gogo/protobuf extends
4GoProtobuf http://github.com/golang/protobuf
5
6Copyright (c) 2013, The GoGo Authors. All rights reserved.
7
8
9# Go support for Protocol Buffers
10
11Google's data interchange format.
12Copyright 2010 The Go Authors.
13https://github.com/golang/protobuf
14
15This package and the code it generates requires at least Go 1.6.
16
17This software implements Go bindings for protocol buffers. For
18information about protocol buffers themselves, see
19 https://developers.google.com/protocol-buffers/
20
21## Installation ##
22
23To use this software, you must:
24- Install the standard C++ implementation of protocol buffers from
25 https://developers.google.com/protocol-buffers/
26- Of course, install the Go compiler and tools from
27 https://golang.org/
28 See
29 https://golang.org/doc/install
30 for details or, if you are using gccgo, follow the instructions at
31 https://golang.org/doc/install/gccgo
32- Grab the code from the repository and install the `proto` package.
33 The simplest way is to run `go get -u github.com/golang/protobuf/protoc-gen-go`.
34 The compiler plugin, `protoc-gen-go`, will be installed in `$GOPATH/bin`
35 unless `$GOBIN` is set. It must be in your `$PATH` for the protocol
36 compiler, `protoc`, to find it.
37- If you need a particular version of `protoc-gen-go` (e.g., to match your
38 `proto` package version), one option is
39 ```shell
40 GIT_TAG="v1.2.0" # change as needed
41 go get -d -u github.com/golang/protobuf/protoc-gen-go
42 git -C "$(go env GOPATH)"/src/github.com/golang/protobuf checkout $GIT_TAG
43 go install github.com/golang/protobuf/protoc-gen-go
44 ```
45
46This software has two parts: a 'protocol compiler plugin' that
47generates Go source files that, once compiled, can access and manage
48protocol buffers; and a library that implements run-time support for
49encoding (marshaling), decoding (unmarshaling), and accessing protocol
50buffers.
51
52There is support for gRPC in Go using protocol buffers.
53See the note at the bottom of this file for details.
54
55There are no insertion points in the plugin.
56
57GoGoProtobuf provides extensions for protocol buffers and GoProtobuf
58see http://github.com/gogo/protobuf/gogoproto/doc.go
59
60## Using protocol buffers with Go ##
61
62Once the software is installed, there are two steps to using it.
63First you must compile the protocol buffer definitions and then import
64them, with the support library, into your program.
65
66To compile the protocol buffer definition, run protoc with the --gogo_out
67parameter set to the directory you want to output the Go code to.
68
69 protoc --gogo_out=. *.proto
70
71The generated files will be suffixed .pb.go. See the Test code below
72for an example using such a file.
73
74## Packages and input paths ##
75
76The protocol buffer language has a concept of "packages" which does not
77correspond well to the Go notion of packages. In generated Go code,
78each source `.proto` file is associated with a single Go package. The
79name and import path for this package is specified with the `go_package`
80proto option:
81
82 option go_package = "github.com/gogo/protobuf/types";
83
84The protocol buffer compiler will attempt to derive a package name and
85import path if a `go_package` option is not present, but it is
86best to always specify one explicitly.
87
88There is a one-to-one relationship between source `.proto` files and
89generated `.pb.go` files, but any number of `.pb.go` files may be
90contained in the same Go package.
91
92The output name of a generated file is produced by replacing the
93`.proto` suffix with `.pb.go` (e.g., `foo.proto` produces `foo.pb.go`).
94However, the output directory is selected in one of two ways. Let
95us say we have `inputs/x.proto` with a `go_package` option of
96`github.com/golang/protobuf/p`. The corresponding output file may
97be:
98
99- Relative to the import path:
100
101 protoc --gogo_out=. inputs/x.proto
102 # writes ./github.com/gogo/protobuf/p/x.pb.go
103
104 (This can work well with `--gogo_out=$GOPATH`.)
105
106- Relative to the input file:
107
108 protoc --gogo_out=paths=source_relative:. inputs/x.proto
109 # generate ./inputs/x.pb.go
110
111## Generated code ##
112
113The package comment for the proto library contains text describing
114the interface provided in Go for protocol buffers. Here is an edited
115version.
116
117If you are using any gogo.proto extensions you will need to specify the
118proto_path to include the descriptor.proto and gogo.proto.
119gogo.proto is located in github.com/gogo/protobuf/gogoproto
120This should be fine, since your import is the same.
121descriptor.proto is located in either github.com/gogo/protobuf/protobuf
122or code.google.com/p/protobuf/trunk/src/
123Its import is google/protobuf/descriptor.proto so it might need some help.
124
125 protoc --gogo_out=. -I=.:github.com/gogo/protobuf/protobuf *.proto
126
127==========
128
129The proto package converts data structures to and from the
130wire format of protocol buffers. It works in concert with the
131Go source code generated for .proto files by the protocol compiler.
132
133A summary of the properties of the protocol buffer interface
134for a protocol buffer variable v:
135
136 - Names are turned from camel_case to CamelCase for export.
137 - There are no methods on v to set fields; just treat
138 them as structure fields.
139 - There are getters that return a field's value if set,
140 and return the field's default value if unset.
141 The getters work even if the receiver is a nil message.
142 - The zero value for a struct is its correct initialization state.
143 All desired fields must be set before marshaling.
144 - A Reset() method will restore a protobuf struct to its zero state.
145 - Non-repeated fields are pointers to the values; nil means unset.
146 That is, optional or required field int32 f becomes F *int32.
147 - Repeated fields are slices.
148 - Helper functions are available to aid the setting of fields.
149 Helpers for getting values are superseded by the
150 GetFoo methods and their use is deprecated.
151 msg.Foo = proto.String("hello") // set field
152 - Constants are defined to hold the default values of all fields that
153 have them. They have the form Default_StructName_FieldName.
154 Because the getter methods handle defaulted values,
155 direct use of these constants should be rare.
156 - Enums are given type names and maps from names to values.
157 Enum values are prefixed with the enum's type name. Enum types have
158 a String method, and a Enum method to assist in message construction.
159 - Nested groups and enums have type names prefixed with the name of
160 the surrounding message type.
161 - Extensions are given descriptor names that start with E_,
162 followed by an underscore-delimited list of the nested messages
163 that contain it (if any) followed by the CamelCased name of the
164 extension field itself. HasExtension, ClearExtension, GetExtension
165 and SetExtension are functions for manipulating extensions.
166 - Oneof field sets are given a single field in their message,
167 with distinguished wrapper types for each possible field value.
168 - Marshal and Unmarshal are functions to encode and decode the wire format.
169
170When the .proto file specifies `syntax="proto3"`, there are some differences:
171
172 - Non-repeated fields of non-message type are values instead of pointers.
173 - Enum types do not get an Enum method.
174
175Consider file test.proto, containing
176
177```proto
178 syntax = "proto2";
179 package example;
180
181 enum FOO { X = 17; };
182
183 message Test {
184 required string label = 1;
185 optional int32 type = 2 [default=77];
186 repeated int64 reps = 3;
187 }
188```
189
190To create and play with a Test object from the example package,
191
192```go
193 package main
194
195 import (
196 "log"
197
198 "github.com/gogo/protobuf/proto"
199 "path/to/example"
200 )
201
202 func main() {
203 test := &example.Test{
204 Label: proto.String("hello"),
205 Type: proto.Int32(17),
206 Reps: []int64{1, 2, 3},
207 }
208 data, err := proto.Marshal(test)
209 if err != nil {
210 log.Fatal("marshaling error: ", err)
211 }
212 newTest := &example.Test{}
213 err = proto.Unmarshal(data, newTest)
214 if err != nil {
215 log.Fatal("unmarshaling error: ", err)
216 }
217 // Now test and newTest contain the same data.
218 if test.GetLabel() != newTest.GetLabel() {
219 log.Fatalf("data mismatch %q != %q", test.GetLabel(), newTest.GetLabel())
220 }
221 // etc.
222 }
223```
224
225
226## Parameters ##
227
228To pass extra parameters to the plugin, use a comma-separated
229parameter list separated from the output directory by a colon:
230
231
232 protoc --gogo_out=plugins=grpc,import_path=mypackage:. *.proto
233
234- `paths=(import | source_relative)` - specifies how the paths of
235 generated files are structured. See the "Packages and imports paths"
236 section above. The default is `import`.
237- `plugins=plugin1+plugin2` - specifies the list of sub-plugins to
238 load. The only plugin in this repo is `grpc`.
239- `Mfoo/bar.proto=quux/shme` - declares that foo/bar.proto is
240 associated with Go package quux/shme. This is subject to the
241 import_prefix parameter.
242
243The following parameters are deprecated and should not be used:
244
245- `import_prefix=xxx` - a prefix that is added onto the beginning of
246 all imports.
247- `import_path=foo/bar` - used as the package if no input files
248 declare `go_package`. If it contains slashes, everything up to the
249 rightmost slash is ignored.
250
251## gRPC Support ##
252
253If a proto file specifies RPC services, protoc-gen-go can be instructed to
254generate code compatible with gRPC (http://www.grpc.io/). To do this, pass
255the `plugins` parameter to protoc-gen-go; the usual way is to insert it into
256the --go_out argument to protoc:
257
258 protoc --gogo_out=plugins=grpc:. *.proto
259
260## Compatibility ##
261
262The library and the generated code are expected to be stable over time.
263However, we reserve the right to make breaking changes without notice for the
264following reasons:
265
266- Security. A security issue in the specification or implementation may come to
267 light whose resolution requires breaking compatibility. We reserve the right
268 to address such security issues.
269- Unspecified behavior. There are some aspects of the Protocol Buffers
270 specification that are undefined. Programs that depend on such unspecified
271 behavior may break in future releases.
272- Specification errors or changes. If it becomes necessary to address an
273 inconsistency, incompleteness, or change in the Protocol Buffers
274 specification, resolving the issue could affect the meaning or legality of
275 existing programs. We reserve the right to address such issues, including
276 updating the implementations.
277- Bugs. If the library has a bug that violates the specification, a program
278 that depends on the buggy behavior may break if the bug is fixed. We reserve
279 the right to fix such bugs.
280- Adding methods or fields to generated structs. These may conflict with field
281 names that already exist in a schema, causing applications to break. When the
282 code generator encounters a field in the schema that would collide with a
283 generated field or method name, the code generator will append an underscore
284 to the generated field or method name.
285- Adding, removing, or changing methods or fields in generated structs that
286 start with `XXX`. These parts of the generated code are exported out of
287 necessity, but should not be considered part of the public API.
288- Adding, removing, or changing unexported symbols in generated code.
289
290Any breaking changes outside of these will be announced 6 months in advance to
291protobuf@googlegroups.com.
292
293You should, whenever possible, use generated code created by the `protoc-gen-go`
294tool built at the same commit as the `proto` package. The `proto` package
295declares package-level constants in the form `ProtoPackageIsVersionX`.
296Application code and generated code may depend on one of these constants to
297ensure that compilation will fail if the available version of the proto library
298is too old. Whenever we make a change to the generated code that requires newer
299library support, in the same commit we will increment the version number of the
300generated code and declare a new package-level constant whose name incorporates
301the latest version number. Removing a compatibility constant is considered a
302breaking change and would be subject to the announcement policy stated above.
303
304The `protoc-gen-go/generator` package exposes a plugin interface,
305which is used by the gRPC code generation. This interface is not
306supported and is subject to incompatible changes without notice.
307
Readme.md
1# Protocol Buffers for Go with Gadgets
2
3[![Build Status](https://travis-ci.org/gogo/protobuf.svg?branch=master)](https://travis-ci.org/gogo/protobuf)
4[![GoDoc](https://godoc.org/github.com/gogo/protobuf?status.svg)](http://godoc.org/github.com/gogo/protobuf)
5
6gogoprotobuf is a fork of <a href="https://github.com/golang/protobuf">golang/protobuf</a> with extra code generation features.
7
8This code generation is used to achieve:
9
10 - fast marshalling and unmarshalling
11 - more canonical Go structures
12 - goprotobuf compatibility
13 - less typing by optionally generating extra helper code
14 - peace of mind by optionally generating test and benchmark code
15 - other serialization formats
16
17Keeping track of how up to date gogoprotobuf is relative to golang/protobuf is done in this
18<a href="https://github.com/gogo/protobuf/issues/191">issue</a>
19
20## Release v1.3.0
21
22The project has updated to release v1.3.0. Check out the release notes <a href="https://github.com/gogo/protobuf/releases/tag/v1.3.0">here</a>.
23
24With this new release comes a new internal library version. This means any newly generated *pb.go files generated with the v1.3.0 library will not be compatible with the old library version (v1.2.1). However, current *pb.go files (generated with v1.2.1) should still work with the new library.
25
26Please make sure you manage your dependencies correctly when upgrading your project. If you are still using v1.2.1 and you update your dependencies, one of which could include a new *pb.go (generated with v1.3.0), you could get a compile time error.
27
28Our upstream repo, golang/protobuf, also had to go through this process in order to update their library version.
29Here is a link explaining <a href="https://github.com/golang/protobuf/issues/763#issuecomment-442434870">hermetic builds</a>.
30
31
32## Users
33
34These projects use gogoprotobuf:
35
36 - <a href="http://godoc.org/github.com/coreos/etcd">etcd</a> - <a href="https://blog.gopheracademy.com/advent-2015/etcd-distributed-key-value-store-with-grpc-http2/">blog</a> - <a href="https://github.com/coreos/etcd/blob/master/etcdserver/etcdserverpb/etcdserver.proto">sample proto file</a>
37 - <a href="https://www.spacemonkey.com/">spacemonkey</a> - <a href="https://www.spacemonkey.com/blog/posts/go-space-monkey">blog</a>
38 - <a href="http://badoo.com">badoo</a> - <a href="https://github.com/badoo/lsd/blob/32061f501c5eca9c76c596d790b450501ba27b2f/proto/lsd.proto">sample proto file</a>
39 - <a href="https://github.com/mesos/mesos-go">mesos-go</a> - <a href="https://github.com/mesos/mesos-go/blob/f9e5fb7c2f50ab5f23299f26b6b07c5d6afdd252/api/v0/mesosproto/authentication.proto">sample proto file</a>
40 - <a href="https://github.com/mozilla-services/heka">heka</a> - <a href="https://github.com/mozilla-services/heka/commit/eb72fbf7d2d28249fbaf8d8dc6607f4eb6f03351">the switch from golang/protobuf to gogo/protobuf when it was still on code.google.com</a>
41 - <a href="https://github.com/cockroachdb/cockroach">cockroachdb</a> - <a href="https://github.com/cockroachdb/cockroach/blob/651d54d393e391a30154e9117ab4b18d9ee6d845/roachpb/metadata.proto">sample proto file</a>
42 - <a href="https://github.com/jbenet/go-ipfs">go-ipfs</a> - <a href="https://github.com/ipfs/go-ipfs/blob/2b6da0c024f28abeb16947fb452787196a6b56a2/merkledag/pb/merkledag.proto">sample proto file</a>
43 - <a href="https://github.com/philhofer/rkive">rkive-go</a> - <a href="https://github.com/philhofer/rkive/blob/e5dd884d3ea07b341321073882ae28aa16dd11be/rpbc/riak_dt.proto">sample proto file</a>
44 - <a href="https://www.dropbox.com">dropbox</a>
45 - <a href="https://srclib.org/">srclib</a> - <a href="https://github.com/sourcegraph/srclib/blob/6538858f0c410cac5c63440317b8d009e889d3fb/graph/def.proto">sample proto file</a>
46 - <a href="http://www.adyoulike.com/">adyoulike</a>
47 - <a href="http://www.cloudfoundry.org/">cloudfoundry</a> - <a href="https://github.com/cloudfoundry/bbs/blob/d673710b8c4211037805129944ee4c5373d6588a/models/events.proto">sample proto file</a>
48 - <a href="http://kubernetes.io/">kubernetes</a> - <a href="https://github.com/kubernetes/kubernetes/tree/88d8628137f94ee816aaa6606ae8cd045dee0bff/cmd/libs/go2idl">go2idl built on top of gogoprotobuf</a>
49 - <a href="https://dgraph.io/">dgraph</a> - <a href="https://github.com/dgraph-io/dgraph/releases/tag/v0.4.3">release notes</a> - <a href="https://discuss.dgraph.io/t/gogoprotobuf-is-extremely-fast/639">benchmarks</a></a>
50 - <a href="https://github.com/centrifugal/centrifugo">centrifugo</a> - <a href="https://forum.golangbridge.org/t/centrifugo-real-time-messaging-websocket-or-sockjs-server-v1-5-0-released/2861">release notes</a> - <a href="https://medium.com/@fzambia/centrifugo-protobuf-inside-json-outside-21d39bdabd68#.o3icmgjqd">blog</a>
51 - <a href="https://github.com/docker/swarmkit">docker swarmkit</a> - <a href="https://github.com/docker/swarmkit/blob/63600e01af3b8da2a0ed1c9fa6e1ae4299d75edb/api/objects.proto">sample proto file</a>
52 - <a href="https://nats.io/">nats.io</a> - <a href="https://github.com/nats-io/go-nats-streaming/blob/master/pb/protocol.proto">go-nats-streaming</a>
53 - <a href="https://github.com/pingcap/tidb">tidb</a> - Communication between <a href="https://github.com/pingcap/tipb/blob/master/generate-go.sh#L4">tidb</a> and <a href="https://github.com/pingcap/kvproto/blob/master/generate_go.sh#L3">tikv</a>
54 - <a href="https://github.com/AsynkronIT/protoactor-go">protoactor-go</a> - <a href="https://github.com/AsynkronIT/protoactor-go/blob/master/protobuf/protoc-gen-protoactor/main.go">vanity command</a> that also generates actors from service definitions
55 - <a href="https://containerd.io/">containerd</a> - <a href="https://github.com/containerd/containerd/tree/master/cmd/protoc-gen-gogoctrd">vanity command with custom field names</a> that conforms to the golang convention.
56 - <a href="https://github.com/heroiclabs/nakama">nakama</a>
57 - <a href="https://github.com/src-d/proteus">proteus</a>
58 - <a href="https://github.com/go-graphite">carbonzipper stack</a>
59 - <a href="https://sendgrid.com/">sendgrid</a>
60 - <a href="https://github.com/zero-os/0-stor">zero-os/0-stor</a>
61 - <a href="https://github.com/spacemeshos/go-spacemesh">go-spacemesh</a>
62 - <a href="https://github.com/weaveworks/cortex">cortex</a> - <a href="https://github.com/weaveworks/cortex/blob/fee02a59729d3771ef888f7bf0fd050e1197c56e/pkg/ingester/client/cortex.proto">sample proto file</a>
63 - <a href="http://skywalking.apache.org/">Apache SkyWalking APM</a> - Istio telemetry receiver based on Mixer bypass protocol
64 - <a href="https://github.com/hyperledger/burrow">Hyperledger Burrow</a> - a permissioned DLT framework
65 - <a href="https://github.com/iov-one/weave">IOV Weave</a> - a blockchain framework - <a href="https://github.com/iov-one/weave/tree/23f9856f1e316f93cb3d45d92c4c6a0c4810f6bf/spec/gogo">sample proto files</a>
66
67Please let us know if you are using gogoprotobuf by posting on our <a href="https://groups.google.com/forum/#!topic/gogoprotobuf/Brw76BxmFpQ">GoogleGroup</a>.
68
69### Mentioned
70
71 - <a href="http://www.slideshare.net/albertstrasheim/serialization-in-go">Cloudflare - go serialization talk - Albert Strasheim</a>
72 - <a href="https://youtu.be/4xB46Xl9O9Q?t=557">GopherCon 2014 Writing High Performance Databases in Go by Ben Johnson</a>
73 - <a href="https://github.com/alecthomas/go_serialization_benchmarks">alecthomas' go serialization benchmarks</a>
74 - <a href="http://agniva.me/go/2017/11/18/gogoproto.html">Go faster with gogoproto - Agniva De Sarker</a>
75 - <a href="https://www.youtube.com/watch?v=CY9T020HLP8">Evolution of protobuf (Gource Visualization) - Landon Wilkins</a>
76 - <a href="https://fosdem.org/2018/schedule/event/gopherjs/">Creating GopherJS Apps with gRPC-Web - Johan Brandhorst</a>
77 - <a href="https://jbrandhorst.com/post/gogoproto/">So you want to use GoGo Protobuf - Johan Brandhorst</a>
78 - <a href="https://jbrandhorst.com/post/grpc-errors/">Advanced gRPC Error Usage - Johan Brandhorst</a>
79 - <a href="https://www.udemy.com/grpc-golang/?couponCode=GITHUB10">gRPC Golang Course on Udemy - Stephane Maarek</a>
80
81## Getting Started
82
83There are several ways to use gogoprotobuf, but for all you need to install go and protoc.
84After that you can choose:
85
86 - Speed
87 - More Speed and more generated code
88 - Most Speed and most customization
89
90### Installation
91
92To install it, you must first have Go (at least version 1.6.3 or 1.9 if you are using gRPC) installed (see [http://golang.org/doc/install](http://golang.org/doc/install)).
93Latest patch versions of 1.10 and 1.11 are continuously tested.
94
95Next, install the standard protocol buffer implementation from [https://github.com/google/protobuf](https://github.com/google/protobuf).
96Most versions from 2.3.1 should not give any problems, but 2.6.1, 3.0.2 and 3.6.1 are continuously tested.
97
98### Speed
99
100Install the protoc-gen-gofast binary
101
102 go get github.com/gogo/protobuf/protoc-gen-gofast
103
104Use it to generate faster marshaling and unmarshaling go code for your protocol buffers.
105
106 protoc --gofast_out=. myproto.proto
107
108This does not allow you to use any of the other gogoprotobuf [extensions](https://github.com/gogo/protobuf/blob/master/extensions.md).
109
110### More Speed and more generated code
111
112Fields without pointers cause less time in the garbage collector.
113More code generation results in more convenient methods.
114
115Other binaries are also included:
116
117 protoc-gen-gogofast (same as gofast, but imports gogoprotobuf)
118 protoc-gen-gogofaster (same as gogofast, without XXX_unrecognized, less pointer fields)
119 protoc-gen-gogoslick (same as gogofaster, but with generated string, gostring and equal methods)
120
121Installing any of these binaries is easy. Simply run:
122
123 go get github.com/gogo/protobuf/proto
124 go get github.com/gogo/protobuf/{binary}
125 go get github.com/gogo/protobuf/gogoproto
126
127These binaries allow you to use gogoprotobuf [extensions](https://github.com/gogo/protobuf/blob/master/extensions.md). You can also use your own binary.
128
129To generate the code, you also need to set the include path properly.
130
131 protoc -I=. -I=$GOPATH/src -I=$GOPATH/src/github.com/gogo/protobuf/protobuf --{binary}_out=. myproto.proto
132
133To use proto files from "google/protobuf" you need to add additional args to protoc.
134
135 protoc -I=. -I=$GOPATH/src -I=$GOPATH/src/github.com/gogo/protobuf/protobuf --{binary}_out=\
136 Mgoogle/protobuf/any.proto=github.com/gogo/protobuf/types,\
137 Mgoogle/protobuf/duration.proto=github.com/gogo/protobuf/types,\
138 Mgoogle/protobuf/struct.proto=github.com/gogo/protobuf/types,\
139 Mgoogle/protobuf/timestamp.proto=github.com/gogo/protobuf/types,\
140 Mgoogle/protobuf/wrappers.proto=github.com/gogo/protobuf/types:. \
141 myproto.proto
142
143Note that in the protoc command, {binary} does not contain the initial prefix of "protoc-gen".
144
145### Most Speed and most customization
146
147Customizing the fields of the messages to be the fields that you actually want to use removes the need to copy between the structs you use and structs you use to serialize.
148gogoprotobuf also offers more serialization formats and generation of tests and even more methods.
149
150Please visit the [extensions](https://github.com/gogo/protobuf/blob/master/extensions.md) page for more documentation.
151
152Install protoc-gen-gogo:
153
154 go get github.com/gogo/protobuf/proto
155 go get github.com/gogo/protobuf/jsonpb
156 go get github.com/gogo/protobuf/protoc-gen-gogo
157 go get github.com/gogo/protobuf/gogoproto
158
159## GRPC
160
161It works the same as golang/protobuf, simply specify the plugin.
162Here is an example using gofast:
163
164 protoc --gofast_out=plugins=grpc:. my.proto
165
166See [https://github.com/gogo/grpc-example](https://github.com/gogo/grpc-example) for an example of using gRPC with gogoprotobuf and the wider grpc-ecosystem.
167
168
169## License
170This software is licensed under the 3-Clause BSD License
171("BSD License 2.0", "Revised BSD License", "New BSD License", or "Modified BSD License").
172
173
174