README.md
1# Codec
2
3High Performance, Feature-Rich Idiomatic Go codec/encoding library for
4binc, msgpack, cbor, json.
5
6Supported Serialization formats are:
7
8 - msgpack: https://github.com/msgpack/msgpack
9 - binc: http://github.com/ugorji/binc
10 - cbor: http://cbor.io http://tools.ietf.org/html/rfc7049
11 - json: http://json.org http://tools.ietf.org/html/rfc7159
12 - simple:
13
14To install:
15
16 go get github.com/ugorji/go/codec
17
18This package will carefully use 'unsafe' for performance reasons in specific places.
19You can build without unsafe use by passing the safe or appengine tag
20i.e. 'go install -tags=safe ...'. Note that unsafe is only supported for the last 3
21go sdk versions e.g. current go release is go 1.9, so we support unsafe use only from
22go 1.7+ . This is because supporting unsafe requires knowledge of implementation details.
23
24Online documentation: http://godoc.org/github.com/ugorji/go/codec
25Detailed Usage/How-to Primer: http://ugorji.net/blog/go-codec-primer
26
27The idiomatic Go support is as seen in other encoding packages in
28the standard library (ie json, xml, gob, etc).
29
30Rich Feature Set includes:
31
32 - Simple but extremely powerful and feature-rich API
33 - Support for go1.4 and above, while selectively using newer APIs for later releases
34 - Excellent code coverage ( > 90% )
35 - Very High Performance.
36 Our extensive benchmarks show us outperforming Gob, Json, Bson, etc by 2-4X.
37 - Careful selected use of 'unsafe' for targeted performance gains.
38 100% mode exists where 'unsafe' is not used at all.
39 - Lock-free (sans mutex) concurrency for scaling to 100's of cores
40 - Coerce types where appropriate
41 e.g. decode an int in the stream into a float, decode numbers from formatted strings, etc
42 - Corner Cases:
43 Overflows, nil maps/slices, nil values in streams are handled correctly
44 - Standard field renaming via tags
45 - Support for omitting empty fields during an encoding
46 - Encoding from any value and decoding into pointer to any value
47 (struct, slice, map, primitives, pointers, interface{}, etc)
48 - Extensions to support efficient encoding/decoding of any named types
49 - Support encoding.(Binary|Text)(M|Unm)arshaler interfaces
50 - Support IsZero() bool to determine if a value is a zero value.
51 Analogous to time.Time.IsZero() bool.
52 - Decoding without a schema (into a interface{}).
53 Includes Options to configure what specific map or slice type to use
54 when decoding an encoded list or map into a nil interface{}
55 - Mapping a non-interface type to an interface, so we can decode appropriately
56 into any interface type with a correctly configured non-interface value.
57 - Encode a struct as an array, and decode struct from an array in the data stream
58 - Option to encode struct keys as numbers (instead of strings)
59 (to support structured streams with fields encoded as numeric codes)
60 - Comprehensive support for anonymous fields
61 - Fast (no-reflection) encoding/decoding of common maps and slices
62 - Code-generation for faster performance.
63 - Support binary (e.g. messagepack, cbor) and text (e.g. json) formats
64 - Support indefinite-length formats to enable true streaming
65 (for formats which support it e.g. json, cbor)
66 - Support canonical encoding, where a value is ALWAYS encoded as same sequence of bytes.
67 This mostly applies to maps, where iteration order is non-deterministic.
68 - NIL in data stream decoded as zero value
69 - Never silently skip data when decoding.
70 User decides whether to return an error or silently skip data when keys or indexes
71 in the data stream do not map to fields in the struct.
72 - Encode/Decode from/to chan types (for iterative streaming support)
73 - Drop-in replacement for encoding/json. `json:` key in struct tag supported.
74 - Provides a RPC Server and Client Codec for net/rpc communication protocol.
75 - Handle unique idiosyncrasies of codecs e.g.
76 - For messagepack, configure how ambiguities in handling raw bytes are resolved
77 - For messagepack, provide rpc server/client codec to support
78 msgpack-rpc protocol defined at:
79 https://github.com/msgpack-rpc/msgpack-rpc/blob/master/spec.md
80
81## Extension Support
82
83Users can register a function to handle the encoding or decoding of
84their custom types.
85
86There are no restrictions on what the custom type can be. Some examples:
87
88 type BisSet []int
89 type BitSet64 uint64
90 type UUID string
91 type MyStructWithUnexportedFields struct { a int; b bool; c []int; }
92 type GifImage struct { ... }
93
94As an illustration, MyStructWithUnexportedFields would normally be
95encoded as an empty map because it has no exported fields, while UUID
96would be encoded as a string. However, with extension support, you can
97encode any of these however you like.
98
99## Custom Encoding and Decoding
100
101This package maintains symmetry in the encoding and decoding halfs.
102We determine how to encode or decode by walking this decision tree
103
104 - is type a codec.Selfer?
105 - is there an extension registered for the type?
106 - is format binary, and is type a encoding.BinaryMarshaler and BinaryUnmarshaler?
107 - is format specifically json, and is type a encoding/json.Marshaler and Unmarshaler?
108 - is format text-based, and type an encoding.TextMarshaler?
109 - else we use a pair of functions based on the "kind" of the type e.g. map, slice, int64, etc
110
111This symmetry is important to reduce chances of issues happening because the
112encoding and decoding sides are out of sync e.g. decoded via very specific
113encoding.TextUnmarshaler but encoded via kind-specific generalized mode.
114
115Consequently, if a type only defines one-half of the symmetry
116(e.g. it implements UnmarshalJSON() but not MarshalJSON() ),
117then that type doesn't satisfy the check and we will continue walking down the
118decision tree.
119
120## RPC
121
122RPC Client and Server Codecs are implemented, so the codecs can be used
123with the standard net/rpc package.
124
125## Usage
126
127Typical usage model:
128
129 // create and configure Handle
130 var (
131 bh codec.BincHandle
132 mh codec.MsgpackHandle
133 ch codec.CborHandle
134 )
135
136 mh.MapType = reflect.TypeOf(map[string]interface{}(nil))
137
138 // configure extensions
139 // e.g. for msgpack, define functions and enable Time support for tag 1
140 // mh.SetExt(reflect.TypeOf(time.Time{}), 1, myExt)
141
142 // create and use decoder/encoder
143 var (
144 r io.Reader
145 w io.Writer
146 b []byte
147 h = &bh // or mh to use msgpack
148 )
149
150 dec = codec.NewDecoder(r, h)
151 dec = codec.NewDecoderBytes(b, h)
152 err = dec.Decode(&v)
153
154 enc = codec.NewEncoder(w, h)
155 enc = codec.NewEncoderBytes(&b, h)
156 err = enc.Encode(v)
157
158 //RPC Server
159 go func() {
160 for {
161 conn, err := listener.Accept()
162 rpcCodec := codec.GoRpc.ServerCodec(conn, h)
163 //OR rpcCodec := codec.MsgpackSpecRpc.ServerCodec(conn, h)
164 rpc.ServeCodec(rpcCodec)
165 }
166 }()
167
168 //RPC Communication (client side)
169 conn, err = net.Dial("tcp", "localhost:5555")
170 rpcCodec := codec.GoRpc.ClientCodec(conn, h)
171 //OR rpcCodec := codec.MsgpackSpecRpc.ClientCodec(conn, h)
172 client := rpc.NewClientWithCodec(rpcCodec)
173
174## Running Tests
175
176To run tests, use the following:
177
178 go test
179
180To run the full suite of tests, use the following:
181
182 go test -tags alltests -run Suite
183
184You can run the tag 'safe' to run tests or build in safe mode. e.g.
185
186 go test -tags safe -run Json
187 go test -tags "alltests safe" -run Suite
188
189## Running Benchmarks
190
191Please see http://github.com/ugorji/go-codec-bench .
192
193## Caveats
194
195Struct fields matching the following are ignored during encoding and decoding
196
197 - struct tag value set to -
198 - func, complex numbers, unsafe pointers
199 - unexported and not embedded
200 - unexported and embedded and not struct kind
201 - unexported and embedded pointers (from go1.10)
202
203Every other field in a struct will be encoded/decoded.
204
205Embedded fields are encoded as if they exist in the top-level struct,
206with some caveats. See Encode documentation.
207