1// Copyright (c) 2012-2018 Ugorji Nwoke. All rights reserved. 2// Use of this source code is governed by a MIT license found in the LICENSE file. 3 4/* 5Package codec provides a 6High Performance, Feature-Rich Idiomatic Go 1.4+ codec/encoding library 7for binc, msgpack, cbor, json. 8 9Supported Serialization formats are: 10 11 - msgpack: https://github.com/msgpack/msgpack 12 - binc: http://github.com/ugorji/binc 13 - cbor: http://cbor.io http://tools.ietf.org/html/rfc7049 14 - json: http://json.org http://tools.ietf.org/html/rfc7159 15 - simple: 16 17This package will carefully use 'package unsafe' for performance reasons in specific places. 18You can build without unsafe use by passing the safe or appengine tag 19i.e. 'go install -tags=safe ...'. Note that unsafe is only supported for the last 4 20go releases e.g. current go release is go 1.12, so we support unsafe use only from 21go 1.9+ . This is because supporting unsafe requires knowledge of implementation details. 22 23For detailed usage information, read the primer at http://ugorji.net/blog/go-codec-primer . 24 25The idiomatic Go support is as seen in other encoding packages in 26the standard library (ie json, xml, gob, etc). 27 28Rich Feature Set includes: 29 30 - Simple but extremely powerful and feature-rich API 31 - Support for go1.4 and above, while selectively using newer APIs for later releases 32 - Excellent code coverage ( > 90% ) 33 - Very High Performance. 34 Our extensive benchmarks show us outperforming Gob, Json, Bson, etc by 2-4X. 35 - Careful selected use of 'unsafe' for targeted performance gains. 36 100% mode exists where 'unsafe' is not used at all. 37 - Lock-free (sans mutex) concurrency for scaling to 100's of cores 38 - In-place updates during decode, with option to zero value in maps and slices prior to decode 39 - Coerce types where appropriate 40 e.g. decode an int in the stream into a float, decode numbers from formatted strings, etc 41 - Corner Cases: 42 Overflows, nil maps/slices, nil values in streams are handled correctly 43 - Standard field renaming via tags 44 - Support for omitting empty fields during an encoding 45 - Encoding from any value and decoding into pointer to any value 46 (struct, slice, map, primitives, pointers, interface{}, etc) 47 - Extensions to support efficient encoding/decoding of any named types 48 - Support encoding.(Binary|Text)(M|Unm)arshaler interfaces 49 - Support IsZero() bool to determine if a value is a zero value. 50 Analogous to time.Time.IsZero() bool. 51 - Decoding without a schema (into a interface{}). 52 Includes Options to configure what specific map or slice type to use 53 when decoding an encoded list or map into a nil interface{} 54 - Mapping a non-interface type to an interface, so we can decode appropriately 55 into any interface type with a correctly configured non-interface value. 56 - Encode a struct as an array, and decode struct from an array in the data stream 57 - Option to encode struct keys as numbers (instead of strings) 58 (to support structured streams with fields encoded as numeric codes) 59 - Comprehensive support for anonymous fields 60 - Fast (no-reflection) encoding/decoding of common maps and slices 61 - Code-generation for faster performance. 62 - Support binary (e.g. messagepack, cbor) and text (e.g. json) formats 63 - Support indefinite-length formats to enable true streaming 64 (for formats which support it e.g. json, cbor) 65 - Support canonical encoding, where a value is ALWAYS encoded as same sequence of bytes. 66 This mostly applies to maps, where iteration order is non-deterministic. 67 - NIL in data stream decoded as zero value 68 - Never silently skip data when decoding. 69 User decides whether to return an error or silently skip data when keys or indexes 70 in the data stream do not map to fields in the struct. 71 - Detect and error when encoding a cyclic reference (instead of stack overflow shutdown) 72 - Encode/Decode from/to chan types (for iterative streaming support) 73 - Drop-in replacement for encoding/json. `json:` key in struct tag supported. 74 - Provides a RPC Server and Client Codec for net/rpc communication protocol. 75 - Handle unique idiosyncrasies of codecs e.g. 76 - For messagepack, configure how ambiguities in handling raw bytes are resolved 77 - For messagepack, provide rpc server/client codec to support 78 msgpack-rpc protocol defined at: 79 https://github.com/msgpack-rpc/msgpack-rpc/blob/master/spec.md 80 81Extension Support 82 83Users can register a function to handle the encoding or decoding of 84their custom types. 85 86There are no restrictions on what the custom type can be. Some examples: 87 88 type BisSet []int 89 type BitSet64 uint64 90 type UUID string 91 type MyStructWithUnexportedFields struct { a int; b bool; c []int; } 92 type GifImage struct { ... } 93 94As an illustration, MyStructWithUnexportedFields would normally be 95encoded as an empty map because it has no exported fields, while UUID 96would be encoded as a string. However, with extension support, you can 97encode any of these however you like. 98 99Custom Encoding and Decoding 100 101This package maintains symmetry in the encoding and decoding halfs. 102We determine how to encode or decode by walking this decision tree 103 104 - is type a codec.Selfer? 105 - is there an extension registered for the type? 106 - is format binary, and is type a encoding.BinaryMarshaler and BinaryUnmarshaler? 107 - is format specifically json, and is type a encoding/json.Marshaler and Unmarshaler? 108 - is format text-based, and type an encoding.TextMarshaler and TextUnmarshaler? 109 - else we use a pair of functions based on the "kind" of the type e.g. map, slice, int64, etc 110 111This symmetry is important to reduce chances of issues happening because the 112encoding and decoding sides are out of sync e.g. decoded via very specific 113encoding.TextUnmarshaler but encoded via kind-specific generalized mode. 114 115Consequently, if a type only defines one-half of the symmetry 116(e.g. it implements UnmarshalJSON() but not MarshalJSON() ), 117then that type doesn't satisfy the check and we will continue walking down the 118decision tree. 119 120RPC 121 122RPC Client and Server Codecs are implemented, so the codecs can be used 123with the standard net/rpc package. 124 125Usage 126 127The Handle is SAFE for concurrent READ, but NOT SAFE for concurrent modification. 128 129The Encoder and Decoder are NOT safe for concurrent use. 130 131Consequently, the usage model is basically: 132 133 - Create and initialize the Handle before any use. 134 Once created, DO NOT modify it. 135 - Multiple Encoders or Decoders can now use the Handle concurrently. 136 They only read information off the Handle (never write). 137 - However, each Encoder or Decoder MUST not be used concurrently 138 - To re-use an Encoder/Decoder, call Reset(...) on it first. 139 This allows you use state maintained on the Encoder/Decoder. 140 141Sample usage model: 142 143 // create and configure Handle 144 var ( 145 bh codec.BincHandle 146 mh codec.MsgpackHandle 147 ch codec.CborHandle 148 ) 149 150 mh.MapType = reflect.TypeOf(map[string]interface{}(nil)) 151 152 // configure extensions 153 // e.g. for msgpack, define functions and enable Time support for tag 1 154 // mh.SetExt(reflect.TypeOf(time.Time{}), 1, myExt) 155 156 // create and use decoder/encoder 157 var ( 158 r io.Reader 159 w io.Writer 160 b []byte 161 h = &bh // or mh to use msgpack 162 ) 163 164 dec = codec.NewDecoder(r, h) 165 dec = codec.NewDecoderBytes(b, h) 166 err = dec.Decode(&v) 167 168 enc = codec.NewEncoder(w, h) 169 enc = codec.NewEncoderBytes(&b, h) 170 err = enc.Encode(v) 171 172 //RPC Server 173 go func() { 174 for { 175 conn, err := listener.Accept() 176 rpcCodec := codec.GoRpc.ServerCodec(conn, h) 177 //OR rpcCodec := codec.MsgpackSpecRpc.ServerCodec(conn, h) 178 rpc.ServeCodec(rpcCodec) 179 } 180 }() 181 182 //RPC Communication (client side) 183 conn, err = net.Dial("tcp", "localhost:5555") 184 rpcCodec := codec.GoRpc.ClientCodec(conn, h) 185 //OR rpcCodec := codec.MsgpackSpecRpc.ClientCodec(conn, h) 186 client := rpc.NewClientWithCodec(rpcCodec) 187 188Running Tests 189 190To run tests, use the following: 191 192 go test 193 194To run the full suite of tests, use the following: 195 196 go test -tags alltests -run Suite 197 198You can run the tag 'safe' to run tests or build in safe mode. e.g. 199 200 go test -tags safe -run Json 201 go test -tags "alltests safe" -run Suite 202 203Running Benchmarks 204 205 cd bench 206 go test -bench . -benchmem -benchtime 1s 207 208Please see http://github.com/ugorji/go-codec-bench . 209 210Managing Binary Size 211 212This package could add up to 10MB to the size of your binaries. 213 214This is because we include some a auto-generated file: `fast-path.generated.go` 215to help with performance when encoding/decoding slices and maps of 216built in numeric, boolean, string and interface{} types. 217 218You can override this by building (or running tests and benchmarks) 219with the tag: `notfastpath`. 220 221 go install -tags notfastpath 222 go build -tags notfastpath 223 go test -tags notfastpath 224 225Be aware that, at least in our representative microbenchmarks for cbor (for example), 226we see up to 33% increase in decoding and 50% increase in encoding speeds. 227YMMV. 228 229Caveats 230 231Struct fields matching the following are ignored during encoding and decoding 232 - struct tag value set to - 233 - func, complex numbers, unsafe pointers 234 - unexported and not embedded 235 - unexported and embedded and not struct kind 236 - unexported and embedded pointers (from go1.10) 237 238Every other field in a struct will be encoded/decoded. 239 240Embedded fields are encoded as if they exist in the top-level struct, 241with some caveats. See Encode documentation. 242 243*/ 244package codec 245 246