1# pbf 2 3[![build status](https://secure.travis-ci.org/mapbox/pbf.svg)](http://travis-ci.org/mapbox/pbf) [![Coverage Status](https://coveralls.io/repos/mapbox/pbf/badge.svg)](https://coveralls.io/r/mapbox/pbf) 4 5A low-level, fast, ultra-lightweight (3KB gzipped) JavaScript library for decoding and encoding [protocol buffers](https://developers.google.com/protocol-buffers), a compact binary format for structured data serialization. Works both in Node and the browser. Supports lazy decoding and detailed customization of the reading/writing code. 6 7## Performance 8 9This library is extremely fast — much faster than native `JSON.parse`/`JSON.stringify` 10and the [protocol-buffers](https://github.com/mafintosh/protocol-buffers) module. 11Here's a result from running a real-world benchmark on Node v6.5 12(decoding and encoding a sample of 439 vector tiles, 22.6 MB total): 13 14- **pbf** decode: 387ms, or 57 MB/s 15- **pbf** encode: 396ms, or 56 MB/s 16- **protocol-buffers** decode: 837ms, or 26 MB/s 17- **protocol-buffers** encode: 4197ms, or 5 MB/s 18- **JSON.parse**: 1540ms, or 125 MB/s (parsing an equivalent 77.5 MB JSON file) 19- **JSON.stringify**: 607ms, or 49 MB/s 20 21## Examples 22 23#### Using Compiled Code 24 25Install `pbf` and compile a JavaScript module from a `.proto` file: 26 27```bash 28$ npm install -g pbf 29$ pbf example.proto > example.js 30``` 31 32Then read and write objects using the module like this: 33 34```js 35var Pbf = require('pbf'); 36var Example = require('./example.js').Example; 37 38// read 39var pbf = new Pbf(buffer); 40var obj = Example.read(pbf); 41 42// write 43var pbf = new Pbf(); 44Example.write(obj, pbf); 45var buffer = pbf.finish(); 46``` 47 48Alternatively, you can compile a module directly in the code: 49 50```js 51var compile = require('pbf/compile'); 52var schema = require('protocol-buffers-schema'); 53 54var proto = schema.parse(fs.readFileSync('example.proto')); 55var Test = compile(proto).Test; 56``` 57 58If you use `webpack` as your module bundler, you can use [pbf-loader](https://github.com/trivago/pbf-loader) 59to load .proto files directly. It returns a compiled module ready to be used. 60 61Given you already configured your `webpack.config.js`, the code above would look like: 62```js 63var Pbf = require('pbf'); 64var proto = require('./example.proto'); 65 66var Test = proto.Test; 67``` 68 69#### Custom Reading 70 71```js 72var data = new Pbf(buffer).readFields(readData, {}); 73 74function readData(tag, data, pbf) { 75 if (tag === 1) data.name = pbf.readString(); 76 else if (tag === 2) data.version = pbf.readVarint(); 77 else if (tag === 3) data.layer = pbf.readMessage(readLayer, {}); 78} 79function readLayer(tag, layer, pbf) { 80 if (tag === 1) layer.name = pbf.readString(); 81 else if (tag === 3) layer.size = pbf.readVarint(); 82} 83``` 84 85#### Custom Writing 86 87```js 88var pbf = new Pbf(); 89writeData(data, pbf); 90var buffer = pbf.finish(); 91 92function writeData(data, pbf) { 93 pbf.writeStringField(1, data.name); 94 pbf.writeVarintField(2, data.version); 95 pbf.writeMessage(3, writeLayer, data.layer); 96} 97function writeLayer(layer, pbf) { 98 pbf.writeStringField(1, layer.name); 99 pbf.writeVarintField(2, layer.size); 100} 101``` 102 103## Install 104 105Node and Browserify: 106 107```bash 108npm install pbf 109``` 110 111Making a browser build: 112 113```bash 114npm install 115npm run build-dev # dist/pbf-dev.js (development build) 116npm run build-min # dist/pbf.js (minified production build) 117``` 118 119CDN link: https://unpkg.com/pbf@3.0.5/dist/pbf.js 120 121## API 122 123Create a `Pbf` object, optionally given a `Buffer` or `Uint8Array` as input data: 124 125```js 126// parse a pbf file from disk in Node 127var pbf = new Pbf(fs.readFileSync('data.pbf')); 128 129// parse a pbf file in a browser after an ajax request with responseType="arraybuffer" 130var pbf = new Pbf(new Uint8Array(xhr.response)); 131``` 132 133`Pbf` object properties: 134 135```js 136pbf.length; // length of the underlying buffer 137pbf.pos; // current offset for reading or writing 138``` 139 140#### Reading 141 142Read a sequence of fields: 143 144```js 145pbf.readFields(function (tag) { 146 if (tag === 1) pbf.readVarint(); 147 else if (tag === 2) pbf.readString(); 148 else ... 149}); 150``` 151 152It optionally accepts an object that will be passed to the reading function for easier construction of decoded data, 153and also passes the `Pbf` object as a third argument: 154 155```js 156var result = pbf.readFields(callback, {}) 157 158function callback(tag, result, pbf) { 159 if (tag === 1) result.id = pbf.readVarint(); 160} 161``` 162 163To read an embedded message, use `pbf.readMessage(fn[, obj])` (in the same way as `read`). 164 165Read values: 166 167```js 168var value = pbf.readVarint(); 169var str = pbf.readString(); 170var numbers = pbf.readPackedVarint(); 171``` 172 173For lazy or partial decoding, simply save the position instead of reading a value, 174then later set it back to the saved value and read: 175 176```js 177var fooPos = -1; 178pbf.readFields(function (tag) { 179 if (tag === 1) fooPos = pbf.pos; 180}); 181... 182pbf.pos = fooPos; 183pbf.readMessage(readFoo); 184``` 185 186Scalar reading methods: 187 188* `readVarint(isSigned)` (pass `true` if you expect negative varints) 189* `readSVarint()` 190* `readFixed32()` 191* `readFixed64()` 192* `readSFixed32()` 193* `readSFixed64()` 194* `readBoolean()` 195* `readFloat()` 196* `readDouble()` 197* `readString()` 198* `readBytes()` 199* `skip(value)` 200 201Packed reading methods: 202 203* `readPackedVarint(arr, isSigned)` (appends read items to `arr`) 204* `readPackedSVarint(arr)` 205* `readPackedFixed32(arr)` 206* `readPackedFixed64(arr)` 207* `readPackedSFixed32(arr)` 208* `readPackedSFixed64(arr)` 209* `readPackedBoolean(arr)` 210* `readPackedFloat(arr)` 211* `readPackedDouble(arr)` 212 213#### Writing 214 215Write values: 216 217```js 218pbf.writeVarint(123); 219pbf.writeString("Hello world"); 220``` 221 222Write an embedded message: 223 224```js 225pbf.writeMessage(1, writeObj, obj); 226 227function writeObj(obj, pbf) { 228 pbf.writeStringField(obj.name); 229 pbf.writeVarintField(obj.version); 230} 231``` 232 233Field writing methods: 234 235* `writeVarintField(tag, val)` 236* `writeSVarintField(tag, val)` 237* `writeFixed32Field(tag, val)` 238* `writeFixed64Field(tag, val)` 239* `writeSFixed32Field(tag, val)` 240* `writeSFixed64Field(tag, val)` 241* `writeBooleanField(tag, val)` 242* `writeFloatField(tag, val)` 243* `writeDoubleField(tag, val)` 244* `writeStringField(tag, val)` 245* `writeBytesField(tag, buffer)` 246 247Packed field writing methods: 248 249* `writePackedVarint(tag, val)` 250* `writePackedSVarint(tag, val)` 251* `writePackedSFixed32(tag, val)` 252* `writePackedSFixed64(tag, val)` 253* `writePackedBoolean(tag, val)` 254* `writePackedFloat(tag, val)` 255* `writePackedDouble(tag, val)` 256 257Scalar writing methods: 258 259* `writeVarint(val)` 260* `writeSVarint(val)` 261* `writeSFixed32(val)` 262* `writeSFixed64(val)` 263* `writeBoolean(val)` 264* `writeFloat(val)` 265* `writeDouble(val)` 266* `writeString(val)` 267* `writeBytes(buffer)` 268 269Message writing methods: 270 271* `writeMessage(tag, fn[, obj])` 272* `writeRawMessage(fn[, obj])` 273 274Misc methods: 275 276* `realloc(minBytes)` - pad the underlying buffer size to accommodate the given number of bytes; 277 note that the size increases exponentially, so it won't necessarily equal the size of data written 278* `finish()` - make the current buffer ready for reading and return the data as a buffer slice 279* `destroy()` - dispose the buffer 280 281For an example of a real-world usage of the library, see [vector-tile-js](https://github.com/mapbox/vector-tile-js). 282 283 284## Proto Schema to JavaScript 285 286If installed globally, `pbf` provides a binary that compiles `proto` files into JavaScript modules. Usage: 287 288```bash 289$ pbf <proto_path> [--no-write] [--no-read] [--browser] 290``` 291 292The `--no-write` and `--no-read` switches remove corresponding code in the output. 293The `--browser` switch makes the module work in browsers instead of Node. 294 295The resulting module exports each message by name with the following methods: 296 297* `read(pbf)` - decodes an object from the given `Pbf` instance 298* `write(obj, pbf)` - encodes an object into the given `Pbf` instance (usually empty) 299 300The resulting code is clean and simple, so feel free to customize it. 301 302## Changelog 303 304#### 3.0.5 (Nov 30, 2016) 305 306- Fixed an error appearing in some versions of IE11 and old Android browsers. 307 308#### 3.0.4 (Nov 14, 2016) 309 310- Fixed compiling repeated packed enum fields. 311 312#### 3.0.3 (Nov 14, 2016) 313 314- Fixed a regression that broke compiling repeated enum fields with defaults. 315 316#### 3.0.2 (Sep 30, 2016) 317 318- Fixed a regression that broke decoding of packed fields with a tag that didn't fit into one byte. 319 320#### 3.0.1 (Sep 20, 2016) 321 322- Fixed a regression that broke encoding of long strings. 323 324#### 3.0.0 (Aug 30, 2016) 325 326This release include tons of compatibility/robustness fixes, and a more reliable Node implementation. Decoding performance is expected to get up to ~15% slower than v2.0 in Node (browsers are unaffected), but encoding got faster by ~15% in return. 327 328##### Encoder/decoder 329 330- **Breaking**: changed Node implementation to use `Uint8Array` instead of `Buffer` internally (and produce corresponding result on `finish()`), making it fully match the browser implementation for consistency and simplicity. 331- Fixed `writeVarint` to write `0` when given `NaN` or other non-number to avoid producing a broken Protobuf message. 332- Changed `readPacked*` methods signature to accept an optional `arr` argument to append the results to (to support messages with repeated fields that mix packed/non-packed encoding). 333- Added an optional `isSigned` argument to `readVarint` that enables proper reading of negative varints. 334- Deprecated `readVarint64()` (it still works, but it's recommended to be changed to `readVarint(true)`). 335- Faster string encoding. 336 337##### Proto compiler 338 339- **Breaking:** Full support for defaults field values (both implicit and explicit); they're now included in the decoded JSON objects. 340- Fixed reading of repeated fields with mixed packed/non-packed encoding for compatibility. 341- Fixed proto3 compiler to use packed by default for repeated scalar fields. 342- Fixed reading of negative varint types. 343- Fixed packed fields to decode into `[]` if they're not present. 344- Fixed nested message references handling. 345- Fixed `packed=false` being interpreted as packed. 346- Added a comment to generated code with pbf version number. 347 348#### 2.0.1 (May 28, 2016) 349 350- Fixed a regression with `writeVarint` that affected certain numbers. 351 352#### 2.0.0 (May 28, 2016) 353 354- Significantly improved the proto compiler, which now produces a much safer reading/writing code. 355- Added the ability to compile a read/write module from a protobuf schema directly in the code. 356- Proto compiler: fixed name resolutions and collisions in schemas with nested messages. 357- Proto compiler: fixed broken top-level enums. 358 359#### 1.3.7 (May 28, 2016) 360 361- Fixed a regression with `writeVarint` that affected certain numbers. 362 363#### 1.3.6 (May 27, 2016) 364 365- Improved read and write performance (both ~15% faster). 366- Improved generated code for default values. 367 368#### 1.3.5 (Oct 5, 2015) 369 370- Added support for `syntax` keyword proto files (by updating `resolve-protobuf-schema` dependency). 371 372#### 1.3.4 (Jul 31, 2015) 373 374- Added `writeRawMessage` method for writing a message without a tag, useful for creating pbfs with multiple top-level messages. 375 376#### 1.3.2 (Mar 5, 2015) 377 378- Added `readVarint64` method for proper decoding of negative `int64`-encoded values. 379 380#### 1.3.1 (Feb 20, 2015) 381 382- Fixed pbf proto compile tool generating broken writing code. 383 384#### 1.3.0 (Feb 5, 2015) 385 386- Added `pbf` binary that compiles `.proto` files into `Pbf`-based JavaScript modules. 387 388#### 1.2.0 (Jan 5, 2015) 389 390##### Breaking API changes 391 392- Changed `writeMessage` signature to `(tag, fn, obj)` (see example in the docs) 393 for a huge encoding performance improvement. 394- Replaced `readPacked` and `writePacked` methods that accept type as a string 395 with `readPackedVarint`, etc. for each type (better performance and simpler API). 396 397##### Improvements 398 399- 5x faster encoding in Node (vector tile benchmark). 400- 40x faster encoding and 3x faster decoding in the browser (vector tile benchmark). 401 402#### 1.1.4 (Jan 2, 2015) 403 404- Significantly improved `readPacked` and `writePacked` performance (the tile reading benchmark is now 70% faster). 405 406#### 1.1.3 (Dec 26, 2014) 407 408Brings tons of improvements and fixes over the previous version (`0.0.2`). 409Basically makes the library complete. 410 411##### Improvements 412 413- Improved performance of both reading and writing. 414- Made the browser build 3 times smaller. 415- Added convenience `readFields` and `readMessage` methods for a much easier reading API. 416- Added reading methods: `readFloat`, `readBoolean`, `readSFixed32`, `readSFixed64`. 417- Added writing methods: `writeUInt64`, `writeSFixed32`, `writeSFixed64`. 418- Improved `readDouble` and `readString` to use native Buffer methods under Node. 419- Improved `readString` and `writeString` to use HTML5 `TextEncoder` and `TextDecoder` where available. 420- Made `Pbf` `buffer` argument optional. 421- Added extensive docs and examples in the readme. 422- Added an extensive test suite that brings test coverage up to 100%. 423 424##### Breaking API changes 425 426- Renamed `readBuffer`/`writeBuffer` to `readBytes`/`writeBytes`. 427- Renamed `readUInt32`/`writeUInt32` to `readFixed32`/`writeFixed32`, etc. 428- Renamed `writeTaggedVarint` to `writeVarintField`, etc. 429- Changed `writePacked` signature from `(type, tag, items)` to `(tag, type, items)`. 430 431##### Bugfixes 432 433- Fixed `readVarint` to handle varints bigger than 6 bytes. 434- Fixed `readSVarint` to handle number bigger than `2^30`. 435- Fixed `writeVarint` failing on some integers. 436- Fixed `writeVarint` not throwing an error on numbers that are too big. 437- Fixed `readUInt64` always failing. 438- Fixed writing to an empty buffer always failing. 439