1# zstd 2 3[Zstandard](https://facebook.github.io/zstd/) is a real-time compression algorithm, providing high compression ratios. 4It offers a very wide range of compression / speed trade-off, while being backed by a very fast decoder. 5A high performance compression algorithm is implemented. For now focused on speed. 6 7This package provides [compression](#Compressor) to and [decompression](#Decompressor) of Zstandard content. 8 9This package is pure Go and without use of "unsafe". 10 11The `zstd` package is provided as open source software using a Go standard license. 12 13Currently the package is heavily optimized for 64 bit processors and will be significantly slower on 32 bit processors. 14 15## Installation 16 17Install using `go get -u github.com/klauspost/compress`. The package is located in `github.com/klauspost/compress/zstd`. 18 19Godoc Documentation: https://godoc.org/github.com/klauspost/compress/zstd 20 21 22## Compressor 23 24### Status: 25 26STABLE - there may always be subtle bugs, a wide variety of content has been tested and the library is actively 27used by several projects. This library is being [fuzz-tested](https://github.com/klauspost/compress-fuzz) for all updates. 28 29There may still be specific combinations of data types/size/settings that could lead to edge cases, 30so as always, testing is recommended. 31 32For now, a high speed (fastest) and medium-fast (default) compressor has been implemented. 33 34* The "Fastest" compression ratio is roughly equivalent to zstd level 1. 35* The "Default" compression ratio is roughly equivalent to zstd level 3 (default). 36* The "Better" compression ratio is roughly equivalent to zstd level 7. 37* The "Best" compression ratio is roughly equivalent to zstd level 11. 38 39In terms of speed, it is typically 2x as fast as the stdlib deflate/gzip in its fastest mode. 40The compression ratio compared to stdlib is around level 3, but usually 3x as fast. 41 42 43### Usage 44 45An Encoder can be used for either compressing a stream via the 46`io.WriteCloser` interface supported by the Encoder or as multiple independent 47tasks via the `EncodeAll` function. 48Smaller encodes are encouraged to use the EncodeAll function. 49Use `NewWriter` to create a new instance that can be used for both. 50 51To create a writer with default options, do like this: 52 53```Go 54// Compress input to output. 55func Compress(in io.Reader, out io.Writer) error { 56 enc, err := zstd.NewWriter(out) 57 if err != nil { 58 return err 59 } 60 _, err = io.Copy(enc, in) 61 if err != nil { 62 enc.Close() 63 return err 64 } 65 return enc.Close() 66} 67``` 68 69Now you can encode by writing data to `enc`. The output will be finished writing when `Close()` is called. 70Even if your encode fails, you should still call `Close()` to release any resources that may be held up. 71 72The above is fine for big encodes. However, whenever possible try to *reuse* the writer. 73 74To reuse the encoder, you can use the `Reset(io.Writer)` function to change to another output. 75This will allow the encoder to reuse all resources and avoid wasteful allocations. 76 77Currently stream encoding has 'light' concurrency, meaning up to 2 goroutines can be working on part 78of a stream. This is independent of the `WithEncoderConcurrency(n)`, but that is likely to change 79in the future. So if you want to limit concurrency for future updates, specify the concurrency 80you would like. 81 82You can specify your desired compression level using `WithEncoderLevel()` option. Currently only pre-defined 83compression settings can be specified. 84 85#### Future Compatibility Guarantees 86 87This will be an evolving project. When using this package it is important to note that both the compression efficiency and speed may change. 88 89The goal will be to keep the default efficiency at the default zstd (level 3). 90However the encoding should never be assumed to remain the same, 91and you should not use hashes of compressed output for similarity checks. 92 93The Encoder can be assumed to produce the same output from the exact same code version. 94However, the may be modes in the future that break this, 95although they will not be enabled without an explicit option. 96 97This encoder is not designed to (and will probably never) output the exact same bitstream as the reference encoder. 98 99Also note, that the cgo decompressor currently does not [report all errors on invalid input](https://github.com/DataDog/zstd/issues/59), 100[omits error checks](https://github.com/DataDog/zstd/issues/61), [ignores checksums](https://github.com/DataDog/zstd/issues/43) 101and seems to ignore concatenated streams, even though [it is part of the spec](https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#frames). 102 103#### Blocks 104 105For compressing small blocks, the returned encoder has a function called `EncodeAll(src, dst []byte) []byte`. 106 107`EncodeAll` will encode all input in src and append it to dst. 108This function can be called concurrently, but each call will only run on a single goroutine. 109 110Encoded blocks can be concatenated and the result will be the combined input stream. 111Data compressed with EncodeAll can be decoded with the Decoder, using either a stream or `DecodeAll`. 112 113Especially when encoding blocks you should take special care to reuse the encoder. 114This will effectively make it run without allocations after a warmup period. 115To make it run completely without allocations, supply a destination buffer with space for all content. 116 117```Go 118import "github.com/klauspost/compress/zstd" 119 120// Create a writer that caches compressors. 121// For this operation type we supply a nil Reader. 122var encoder, _ = zstd.NewWriter(nil) 123 124// Compress a buffer. 125// If you have a destination buffer, the allocation in the call can also be eliminated. 126func Compress(src []byte) []byte { 127 return encoder.EncodeAll(src, make([]byte, 0, len(src))) 128} 129``` 130 131You can control the maximum number of concurrent encodes using the `WithEncoderConcurrency(n)` 132option when creating the writer. 133 134Using the Encoder for both a stream and individual blocks concurrently is safe. 135 136### Performance 137 138I have collected some speed examples to compare speed and compression against other compressors. 139 140* `file` is the input file. 141* `out` is the compressor used. `zskp` is this package. `zstd` is the Datadog cgo library. `gzstd/gzkp` is gzip standard and this library. 142* `level` is the compression level used. For `zskp` level 1 is "fastest", level 2 is "default"; 3 is "better", 4 is "best". 143* `insize`/`outsize` is the input/output size. 144* `millis` is the number of milliseconds used for compression. 145* `mb/s` is megabytes (2^20 bytes) per second. 146 147``` 148Silesia Corpus: 149http://sun.aei.polsl.pl/~sdeor/corpus/silesia.zip 150 151This package: 152file out level insize outsize millis mb/s 153silesia.tar zskp 1 211947520 73101992 643 313.87 154silesia.tar zskp 2 211947520 67504318 969 208.38 155silesia.tar zskp 3 211947520 65177448 1899 106.44 156silesia.tar zskp 4 211947520 61381950 8115 24.91 157 158cgo zstd: 159silesia.tar zstd 1 211947520 73605392 543 371.56 160silesia.tar zstd 3 211947520 66793289 864 233.68 161silesia.tar zstd 6 211947520 62916450 1913 105.66 162silesia.tar zstd 9 211947520 60212393 5063 39.92 163 164gzip, stdlib/this package: 165silesia.tar gzstd 1 211947520 80007735 1654 122.21 166silesia.tar gzkp 1 211947520 80369488 1168 173.06 167 168GOB stream of binary data. Highly compressible. 169https://files.klauspost.com/compress/gob-stream.7z 170 171file out level insize outsize millis mb/s 172gob-stream zskp 1 1911399616 235022249 3088 590.30 173gob-stream zskp 2 1911399616 205669791 3786 481.34 174gob-stream zskp 3 1911399616 185792019 9324 195.48 175gob-stream zskp 4 1911399616 171537212 32113 56.76 176gob-stream zstd 1 1911399616 249810424 2637 691.26 177gob-stream zstd 3 1911399616 208192146 3490 522.31 178gob-stream zstd 6 1911399616 193632038 6687 272.56 179gob-stream zstd 9 1911399616 177620386 16175 112.70 180gob-stream gzstd 1 1911399616 357382641 10251 177.82 181gob-stream gzkp 1 1911399616 362156523 5695 320.08 182 183The test data for the Large Text Compression Benchmark is the first 18410^9 bytes of the English Wikipedia dump on Mar. 3, 2006. 185http://mattmahoney.net/dc/textdata.html 186 187file out level insize outsize millis mb/s 188enwik9 zskp 1 1000000000 343848582 3609 264.18 189enwik9 zskp 2 1000000000 317276632 5746 165.97 190enwik9 zskp 3 1000000000 294540704 11725 81.34 191enwik9 zskp 4 1000000000 276609671 44029 21.66 192enwik9 zstd 1 1000000000 358072021 3110 306.65 193enwik9 zstd 3 1000000000 313734672 4784 199.35 194enwik9 zstd 6 1000000000 295138875 10290 92.68 195enwik9 zstd 9 1000000000 278348700 28549 33.40 196enwik9 gzstd 1 1000000000 382578136 9604 99.30 197enwik9 gzkp 1 1000000000 383825945 6544 145.73 198 199Highly compressible JSON file. 200https://files.klauspost.com/compress/github-june-2days-2019.json.zst 201 202file out level insize outsize millis mb/s 203github-june-2days-2019.json zskp 1 6273951764 699045015 10620 563.40 204github-june-2days-2019.json zskp 2 6273951764 617881763 11687 511.96 205github-june-2days-2019.json zskp 3 6273951764 537511906 29252 204.54 206github-june-2days-2019.json zskp 4 6273951764 512796117 97791 61.18 207github-june-2days-2019.json zstd 1 6273951764 766284037 8450 708.00 208github-june-2days-2019.json zstd 3 6273951764 661889476 10927 547.57 209github-june-2days-2019.json zstd 6 6273951764 642756859 22996 260.18 210github-june-2days-2019.json zstd 9 6273951764 601974523 52413 114.16 211github-june-2days-2019.json gzstd 1 6273951764 1164400847 29948 199.79 212github-june-2days-2019.json gzkp 1 6273951764 1128755542 19236 311.03 213 214VM Image, Linux mint with a few installed applications: 215https://files.klauspost.com/compress/rawstudio-mint14.7z 216 217file out level insize outsize millis mb/s 218rawstudio-mint14.tar zskp 1 8558382592 3667489370 20210 403.84 219rawstudio-mint14.tar zskp 2 8558382592 3364592300 31873 256.07 220rawstudio-mint14.tar zskp 3 8558382592 3224594213 71751 113.75 221rawstudio-mint14.tar zskp 4 8558382592 3027332295 486243 16.79 222rawstudio-mint14.tar zstd 1 8558382592 3609250104 17136 476.27 223rawstudio-mint14.tar zstd 3 8558382592 3341679997 29262 278.92 224rawstudio-mint14.tar zstd 6 8558382592 3235846406 77904 104.77 225rawstudio-mint14.tar zstd 9 8558382592 3160778861 140946 57.91 226rawstudio-mint14.tar gzstd 1 8558382592 3926257486 57722 141.40 227rawstudio-mint14.tar gzkp 1 8558382592 3970463184 41749 195.49 228 229CSV data: 230https://files.klauspost.com/compress/nyc-taxi-data-10M.csv.zst 231 232file out level insize outsize millis mb/s 233nyc-taxi-data-10M.csv zskp 1 3325605752 641339945 8925 355.35 234nyc-taxi-data-10M.csv zskp 2 3325605752 591748091 11268 281.44 235nyc-taxi-data-10M.csv zskp 3 3325605752 538490114 19880 159.53 236nyc-taxi-data-10M.csv zskp 4 3325605752 495986829 89368 35.49 237nyc-taxi-data-10M.csv zstd 1 3325605752 687399637 8233 385.18 238nyc-taxi-data-10M.csv zstd 3 3325605752 598514411 10065 315.07 239nyc-taxi-data-10M.csv zstd 6 3325605752 570522953 20038 158.27 240nyc-taxi-data-10M.csv zstd 9 3325605752 517554797 64565 49.12 241nyc-taxi-data-10M.csv gzstd 1 3325605752 928656485 23876 132.83 242nyc-taxi-data-10M.csv gzkp 1 3325605752 924718719 16388 193.53 243``` 244 245## Decompressor 246 247Staus: STABLE - there may still be subtle bugs, but a wide variety of content has been tested. 248 249This library is being continuously [fuzz-tested](https://github.com/klauspost/compress-fuzz), 250kindly supplied by [fuzzit.dev](https://fuzzit.dev/). 251The main purpose of the fuzz testing is to ensure that it is not possible to crash the decoder, 252or run it past its limits with ANY input provided. 253 254### Usage 255 256The package has been designed for two main usages, big streams of data and smaller in-memory buffers. 257There are two main usages of the package for these. Both of them are accessed by creating a `Decoder`. 258 259For streaming use a simple setup could look like this: 260 261```Go 262import "github.com/klauspost/compress/zstd" 263 264func Decompress(in io.Reader, out io.Writer) error { 265 d, err := zstd.NewReader(in) 266 if err != nil { 267 return err 268 } 269 defer d.Close() 270 271 // Copy content... 272 _, err = io.Copy(out, d) 273 return err 274} 275``` 276 277It is important to use the "Close" function when you no longer need the Reader to stop running goroutines. 278See "Allocation-less operation" below. 279 280For decoding buffers, it could look something like this: 281 282```Go 283import "github.com/klauspost/compress/zstd" 284 285// Create a reader that caches decompressors. 286// For this operation type we supply a nil Reader. 287var decoder, _ = zstd.NewReader(nil) 288 289// Decompress a buffer. We don't supply a destination buffer, 290// so it will be allocated by the decoder. 291func Decompress(src []byte) ([]byte, error) { 292 return decoder.DecodeAll(src, nil) 293} 294``` 295 296Both of these cases should provide the functionality needed. 297The decoder can be used for *concurrent* decompression of multiple buffers. 298It will only allow a certain number of concurrent operations to run. 299To tweak that yourself use the `WithDecoderConcurrency(n)` option when creating the decoder. 300 301### Dictionaries 302 303Data compressed with [dictionaries](https://github.com/facebook/zstd#the-case-for-small-data-compression) can be decompressed. 304 305Dictionaries are added individually to Decoders. 306Dictionaries are generated by the `zstd --train` command and contains an initial state for the decoder. 307To add a dictionary use the `WithDecoderDicts(dicts ...[]byte)` option with the dictionary data. 308Several dictionaries can be added at once. 309 310The dictionary will be used automatically for the data that specifies them. 311A re-used Decoder will still contain the dictionaries registered. 312 313When registering multiple dictionaries with the same ID, the last one will be used. 314 315It is possible to use dictionaries when compressing data. 316 317To enable a dictionary use `WithEncoderDict(dict []byte)`. Here only one dictionary will be used 318and it will likely be used even if it doesn't improve compression. 319 320The used dictionary must be used to decompress the content. 321 322For any real gains, the dictionary should be built with similar data. 323If an unsuitable dictionary is used the output may be slightly larger than using no dictionary. 324Use the [zstd commandline tool](https://github.com/facebook/zstd/releases) to build a dictionary from sample data. 325For information see [zstd dictionary information](https://github.com/facebook/zstd#the-case-for-small-data-compression). 326 327For now there is a fixed startup performance penalty for compressing content with dictionaries. 328This will likely be improved over time. Just be aware to test performance when implementing. 329 330### Allocation-less operation 331 332The decoder has been designed to operate without allocations after a warmup. 333 334This means that you should *store* the decoder for best performance. 335To re-use a stream decoder, use the `Reset(r io.Reader) error` to switch to another stream. 336A decoder can safely be re-used even if the previous stream failed. 337 338To release the resources, you must call the `Close()` function on a decoder. 339After this it can *no longer be reused*, but all running goroutines will be stopped. 340So you *must* use this if you will no longer need the Reader. 341 342For decompressing smaller buffers a single decoder can be used. 343When decoding buffers, you can supply a destination slice with length 0 and your expected capacity. 344In this case no unneeded allocations should be made. 345 346### Concurrency 347 348The buffer decoder does everything on the same goroutine and does nothing concurrently. 349It can however decode several buffers concurrently. Use `WithDecoderConcurrency(n)` to limit that. 350 351The stream decoder operates on 352 353* One goroutine reads input and splits the input to several block decoders. 354* A number of decoders will decode blocks. 355* A goroutine coordinates these blocks and sends history from one to the next. 356 357So effectively this also means the decoder will "read ahead" and prepare data to always be available for output. 358 359Since "blocks" are quite dependent on the output of the previous block stream decoding will only have limited concurrency. 360 361In practice this means that concurrency is often limited to utilizing about 2 cores effectively. 362 363 364### Benchmarks 365 366These are some examples of performance compared to [datadog cgo library](https://github.com/DataDog/zstd). 367 368The first two are streaming decodes and the last are smaller inputs. 369 370``` 371BenchmarkDecoderSilesia-8 3 385000067 ns/op 550.51 MB/s 5498 B/op 8 allocs/op 372BenchmarkDecoderSilesiaCgo-8 6 197666567 ns/op 1072.25 MB/s 270672 B/op 8 allocs/op 373 374BenchmarkDecoderEnwik9-8 1 2027001600 ns/op 493.34 MB/s 10496 B/op 18 allocs/op 375BenchmarkDecoderEnwik9Cgo-8 2 979499200 ns/op 1020.93 MB/s 270672 B/op 8 allocs/op 376 377Concurrent performance: 378 379BenchmarkDecoder_DecodeAllParallel/kppkn.gtb.zst-16 28915 42469 ns/op 4340.07 MB/s 114 B/op 0 allocs/op 380BenchmarkDecoder_DecodeAllParallel/geo.protodata.zst-16 116505 9965 ns/op 11900.16 MB/s 16 B/op 0 allocs/op 381BenchmarkDecoder_DecodeAllParallel/plrabn12.txt.zst-16 8952 134272 ns/op 3588.70 MB/s 915 B/op 0 allocs/op 382BenchmarkDecoder_DecodeAllParallel/lcet10.txt.zst-16 11820 102538 ns/op 4161.90 MB/s 594 B/op 0 allocs/op 383BenchmarkDecoder_DecodeAllParallel/asyoulik.txt.zst-16 34782 34184 ns/op 3661.88 MB/s 60 B/op 0 allocs/op 384BenchmarkDecoder_DecodeAllParallel/alice29.txt.zst-16 27712 43447 ns/op 3500.58 MB/s 99 B/op 0 allocs/op 385BenchmarkDecoder_DecodeAllParallel/html_x_4.zst-16 62826 18750 ns/op 21845.10 MB/s 104 B/op 0 allocs/op 386BenchmarkDecoder_DecodeAllParallel/paper-100k.pdf.zst-16 631545 1794 ns/op 57078.74 MB/s 2 B/op 0 allocs/op 387BenchmarkDecoder_DecodeAllParallel/fireworks.jpeg.zst-16 1690140 712 ns/op 172938.13 MB/s 1 B/op 0 allocs/op 388BenchmarkDecoder_DecodeAllParallel/urls.10K.zst-16 10432 113593 ns/op 6180.73 MB/s 1143 B/op 0 allocs/op 389BenchmarkDecoder_DecodeAllParallel/html.zst-16 113206 10671 ns/op 9596.27 MB/s 15 B/op 0 allocs/op 390BenchmarkDecoder_DecodeAllParallel/comp-data.bin.zst-16 1530615 779 ns/op 5229.49 MB/s 0 B/op 0 allocs/op 391 392BenchmarkDecoder_DecodeAllParallelCgo/kppkn.gtb.zst-16 65217 16192 ns/op 11383.34 MB/s 46 B/op 0 allocs/op 393BenchmarkDecoder_DecodeAllParallelCgo/geo.protodata.zst-16 292671 4039 ns/op 29363.19 MB/s 6 B/op 0 allocs/op 394BenchmarkDecoder_DecodeAllParallelCgo/plrabn12.txt.zst-16 26314 46021 ns/op 10470.43 MB/s 293 B/op 0 allocs/op 395BenchmarkDecoder_DecodeAllParallelCgo/lcet10.txt.zst-16 33897 34900 ns/op 12227.96 MB/s 205 B/op 0 allocs/op 396BenchmarkDecoder_DecodeAllParallelCgo/asyoulik.txt.zst-16 104348 11433 ns/op 10949.01 MB/s 20 B/op 0 allocs/op 397BenchmarkDecoder_DecodeAllParallelCgo/alice29.txt.zst-16 75949 15510 ns/op 9805.60 MB/s 32 B/op 0 allocs/op 398BenchmarkDecoder_DecodeAllParallelCgo/html_x_4.zst-16 173910 6756 ns/op 60624.29 MB/s 37 B/op 0 allocs/op 399BenchmarkDecoder_DecodeAllParallelCgo/paper-100k.pdf.zst-16 923076 1339 ns/op 76474.87 MB/s 1 B/op 0 allocs/op 400BenchmarkDecoder_DecodeAllParallelCgo/fireworks.jpeg.zst-16 922920 1351 ns/op 91102.57 MB/s 2 B/op 0 allocs/op 401BenchmarkDecoder_DecodeAllParallelCgo/urls.10K.zst-16 27649 43618 ns/op 16096.19 MB/s 407 B/op 0 allocs/op 402BenchmarkDecoder_DecodeAllParallelCgo/html.zst-16 279073 4160 ns/op 24614.18 MB/s 6 B/op 0 allocs/op 403BenchmarkDecoder_DecodeAllParallelCgo/comp-data.bin.zst-16 749938 1579 ns/op 2581.71 MB/s 0 B/op 0 allocs/op 404``` 405 406This reflects the performance around May 2020, but this may be out of date. 407 408# Contributions 409 410Contributions are always welcome. 411For new features/fixes, remember to add tests and for performance enhancements include benchmarks. 412 413For sending files for reproducing errors use a service like [goobox](https://goobox.io/#/upload) or similar to share your files. 414 415For general feedback and experience reports, feel free to open an issue or write me on [Twitter](https://twitter.com/sh0dan). 416 417This package includes the excellent [`github.com/cespare/xxhash`](https://github.com/cespare/xxhash) package Copyright (c) 2016 Caleb Spare. 418