README.md
1# zstd
2
3[Zstandard](https://facebook.github.io/zstd/) is a real-time compression algorithm, providing high compression ratios.
4It offers a very wide range of compression / speed trade-off, while being backed by a very fast decoder.
5A high performance compression algorithm is implemented. For now focused on speed.
6
7This package provides [compression](#Compressor) to and [decompression](#Decompressor) of Zstandard content.
8
9This package is pure Go and without use of "unsafe".
10
11The `zstd` package is provided as open source software using a Go standard license.
12
13Currently the package is heavily optimized for 64 bit processors and will be significantly slower on 32 bit processors.
14
15## Installation
16
17Install using `go get -u github.com/klauspost/compress`. The package is located in `github.com/klauspost/compress/zstd`.
18
19Godoc Documentation: https://godoc.org/github.com/klauspost/compress/zstd
20
21
22## Compressor
23
24### Status:
25
26STABLE - there may always be subtle bugs, a wide variety of content has been tested and the library is actively
27used by several projects. This library is being [fuzz-tested](https://github.com/klauspost/compress-fuzz) for all updates.
28
29There may still be specific combinations of data types/size/settings that could lead to edge cases,
30so as always, testing is recommended.
31
32For now, a high speed (fastest) and medium-fast (default) compressor has been implemented.
33
34* The "Fastest" compression ratio is roughly equivalent to zstd level 1.
35* The "Default" compression ratio is roughly equivalent to zstd level 3 (default).
36* The "Better" compression ratio is roughly equivalent to zstd level 7.
37* The "Best" compression ratio is roughly equivalent to zstd level 11.
38
39In terms of speed, it is typically 2x as fast as the stdlib deflate/gzip in its fastest mode.
40The compression ratio compared to stdlib is around level 3, but usually 3x as fast.
41
42
43### Usage
44
45An Encoder can be used for either compressing a stream via the
46`io.WriteCloser` interface supported by the Encoder or as multiple independent
47tasks via the `EncodeAll` function.
48Smaller encodes are encouraged to use the EncodeAll function.
49Use `NewWriter` to create a new instance that can be used for both.
50
51To create a writer with default options, do like this:
52
53```Go
54// Compress input to output.
55func Compress(in io.Reader, out io.Writer) error {
56 enc, err := zstd.NewWriter(out)
57 if err != nil {
58 return err
59 }
60 _, err = io.Copy(enc, in)
61 if err != nil {
62 enc.Close()
63 return err
64 }
65 return enc.Close()
66}
67```
68
69Now you can encode by writing data to `enc`. The output will be finished writing when `Close()` is called.
70Even if your encode fails, you should still call `Close()` to release any resources that may be held up.
71
72The above is fine for big encodes. However, whenever possible try to *reuse* the writer.
73
74To reuse the encoder, you can use the `Reset(io.Writer)` function to change to another output.
75This will allow the encoder to reuse all resources and avoid wasteful allocations.
76
77Currently stream encoding has 'light' concurrency, meaning up to 2 goroutines can be working on part
78of a stream. This is independent of the `WithEncoderConcurrency(n)`, but that is likely to change
79in the future. So if you want to limit concurrency for future updates, specify the concurrency
80you would like.
81
82You can specify your desired compression level using `WithEncoderLevel()` option. Currently only pre-defined
83compression settings can be specified.
84
85#### Future Compatibility Guarantees
86
87This will be an evolving project. When using this package it is important to note that both the compression efficiency and speed may change.
88
89The goal will be to keep the default efficiency at the default zstd (level 3).
90However the encoding should never be assumed to remain the same,
91and you should not use hashes of compressed output for similarity checks.
92
93The Encoder can be assumed to produce the same output from the exact same code version.
94However, the may be modes in the future that break this,
95although they will not be enabled without an explicit option.
96
97This encoder is not designed to (and will probably never) output the exact same bitstream as the reference encoder.
98
99Also note, that the cgo decompressor currently does not [report all errors on invalid input](https://github.com/DataDog/zstd/issues/59),
100[omits error checks](https://github.com/DataDog/zstd/issues/61), [ignores checksums](https://github.com/DataDog/zstd/issues/43)
101and seems to ignore concatenated streams, even though [it is part of the spec](https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#frames).
102
103#### Blocks
104
105For compressing small blocks, the returned encoder has a function called `EncodeAll(src, dst []byte) []byte`.
106
107`EncodeAll` will encode all input in src and append it to dst.
108This function can be called concurrently, but each call will only run on a single goroutine.
109
110Encoded blocks can be concatenated and the result will be the combined input stream.
111Data compressed with EncodeAll can be decoded with the Decoder, using either a stream or `DecodeAll`.
112
113Especially when encoding blocks you should take special care to reuse the encoder.
114This will effectively make it run without allocations after a warmup period.
115To make it run completely without allocations, supply a destination buffer with space for all content.
116
117```Go
118import "github.com/klauspost/compress/zstd"
119
120// Create a writer that caches compressors.
121// For this operation type we supply a nil Reader.
122var encoder, _ = zstd.NewWriter(nil)
123
124// Compress a buffer.
125// If you have a destination buffer, the allocation in the call can also be eliminated.
126func Compress(src []byte) []byte {
127 return encoder.EncodeAll(src, make([]byte, 0, len(src)))
128}
129```
130
131You can control the maximum number of concurrent encodes using the `WithEncoderConcurrency(n)`
132option when creating the writer.
133
134Using the Encoder for both a stream and individual blocks concurrently is safe.
135
136### Performance
137
138I have collected some speed examples to compare speed and compression against other compressors.
139
140* `file` is the input file.
141* `out` is the compressor used. `zskp` is this package. `zstd` is the Datadog cgo library. `gzstd/gzkp` is gzip standard and this library.
142* `level` is the compression level used. For `zskp` level 1 is "fastest", level 2 is "default"; 3 is "better", 4 is "best".
143* `insize`/`outsize` is the input/output size.
144* `millis` is the number of milliseconds used for compression.
145* `mb/s` is megabytes (2^20 bytes) per second.
146
147```
148Silesia Corpus:
149http://sun.aei.polsl.pl/~sdeor/corpus/silesia.zip
150
151This package:
152file out level insize outsize millis mb/s
153silesia.tar zskp 1 211947520 73101992 643 313.87
154silesia.tar zskp 2 211947520 67504318 969 208.38
155silesia.tar zskp 3 211947520 65177448 1899 106.44
156silesia.tar zskp 4 211947520 61381950 8115 24.91
157
158cgo zstd:
159silesia.tar zstd 1 211947520 73605392 543 371.56
160silesia.tar zstd 3 211947520 66793289 864 233.68
161silesia.tar zstd 6 211947520 62916450 1913 105.66
162silesia.tar zstd 9 211947520 60212393 5063 39.92
163
164gzip, stdlib/this package:
165silesia.tar gzstd 1 211947520 80007735 1654 122.21
166silesia.tar gzkp 1 211947520 80369488 1168 173.06
167
168GOB stream of binary data. Highly compressible.
169https://files.klauspost.com/compress/gob-stream.7z
170
171file out level insize outsize millis mb/s
172gob-stream zskp 1 1911399616 235022249 3088 590.30
173gob-stream zskp 2 1911399616 205669791 3786 481.34
174gob-stream zskp 3 1911399616 185792019 9324 195.48
175gob-stream zskp 4 1911399616 171537212 32113 56.76
176gob-stream zstd 1 1911399616 249810424 2637 691.26
177gob-stream zstd 3 1911399616 208192146 3490 522.31
178gob-stream zstd 6 1911399616 193632038 6687 272.56
179gob-stream zstd 9 1911399616 177620386 16175 112.70
180gob-stream gzstd 1 1911399616 357382641 10251 177.82
181gob-stream gzkp 1 1911399616 362156523 5695 320.08
182
183The test data for the Large Text Compression Benchmark is the first
18410^9 bytes of the English Wikipedia dump on Mar. 3, 2006.
185http://mattmahoney.net/dc/textdata.html
186
187file out level insize outsize millis mb/s
188enwik9 zskp 1 1000000000 343848582 3609 264.18
189enwik9 zskp 2 1000000000 317276632 5746 165.97
190enwik9 zskp 3 1000000000 294540704 11725 81.34
191enwik9 zskp 4 1000000000 276609671 44029 21.66
192enwik9 zstd 1 1000000000 358072021 3110 306.65
193enwik9 zstd 3 1000000000 313734672 4784 199.35
194enwik9 zstd 6 1000000000 295138875 10290 92.68
195enwik9 zstd 9 1000000000 278348700 28549 33.40
196enwik9 gzstd 1 1000000000 382578136 9604 99.30
197enwik9 gzkp 1 1000000000 383825945 6544 145.73
198
199Highly compressible JSON file.
200https://files.klauspost.com/compress/github-june-2days-2019.json.zst
201
202file out level insize outsize millis mb/s
203github-june-2days-2019.json zskp 1 6273951764 699045015 10620 563.40
204github-june-2days-2019.json zskp 2 6273951764 617881763 11687 511.96
205github-june-2days-2019.json zskp 3 6273951764 537511906 29252 204.54
206github-june-2days-2019.json zskp 4 6273951764 512796117 97791 61.18
207github-june-2days-2019.json zstd 1 6273951764 766284037 8450 708.00
208github-june-2days-2019.json zstd 3 6273951764 661889476 10927 547.57
209github-june-2days-2019.json zstd 6 6273951764 642756859 22996 260.18
210github-june-2days-2019.json zstd 9 6273951764 601974523 52413 114.16
211github-june-2days-2019.json gzstd 1 6273951764 1164400847 29948 199.79
212github-june-2days-2019.json gzkp 1 6273951764 1128755542 19236 311.03
213
214VM Image, Linux mint with a few installed applications:
215https://files.klauspost.com/compress/rawstudio-mint14.7z
216
217file out level insize outsize millis mb/s
218rawstudio-mint14.tar zskp 1 8558382592 3667489370 20210 403.84
219rawstudio-mint14.tar zskp 2 8558382592 3364592300 31873 256.07
220rawstudio-mint14.tar zskp 3 8558382592 3224594213 71751 113.75
221rawstudio-mint14.tar zskp 4 8558382592 3027332295 486243 16.79
222rawstudio-mint14.tar zstd 1 8558382592 3609250104 17136 476.27
223rawstudio-mint14.tar zstd 3 8558382592 3341679997 29262 278.92
224rawstudio-mint14.tar zstd 6 8558382592 3235846406 77904 104.77
225rawstudio-mint14.tar zstd 9 8558382592 3160778861 140946 57.91
226rawstudio-mint14.tar gzstd 1 8558382592 3926257486 57722 141.40
227rawstudio-mint14.tar gzkp 1 8558382592 3970463184 41749 195.49
228
229CSV data:
230https://files.klauspost.com/compress/nyc-taxi-data-10M.csv.zst
231
232file out level insize outsize millis mb/s
233nyc-taxi-data-10M.csv zskp 1 3325605752 641339945 8925 355.35
234nyc-taxi-data-10M.csv zskp 2 3325605752 591748091 11268 281.44
235nyc-taxi-data-10M.csv zskp 3 3325605752 538490114 19880 159.53
236nyc-taxi-data-10M.csv zskp 4 3325605752 495986829 89368 35.49
237nyc-taxi-data-10M.csv zstd 1 3325605752 687399637 8233 385.18
238nyc-taxi-data-10M.csv zstd 3 3325605752 598514411 10065 315.07
239nyc-taxi-data-10M.csv zstd 6 3325605752 570522953 20038 158.27
240nyc-taxi-data-10M.csv zstd 9 3325605752 517554797 64565 49.12
241nyc-taxi-data-10M.csv gzstd 1 3325605752 928656485 23876 132.83
242nyc-taxi-data-10M.csv gzkp 1 3325605752 924718719 16388 193.53
243```
244
245## Decompressor
246
247Staus: STABLE - there may still be subtle bugs, but a wide variety of content has been tested.
248
249This library is being continuously [fuzz-tested](https://github.com/klauspost/compress-fuzz),
250kindly supplied by [fuzzit.dev](https://fuzzit.dev/).
251The main purpose of the fuzz testing is to ensure that it is not possible to crash the decoder,
252or run it past its limits with ANY input provided.
253
254### Usage
255
256The package has been designed for two main usages, big streams of data and smaller in-memory buffers.
257There are two main usages of the package for these. Both of them are accessed by creating a `Decoder`.
258
259For streaming use a simple setup could look like this:
260
261```Go
262import "github.com/klauspost/compress/zstd"
263
264func Decompress(in io.Reader, out io.Writer) error {
265 d, err := zstd.NewReader(in)
266 if err != nil {
267 return err
268 }
269 defer d.Close()
270
271 // Copy content...
272 _, err = io.Copy(out, d)
273 return err
274}
275```
276
277It is important to use the "Close" function when you no longer need the Reader to stop running goroutines.
278See "Allocation-less operation" below.
279
280For decoding buffers, it could look something like this:
281
282```Go
283import "github.com/klauspost/compress/zstd"
284
285// Create a reader that caches decompressors.
286// For this operation type we supply a nil Reader.
287var decoder, _ = zstd.NewReader(nil)
288
289// Decompress a buffer. We don't supply a destination buffer,
290// so it will be allocated by the decoder.
291func Decompress(src []byte) ([]byte, error) {
292 return decoder.DecodeAll(src, nil)
293}
294```
295
296Both of these cases should provide the functionality needed.
297The decoder can be used for *concurrent* decompression of multiple buffers.
298It will only allow a certain number of concurrent operations to run.
299To tweak that yourself use the `WithDecoderConcurrency(n)` option when creating the decoder.
300
301### Dictionaries
302
303Data compressed with [dictionaries](https://github.com/facebook/zstd#the-case-for-small-data-compression) can be decompressed.
304
305Dictionaries are added individually to Decoders.
306Dictionaries are generated by the `zstd --train` command and contains an initial state for the decoder.
307To add a dictionary use the `WithDecoderDicts(dicts ...[]byte)` option with the dictionary data.
308Several dictionaries can be added at once.
309
310The dictionary will be used automatically for the data that specifies them.
311A re-used Decoder will still contain the dictionaries registered.
312
313When registering multiple dictionaries with the same ID, the last one will be used.
314
315It is possible to use dictionaries when compressing data.
316
317To enable a dictionary use `WithEncoderDict(dict []byte)`. Here only one dictionary will be used
318and it will likely be used even if it doesn't improve compression.
319
320The used dictionary must be used to decompress the content.
321
322For any real gains, the dictionary should be built with similar data.
323If an unsuitable dictionary is used the output may be slightly larger than using no dictionary.
324Use the [zstd commandline tool](https://github.com/facebook/zstd/releases) to build a dictionary from sample data.
325For information see [zstd dictionary information](https://github.com/facebook/zstd#the-case-for-small-data-compression).
326
327For now there is a fixed startup performance penalty for compressing content with dictionaries.
328This will likely be improved over time. Just be aware to test performance when implementing.
329
330### Allocation-less operation
331
332The decoder has been designed to operate without allocations after a warmup.
333
334This means that you should *store* the decoder for best performance.
335To re-use a stream decoder, use the `Reset(r io.Reader) error` to switch to another stream.
336A decoder can safely be re-used even if the previous stream failed.
337
338To release the resources, you must call the `Close()` function on a decoder.
339After this it can *no longer be reused*, but all running goroutines will be stopped.
340So you *must* use this if you will no longer need the Reader.
341
342For decompressing smaller buffers a single decoder can be used.
343When decoding buffers, you can supply a destination slice with length 0 and your expected capacity.
344In this case no unneeded allocations should be made.
345
346### Concurrency
347
348The buffer decoder does everything on the same goroutine and does nothing concurrently.
349It can however decode several buffers concurrently. Use `WithDecoderConcurrency(n)` to limit that.
350
351The stream decoder operates on
352
353* One goroutine reads input and splits the input to several block decoders.
354* A number of decoders will decode blocks.
355* A goroutine coordinates these blocks and sends history from one to the next.
356
357So effectively this also means the decoder will "read ahead" and prepare data to always be available for output.
358
359Since "blocks" are quite dependent on the output of the previous block stream decoding will only have limited concurrency.
360
361In practice this means that concurrency is often limited to utilizing about 2 cores effectively.
362
363
364### Benchmarks
365
366These are some examples of performance compared to [datadog cgo library](https://github.com/DataDog/zstd).
367
368The first two are streaming decodes and the last are smaller inputs.
369
370```
371BenchmarkDecoderSilesia-8 3 385000067 ns/op 550.51 MB/s 5498 B/op 8 allocs/op
372BenchmarkDecoderSilesiaCgo-8 6 197666567 ns/op 1072.25 MB/s 270672 B/op 8 allocs/op
373
374BenchmarkDecoderEnwik9-8 1 2027001600 ns/op 493.34 MB/s 10496 B/op 18 allocs/op
375BenchmarkDecoderEnwik9Cgo-8 2 979499200 ns/op 1020.93 MB/s 270672 B/op 8 allocs/op
376
377Concurrent performance:
378
379BenchmarkDecoder_DecodeAllParallel/kppkn.gtb.zst-16 28915 42469 ns/op 4340.07 MB/s 114 B/op 0 allocs/op
380BenchmarkDecoder_DecodeAllParallel/geo.protodata.zst-16 116505 9965 ns/op 11900.16 MB/s 16 B/op 0 allocs/op
381BenchmarkDecoder_DecodeAllParallel/plrabn12.txt.zst-16 8952 134272 ns/op 3588.70 MB/s 915 B/op 0 allocs/op
382BenchmarkDecoder_DecodeAllParallel/lcet10.txt.zst-16 11820 102538 ns/op 4161.90 MB/s 594 B/op 0 allocs/op
383BenchmarkDecoder_DecodeAllParallel/asyoulik.txt.zst-16 34782 34184 ns/op 3661.88 MB/s 60 B/op 0 allocs/op
384BenchmarkDecoder_DecodeAllParallel/alice29.txt.zst-16 27712 43447 ns/op 3500.58 MB/s 99 B/op 0 allocs/op
385BenchmarkDecoder_DecodeAllParallel/html_x_4.zst-16 62826 18750 ns/op 21845.10 MB/s 104 B/op 0 allocs/op
386BenchmarkDecoder_DecodeAllParallel/paper-100k.pdf.zst-16 631545 1794 ns/op 57078.74 MB/s 2 B/op 0 allocs/op
387BenchmarkDecoder_DecodeAllParallel/fireworks.jpeg.zst-16 1690140 712 ns/op 172938.13 MB/s 1 B/op 0 allocs/op
388BenchmarkDecoder_DecodeAllParallel/urls.10K.zst-16 10432 113593 ns/op 6180.73 MB/s 1143 B/op 0 allocs/op
389BenchmarkDecoder_DecodeAllParallel/html.zst-16 113206 10671 ns/op 9596.27 MB/s 15 B/op 0 allocs/op
390BenchmarkDecoder_DecodeAllParallel/comp-data.bin.zst-16 1530615 779 ns/op 5229.49 MB/s 0 B/op 0 allocs/op
391
392BenchmarkDecoder_DecodeAllParallelCgo/kppkn.gtb.zst-16 65217 16192 ns/op 11383.34 MB/s 46 B/op 0 allocs/op
393BenchmarkDecoder_DecodeAllParallelCgo/geo.protodata.zst-16 292671 4039 ns/op 29363.19 MB/s 6 B/op 0 allocs/op
394BenchmarkDecoder_DecodeAllParallelCgo/plrabn12.txt.zst-16 26314 46021 ns/op 10470.43 MB/s 293 B/op 0 allocs/op
395BenchmarkDecoder_DecodeAllParallelCgo/lcet10.txt.zst-16 33897 34900 ns/op 12227.96 MB/s 205 B/op 0 allocs/op
396BenchmarkDecoder_DecodeAllParallelCgo/asyoulik.txt.zst-16 104348 11433 ns/op 10949.01 MB/s 20 B/op 0 allocs/op
397BenchmarkDecoder_DecodeAllParallelCgo/alice29.txt.zst-16 75949 15510 ns/op 9805.60 MB/s 32 B/op 0 allocs/op
398BenchmarkDecoder_DecodeAllParallelCgo/html_x_4.zst-16 173910 6756 ns/op 60624.29 MB/s 37 B/op 0 allocs/op
399BenchmarkDecoder_DecodeAllParallelCgo/paper-100k.pdf.zst-16 923076 1339 ns/op 76474.87 MB/s 1 B/op 0 allocs/op
400BenchmarkDecoder_DecodeAllParallelCgo/fireworks.jpeg.zst-16 922920 1351 ns/op 91102.57 MB/s 2 B/op 0 allocs/op
401BenchmarkDecoder_DecodeAllParallelCgo/urls.10K.zst-16 27649 43618 ns/op 16096.19 MB/s 407 B/op 0 allocs/op
402BenchmarkDecoder_DecodeAllParallelCgo/html.zst-16 279073 4160 ns/op 24614.18 MB/s 6 B/op 0 allocs/op
403BenchmarkDecoder_DecodeAllParallelCgo/comp-data.bin.zst-16 749938 1579 ns/op 2581.71 MB/s 0 B/op 0 allocs/op
404```
405
406This reflects the performance around May 2020, but this may be out of date.
407
408# Contributions
409
410Contributions are always welcome.
411For new features/fixes, remember to add tests and for performance enhancements include benchmarks.
412
413For sending files for reproducing errors use a service like [goobox](https://goobox.io/#/upload) or similar to share your files.
414
415For general feedback and experience reports, feel free to open an issue or write me on [Twitter](https://twitter.com/sh0dan).
416
417This package includes the excellent [`github.com/cespare/xxhash`](https://github.com/cespare/xxhash) package Copyright (c) 2016 Caleb Spare.
418