README.md
1# Parallel Zstandard (PZstandard)
2
3Parallel Zstandard is a Pigz-like tool for Zstandard.
4It provides Zstandard format compatible compression and decompression that is able to utilize multiple cores.
5It breaks the input up into equal sized chunks and compresses each chunk independently into a Zstandard frame.
6It then concatenates the frames together to produce the final compressed output.
7Pzstandard will write a 12 byte header for each frame that is a skippable frame in the Zstandard format, which tells PZstandard the size of the next compressed frame.
8PZstandard supports parallel decompression of files compressed with PZstandard.
9When decompressing files compressed with Zstandard, PZstandard does IO in one thread, and decompression in another.
10
11## Usage
12
13PZstandard supports the same command line interface as Zstandard, but also provides the `-p` option to specify the number of threads.
14Dictionary mode is not currently supported.
15
16Basic usage
17
18 pzstd input-file -o output-file -p num-threads -# # Compression
19 pzstd -d input-file -o output-file -p num-threads # Decompression
20
21PZstandard also supports piping and fifo pipes
22
23 cat input-file | pzstd -p num-threads -# -c > /dev/null
24
25For more options
26
27 pzstd --help
28
29PZstandard tries to pick a smart default number of threads if not specified (displayed in `pzstd --help`).
30If this number is not suitable, during compilation you can define `PZSTD_NUM_THREADS` to the number of threads you prefer.
31
32## Benchmarks
33
34As a reference, PZstandard and Pigz were compared on an Intel Core i7 @ 3.1 GHz, each using 4 threads, with the [Silesia compression corpus](http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia).
35
36Compression Speed vs Ratio with 4 Threads | Decompression Speed with 4 Threads
37------------------------------------------|-----------------------------------
38 | 
39
40The test procedure was to run each of the following commands 2 times for each compression level, and take the minimum time.
41
42 time pzstd -# -p 4 -c silesia.tar > silesia.tar.zst
43 time pzstd -d -p 4 -c silesia.tar.zst > /dev/null
44
45 time pigz -# -p 4 -k -c silesia.tar > silesia.tar.gz
46 time pigz -d -p 4 -k -c silesia.tar.gz > /dev/null
47
48PZstandard was tested using compression levels 1-19, and Pigz was tested using compression levels 1-9.
49Pigz cannot do parallel decompression, it simply does each of reading, decompression, and writing on separate threads.
50
51## Tests
52
53Tests require that you have [gtest](https://github.com/google/googletest) installed.
54Set `GTEST_INC` and `GTEST_LIB` in `Makefile` to specify the location of the gtest headers and libraries.
55Alternatively, run `make googletest`, which will clone googletest and build it.
56Run `make tests && make check` to run tests.
57
