1zstd(1) -- zstd, zstdmt, unzstd, zstdcat - Compress or decompress .zst files 2============================================================================ 3 4SYNOPSIS 5-------- 6 7`zstd` [*OPTIONS*] [-|_INPUT-FILE_] [-o _OUTPUT-FILE_] 8 9`zstdmt` is equivalent to `zstd -T0` 10 11`unzstd` is equivalent to `zstd -d` 12 13`zstdcat` is equivalent to `zstd -dcf` 14 15 16DESCRIPTION 17----------- 18`zstd` is a fast lossless compression algorithm and data compression tool, 19with command line syntax similar to `gzip (1)` and `xz (1)`. 20It is based on the **LZ77** family, with further FSE & huff0 entropy stages. 21`zstd` offers highly configurable compression speed, 22with fast modes at > 200 MB/s per core, 23and strong modes nearing lzma compression ratios. 24It also features a very fast decoder, with speeds > 500 MB/s per core. 25 26`zstd` command line syntax is generally similar to gzip, 27but features the following differences : 28 29 - Source files are preserved by default. 30 It's possible to remove them automatically by using the `--rm` command. 31 - When compressing a single file, `zstd` displays progress notifications 32 and result summary by default. 33 Use `-q` to turn them off. 34 - `zstd` does not accept input from console, 35 but it properly accepts `stdin` when it's not the console. 36 - `zstd` displays a short help page when command line is an error. 37 Use `-q` to turn it off. 38 39`zstd` compresses or decompresses each _file_ according to the selected 40operation mode. 41If no _files_ are given or _file_ is `-`, `zstd` reads from standard input 42and writes the processed data to standard output. 43`zstd` will refuse to write compressed data to standard output 44if it is a terminal : it will display an error message and skip the _file_. 45Similarly, `zstd` will refuse to read compressed data from standard input 46if it is a terminal. 47 48Unless `--stdout` or `-o` is specified, _files_ are written to a new file 49whose name is derived from the source _file_ name: 50 51* When compressing, the suffix `.zst` is appended to the source filename to 52 get the target filename. 53* When decompressing, the `.zst` suffix is removed from the source filename to 54 get the target filename 55 56### Concatenation with .zst files 57It is possible to concatenate `.zst` files as is. 58`zstd` will decompress such files as if they were a single `.zst` file. 59 60OPTIONS 61------- 62 63### Integer suffixes and special values 64In most places where an integer argument is expected, 65an optional suffix is supported to easily indicate large integers. 66There must be no space between the integer and the suffix. 67 68* `KiB`: 69 Multiply the integer by 1,024 (2\^10). 70 `Ki`, `K`, and `KB` are accepted as synonyms for `KiB`. 71* `MiB`: 72 Multiply the integer by 1,048,576 (2\^20). 73 `Mi`, `M`, and `MB` are accepted as synonyms for `MiB`. 74 75### Operation mode 76If multiple operation mode options are given, 77the last one takes effect. 78 79* `-z`, `--compress`: 80 Compress. 81 This is the default operation mode when no operation mode option is specified 82 and no other operation mode is implied from the command name 83 (for example, `unzstd` implies `--decompress`). 84* `-d`, `--decompress`, `--uncompress`: 85 Decompress. 86* `-t`, `--test`: 87 Test the integrity of compressed _files_. 88 This option is equivalent to `--decompress --stdout` except that the 89 decompressed data is discarded instead of being written to standard output. 90 No files are created or removed. 91* `-b#`: 92 Benchmark file(s) using compression level # 93* `--train FILEs`: 94 Use FILEs as a training set to create a dictionary. 95 The training set should contain a lot of small files (> 100). 96* `-l`, `--list`: 97 Display information related to a zstd compressed file, such as size, ratio, and checksum. 98 Some of these fields may not be available. 99 This command can be augmented with the `-v` modifier. 100 101### Operation modifiers 102 103* `-#`: 104 `#` compression level \[1-19] (default: 3) 105* `--fast[=#]`: 106 switch to ultra-fast compression levels. 107 If `=#` is not present, it defaults to `1`. 108 The higher the value, the faster the compression speed, 109 at the cost of some compression ratio. 110 This setting overwrites compression level if one was set previously. 111 Similarly, if a compression level is set after `--fast`, it overrides it. 112* `--ultra`: 113 unlocks high compression levels 20+ (maximum 22), using a lot more memory. 114 Note that decompression will also require more memory when using these levels. 115* `--long[=#]`: 116 enables long distance matching with `#` `windowLog`, if not `#` is not 117 present it defaults to `27`. 118 This increases the window size (`windowLog`) and memory usage for both the 119 compressor and decompressor. 120 This setting is designed to improve the compression ratio for files with 121 long matches at a large distance. 122 123 Note: If `windowLog` is set to larger than 27, `--long=windowLog` or 124 `--memory=windowSize` needs to be passed to the decompressor. 125* `-T#`, `--threads=#`: 126 Compress using `#` working threads (default: 1). 127 If `#` is 0, attempt to detect and use the number of physical CPU cores. 128 In all cases, the nb of threads is capped to ZSTDMT_NBTHREADS_MAX==200. 129 This modifier does nothing if `zstd` is compiled without multithread support. 130* `--single-thread`: 131 Does not spawn a thread for compression, use a single thread for both I/O and compression. 132 In this mode, compression is serialized with I/O, which is slightly slower. 133 (This is different from `-T1`, which spawns 1 compression thread in parallel of I/O). 134 This mode is the only one available when multithread support is disabled. 135 Single-thread mode features lower memory usage. 136 Final compressed result is slightly different from `-T1`. 137* `--adapt[=min=#,max=#]` : 138 `zstd` will dynamically adapt compression level to perceived I/O conditions. 139 Compression level adaptation can be observed live by using command `-v`. 140 Adaptation can be constrained between supplied `min` and `max` levels. 141 The feature works when combined with multi-threading and `--long` mode. 142 It does not work with `--single-thread`. 143 It sets window size to 8 MB by default (can be changed manually, see `wlog`). 144 Due to the chaotic nature of dynamic adaptation, compressed result is not reproducible. 145 _note_ : at the time of this writing, `--adapt` can remain stuck at low speed 146 when combined with multiple worker threads (>=2). 147* `--rsyncable` : 148 `zstd` will periodically synchronize the compression state to make the 149 compressed file more rsync-friendly. There is a negligible impact to 150 compression ratio, and the faster compression levels will see a small 151 compression speed hit. 152 This feature does not work with `--single-thread`. You probably don't want 153 to use it with long range mode, since it will decrease the effectiveness of 154 the synchronization points, but your milage may vary. 155* `-D file`: 156 use `file` as Dictionary to compress or decompress FILE(s) 157* `--no-dictID`: 158 do not store dictionary ID within frame header (dictionary compression). 159 The decoder will have to rely on implicit knowledge about which dictionary to use, 160 it won't be able to check if it's correct. 161* `-o file`: 162 save result into `file` (only possible with a single _INPUT-FILE_) 163* `-f`, `--force`: 164 overwrite output without prompting, and (de)compress symbolic links 165* `-c`, `--stdout`: 166 force write to standard output, even if it is the console 167* `--[no-]sparse`: 168 enable / disable sparse FS support, 169 to make files with many zeroes smaller on disk. 170 Creating sparse files may save disk space and speed up decompression by 171 reducing the amount of disk I/O. 172 default: enabled when output is into a file, 173 and disabled when output is stdout. 174 This setting overrides default and can force sparse mode over stdout. 175* `--rm`: 176 remove source file(s) after successful compression or decompression 177* `-k`, `--keep`: 178 keep source file(s) after successful compression or decompression. 179 This is the default behavior. 180* `-r`: 181 operate recursively on directories 182* `--format=FORMAT`: 183 compress and decompress in other formats. If compiled with 184 support, zstd can compress to or decompress from other compression algorithm 185 formats. Possibly available options are `zstd`, `gzip`, `xz`, `lzma`, and `lz4`. 186 If no such format is provided, `zstd` is the default. 187* `-h`/`-H`, `--help`: 188 display help/long help and exit 189* `-V`, `--version`: 190 display version number and exit. 191 Advanced : `-vV` also displays supported formats. 192 `-vvV` also displays POSIX support. 193* `-v`: 194 verbose mode 195* `-q`, `--quiet`: 196 suppress warnings, interactivity, and notifications. 197 specify twice to suppress errors too. 198* `--no-progress`: 199 do not display the progress bar, but keep all other messages. 200* `-C`, `--[no-]check`: 201 add integrity check computed from uncompressed data (default: enabled) 202* `--`: 203 All arguments after `--` are treated as files 204 205 206DICTIONARY BUILDER 207------------------ 208`zstd` offers _dictionary_ compression, 209which greatly improves efficiency on small files and messages. 210It's possible to train `zstd` with a set of samples, 211the result of which is saved into a file called a `dictionary`. 212Then during compression and decompression, reference the same dictionary, 213using command `-D dictionaryFileName`. 214Compression of small files similar to the sample set will be greatly improved. 215 216* `--train FILEs`: 217 Use FILEs as training set to create a dictionary. 218 The training set should contain a lot of small files (> 100), 219 and weight typically 100x the target dictionary size 220 (for example, 10 MB for a 100 KB dictionary). 221 222 Supports multithreading if `zstd` is compiled with threading support. 223 Additional parameters can be specified with `--train-fastcover`. 224 The legacy dictionary builder can be accessed with `--train-legacy`. 225 The cover dictionary builder can be accessed with `--train-cover`. 226 Equivalent to `--train-fastcover=d=8,steps=4`. 227* `-o file`: 228 Dictionary saved into `file` (default name: dictionary). 229* `--maxdict=#`: 230 Limit dictionary to specified size (default: 112640). 231* `-#`: 232 Use `#` compression level during training (optional). 233 Will generate statistics more tuned for selected compression level, 234 resulting in a _small_ compression ratio improvement for this level. 235* `-B#`: 236 Split input files in blocks of size # (default: no split) 237* `--dictID=#`: 238 A dictionary ID is a locally unique ID that a decoder can use to verify it is 239 using the right dictionary. 240 By default, zstd will create a 4-bytes random number ID. 241 It's possible to give a precise number instead. 242 Short numbers have an advantage : an ID < 256 will only need 1 byte in the 243 compressed frame header, and an ID < 65536 will only need 2 bytes. 244 This compares favorably to 4 bytes default. 245 However, it's up to the dictionary manager to not assign twice the same ID to 246 2 different dictionaries. 247* `--train-cover[=k#,d=#,steps=#,split=#,shrink[=#]]`: 248 Select parameters for the default dictionary builder algorithm named cover. 249 If _d_ is not specified, then it tries _d_ = 6 and _d_ = 8. 250 If _k_ is not specified, then it tries _steps_ values in the range [50, 2000]. 251 If _steps_ is not specified, then the default value of 40 is used. 252 If _split_ is not specified or split <= 0, then the default value of 100 is used. 253 Requires that _d_ <= _k_. 254 If _shrink_ flag is not used, then the default value for _shrinkDict_ of 0 is used. 255 If _shrink_ is not specified, then the default value for _shrinkDictMaxRegression_ of 1 is used. 256 257 Selects segments of size _k_ with highest score to put in the dictionary. 258 The score of a segment is computed by the sum of the frequencies of all the 259 subsegments of size _d_. 260 Generally _d_ should be in the range [6, 8], occasionally up to 16, but the 261 algorithm will run faster with d <= _8_. 262 Good values for _k_ vary widely based on the input data, but a safe range is 263 [2 * _d_, 2000]. 264 If _split_ is 100, all input samples are used for both training and testing 265 to find optimal _d_ and _k_ to build dictionary. 266 Supports multithreading if `zstd` is compiled with threading support. 267 Having _shrink_ enabled takes a truncated dictionary of minimum size and doubles 268 in size until compression ratio of the truncated dictionary is at most 269 _shrinkDictMaxRegression%_ worse than the compression ratio of the largest dictionary. 270 271 Examples: 272 273 `zstd --train-cover FILEs` 274 275 `zstd --train-cover=k=50,d=8 FILEs` 276 277 `zstd --train-cover=d=8,steps=500 FILEs` 278 279 `zstd --train-cover=k=50 FILEs` 280 281 `zstd --train-cover=k=50,split=60 FILEs` 282 283 `zstd --train-cover=shrink FILEs` 284 285 `zstd --train-cover=shrink=2 FILEs` 286 287* `--train-fastcover[=k#,d=#,f=#,steps=#,split=#,accel=#]`: 288 Same as cover but with extra parameters _f_ and _accel_ and different default value of split 289 If _split_ is not specified, then it tries _split_ = 75. 290 If _f_ is not specified, then it tries _f_ = 20. 291 Requires that 0 < _f_ < 32. 292 If _accel_ is not specified, then it tries _accel_ = 1. 293 Requires that 0 < _accel_ <= 10. 294 Requires that _d_ = 6 or _d_ = 8. 295 296 _f_ is log of size of array that keeps track of frequency of subsegments of size _d_. 297 The subsegment is hashed to an index in the range [0,2^_f_ - 1]. 298 It is possible that 2 different subsegments are hashed to the same index, and they are considered as the same subsegment when computing frequency. 299 Using a higher _f_ reduces collision but takes longer. 300 301 Examples: 302 303 `zstd --train-fastcover FILEs` 304 305 `zstd --train-fastcover=d=8,f=15,accel=2 FILEs` 306 307* `--train-legacy[=selectivity=#]`: 308 Use legacy dictionary builder algorithm with the given dictionary 309 _selectivity_ (default: 9). 310 The smaller the _selectivity_ value, the denser the dictionary, 311 improving its efficiency but reducing its possible maximum size. 312 `--train-legacy=s=#` is also accepted. 313 314 Examples: 315 316 `zstd --train-legacy FILEs` 317 318 `zstd --train-legacy=selectivity=8 FILEs` 319 320 321BENCHMARK 322--------- 323 324* `-b#`: 325 benchmark file(s) using compression level # 326* `-e#`: 327 benchmark file(s) using multiple compression levels, from `-b#` to `-e#` (inclusive) 328* `-i#`: 329 minimum evaluation time, in seconds (default: 3s), benchmark mode only 330* `-B#`, `--block-size=#`: 331 cut file(s) into independent blocks of size # (default: no block) 332* `--priority=rt`: 333 set process priority to real-time 334 335**Output Format:** CompressionLevel#Filename : IntputSize -> OutputSize (CompressionRatio), CompressionSpeed, DecompressionSpeed 336 337**Methodology:** For both compression and decompression speed, the entire input is compressed/decompressed in-memory to measure speed. A run lasts at least 1 sec, so when files are small, they are compressed/decompressed several times per run, in order to improve measurement accuracy. 338 339ADVANCED COMPRESSION OPTIONS 340---------------------------- 341### --zstd[=options]: 342`zstd` provides 22 predefined compression levels. 343The selected or default predefined compression level can be changed with 344advanced compression options. 345The _options_ are provided as a comma-separated list. 346You may specify only the options you want to change and the rest will be 347taken from the selected or default compression level. 348The list of available _options_: 349 350- `strategy`=_strat_, `strat`=_strat_: 351 Specify a strategy used by a match finder. 352 353 There are 9 strategies numbered from 1 to 9, from faster to stronger: 354 1=ZSTD\_fast, 2=ZSTD\_dfast, 3=ZSTD\_greedy, 355 4=ZSTD\_lazy, 5=ZSTD\_lazy2, 6=ZSTD\_btlazy2, 356 7=ZSTD\_btopt, 8=ZSTD\_btultra, 9=ZSTD\_btultra2. 357 358- `windowLog`=_wlog_, `wlog`=_wlog_: 359 Specify the maximum number of bits for a match distance. 360 361 The higher number of increases the chance to find a match which usually 362 improves compression ratio. 363 It also increases memory requirements for the compressor and decompressor. 364 The minimum _wlog_ is 10 (1 KiB) and the maximum is 30 (1 GiB) on 32-bit 365 platforms and 31 (2 GiB) on 64-bit platforms. 366 367 Note: If `windowLog` is set to larger than 27, `--long=windowLog` or 368 `--memory=windowSize` needs to be passed to the decompressor. 369 370- `hashLog`=_hlog_, `hlog`=_hlog_: 371 Specify the maximum number of bits for a hash table. 372 373 Bigger hash tables cause less collisions which usually makes compression 374 faster, but requires more memory during compression. 375 376 The minimum _hlog_ is 6 (64 B) and the maximum is 26 (128 MiB). 377 378- `chainLog`=_clog_, `clog`=_clog_: 379 Specify the maximum number of bits for a hash chain or a binary tree. 380 381 Higher numbers of bits increases the chance to find a match which usually 382 improves compression ratio. 383 It also slows down compression speed and increases memory requirements for 384 compression. 385 This option is ignored for the ZSTD_fast strategy. 386 387 The minimum _clog_ is 6 (64 B) and the maximum is 28 (256 MiB). 388 389- `searchLog`=_slog_, `slog`=_slog_: 390 Specify the maximum number of searches in a hash chain or a binary tree 391 using logarithmic scale. 392 393 More searches increases the chance to find a match which usually increases 394 compression ratio but decreases compression speed. 395 396 The minimum _slog_ is 1 and the maximum is 26. 397 398- `minMatch`=_mml_, `mml`=_mml_: 399 Specify the minimum searched length of a match in a hash table. 400 401 Larger search lengths usually decrease compression ratio but improve 402 decompression speed. 403 404 The minimum _mml_ is 3 and the maximum is 7. 405 406- `targetLen`=_tlen_, `tlen`=_tlen_: 407 The impact of this field vary depending on selected strategy. 408 409 For ZSTD\_btopt, ZSTD\_btultra and ZSTD\_btultra2, it specifies 410 the minimum match length that causes match finder to stop searching. 411 A larger `targetLen` usually improves compression ratio 412 but decreases compression speed. 413 414 For ZSTD\_fast, it triggers ultra-fast mode when > 0. 415 The value represents the amount of data skipped between match sampling. 416 Impact is reversed : a larger `targetLen` increases compression speed 417 but decreases compression ratio. 418 419 For all other strategies, this field has no impact. 420 421 The minimum _tlen_ is 0 and the maximum is 999. 422 423- `overlapLog`=_ovlog_, `ovlog`=_ovlog_: 424 Determine `overlapSize`, amount of data reloaded from previous job. 425 This parameter is only available when multithreading is enabled. 426 Reloading more data improves compression ratio, but decreases speed. 427 428 The minimum _ovlog_ is 0, and the maximum is 9. 429 1 means "no overlap", hence completely independent jobs. 430 9 means "full overlap", meaning up to `windowSize` is reloaded from previous job. 431 Reducing _ovlog_ by 1 reduces the reloaded amount by a factor 2. 432 For example, 8 means "windowSize/2", and 6 means "windowSize/8". 433 Value 0 is special and means "default" : _ovlog_ is automatically determined by `zstd`. 434 In which case, _ovlog_ will range from 6 to 9, depending on selected _strat_. 435 436- `ldmHashLog`=_lhlog_, `lhlog`=_lhlog_: 437 Specify the maximum size for a hash table used for long distance matching. 438 439 This option is ignored unless long distance matching is enabled. 440 441 Bigger hash tables usually improve compression ratio at the expense of more 442 memory during compression and a decrease in compression speed. 443 444 The minimum _lhlog_ is 6 and the maximum is 26 (default: 20). 445 446- `ldmMinMatch`=_lmml_, `lmml`=_lmml_: 447 Specify the minimum searched length of a match for long distance matching. 448 449 This option is ignored unless long distance matching is enabled. 450 451 Larger/very small values usually decrease compression ratio. 452 453 The minimum _lmml_ is 4 and the maximum is 4096 (default: 64). 454 455- `ldmBucketSizeLog`=_lblog_, `lblog`=_lblog_: 456 Specify the size of each bucket for the hash table used for long distance 457 matching. 458 459 This option is ignored unless long distance matching is enabled. 460 461 Larger bucket sizes improve collision resolution but decrease compression 462 speed. 463 464 The minimum _lblog_ is 0 and the maximum is 8 (default: 3). 465 466- `ldmHashRateLog`=_lhrlog_, `lhrlog`=_lhrlog_: 467 Specify the frequency of inserting entries into the long distance matching 468 hash table. 469 470 This option is ignored unless long distance matching is enabled. 471 472 Larger values will improve compression speed. Deviating far from the 473 default value will likely result in a decrease in compression ratio. 474 475 The default value is `wlog - lhlog`. 476 477### Example 478The following parameters sets advanced compression options to something 479similar to predefined level 19 for files bigger than 256 KB: 480 481`--zstd`=wlog=23,clog=23,hlog=22,slog=6,mml=3,tlen=48,strat=6 482 483### -B#: 484Select the size of each compression job. 485This parameter is available only when multi-threading is enabled. 486Default value is `4 * windowSize`, which means it varies depending on compression level. 487`-B#` makes it possible to select a custom value. 488Note that job size must respect a minimum value which is enforced transparently. 489This minimum is either 1 MB, or `overlapSize`, whichever is largest. 490 491BUGS 492---- 493Report bugs at: https://github.com/facebook/zstd/issues 494 495AUTHOR 496------ 497Yann Collet 498