1Zstandard Compression Format 2============================ 3 4### Notices 5 6Copyright (c) 2016-2021 Yann Collet, Facebook, Inc. 7 8Permission is granted to copy and distribute this document 9for any purpose and without charge, 10including translations into other languages 11and incorporation into compilations, 12provided that the copyright notice and this notice are preserved, 13and that any substantive changes or deletions from the original 14are clearly marked. 15Distribution of this document is unlimited. 16 17### Version 18 190.3.7 (2020-12-09) 20 21 22Introduction 23------------ 24 25The purpose of this document is to define a lossless compressed data format, 26that is independent of CPU type, operating system, 27file system and character set, suitable for 28file compression, pipe and streaming compression, 29using the [Zstandard algorithm](http://www.zstandard.org). 30The text of the specification assumes a basic background in programming 31at the level of bits and other primitive data representations. 32 33The data can be produced or consumed, 34even for an arbitrarily long sequentially presented input data stream, 35using only an a priori bounded amount of intermediate storage, 36and hence can be used in data communications. 37The format uses the Zstandard compression method, 38and optional [xxHash-64 checksum method](http://www.xxhash.org), 39for detection of data corruption. 40 41The data format defined by this specification 42does not attempt to allow random access to compressed data. 43 44Unless otherwise indicated below, 45a compliant compressor must produce data sets 46that conform to the specifications presented here. 47It doesn’t need to support all options though. 48 49A compliant decompressor must be able to decompress 50at least one working set of parameters 51that conforms to the specifications presented here. 52It may also ignore informative fields, such as checksum. 53Whenever it does not support a parameter defined in the compressed stream, 54it must produce a non-ambiguous error code and associated error message 55explaining which parameter is unsupported. 56 57This specification is intended for use by implementers of software 58to compress data into Zstandard format and/or decompress data from Zstandard format. 59The Zstandard format is supported by an open source reference implementation, 60written in portable C, and available at : https://github.com/facebook/zstd . 61 62 63### Overall conventions 64In this document: 65- square brackets i.e. `[` and `]` are used to indicate optional fields or parameters. 66- the naming convention for identifiers is `Mixed_Case_With_Underscores` 67 68### Definitions 69Content compressed by Zstandard is transformed into a Zstandard __frame__. 70Multiple frames can be appended into a single file or stream. 71A frame is completely independent, has a defined beginning and end, 72and a set of parameters which tells the decoder how to decompress it. 73 74A frame encapsulates one or multiple __blocks__. 75Each block contains arbitrary content, which is described by its header, 76and has a guaranteed maximum content size, which depends on frame parameters. 77Unlike frames, each block depends on previous blocks for proper decoding. 78However, each block can be decompressed without waiting for its successor, 79allowing streaming operations. 80 81Overview 82--------- 83- [Frames](#frames) 84 - [Zstandard frames](#zstandard-frames) 85 - [Blocks](#blocks) 86 - [Literals Section](#literals-section) 87 - [Sequences Section](#sequences-section) 88 - [Sequence Execution](#sequence-execution) 89 - [Skippable frames](#skippable-frames) 90- [Entropy Encoding](#entropy-encoding) 91 - [FSE](#fse) 92 - [Huffman Coding](#huffman-coding) 93- [Dictionary Format](#dictionary-format) 94 95Frames 96------ 97Zstandard compressed data is made of one or more __frames__. 98Each frame is independent and can be decompressed independently of other frames. 99The decompressed content of multiple concatenated frames is the concatenation of 100each frame decompressed content. 101 102There are two frame formats defined by Zstandard: 103 Zstandard frames and Skippable frames. 104Zstandard frames contain compressed data, while 105skippable frames contain custom user metadata. 106 107## Zstandard frames 108The structure of a single Zstandard frame is following: 109 110| `Magic_Number` | `Frame_Header` |`Data_Block`| [More data blocks] | [`Content_Checksum`] | 111|:--------------:|:--------------:|:----------:| ------------------ |:--------------------:| 112| 4 bytes | 2-14 bytes | n bytes | | 0-4 bytes | 113 114__`Magic_Number`__ 115 1164 Bytes, __little-endian__ format. 117Value : 0xFD2FB528 118Note: This value was selected to be less probable to find at the beginning of some random file. 119It avoids trivial patterns (0x00, 0xFF, repeated bytes, increasing bytes, etc.), 120contains byte values outside of ASCII range, 121and doesn't map into UTF8 space. 122It reduces the chances that a text file represent this value by accident. 123 124__`Frame_Header`__ 125 1262 to 14 Bytes, detailed in [`Frame_Header`](#frame_header). 127 128__`Data_Block`__ 129 130Detailed in [`Blocks`](#blocks). 131That’s where compressed data is stored. 132 133__`Content_Checksum`__ 134 135An optional 32-bit checksum, only present if `Content_Checksum_flag` is set. 136The content checksum is the result 137of [xxh64() hash function](http://www.xxhash.org) 138digesting the original (decoded) data as input, and a seed of zero. 139The low 4 bytes of the checksum are stored in __little-endian__ format. 140 141### `Frame_Header` 142 143The `Frame_Header` has a variable size, with a minimum of 2 bytes, 144and up to 14 bytes depending on optional parameters. 145The structure of `Frame_Header` is following: 146 147| `Frame_Header_Descriptor` | [`Window_Descriptor`] | [`Dictionary_ID`] | [`Frame_Content_Size`] | 148| ------------------------- | --------------------- | ----------------- | ---------------------- | 149| 1 byte | 0-1 byte | 0-4 bytes | 0-8 bytes | 150 151#### `Frame_Header_Descriptor` 152 153The first header's byte is called the `Frame_Header_Descriptor`. 154It describes which other fields are present. 155Decoding this byte is enough to tell the size of `Frame_Header`. 156 157| Bit number | Field name | 158| ---------- | ---------- | 159| 7-6 | `Frame_Content_Size_flag` | 160| 5 | `Single_Segment_flag` | 161| 4 | `Unused_bit` | 162| 3 | `Reserved_bit` | 163| 2 | `Content_Checksum_flag` | 164| 1-0 | `Dictionary_ID_flag` | 165 166In this table, bit 7 is the highest bit, while bit 0 is the lowest one. 167 168__`Frame_Content_Size_flag`__ 169 170This is a 2-bits flag (`= Frame_Header_Descriptor >> 6`), 171specifying if `Frame_Content_Size` (the decompressed data size) 172is provided within the header. 173`Flag_Value` provides `FCS_Field_Size`, 174which is the number of bytes used by `Frame_Content_Size` 175according to the following table: 176 177| `Flag_Value` | 0 | 1 | 2 | 3 | 178| -------------- | ------ | --- | --- | --- | 179|`FCS_Field_Size`| 0 or 1 | 2 | 4 | 8 | 180 181When `Flag_Value` is `0`, `FCS_Field_Size` depends on `Single_Segment_flag` : 182if `Single_Segment_flag` is set, `FCS_Field_Size` is 1. 183Otherwise, `FCS_Field_Size` is 0 : `Frame_Content_Size` is not provided. 184 185__`Single_Segment_flag`__ 186 187If this flag is set, 188data must be regenerated within a single continuous memory segment. 189 190In this case, `Window_Descriptor` byte is skipped, 191but `Frame_Content_Size` is necessarily present. 192As a consequence, the decoder must allocate a memory segment 193of size equal or larger than `Frame_Content_Size`. 194 195In order to preserve the decoder from unreasonable memory requirements, 196a decoder is allowed to reject a compressed frame 197which requests a memory size beyond decoder's authorized range. 198 199For broader compatibility, decoders are recommended to support 200memory sizes of at least 8 MB. 201This is only a recommendation, 202each decoder is free to support higher or lower limits, 203depending on local limitations. 204 205__`Unused_bit`__ 206 207A decoder compliant with this specification version shall not interpret this bit. 208It might be used in any future version, 209to signal a property which is transparent to properly decode the frame. 210An encoder compliant with this specification version must set this bit to zero. 211 212__`Reserved_bit`__ 213 214This bit is reserved for some future feature. 215Its value _must be zero_. 216A decoder compliant with this specification version must ensure it is not set. 217This bit may be used in a future revision, 218to signal a feature that must be interpreted to decode the frame correctly. 219 220__`Content_Checksum_flag`__ 221 222If this flag is set, a 32-bits `Content_Checksum` will be present at frame's end. 223See `Content_Checksum` paragraph. 224 225__`Dictionary_ID_flag`__ 226 227This is a 2-bits flag (`= FHD & 3`), 228telling if a dictionary ID is provided within the header. 229It also specifies the size of this field as `DID_Field_Size`. 230 231|`Flag_Value` | 0 | 1 | 2 | 3 | 232| -------------- | --- | --- | --- | --- | 233|`DID_Field_Size`| 0 | 1 | 2 | 4 | 234 235#### `Window_Descriptor` 236 237Provides guarantees on minimum memory buffer required to decompress a frame. 238This information is important for decoders to allocate enough memory. 239 240The `Window_Descriptor` byte is optional. 241When `Single_Segment_flag` is set, `Window_Descriptor` is not present. 242In this case, `Window_Size` is `Frame_Content_Size`, 243which can be any value from 0 to 2^64-1 bytes (16 ExaBytes). 244 245| Bit numbers | 7-3 | 2-0 | 246| ----------- | ---------- | ---------- | 247| Field name | `Exponent` | `Mantissa` | 248 249The minimum memory buffer size is called `Window_Size`. 250It is described by the following formulas : 251``` 252windowLog = 10 + Exponent; 253windowBase = 1 << windowLog; 254windowAdd = (windowBase / 8) * Mantissa; 255Window_Size = windowBase + windowAdd; 256``` 257The minimum `Window_Size` is 1 KB. 258The maximum `Window_Size` is `(1<<41) + 7*(1<<38)` bytes, which is 3.75 TB. 259 260In general, larger `Window_Size` tend to improve compression ratio, 261but at the cost of memory usage. 262 263To properly decode compressed data, 264a decoder will need to allocate a buffer of at least `Window_Size` bytes. 265 266In order to preserve decoder from unreasonable memory requirements, 267a decoder is allowed to reject a compressed frame 268which requests a memory size beyond decoder's authorized range. 269 270For improved interoperability, 271it's recommended for decoders to support `Window_Size` of up to 8 MB, 272and it's recommended for encoders to not generate frame requiring `Window_Size` larger than 8 MB. 273It's merely a recommendation though, 274decoders are free to support larger or lower limits, 275depending on local limitations. 276 277#### `Dictionary_ID` 278 279This is a variable size field, which contains 280the ID of the dictionary required to properly decode the frame. 281`Dictionary_ID` field is optional. When it's not present, 282it's up to the decoder to know which dictionary to use. 283 284`Dictionary_ID` field size is provided by `DID_Field_Size`. 285`DID_Field_Size` is directly derived from value of `Dictionary_ID_flag`. 2861 byte can represent an ID 0-255. 2872 bytes can represent an ID 0-65535. 2884 bytes can represent an ID 0-4294967295. 289Format is __little-endian__. 290 291It's allowed to represent a small ID (for example `13`) 292with a large 4-bytes dictionary ID, even if it is less efficient. 293 294A value of `0` has same meaning as no `Dictionary_ID`, 295in which case the frame may or may not need a dictionary to be decoded, 296and the ID of such a dictionary is not specified. 297The decoder must know this information by other means. 298 299#### `Frame_Content_Size` 300 301This is the original (uncompressed) size. This information is optional. 302`Frame_Content_Size` uses a variable number of bytes, provided by `FCS_Field_Size`. 303`FCS_Field_Size` is provided by the value of `Frame_Content_Size_flag`. 304`FCS_Field_Size` can be equal to 0 (not present), 1, 2, 4 or 8 bytes. 305 306| `FCS_Field_Size` | Range | 307| ---------------- | ---------- | 308| 0 | unknown | 309| 1 | 0 - 255 | 310| 2 | 256 - 65791| 311| 4 | 0 - 2^32-1 | 312| 8 | 0 - 2^64-1 | 313 314`Frame_Content_Size` format is __little-endian__. 315When `FCS_Field_Size` is 1, 4 or 8 bytes, the value is read directly. 316When `FCS_Field_Size` is 2, _the offset of 256 is added_. 317It's allowed to represent a small size (for example `18`) using any compatible variant. 318 319 320Blocks 321------- 322 323After `Magic_Number` and `Frame_Header`, there are some number of blocks. 324Each frame must have at least one block, 325but there is no upper limit on the number of blocks per frame. 326 327The structure of a block is as follows: 328 329| `Block_Header` | `Block_Content` | 330|:--------------:|:---------------:| 331| 3 bytes | n bytes | 332 333__`Block_Header`__ 334 335`Block_Header` uses 3 bytes, written using __little-endian__ convention. 336It contains 3 fields : 337 338| `Last_Block` | `Block_Type` | `Block_Size` | 339|:------------:|:------------:|:------------:| 340| bit 0 | bits 1-2 | bits 3-23 | 341 342__`Last_Block`__ 343 344The lowest bit signals if this block is the last one. 345The frame will end after this last block. 346It may be followed by an optional `Content_Checksum` 347(see [Zstandard Frames](#zstandard-frames)). 348 349__`Block_Type`__ 350 351The next 2 bits represent the `Block_Type`. 352`Block_Type` influences the meaning of `Block_Size`. 353There are 4 block types : 354 355| Value | 0 | 1 | 2 | 3 | 356| ------------ | ----------- | ----------- | ------------------ | --------- | 357| `Block_Type` | `Raw_Block` | `RLE_Block` | `Compressed_Block` | `Reserved`| 358 359- `Raw_Block` - this is an uncompressed block. 360 `Block_Content` contains `Block_Size` bytes. 361 362- `RLE_Block` - this is a single byte, repeated `Block_Size` times. 363 `Block_Content` consists of a single byte. 364 On the decompression side, this byte must be repeated `Block_Size` times. 365 366- `Compressed_Block` - this is a [Zstandard compressed block](#compressed-blocks), 367 explained later on. 368 `Block_Size` is the length of `Block_Content`, the compressed data. 369 The decompressed size is not known, 370 but its maximum possible value is guaranteed (see below) 371 372- `Reserved` - this is not a block. 373 This value cannot be used with current version of this specification. 374 If such a value is present, it is considered corrupted data. 375 376__`Block_Size`__ 377 378The upper 21 bits of `Block_Header` represent the `Block_Size`. 379 380When `Block_Type` is `Compressed_Block` or `Raw_Block`, 381`Block_Size` is the size of `Block_Content` (hence excluding `Block_Header`). 382 383When `Block_Type` is `RLE_Block`, since `Block_Content`’s size is always 1, 384`Block_Size` represents the number of times this byte must be repeated. 385 386`Block_Size` is limited by `Block_Maximum_Size` (see below). 387 388__`Block_Content`__ and __`Block_Maximum_Size`__ 389 390The size of `Block_Content` is limited by `Block_Maximum_Size`, 391which is the smallest of: 392- `Window_Size` 393- 128 KB 394 395`Block_Maximum_Size` is constant for a given frame. 396This maximum is applicable to both the decompressed size 397and the compressed size of any block in the frame. 398 399The reasoning for this limit is that a decoder can read this information 400at the beginning of a frame and use it to allocate buffers. 401The guarantees on the size of blocks ensure that 402the buffers will be large enough for any following block of the valid frame. 403 404 405Compressed Blocks 406----------------- 407To decompress a compressed block, the compressed size must be provided 408from `Block_Size` field within `Block_Header`. 409 410A compressed block consists of 2 sections : 411- [Literals Section](#literals-section) 412- [Sequences Section](#sequences-section) 413 414The results of the two sections are then combined to produce the decompressed 415data in [Sequence Execution](#sequence-execution) 416 417#### Prerequisites 418To decode a compressed block, the following elements are necessary : 419- Previous decoded data, up to a distance of `Window_Size`, 420 or beginning of the Frame, whichever is smaller. 421- List of "recent offsets" from previous `Compressed_Block`. 422- The previous Huffman tree, required by `Treeless_Literals_Block` type 423- Previous FSE decoding tables, required by `Repeat_Mode` 424 for each symbol type (literals lengths, match lengths, offsets) 425 426Note that decoding tables aren't always from the previous `Compressed_Block`. 427 428- Every decoding table can come from a dictionary. 429- The Huffman tree comes from the previous `Compressed_Literals_Block`. 430 431Literals Section 432---------------- 433All literals are regrouped in the first part of the block. 434They can be decoded first, and then copied during [Sequence Execution], 435or they can be decoded on the flow during [Sequence Execution]. 436 437Literals can be stored uncompressed or compressed using Huffman prefix codes. 438When compressed, an optional tree description can be present, 439followed by 1 or 4 streams. 440 441| `Literals_Section_Header` | [`Huffman_Tree_Description`] | [jumpTable] | Stream1 | [Stream2] | [Stream3] | [Stream4] | 442| ------------------------- | ---------------------------- | ----------- | ------- | --------- | --------- | --------- | 443 444 445### `Literals_Section_Header` 446 447Header is in charge of describing how literals are packed. 448It's a byte-aligned variable-size bitfield, ranging from 1 to 5 bytes, 449using __little-endian__ convention. 450 451| `Literals_Block_Type` | `Size_Format` | `Regenerated_Size` | [`Compressed_Size`] | 452| --------------------- | ------------- | ------------------ | ------------------- | 453| 2 bits | 1 - 2 bits | 5 - 20 bits | 0 - 18 bits | 454 455In this representation, bits on the left are the lowest bits. 456 457__`Literals_Block_Type`__ 458 459This field uses 2 lowest bits of first byte, describing 4 different block types : 460 461| `Literals_Block_Type` | Value | 462| --------------------------- | ----- | 463| `Raw_Literals_Block` | 0 | 464| `RLE_Literals_Block` | 1 | 465| `Compressed_Literals_Block` | 2 | 466| `Treeless_Literals_Block` | 3 | 467 468- `Raw_Literals_Block` - Literals are stored uncompressed. 469- `RLE_Literals_Block` - Literals consist of a single byte value 470 repeated `Regenerated_Size` times. 471- `Compressed_Literals_Block` - This is a standard Huffman-compressed block, 472 starting with a Huffman tree description. 473 See details below. 474- `Treeless_Literals_Block` - This is a Huffman-compressed block, 475 using Huffman tree _from previous Huffman-compressed literals block_. 476 `Huffman_Tree_Description` will be skipped. 477 Note: If this mode is triggered without any previous Huffman-table in the frame 478 (or [dictionary](#dictionary-format)), this should be treated as data corruption. 479 480__`Size_Format`__ 481 482`Size_Format` is divided into 2 families : 483 484- For `Raw_Literals_Block` and `RLE_Literals_Block`, 485 it's only necessary to decode `Regenerated_Size`. 486 There is no `Compressed_Size` field. 487- For `Compressed_Block` and `Treeless_Literals_Block`, 488 it's required to decode both `Compressed_Size` 489 and `Regenerated_Size` (the decompressed size). 490 It's also necessary to decode the number of streams (1 or 4). 491 492For values spanning several bytes, convention is __little-endian__. 493 494__`Size_Format` for `Raw_Literals_Block` and `RLE_Literals_Block`__ : 495 496`Size_Format` uses 1 _or_ 2 bits. 497Its value is : `Size_Format = (Literals_Section_Header[0]>>2) & 3` 498 499- `Size_Format` == 00 or 10 : `Size_Format` uses 1 bit. 500 `Regenerated_Size` uses 5 bits (0-31). 501 `Literals_Section_Header` uses 1 byte. 502 `Regenerated_Size = Literals_Section_Header[0]>>3` 503- `Size_Format` == 01 : `Size_Format` uses 2 bits. 504 `Regenerated_Size` uses 12 bits (0-4095). 505 `Literals_Section_Header` uses 2 bytes. 506 `Regenerated_Size = (Literals_Section_Header[0]>>4) + (Literals_Section_Header[1]<<4)` 507- `Size_Format` == 11 : `Size_Format` uses 2 bits. 508 `Regenerated_Size` uses 20 bits (0-1048575). 509 `Literals_Section_Header` uses 3 bytes. 510 `Regenerated_Size = (Literals_Section_Header[0]>>4) + (Literals_Section_Header[1]<<4) + (Literals_Section_Header[2]<<12)` 511 512Only Stream1 is present for these cases. 513Note : it's allowed to represent a short value (for example `13`) 514using a long format, even if it's less efficient. 515 516__`Size_Format` for `Compressed_Literals_Block` and `Treeless_Literals_Block`__ : 517 518`Size_Format` always uses 2 bits. 519 520- `Size_Format` == 00 : _A single stream_. 521 Both `Regenerated_Size` and `Compressed_Size` use 10 bits (0-1023). 522 `Literals_Section_Header` uses 3 bytes. 523- `Size_Format` == 01 : 4 streams. 524 Both `Regenerated_Size` and `Compressed_Size` use 10 bits (0-1023). 525 `Literals_Section_Header` uses 3 bytes. 526- `Size_Format` == 10 : 4 streams. 527 Both `Regenerated_Size` and `Compressed_Size` use 14 bits (0-16383). 528 `Literals_Section_Header` uses 4 bytes. 529- `Size_Format` == 11 : 4 streams. 530 Both `Regenerated_Size` and `Compressed_Size` use 18 bits (0-262143). 531 `Literals_Section_Header` uses 5 bytes. 532 533Both `Compressed_Size` and `Regenerated_Size` fields follow __little-endian__ convention. 534Note: `Compressed_Size` __includes__ the size of the Huffman Tree description 535_when_ it is present. 536 537#### Raw Literals Block 538The data in Stream1 is `Regenerated_Size` bytes long, 539it contains the raw literals data to be used during [Sequence Execution]. 540 541#### RLE Literals Block 542Stream1 consists of a single byte which should be repeated `Regenerated_Size` times 543to generate the decoded literals. 544 545#### Compressed Literals Block and Treeless Literals Block 546Both of these modes contain Huffman encoded data. 547 548For `Treeless_Literals_Block`, 549the Huffman table comes from previously compressed literals block, 550or from a dictionary. 551 552 553### `Huffman_Tree_Description` 554This section is only present when `Literals_Block_Type` type is `Compressed_Literals_Block` (`2`). 555The format of the Huffman tree description can be found at [Huffman Tree description](#huffman-tree-description). 556The size of `Huffman_Tree_Description` is determined during decoding process, 557it must be used to determine where streams begin. 558`Total_Streams_Size = Compressed_Size - Huffman_Tree_Description_Size`. 559 560 561### Jump Table 562The Jump Table is only present when there are 4 Huffman-coded streams. 563 564Reminder : Huffman compressed data consists of either 1 or 4 Huffman-coded streams. 565 566If only one stream is present, it is a single bitstream occupying the entire 567remaining portion of the literals block, encoded as described within 568[Huffman-Coded Streams](#huffman-coded-streams). 569 570If there are four streams, `Literals_Section_Header` only provided 571enough information to know the decompressed and compressed sizes 572of all four streams _combined_. 573The decompressed size of _each_ stream is equal to `(Regenerated_Size+3)/4`, 574except for the last stream which may be up to 3 bytes smaller, 575to reach a total decompressed size as specified in `Regenerated_Size`. 576 577The compressed size of each stream is provided explicitly in the Jump Table. 578Jump Table is 6 bytes long, and consist of three 2-byte __little-endian__ fields, 579describing the compressed sizes of the first three streams. 580`Stream4_Size` is computed from total `Total_Streams_Size` minus sizes of other streams. 581 582`Stream4_Size = Total_Streams_Size - 6 - Stream1_Size - Stream2_Size - Stream3_Size`. 583 584Note: if `Stream1_Size + Stream2_Size + Stream3_Size > Total_Streams_Size`, 585data is considered corrupted. 586 587Each of these 4 bitstreams is then decoded independently as a Huffman-Coded stream, 588as described at [Huffman-Coded Streams](#huffman-coded-streams) 589 590 591Sequences Section 592----------------- 593A compressed block is a succession of _sequences_ . 594A sequence is a literal copy command, followed by a match copy command. 595A literal copy command specifies a length. 596It is the number of bytes to be copied (or extracted) from the Literals Section. 597A match copy command specifies an offset and a length. 598 599When all _sequences_ are decoded, 600if there are literals left in the _literals section_, 601these bytes are added at the end of the block. 602 603This is described in more detail in [Sequence Execution](#sequence-execution). 604 605The `Sequences_Section` regroup all symbols required to decode commands. 606There are 3 symbol types : literals lengths, offsets and match lengths. 607They are encoded together, interleaved, in a single _bitstream_. 608 609The `Sequences_Section` starts by a header, 610followed by optional probability tables for each symbol type, 611followed by the bitstream. 612 613| `Sequences_Section_Header` | [`Literals_Length_Table`] | [`Offset_Table`] | [`Match_Length_Table`] | bitStream | 614| -------------------------- | ------------------------- | ---------------- | ---------------------- | --------- | 615 616To decode the `Sequences_Section`, it's required to know its size. 617Its size is deduced from the size of `Literals_Section`: 618`Sequences_Section_Size = Block_Size - Literals_Section_Size`. 619 620 621#### `Sequences_Section_Header` 622 623Consists of 2 items: 624- `Number_of_Sequences` 625- Symbol compression modes 626 627__`Number_of_Sequences`__ 628 629This is a variable size field using between 1 and 3 bytes. 630Let's call its first byte `byte0`. 631- `if (byte0 == 0)` : there are no sequences. 632 The sequence section stops there. 633 Decompressed content is defined entirely as Literals Section content. 634 The FSE tables used in `Repeat_Mode` aren't updated. 635- `if (byte0 < 128)` : `Number_of_Sequences = byte0` . Uses 1 byte. 636- `if (byte0 < 255)` : `Number_of_Sequences = ((byte0-128) << 8) + byte1` . Uses 2 bytes. 637- `if (byte0 == 255)`: `Number_of_Sequences = byte1 + (byte2<<8) + 0x7F00` . Uses 3 bytes. 638 639__Symbol compression modes__ 640 641This is a single byte, defining the compression mode of each symbol type. 642 643|Bit number| 7-6 | 5-4 | 3-2 | 1-0 | 644| -------- | ----------------------- | -------------- | -------------------- | ---------- | 645|Field name| `Literals_Lengths_Mode` | `Offsets_Mode` | `Match_Lengths_Mode` | `Reserved` | 646 647The last field, `Reserved`, must be all-zeroes. 648 649`Literals_Lengths_Mode`, `Offsets_Mode` and `Match_Lengths_Mode` define the `Compression_Mode` of 650literals lengths, offsets, and match lengths symbols respectively. 651 652They follow the same enumeration : 653 654| Value | 0 | 1 | 2 | 3 | 655| ------------------ | ----------------- | ---------- | --------------------- | ------------- | 656| `Compression_Mode` | `Predefined_Mode` | `RLE_Mode` | `FSE_Compressed_Mode` | `Repeat_Mode` | 657 658- `Predefined_Mode` : A predefined FSE distribution table is used, defined in 659 [default distributions](#default-distributions). 660 No distribution table will be present. 661- `RLE_Mode` : The table description consists of a single byte, which contains the symbol's value. 662 This symbol will be used for all sequences. 663- `FSE_Compressed_Mode` : standard FSE compression. 664 A distribution table will be present. 665 The format of this distribution table is described in [FSE Table Description](#fse-table-description). 666 Note that the maximum allowed accuracy log for literals length and match length tables is 9, 667 and the maximum accuracy log for the offsets table is 8. 668 `FSE_Compressed_Mode` must not be used when only one symbol is present, 669 `RLE_Mode` should be used instead (although any other mode will work). 670- `Repeat_Mode` : The table used in the previous `Compressed_Block` with `Number_of_Sequences > 0` will be used again, 671 or if this is the first block, table in the dictionary will be used. 672 Note that this includes `RLE_mode`, so if `Repeat_Mode` follows `RLE_Mode`, the same symbol will be repeated. 673 It also includes `Predefined_Mode`, in which case `Repeat_Mode` will have same outcome as `Predefined_Mode`. 674 No distribution table will be present. 675 If this mode is used without any previous sequence table in the frame 676 (nor [dictionary](#dictionary-format)) to repeat, this should be treated as corruption. 677 678#### The codes for literals lengths, match lengths, and offsets. 679 680Each symbol is a _code_ in its own context, 681which specifies `Baseline` and `Number_of_Bits` to add. 682_Codes_ are FSE compressed, 683and interleaved with raw additional bits in the same bitstream. 684 685##### Literals length codes 686 687Literals length codes are values ranging from `0` to `35` included. 688They define lengths from 0 to 131071 bytes. 689The literals length is equal to the decoded `Baseline` plus 690the result of reading `Number_of_Bits` bits from the bitstream, 691as a __little-endian__ value. 692 693| `Literals_Length_Code` | 0-15 | 694| ---------------------- | ---------------------- | 695| length | `Literals_Length_Code` | 696| `Number_of_Bits` | 0 | 697 698| `Literals_Length_Code` | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 699| ---------------------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | 700| `Baseline` | 16 | 18 | 20 | 22 | 24 | 28 | 32 | 40 | 701| `Number_of_Bits` | 1 | 1 | 1 | 1 | 2 | 2 | 3 | 3 | 702 703| `Literals_Length_Code` | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 704| ---------------------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | 705| `Baseline` | 48 | 64 | 128 | 256 | 512 | 1024 | 2048 | 4096 | 706| `Number_of_Bits` | 4 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 707 708| `Literals_Length_Code` | 32 | 33 | 34 | 35 | 709| ---------------------- | ---- | ---- | ---- | ---- | 710| `Baseline` | 8192 |16384 |32768 |65536 | 711| `Number_of_Bits` | 13 | 14 | 15 | 16 | 712 713 714##### Match length codes 715 716Match length codes are values ranging from `0` to `52` included. 717They define lengths from 3 to 131074 bytes. 718The match length is equal to the decoded `Baseline` plus 719the result of reading `Number_of_Bits` bits from the bitstream, 720as a __little-endian__ value. 721 722| `Match_Length_Code` | 0-31 | 723| ------------------- | ----------------------- | 724| value | `Match_Length_Code` + 3 | 725| `Number_of_Bits` | 0 | 726 727| `Match_Length_Code` | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 728| ------------------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | 729| `Baseline` | 35 | 37 | 39 | 41 | 43 | 47 | 51 | 59 | 730| `Number_of_Bits` | 1 | 1 | 1 | 1 | 2 | 2 | 3 | 3 | 731 732| `Match_Length_Code` | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 733| ------------------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | 734| `Baseline` | 67 | 83 | 99 | 131 | 259 | 515 | 1027 | 2051 | 735| `Number_of_Bits` | 4 | 4 | 5 | 7 | 8 | 9 | 10 | 11 | 736 737| `Match_Length_Code` | 48 | 49 | 50 | 51 | 52 | 738| ------------------- | ---- | ---- | ---- | ---- | ---- | 739| `Baseline` | 4099 | 8195 |16387 |32771 |65539 | 740| `Number_of_Bits` | 12 | 13 | 14 | 15 | 16 | 741 742##### Offset codes 743 744Offset codes are values ranging from `0` to `N`. 745 746A decoder is free to limit its maximum `N` supported. 747Recommendation is to support at least up to `22`. 748For information, at the time of this writing. 749the reference decoder supports a maximum `N` value of `31`. 750 751An offset code is also the number of additional bits to read in __little-endian__ fashion, 752and can be translated into an `Offset_Value` using the following formulas : 753 754``` 755Offset_Value = (1 << offsetCode) + readNBits(offsetCode); 756if (Offset_Value > 3) offset = Offset_Value - 3; 757``` 758It means that maximum `Offset_Value` is `(2^(N+1))-1` 759supporting back-reference distances up to `(2^(N+1))-4`, 760but is limited by [maximum back-reference distance](#window_descriptor). 761 762`Offset_Value` from 1 to 3 are special : they define "repeat codes". 763This is described in more detail in [Repeat Offsets](#repeat-offsets). 764 765#### Decoding Sequences 766FSE bitstreams are read in reverse direction than written. In zstd, 767the compressor writes bits forward into a block and the decompressor 768must read the bitstream _backwards_. 769 770To find the start of the bitstream it is therefore necessary to 771know the offset of the last byte of the block which can be found 772by counting `Block_Size` bytes after the block header. 773 774After writing the last bit containing information, the compressor 775writes a single `1`-bit and then fills the byte with 0-7 `0` bits of 776padding. The last byte of the compressed bitstream cannot be `0` for 777that reason. 778 779When decompressing, the last byte containing the padding is the first 780byte to read. The decompressor needs to skip 0-7 initial `0`-bits and 781the first `1`-bit it occurs. Afterwards, the useful part of the bitstream 782begins. 783 784FSE decoding requires a 'state' to be carried from symbol to symbol. 785For more explanation on FSE decoding, see the [FSE section](#fse). 786 787For sequence decoding, a separate state keeps track of each 788literal lengths, offsets, and match lengths symbols. 789Some FSE primitives are also used. 790For more details on the operation of these primitives, see the [FSE section](#fse). 791 792##### Starting states 793The bitstream starts with initial FSE state values, 794each using the required number of bits in their respective _accuracy_, 795decoded previously from their normalized distribution. 796 797It starts by `Literals_Length_State`, 798followed by `Offset_State`, 799and finally `Match_Length_State`. 800 801Reminder : always keep in mind that all values are read _backward_, 802so the 'start' of the bitstream is at the highest position in memory, 803immediately before the last `1`-bit for padding. 804 805After decoding the starting states, a single sequence is decoded 806`Number_Of_Sequences` times. 807These sequences are decoded in order from first to last. 808Since the compressor writes the bitstream in the forward direction, 809this means the compressor must encode the sequences starting with the last 810one and ending with the first. 811 812##### Decoding a sequence 813For each of the symbol types, the FSE state can be used to determine the appropriate code. 814The code then defines the `Baseline` and `Number_of_Bits` to read for each type. 815See the [description of the codes] for how to determine these values. 816 817[description of the codes]: #the-codes-for-literals-lengths-match-lengths-and-offsets 818 819Decoding starts by reading the `Number_of_Bits` required to decode `Offset`. 820It then does the same for `Match_Length`, and then for `Literals_Length`. 821This sequence is then used for [sequence execution](#sequence-execution). 822 823If it is not the last sequence in the block, 824the next operation is to update states. 825Using the rules pre-calculated in the decoding tables, 826`Literals_Length_State` is updated, 827followed by `Match_Length_State`, 828and then `Offset_State`. 829See the [FSE section](#fse) for details on how to update states from the bitstream. 830 831This operation will be repeated `Number_of_Sequences` times. 832At the end, the bitstream shall be entirely consumed, 833otherwise the bitstream is considered corrupted. 834 835#### Default Distributions 836If `Predefined_Mode` is selected for a symbol type, 837its FSE decoding table is generated from a predefined distribution table defined here. 838For details on how to convert this distribution into a decoding table, see the [FSE section]. 839 840[FSE section]: #from-normalized-distribution-to-decoding-tables 841 842##### Literals Length 843The decoding table uses an accuracy log of 6 bits (64 states). 844``` 845short literalsLength_defaultDistribution[36] = 846 { 4, 3, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 847 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 2, 1, 1, 1, 1, 1, 848 -1,-1,-1,-1 }; 849``` 850 851##### Match Length 852The decoding table uses an accuracy log of 6 bits (64 states). 853``` 854short matchLengths_defaultDistribution[53] = 855 { 1, 4, 3, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 856 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 857 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,-1,-1, 858 -1,-1,-1,-1,-1 }; 859``` 860 861##### Offset Codes 862The decoding table uses an accuracy log of 5 bits (32 states), 863and supports a maximum `N` value of 28, allowing offset values up to 536,870,908 . 864 865If any sequence in the compressed block requires a larger offset than this, 866it's not possible to use the default distribution to represent it. 867``` 868short offsetCodes_defaultDistribution[29] = 869 { 1, 1, 1, 1, 1, 1, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 870 1, 1, 1, 1, 1, 1, 1, 1,-1,-1,-1,-1,-1 }; 871``` 872 873 874Sequence Execution 875------------------ 876Once literals and sequences have been decoded, 877they are combined to produce the decoded content of a block. 878 879Each sequence consists of a tuple of (`literals_length`, `offset_value`, `match_length`), 880decoded as described in the [Sequences Section](#sequences-section). 881To execute a sequence, first copy `literals_length` bytes 882from the decoded literals to the output. 883 884Then `match_length` bytes are copied from previous decoded data. 885The offset to copy from is determined by `offset_value`: 886if `offset_value > 3`, then the offset is `offset_value - 3`. 887If `offset_value` is from 1-3, the offset is a special repeat offset value. 888See the [repeat offset](#repeat-offsets) section for how the offset is determined 889in this case. 890 891The offset is defined as from the current position, so an offset of 6 892and a match length of 3 means that 3 bytes should be copied from 6 bytes back. 893Note that all offsets leading to previously decoded data 894must be smaller than `Window_Size` defined in `Frame_Header_Descriptor`. 895 896#### Repeat offsets 897As seen in [Sequence Execution](#sequence-execution), 898the first 3 values define a repeated offset and we will call them 899`Repeated_Offset1`, `Repeated_Offset2`, and `Repeated_Offset3`. 900They are sorted in recency order, with `Repeated_Offset1` meaning "most recent one". 901 902If `offset_value == 1`, then the offset used is `Repeated_Offset1`, etc. 903 904There is an exception though, when current sequence's `literals_length = 0`. 905In this case, repeated offsets are shifted by one, 906so an `offset_value` of 1 means `Repeated_Offset2`, 907an `offset_value` of 2 means `Repeated_Offset3`, 908and an `offset_value` of 3 means `Repeated_Offset1 - 1_byte`. 909 910For the first block, the starting offset history is populated with following values : 911`Repeated_Offset1`=1, `Repeated_Offset2`=4, `Repeated_Offset3`=8, 912unless a dictionary is used, in which case they come from the dictionary. 913 914Then each block gets its starting offset history from the ending values of the most recent `Compressed_Block`. 915Note that blocks which are not `Compressed_Block` are skipped, they do not contribute to offset history. 916 917[Offset Codes]: #offset-codes 918 919###### Offset updates rules 920 921During the execution of the sequences of a `Compressed_Block`, the 922`Repeated_Offsets`' values are kept up to date, so that they always represent 923the three most-recently used offsets. In order to achieve that, they are 924updated after executing each sequence in the following way: 925 926When the sequence's `offset_value` does not refer to one of the 927`Repeated_Offsets`--when it has value greater than 3, or when it has value 3 928and the sequence's `literals_length` is zero--the `Repeated_Offsets`' values 929are shifted back one, and `Repeated_Offset1` takes on the value of the 930just-used offset. 931 932Otherwise, when the sequence's `offset_value` refers to one of the 933`Repeated_Offsets`--when it has value 1 or 2, or when it has value 3 and the 934sequence's `literals_length` is non-zero--the `Repeated_Offsets` are re-ordered 935so that `Repeated_Offset1` takes on the value of the used Repeated_Offset, and 936the existing values are pushed back from the first `Repeated_Offset` through to 937the `Repeated_Offset` selected by the `offset_value`. This effectively performs 938a single-stepped wrapping rotation of the values of these offsets, so that 939their order again reflects the recency of their use. 940 941The following table shows the values of the `Repeated_Offsets` as a series of 942sequences are applied to them: 943 944| `offset_value` | `literals_length` | `Repeated_Offset1` | `Repeated_Offset2` | `Repeated_Offset3` | Comment | 945|:--------------:|:-----------------:|:------------------:|:------------------:|:------------------:|:-----------------------:| 946| | | 1 | 4 | 8 | starting values | 947| 1114 | 11 | 1111 | 1 | 4 | non-repeat | 948| 1 | 22 | 1111 | 1 | 4 | repeat 1; no change | 949| 2225 | 22 | 2222 | 1111 | 1 | non-repeat | 950| 1114 | 111 | 1111 | 2222 | 1111 | non-repeat | 951| 3336 | 33 | 3333 | 1111 | 2222 | non-repeat | 952| 2 | 22 | 1111 | 3333 | 2222 | repeat 2; swap 1 & 2 | 953| 3 | 33 | 2222 | 1111 | 3333 | repeat 3; rotate 3 to 1 | 954| 3 | 0 | 2221 | 2222 | 1111 | insert resolved offset | 955| 1 | 0 | 2222 | 2221 | 3333 | repeat 2 | 956 957 958Skippable Frames 959---------------- 960 961| `Magic_Number` | `Frame_Size` | `User_Data` | 962|:--------------:|:------------:|:-----------:| 963| 4 bytes | 4 bytes | n bytes | 964 965Skippable frames allow the insertion of user-defined metadata 966into a flow of concatenated frames. 967 968Skippable frames defined in this specification are compatible with [LZ4] ones. 969 970[LZ4]:http://www.lz4.org 971 972From a compliant decoder perspective, skippable frames need just be skipped, 973and their content ignored, resuming decoding after the skippable frame. 974 975It can be noted that a skippable frame 976can be used to watermark a stream of concatenated frames 977embedding any kind of tracking information (even just an UUID). 978Users wary of such possibility should scan the stream of concatenated frames 979in an attempt to detect such frame for analysis or removal. 980 981__`Magic_Number`__ 982 9834 Bytes, __little-endian__ format. 984Value : 0x184D2A5?, which means any value from 0x184D2A50 to 0x184D2A5F. 985All 16 values are valid to identify a skippable frame. 986This specification doesn't detail any specific tagging for skippable frames. 987 988__`Frame_Size`__ 989 990This is the size, in bytes, of the following `User_Data` 991(without including the magic number nor the size field itself). 992This field is represented using 4 Bytes, __little-endian__ format, unsigned 32-bits. 993This means `User_Data` can’t be bigger than (2^32-1) bytes. 994 995__`User_Data`__ 996 997The `User_Data` can be anything. Data will just be skipped by the decoder. 998 999 1000 1001Entropy Encoding 1002---------------- 1003Two types of entropy encoding are used by the Zstandard format: 1004FSE, and Huffman coding. 1005Huffman is used to compress literals, 1006while FSE is used for all other symbols 1007(`Literals_Length_Code`, `Match_Length_Code`, offset codes) 1008and to compress Huffman headers. 1009 1010 1011FSE 1012--- 1013FSE, short for Finite State Entropy, is an entropy codec based on [ANS]. 1014FSE encoding/decoding involves a state that is carried over between symbols, 1015so decoding must be done in the opposite direction as encoding. 1016Therefore, all FSE bitstreams are read from end to beginning. 1017Note that the order of the bits in the stream is not reversed, 1018we just read the elements in the reverse order they are written. 1019 1020For additional details on FSE, see [Finite State Entropy]. 1021 1022[Finite State Entropy]:https://github.com/Cyan4973/FiniteStateEntropy/ 1023 1024FSE decoding involves a decoding table which has a power of 2 size, and contain three elements: 1025`Symbol`, `Num_Bits`, and `Baseline`. 1026The `log2` of the table size is its `Accuracy_Log`. 1027An FSE state value represents an index in this table. 1028 1029To obtain the initial state value, consume `Accuracy_Log` bits from the stream as a __little-endian__ value. 1030The next symbol in the stream is the `Symbol` indicated in the table for that state. 1031To obtain the next state value, 1032the decoder should consume `Num_Bits` bits from the stream as a __little-endian__ value and add it to `Baseline`. 1033 1034[ANS]: https://en.wikipedia.org/wiki/Asymmetric_Numeral_Systems 1035 1036### FSE Table Description 1037To decode FSE streams, it is necessary to construct the decoding table. 1038The Zstandard format encodes FSE table descriptions as follows: 1039 1040An FSE distribution table describes the probabilities of all symbols 1041from `0` to the last present one (included) 1042on a normalized scale of `1 << Accuracy_Log` . 1043Note that there must be two or more symbols with nonzero probability. 1044 1045It's a bitstream which is read forward, in __little-endian__ fashion. 1046It's not necessary to know bitstream exact size, 1047it will be discovered and reported by the decoding process. 1048 1049The bitstream starts by reporting on which scale it operates. 1050Let's `low4Bits` designate the lowest 4 bits of the first byte : 1051`Accuracy_Log = low4bits + 5`. 1052 1053Then follows each symbol value, from `0` to last present one. 1054The number of bits used by each field is variable. 1055It depends on : 1056 1057- Remaining probabilities + 1 : 1058 __example__ : 1059 Presuming an `Accuracy_Log` of 8, 1060 and presuming 100 probabilities points have already been distributed, 1061 the decoder may read any value from `0` to `256 - 100 + 1 == 157` (inclusive). 1062 Therefore, it must read `log2sup(157) == 8` bits. 1063 1064- Value decoded : small values use 1 less bit : 1065 __example__ : 1066 Presuming values from 0 to 157 (inclusive) are possible, 1067 255-157 = 98 values are remaining in an 8-bits field. 1068 They are used this way : 1069 first 98 values (hence from 0 to 97) use only 7 bits, 1070 values from 98 to 157 use 8 bits. 1071 This is achieved through this scheme : 1072 1073 | Value read | Value decoded | Number of bits used | 1074 | ---------- | ------------- | ------------------- | 1075 | 0 - 97 | 0 - 97 | 7 | 1076 | 98 - 127 | 98 - 127 | 8 | 1077 | 128 - 225 | 0 - 97 | 7 | 1078 | 226 - 255 | 128 - 157 | 8 | 1079 1080Symbols probabilities are read one by one, in order. 1081 1082Probability is obtained from Value decoded by following formula : 1083`Proba = value - 1` 1084 1085It means value `0` becomes negative probability `-1`. 1086`-1` is a special probability, which means "less than 1". 1087Its effect on distribution table is described in the [next section]. 1088For the purpose of calculating total allocated probability points, it counts as one. 1089 1090[next section]:#from-normalized-distribution-to-decoding-tables 1091 1092When a symbol has a __probability__ of `zero`, 1093it is followed by a 2-bits repeat flag. 1094This repeat flag tells how many probabilities of zeroes follow the current one. 1095It provides a number ranging from 0 to 3. 1096If it is a 3, another 2-bits repeat flag follows, and so on. 1097 1098When last symbol reaches cumulated total of `1 << Accuracy_Log`, 1099decoding is complete. 1100If the last symbol makes cumulated total go above `1 << Accuracy_Log`, 1101distribution is considered corrupted. 1102 1103Then the decoder can tell how many bytes were used in this process, 1104and how many symbols are present. 1105The bitstream consumes a round number of bytes. 1106Any remaining bit within the last byte is just unused. 1107 1108#### From normalized distribution to decoding tables 1109 1110The distribution of normalized probabilities is enough 1111to create a unique decoding table. 1112 1113It follows the following build rule : 1114 1115The table has a size of `Table_Size = 1 << Accuracy_Log`. 1116Each cell describes the symbol decoded, 1117and instructions to get the next state (`Number_of_Bits` and `Baseline`). 1118 1119Symbols are scanned in their natural order for "less than 1" probabilities. 1120Symbols with this probability are being attributed a single cell, 1121starting from the end of the table and retreating. 1122These symbols define a full state reset, reading `Accuracy_Log` bits. 1123 1124Then, all remaining symbols, sorted in natural order, are allocated cells. 1125Starting from symbol `0` (if it exists), and table position `0`, 1126each symbol gets allocated as many cells as its probability. 1127Cell allocation is spread, not linear : 1128each successor position follows this rule : 1129 1130``` 1131position += (tableSize>>1) + (tableSize>>3) + 3; 1132position &= tableSize-1; 1133``` 1134 1135A position is skipped if already occupied by a "less than 1" probability symbol. 1136`position` does not reset between symbols, it simply iterates through 1137each position in the table, switching to the next symbol when enough 1138states have been allocated to the current one. 1139 1140The process guarantees that the table is entirely filled. 1141Each cell corresponds to a state value, which contains the symbol being decoded. 1142 1143To add the `Number_of_Bits` and `Baseline` required to retrieve next state, 1144it's first necessary to sort all occurrences of each symbol in state order. 1145Lower states will need 1 more bit than higher ones. 1146The process is repeated for each symbol. 1147 1148__Example__ : 1149Presuming a symbol has a probability of 5, 1150it receives 5 cells, corresponding to 5 state values. 1151These state values are then sorted in natural order. 1152 1153Next power of 2 after 5 is 8. 1154Space of probabilities must be divided into 8 equal parts. 1155Presuming the `Accuracy_Log` is 7, it defines a space of 128 states. 1156Divided by 8, each share is 16 large. 1157 1158In order to reach 8 shares, 8-5=3 lowest states will count "double", 1159doubling their shares (32 in width), hence requiring one more bit. 1160 1161Baseline is assigned starting from the higher states using fewer bits, 1162increasing at each state, then resuming at the first state, 1163each state takes its allocated width from Baseline. 1164 1165| state value | 1 | 39 | 77 | 84 | 122 | 1166| state order | 0 | 1 | 2 | 3 | 4 | 1167| ---------------- | ----- | ----- | ------ | ---- | ------ | 1168| width | 32 | 32 | 32 | 16 | 16 | 1169| `Number_of_Bits` | 5 | 5 | 5 | 4 | 4 | 1170| range number | 2 | 4 | 6 | 0 | 1 | 1171| `Baseline` | 32 | 64 | 96 | 0 | 16 | 1172| range | 32-63 | 64-95 | 96-127 | 0-15 | 16-31 | 1173 1174During decoding, the next state value is determined from current state value, 1175by reading the required `Number_of_Bits`, and adding the specified `Baseline`. 1176 1177See [Appendix A] for the results of this process applied to the default distributions. 1178 1179[Appendix A]: #appendix-a---decoding-tables-for-predefined-codes 1180 1181 1182Huffman Coding 1183-------------- 1184Zstandard Huffman-coded streams are read backwards, 1185similar to the FSE bitstreams. 1186Therefore, to find the start of the bitstream, it is therefore to 1187know the offset of the last byte of the Huffman-coded stream. 1188 1189After writing the last bit containing information, the compressor 1190writes a single `1`-bit and then fills the byte with 0-7 `0` bits of 1191padding. The last byte of the compressed bitstream cannot be `0` for 1192that reason. 1193 1194When decompressing, the last byte containing the padding is the first 1195byte to read. The decompressor needs to skip 0-7 initial `0`-bits and 1196the first `1`-bit it occurs. Afterwards, the useful part of the bitstream 1197begins. 1198 1199The bitstream contains Huffman-coded symbols in __little-endian__ order, 1200with the codes defined by the method below. 1201 1202### Huffman Tree Description 1203 1204Prefix coding represents symbols from an a priori known alphabet 1205by bit sequences (codewords), one codeword for each symbol, 1206in a manner such that different symbols may be represented 1207by bit sequences of different lengths, 1208but a parser can always parse an encoded string 1209unambiguously symbol-by-symbol. 1210 1211Given an alphabet with known symbol frequencies, 1212the Huffman algorithm allows the construction of an optimal prefix code 1213using the fewest bits of any possible prefix codes for that alphabet. 1214 1215Prefix code must not exceed a maximum code length. 1216More bits improve accuracy but cost more header size, 1217and require more memory or more complex decoding operations. 1218This specification limits maximum code length to 11 bits. 1219 1220#### Representation 1221 1222All literal values from zero (included) to last present one (excluded) 1223are represented by `Weight` with values from `0` to `Max_Number_of_Bits`. 1224Transformation from `Weight` to `Number_of_Bits` follows this formula : 1225``` 1226Number_of_Bits = Weight ? (Max_Number_of_Bits + 1 - Weight) : 0 1227``` 1228The last symbol's `Weight` is deduced from previously decoded ones, 1229by completing to the nearest power of 2. 1230This power of 2 gives `Max_Number_of_Bits`, the depth of the current tree. 1231`Max_Number_of_Bits` must be <= 11, 1232otherwise the representation is considered corrupted. 1233 1234__Example__ : 1235Let's presume the following Huffman tree must be described : 1236 1237| literal value | 0 | 1 | 2 | 3 | 4 | 5 | 1238| ---------------- | --- | --- | --- | --- | --- | --- | 1239| `Number_of_Bits` | 1 | 2 | 3 | 0 | 4 | 4 | 1240 1241The tree depth is 4, since its longest elements uses 4 bits 1242(longest elements are the one with smallest frequency). 1243Value `5` will not be listed, as it can be determined from values for 0-4, 1244nor will values above `5` as they are all 0. 1245Values from `0` to `4` will be listed using `Weight` instead of `Number_of_Bits`. 1246Weight formula is : 1247``` 1248Weight = Number_of_Bits ? (Max_Number_of_Bits + 1 - Number_of_Bits) : 0 1249``` 1250It gives the following series of weights : 1251 1252| literal value | 0 | 1 | 2 | 3 | 4 | 1253| ------------- | --- | --- | --- | --- | --- | 1254| `Weight` | 4 | 3 | 2 | 0 | 1 | 1255 1256The decoder will do the inverse operation : 1257having collected weights of literal symbols from `0` to `4`, 1258it knows the last literal, `5`, is present with a non-zero `Weight`. 1259The `Weight` of `5` can be determined by advancing to the next power of 2. 1260The sum of `2^(Weight-1)` (excluding 0's) is : 1261`8 + 4 + 2 + 0 + 1 = 15`. 1262Nearest larger power of 2 value is 16. 1263Therefore, `Max_Number_of_Bits = 4` and `Weight[5] = 16-15 = 1`. 1264 1265#### Huffman Tree header 1266 1267This is a single byte value (0-255), 1268which describes how the series of weights is encoded. 1269 1270- if `headerByte` < 128 : 1271 the series of weights is compressed using FSE (see below). 1272 The length of the FSE-compressed series is equal to `headerByte` (0-127). 1273 1274- if `headerByte` >= 128 : 1275 + the series of weights uses a direct representation, 1276 where each `Weight` is encoded directly as a 4 bits field (0-15). 1277 + They are encoded forward, 2 weights to a byte, 1278 first weight taking the top four bits and second one taking the bottom four. 1279 * e.g. the following operations could be used to read the weights: 1280 `Weight[0] = (Byte[0] >> 4), Weight[1] = (Byte[0] & 0xf)`, etc. 1281 + The full representation occupies `Ceiling(Number_of_Weights/2)` bytes, 1282 meaning it uses only full bytes even if `Number_of_Weights` is odd. 1283 + `Number_of_Weights = headerByte - 127`. 1284 * Note that maximum `Number_of_Weights` is 255-127 = 128, 1285 therefore, only up to 128 `Weight` can be encoded using direct representation. 1286 * Since the last non-zero `Weight` is _not_ encoded, 1287 this scheme is compatible with alphabet sizes of up to 129 symbols, 1288 hence including literal symbol 128. 1289 * If any literal symbol > 128 has a non-zero `Weight`, 1290 direct representation is not possible. 1291 In such case, it's necessary to use FSE compression. 1292 1293 1294#### Finite State Entropy (FSE) compression of Huffman weights 1295 1296In this case, the series of Huffman weights is compressed using FSE compression. 1297It's a single bitstream with 2 interleaved states, 1298sharing a single distribution table. 1299 1300To decode an FSE bitstream, it is necessary to know its compressed size. 1301Compressed size is provided by `headerByte`. 1302It's also necessary to know its _maximum possible_ decompressed size, 1303which is `255`, since literal values span from `0` to `255`, 1304and last symbol's `Weight` is not represented. 1305 1306An FSE bitstream starts by a header, describing probabilities distribution. 1307It will create a Decoding Table. 1308For a list of Huffman weights, the maximum accuracy log is 6 bits. 1309For more description see the [FSE header description](#fse-table-description) 1310 1311The Huffman header compression uses 2 states, 1312which share the same FSE distribution table. 1313The first state (`State1`) encodes the even indexed symbols, 1314and the second (`State2`) encodes the odd indexed symbols. 1315`State1` is initialized first, and then `State2`, and they take turns 1316decoding a single symbol and updating their state. 1317For more details on these FSE operations, see the [FSE section](#fse). 1318 1319The number of symbols to decode is determined 1320by tracking bitStream overflow condition: 1321If updating state after decoding a symbol would require more bits than 1322remain in the stream, it is assumed that extra bits are 0. Then, 1323symbols for each of the final states are decoded and the process is complete. 1324 1325#### Conversion from weights to Huffman prefix codes 1326 1327All present symbols shall now have a `Weight` value. 1328It is possible to transform weights into `Number_of_Bits`, using this formula: 1329``` 1330Number_of_Bits = (Weight>0) ? Max_Number_of_Bits + 1 - Weight : 0 1331``` 1332Symbols are sorted by `Weight`. 1333Within same `Weight`, symbols keep natural sequential order. 1334Symbols with a `Weight` of zero are removed. 1335Then, starting from lowest `Weight`, prefix codes are distributed in sequential order. 1336 1337__Example__ : 1338Let's presume the following list of weights has been decoded : 1339 1340| Literal | 0 | 1 | 2 | 3 | 4 | 5 | 1341| -------- | --- | --- | --- | --- | --- | --- | 1342| `Weight` | 4 | 3 | 2 | 0 | 1 | 1 | 1343 1344Sorted by weight and then natural sequential order, 1345it gives the following distribution : 1346 1347| Literal | 3 | 4 | 5 | 2 | 1 | 0 | 1348| ---------------- | --- | --- | --- | --- | --- | ---- | 1349| `Weight` | 0 | 1 | 1 | 2 | 3 | 4 | 1350| `Number_of_Bits` | 0 | 4 | 4 | 3 | 2 | 1 | 1351| prefix codes | N/A | 0000| 0001| 001 | 01 | 1 | 1352 1353### Huffman-coded Streams 1354 1355Given a Huffman decoding table, 1356it's possible to decode a Huffman-coded stream. 1357 1358Each bitstream must be read _backward_, 1359that is starting from the end down to the beginning. 1360Therefore it's necessary to know the size of each bitstream. 1361 1362It's also necessary to know exactly which _bit_ is the last one. 1363This is detected by a final bit flag : 1364the highest bit of latest byte is a final-bit-flag. 1365Consequently, a last byte of `0` is not possible. 1366And the final-bit-flag itself is not part of the useful bitstream. 1367Hence, the last byte contains between 0 and 7 useful bits. 1368 1369Starting from the end, 1370it's possible to read the bitstream in a __little-endian__ fashion, 1371keeping track of already used bits. Since the bitstream is encoded in reverse 1372order, starting from the end read symbols in forward order. 1373 1374For example, if the literal sequence "0145" was encoded using above prefix code, 1375it would be encoded (in reverse order) as: 1376 1377|Symbol | 5 | 4 | 1 | 0 | Padding | 1378|--------|------|------|----|---|---------| 1379|Encoding|`0000`|`0001`|`01`|`1`| `00001` | 1380 1381Resulting in following 2-bytes bitstream : 1382``` 138300010000 00001101 1384``` 1385 1386Here is an alternative representation with the symbol codes separated by underscore: 1387``` 13880001_0000 00001_1_01 1389``` 1390 1391Reading highest `Max_Number_of_Bits` bits, 1392it's possible to compare extracted value to decoding table, 1393determining the symbol to decode and number of bits to discard. 1394 1395The process continues up to reading the required number of symbols per stream. 1396If a bitstream is not entirely and exactly consumed, 1397hence reaching exactly its beginning position with _all_ bits consumed, 1398the decoding process is considered faulty. 1399 1400 1401Dictionary Format 1402----------------- 1403 1404Zstandard is compatible with "raw content" dictionaries, 1405free of any format restriction, except that they must be at least 8 bytes. 1406These dictionaries function as if they were just the `Content` part 1407of a formatted dictionary. 1408 1409But dictionaries created by `zstd --train` follow a format, described here. 1410 1411__Pre-requisites__ : a dictionary has a size, 1412 defined either by a buffer limit, or a file size. 1413 1414| `Magic_Number` | `Dictionary_ID` | `Entropy_Tables` | `Content` | 1415| -------------- | --------------- | ---------------- | --------- | 1416 1417__`Magic_Number`__ : 4 bytes ID, value 0xEC30A437, __little-endian__ format 1418 1419__`Dictionary_ID`__ : 4 bytes, stored in __little-endian__ format. 1420 `Dictionary_ID` can be any value, except 0 (which means no `Dictionary_ID`). 1421 It's used by decoders to check if they use the correct dictionary. 1422 1423_Reserved ranges :_ 1424If the dictionary is going to be distributed in a public environment, 1425the following ranges of `Dictionary_ID` are reserved for some future registrar 1426and shall not be used : 1427 1428 - low range : <= 32767 1429 - high range : >= (2^31) 1430 1431Outside of these ranges, any value of `Dictionary_ID` 1432which is both `>= 32768` and `< (1<<31)` can be used freely, 1433even in public environment. 1434 1435 1436__`Entropy_Tables`__ : follow the same format as tables in [compressed blocks]. 1437 See the relevant [FSE](#fse-table-description) 1438 and [Huffman](#huffman-tree-description) sections for how to decode these tables. 1439 They are stored in following order : 1440 Huffman tables for literals, FSE table for offsets, 1441 FSE table for match lengths, and FSE table for literals lengths. 1442 These tables populate the Repeat Stats literals mode and 1443 Repeat distribution mode for sequence decoding. 1444 It's finally followed by 3 offset values, populating recent offsets (instead of using `{1,4,8}`), 1445 stored in order, 4-bytes __little-endian__ each, for a total of 12 bytes. 1446 Each recent offset must have a value <= dictionary content size, and cannot equal 0. 1447 1448__`Content`__ : The rest of the dictionary is its content. 1449 The content act as a "past" in front of data to compress or decompress, 1450 so it can be referenced in sequence commands. 1451 As long as the amount of data decoded from this frame is less than or 1452 equal to `Window_Size`, sequence commands may specify offsets longer 1453 than the total length of decoded output so far to reference back to the 1454 dictionary, even parts of the dictionary with offsets larger than `Window_Size`. 1455 After the total output has surpassed `Window_Size` however, 1456 this is no longer allowed and the dictionary is no longer accessible. 1457 1458[compressed blocks]: #the-format-of-compressed_block 1459 1460If a dictionary is provided by an external source, 1461it should be loaded with great care, its content considered untrusted. 1462 1463 1464 1465Appendix A - Decoding tables for predefined codes 1466------------------------------------------------- 1467 1468This appendix contains FSE decoding tables 1469for the predefined literal length, match length, and offset codes. 1470The tables have been constructed using the algorithm as given above in chapter 1471"from normalized distribution to decoding tables". 1472The tables here can be used as examples 1473to crosscheck that an implementation build its decoding tables correctly. 1474 1475#### Literal Length Code: 1476 1477| State | Symbol | Number_Of_Bits | Base | 1478| ----- | ------ | -------------- | ---- | 1479| 0 | 0 | 4 | 0 | 1480| 1 | 0 | 4 | 16 | 1481| 2 | 1 | 5 | 32 | 1482| 3 | 3 | 5 | 0 | 1483| 4 | 4 | 5 | 0 | 1484| 5 | 6 | 5 | 0 | 1485| 6 | 7 | 5 | 0 | 1486| 7 | 9 | 5 | 0 | 1487| 8 | 10 | 5 | 0 | 1488| 9 | 12 | 5 | 0 | 1489| 10 | 14 | 6 | 0 | 1490| 11 | 16 | 5 | 0 | 1491| 12 | 18 | 5 | 0 | 1492| 13 | 19 | 5 | 0 | 1493| 14 | 21 | 5 | 0 | 1494| 15 | 22 | 5 | 0 | 1495| 16 | 24 | 5 | 0 | 1496| 17 | 25 | 5 | 32 | 1497| 18 | 26 | 5 | 0 | 1498| 19 | 27 | 6 | 0 | 1499| 20 | 29 | 6 | 0 | 1500| 21 | 31 | 6 | 0 | 1501| 22 | 0 | 4 | 32 | 1502| 23 | 1 | 4 | 0 | 1503| 24 | 2 | 5 | 0 | 1504| 25 | 4 | 5 | 32 | 1505| 26 | 5 | 5 | 0 | 1506| 27 | 7 | 5 | 32 | 1507| 28 | 8 | 5 | 0 | 1508| 29 | 10 | 5 | 32 | 1509| 30 | 11 | 5 | 0 | 1510| 31 | 13 | 6 | 0 | 1511| 32 | 16 | 5 | 32 | 1512| 33 | 17 | 5 | 0 | 1513| 34 | 19 | 5 | 32 | 1514| 35 | 20 | 5 | 0 | 1515| 36 | 22 | 5 | 32 | 1516| 37 | 23 | 5 | 0 | 1517| 38 | 25 | 4 | 0 | 1518| 39 | 25 | 4 | 16 | 1519| 40 | 26 | 5 | 32 | 1520| 41 | 28 | 6 | 0 | 1521| 42 | 30 | 6 | 0 | 1522| 43 | 0 | 4 | 48 | 1523| 44 | 1 | 4 | 16 | 1524| 45 | 2 | 5 | 32 | 1525| 46 | 3 | 5 | 32 | 1526| 47 | 5 | 5 | 32 | 1527| 48 | 6 | 5 | 32 | 1528| 49 | 8 | 5 | 32 | 1529| 50 | 9 | 5 | 32 | 1530| 51 | 11 | 5 | 32 | 1531| 52 | 12 | 5 | 32 | 1532| 53 | 15 | 6 | 0 | 1533| 54 | 17 | 5 | 32 | 1534| 55 | 18 | 5 | 32 | 1535| 56 | 20 | 5 | 32 | 1536| 57 | 21 | 5 | 32 | 1537| 58 | 23 | 5 | 32 | 1538| 59 | 24 | 5 | 32 | 1539| 60 | 35 | 6 | 0 | 1540| 61 | 34 | 6 | 0 | 1541| 62 | 33 | 6 | 0 | 1542| 63 | 32 | 6 | 0 | 1543 1544#### Match Length Code: 1545 1546| State | Symbol | Number_Of_Bits | Base | 1547| ----- | ------ | -------------- | ---- | 1548| 0 | 0 | 6 | 0 | 1549| 1 | 1 | 4 | 0 | 1550| 2 | 2 | 5 | 32 | 1551| 3 | 3 | 5 | 0 | 1552| 4 | 5 | 5 | 0 | 1553| 5 | 6 | 5 | 0 | 1554| 6 | 8 | 5 | 0 | 1555| 7 | 10 | 6 | 0 | 1556| 8 | 13 | 6 | 0 | 1557| 9 | 16 | 6 | 0 | 1558| 10 | 19 | 6 | 0 | 1559| 11 | 22 | 6 | 0 | 1560| 12 | 25 | 6 | 0 | 1561| 13 | 28 | 6 | 0 | 1562| 14 | 31 | 6 | 0 | 1563| 15 | 33 | 6 | 0 | 1564| 16 | 35 | 6 | 0 | 1565| 17 | 37 | 6 | 0 | 1566| 18 | 39 | 6 | 0 | 1567| 19 | 41 | 6 | 0 | 1568| 20 | 43 | 6 | 0 | 1569| 21 | 45 | 6 | 0 | 1570| 22 | 1 | 4 | 16 | 1571| 23 | 2 | 4 | 0 | 1572| 24 | 3 | 5 | 32 | 1573| 25 | 4 | 5 | 0 | 1574| 26 | 6 | 5 | 32 | 1575| 27 | 7 | 5 | 0 | 1576| 28 | 9 | 6 | 0 | 1577| 29 | 12 | 6 | 0 | 1578| 30 | 15 | 6 | 0 | 1579| 31 | 18 | 6 | 0 | 1580| 32 | 21 | 6 | 0 | 1581| 33 | 24 | 6 | 0 | 1582| 34 | 27 | 6 | 0 | 1583| 35 | 30 | 6 | 0 | 1584| 36 | 32 | 6 | 0 | 1585| 37 | 34 | 6 | 0 | 1586| 38 | 36 | 6 | 0 | 1587| 39 | 38 | 6 | 0 | 1588| 40 | 40 | 6 | 0 | 1589| 41 | 42 | 6 | 0 | 1590| 42 | 44 | 6 | 0 | 1591| 43 | 1 | 4 | 32 | 1592| 44 | 1 | 4 | 48 | 1593| 45 | 2 | 4 | 16 | 1594| 46 | 4 | 5 | 32 | 1595| 47 | 5 | 5 | 32 | 1596| 48 | 7 | 5 | 32 | 1597| 49 | 8 | 5 | 32 | 1598| 50 | 11 | 6 | 0 | 1599| 51 | 14 | 6 | 0 | 1600| 52 | 17 | 6 | 0 | 1601| 53 | 20 | 6 | 0 | 1602| 54 | 23 | 6 | 0 | 1603| 55 | 26 | 6 | 0 | 1604| 56 | 29 | 6 | 0 | 1605| 57 | 52 | 6 | 0 | 1606| 58 | 51 | 6 | 0 | 1607| 59 | 50 | 6 | 0 | 1608| 60 | 49 | 6 | 0 | 1609| 61 | 48 | 6 | 0 | 1610| 62 | 47 | 6 | 0 | 1611| 63 | 46 | 6 | 0 | 1612 1613#### Offset Code: 1614 1615| State | Symbol | Number_Of_Bits | Base | 1616| ----- | ------ | -------------- | ---- | 1617| 0 | 0 | 5 | 0 | 1618| 1 | 6 | 4 | 0 | 1619| 2 | 9 | 5 | 0 | 1620| 3 | 15 | 5 | 0 | 1621| 4 | 21 | 5 | 0 | 1622| 5 | 3 | 5 | 0 | 1623| 6 | 7 | 4 | 0 | 1624| 7 | 12 | 5 | 0 | 1625| 8 | 18 | 5 | 0 | 1626| 9 | 23 | 5 | 0 | 1627| 10 | 5 | 5 | 0 | 1628| 11 | 8 | 4 | 0 | 1629| 12 | 14 | 5 | 0 | 1630| 13 | 20 | 5 | 0 | 1631| 14 | 2 | 5 | 0 | 1632| 15 | 7 | 4 | 16 | 1633| 16 | 11 | 5 | 0 | 1634| 17 | 17 | 5 | 0 | 1635| 18 | 22 | 5 | 0 | 1636| 19 | 4 | 5 | 0 | 1637| 20 | 8 | 4 | 16 | 1638| 21 | 13 | 5 | 0 | 1639| 22 | 19 | 5 | 0 | 1640| 23 | 1 | 5 | 0 | 1641| 24 | 6 | 4 | 16 | 1642| 25 | 10 | 5 | 0 | 1643| 26 | 16 | 5 | 0 | 1644| 27 | 28 | 5 | 0 | 1645| 28 | 27 | 5 | 0 | 1646| 29 | 26 | 5 | 0 | 1647| 30 | 25 | 5 | 0 | 1648| 31 | 24 | 5 | 0 | 1649 1650 1651 1652Appendix B - Resources for implementers 1653------------------------------------------------- 1654 1655An open source reference implementation is available on : 1656https://github.com/facebook/zstd 1657 1658The project contains a frame generator, called [decodeCorpus], 1659which can be used by any 3rd-party implementation 1660to verify that a tested decoder is compliant with the specification. 1661 1662[decodeCorpus]: https://github.com/facebook/zstd/tree/v1.3.4/tests#decodecorpus---tool-to-generate-zstandard-frames-for-decoder-testing 1663 1664`decodeCorpus` generates random valid frames. 1665A compliant decoder should be able to decode them all, 1666or at least provide a meaningful error code explaining for which reason it cannot 1667(memory limit restrictions for example). 1668 1669 1670Version changes 1671--------------- 1672- 0.3.7 : clarifications for Repeat_Offsets, matching RFC8878 1673- 0.3.6 : clarifications for Dictionary_ID 1674- 0.3.5 : clarifications for Block_Maximum_Size 1675- 0.3.4 : clarifications for FSE decoding table 1676- 0.3.3 : clarifications for field Block_Size 1677- 0.3.2 : remove additional block size restriction on compressed blocks 1678- 0.3.1 : minor clarification regarding offset history update rules 1679- 0.3.0 : minor edits to match RFC8478 1680- 0.2.9 : clarifications for huffman weights direct representation, by Ulrich Kunitz 1681- 0.2.8 : clarifications for IETF RFC discuss 1682- 0.2.7 : clarifications from IETF RFC review, by Vijay Gurbani and Nick Terrell 1683- 0.2.6 : fixed an error in huffman example, by Ulrich Kunitz 1684- 0.2.5 : minor typos and clarifications 1685- 0.2.4 : section restructuring, by Sean Purcell 1686- 0.2.3 : clarified several details, by Sean Purcell 1687- 0.2.2 : added predefined codes, by Johannes Rudolph 1688- 0.2.1 : clarify field names, by Przemyslaw Skibinski 1689- 0.2.0 : numerous format adjustments for zstd v0.8+ 1690- 0.1.2 : limit Huffman tree depth to 11 bits 1691- 0.1.1 : reserved dictID ranges 1692- 0.1.0 : initial release 1693