1Zstandard Compression Format
2============================
3
4### Notices
5
6Copyright (c) 2016-2020 Yann Collet, Facebook, Inc.
7
8Permission is granted to copy and distribute this document
9for any purpose and without charge,
10including translations into other languages
11and incorporation into compilations,
12provided that the copyright notice and this notice are preserved,
13and that any substantive changes or deletions from the original
14are clearly marked.
15Distribution of this document is unlimited.
16
17### Version
18
190.3.7 (2020-12-09)
20
21
22Introduction
23------------
24
25The purpose of this document is to define a lossless compressed data format,
26that is independent of CPU type, operating system,
27file system and character set, suitable for
28file compression, pipe and streaming compression,
29using the [Zstandard algorithm](http://www.zstandard.org).
30The text of the specification assumes a basic background in programming
31at the level of bits and other primitive data representations.
32
33The data can be produced or consumed,
34even for an arbitrarily long sequentially presented input data stream,
35using only an a priori bounded amount of intermediate storage,
36and hence can be used in data communications.
37The format uses the Zstandard compression method,
38and optional [xxHash-64 checksum method](http://www.xxhash.org),
39for detection of data corruption.
40
41The data format defined by this specification
42does not attempt to allow random access to compressed data.
43
44Unless otherwise indicated below,
45a compliant compressor must produce data sets
46that conform to the specifications presented here.
47It doesn’t need to support all options though.
48
49A compliant decompressor must be able to decompress
50at least one working set of parameters
51that conforms to the specifications presented here.
52It may also ignore informative fields, such as checksum.
53Whenever it does not support a parameter defined in the compressed stream,
54it must produce a non-ambiguous error code and associated error message
55explaining which parameter is unsupported.
56
57This specification is intended for use by implementers of software
58to compress data into Zstandard format and/or decompress data from Zstandard format.
59The Zstandard format is supported by an open source reference implementation,
60written in portable C, and available at : https://github.com/facebook/zstd .
61
62
63### Overall conventions
64In this document:
65- square brackets i.e. `[` and `]` are used to indicate optional fields or parameters.
66- the naming convention for identifiers is `Mixed_Case_With_Underscores`
67
68### Definitions
69Content compressed by Zstandard is transformed into a Zstandard __frame__.
70Multiple frames can be appended into a single file or stream.
71A frame is completely independent, has a defined beginning and end,
72and a set of parameters which tells the decoder how to decompress it.
73
74A frame encapsulates one or multiple __blocks__.
75Each block contains arbitrary content, which is described by its header,
76and has a guaranteed maximum content size, which depends on frame parameters.
77Unlike frames, each block depends on previous blocks for proper decoding.
78However, each block can be decompressed without waiting for its successor,
79allowing streaming operations.
80
81Overview
82---------
83- [Frames](#frames)
84  - [Zstandard frames](#zstandard-frames)
85    - [Blocks](#blocks)
86      - [Literals Section](#literals-section)
87      - [Sequences Section](#sequences-section)
88      - [Sequence Execution](#sequence-execution)
89  - [Skippable frames](#skippable-frames)
90- [Entropy Encoding](#entropy-encoding)
91  - [FSE](#fse)
92  - [Huffman Coding](#huffman-coding)
93- [Dictionary Format](#dictionary-format)
94
95Frames
96------
97Zstandard compressed data is made of one or more __frames__.
98Each frame is independent and can be decompressed independently of other frames.
99The decompressed content of multiple concatenated frames is the concatenation of
100each frame decompressed content.
101
102There are two frame formats defined by Zstandard:
103  Zstandard frames and Skippable frames.
104Zstandard frames contain compressed data, while
105skippable frames contain custom user metadata.
106
107## Zstandard frames
108The structure of a single Zstandard frame is following:
109
110| `Magic_Number` | `Frame_Header` |`Data_Block`| [More data blocks] | [`Content_Checksum`] |
111|:--------------:|:--------------:|:----------:| ------------------ |:--------------------:|
112|  4 bytes       |  2-14 bytes    |  n bytes   |                    |     0-4 bytes        |
113
114__`Magic_Number`__
115
1164 Bytes, __little-endian__ format.
117Value : 0xFD2FB528
118Note: This value was selected to be less probable to find at the beginning of some random file.
119It avoids trivial patterns (0x00, 0xFF, repeated bytes, increasing bytes, etc.),
120contains byte values outside of ASCII range,
121and doesn't map into UTF8 space.
122It reduces the chances that a text file represent this value by accident.
123
124__`Frame_Header`__
125
1262 to 14 Bytes, detailed in [`Frame_Header`](#frame_header).
127
128__`Data_Block`__
129
130Detailed in [`Blocks`](#blocks).
131That’s where compressed data is stored.
132
133__`Content_Checksum`__
134
135An optional 32-bit checksum, only present if `Content_Checksum_flag` is set.
136The content checksum is the result
137of [xxh64() hash function](http://www.xxhash.org)
138digesting the original (decoded) data as input, and a seed of zero.
139The low 4 bytes of the checksum are stored in __little-endian__ format.
140
141### `Frame_Header`
142
143The `Frame_Header` has a variable size, with a minimum of 2 bytes,
144and up to 14 bytes depending on optional parameters.
145The structure of `Frame_Header` is following:
146
147| `Frame_Header_Descriptor` | [`Window_Descriptor`] | [`Dictionary_ID`] | [`Frame_Content_Size`] |
148| ------------------------- | --------------------- | ----------------- | ---------------------- |
149| 1 byte                    | 0-1 byte              | 0-4 bytes         | 0-8 bytes              |
150
151#### `Frame_Header_Descriptor`
152
153The first header's byte is called the `Frame_Header_Descriptor`.
154It describes which other fields are present.
155Decoding this byte is enough to tell the size of `Frame_Header`.
156
157| Bit number | Field name                |
158| ---------- | ----------                |
159| 7-6        | `Frame_Content_Size_flag` |
160| 5          | `Single_Segment_flag`     |
161| 4          | `Unused_bit`              |
162| 3          | `Reserved_bit`            |
163| 2          | `Content_Checksum_flag`   |
164| 1-0        | `Dictionary_ID_flag`      |
165
166In this table, bit 7 is the highest bit, while bit 0 is the lowest one.
167
168__`Frame_Content_Size_flag`__
169
170This is a 2-bits flag (`= Frame_Header_Descriptor >> 6`),
171specifying if `Frame_Content_Size` (the decompressed data size)
172is provided within the header.
173`Flag_Value` provides `FCS_Field_Size`,
174which is the number of bytes used by `Frame_Content_Size`
175according to the following table:
176
177|  `Flag_Value`  |    0   |  1  |  2  |  3  |
178| -------------- | ------ | --- | --- | --- |
179|`FCS_Field_Size`| 0 or 1 |  2  |  4  |  8  |
180
181When `Flag_Value` is `0`, `FCS_Field_Size` depends on `Single_Segment_flag` :
182if `Single_Segment_flag` is set, `FCS_Field_Size` is 1.
183Otherwise, `FCS_Field_Size` is 0 : `Frame_Content_Size` is not provided.
184
185__`Single_Segment_flag`__
186
187If this flag is set,
188data must be regenerated within a single continuous memory segment.
189
190In this case, `Window_Descriptor` byte is skipped,
191but `Frame_Content_Size` is necessarily present.
192As a consequence, the decoder must allocate a memory segment
193of size equal or larger than `Frame_Content_Size`.
194
195In order to preserve the decoder from unreasonable memory requirements,
196a decoder is allowed to reject a compressed frame
197which requests a memory size beyond decoder's authorized range.
198
199For broader compatibility, decoders are recommended to support
200memory sizes of at least 8 MB.
201This is only a recommendation,
202each decoder is free to support higher or lower limits,
203depending on local limitations.
204
205__`Unused_bit`__
206
207A decoder compliant with this specification version shall not interpret this bit.
208It might be used in any future version,
209to signal a property which is transparent to properly decode the frame.
210An encoder compliant with this specification version must set this bit to zero.
211
212__`Reserved_bit`__
213
214This bit is reserved for some future feature.
215Its value _must be zero_.
216A decoder compliant with this specification version must ensure it is not set.
217This bit may be used in a future revision,
218to signal a feature that must be interpreted to decode the frame correctly.
219
220__`Content_Checksum_flag`__
221
222If this flag is set, a 32-bits `Content_Checksum` will be present at frame's end.
223See `Content_Checksum` paragraph.
224
225__`Dictionary_ID_flag`__
226
227This is a 2-bits flag (`= FHD & 3`),
228telling if a dictionary ID is provided within the header.
229It also specifies the size of this field as `DID_Field_Size`.
230
231|`Flag_Value`    |  0  |  1  |  2  |  3  |
232| -------------- | --- | --- | --- | --- |
233|`DID_Field_Size`|  0  |  1  |  2  |  4  |
234
235#### `Window_Descriptor`
236
237Provides guarantees on minimum memory buffer required to decompress a frame.
238This information is important for decoders to allocate enough memory.
239
240The `Window_Descriptor` byte is optional.
241When `Single_Segment_flag` is set, `Window_Descriptor` is not present.
242In this case, `Window_Size` is `Frame_Content_Size`,
243which can be any value from 0 to 2^64-1 bytes (16 ExaBytes).
244
245| Bit numbers |     7-3    |     2-0    |
246| ----------- | ---------- | ---------- |
247| Field name  | `Exponent` | `Mantissa` |
248
249The minimum memory buffer size is called `Window_Size`.
250It is described by the following formulas :
251```
252windowLog = 10 + Exponent;
253windowBase = 1 << windowLog;
254windowAdd = (windowBase / 8) * Mantissa;
255Window_Size = windowBase + windowAdd;
256```
257The minimum `Window_Size` is 1 KB.
258The maximum `Window_Size` is `(1<<41) + 7*(1<<38)` bytes, which is 3.75 TB.
259
260In general, larger `Window_Size` tend to improve compression ratio,
261but at the cost of memory usage.
262
263To properly decode compressed data,
264a decoder will need to allocate a buffer of at least `Window_Size` bytes.
265
266In order to preserve decoder from unreasonable memory requirements,
267a decoder is allowed to reject a compressed frame
268which requests a memory size beyond decoder's authorized range.
269
270For improved interoperability,
271it's recommended for decoders to support `Window_Size` of up to 8 MB,
272and it's recommended for encoders to not generate frame requiring `Window_Size` larger than 8 MB.
273It's merely a recommendation though,
274decoders are free to support larger or lower limits,
275depending on local limitations.
276
277#### `Dictionary_ID`
278
279This is a variable size field, which contains
280the ID of the dictionary required to properly decode the frame.
281`Dictionary_ID` field is optional. When it's not present,
282it's up to the decoder to know which dictionary to use.
283
284`Dictionary_ID` field size is provided by `DID_Field_Size`.
285`DID_Field_Size` is directly derived from value of `Dictionary_ID_flag`.
2861 byte can represent an ID 0-255.
2872 bytes can represent an ID 0-65535.
2884 bytes can represent an ID 0-4294967295.
289Format is __little-endian__.
290
291It's allowed to represent a small ID (for example `13`)
292with a large 4-bytes dictionary ID, even if it is less efficient.
293
294A value of `0` has same meaning as no `Dictionary_ID`,
295in which case the frame may or may not need a dictionary to be decoded,
296and the ID of such a dictionary is not specified.
297The decoder must know this information by other means.
298
299#### `Frame_Content_Size`
300
301This is the original (uncompressed) size. This information is optional.
302`Frame_Content_Size` uses a variable number of bytes, provided by `FCS_Field_Size`.
303`FCS_Field_Size` is provided by the value of `Frame_Content_Size_flag`.
304`FCS_Field_Size` can be equal to 0 (not present), 1, 2, 4 or 8 bytes.
305
306| `FCS_Field_Size` |    Range   |
307| ---------------- | ---------- |
308|        0         |   unknown  |
309|        1         |   0 - 255  |
310|        2         | 256 - 65791|
311|        4         | 0 - 2^32-1 |
312|        8         | 0 - 2^64-1 |
313
314`Frame_Content_Size` format is __little-endian__.
315When `FCS_Field_Size` is 1, 4 or 8 bytes, the value is read directly.
316When `FCS_Field_Size` is 2, _the offset of 256 is added_.
317It's allowed to represent a small size (for example `18`) using any compatible variant.
318
319
320Blocks
321-------
322
323After `Magic_Number` and `Frame_Header`, there are some number of blocks.
324Each frame must have at least one block,
325but there is no upper limit on the number of blocks per frame.
326
327The structure of a block is as follows:
328
329| `Block_Header` | `Block_Content` |
330|:--------------:|:---------------:|
331|    3 bytes     |     n bytes     |
332
333__`Block_Header`__
334
335`Block_Header` uses 3 bytes, written using __little-endian__ convention.
336It contains 3 fields :
337
338| `Last_Block` | `Block_Type` | `Block_Size` |
339|:------------:|:------------:|:------------:|
340|    bit 0     |  bits 1-2    |  bits 3-23   |
341
342__`Last_Block`__
343
344The lowest bit signals if this block is the last one.
345The frame will end after this last block.
346It may be followed by an optional `Content_Checksum`
347(see [Zstandard Frames](#zstandard-frames)).
348
349__`Block_Type`__
350
351The next 2 bits represent the `Block_Type`.
352`Block_Type` influences the meaning of `Block_Size`.
353There are 4 block types :
354
355|    Value     |      0      |      1      |         2          |     3     |
356| ------------ | ----------- | ----------- | ------------------ | --------- |
357| `Block_Type` | `Raw_Block` | `RLE_Block` | `Compressed_Block` | `Reserved`|
358
359- `Raw_Block` - this is an uncompressed block.
360  `Block_Content` contains `Block_Size` bytes.
361
362- `RLE_Block` - this is a single byte, repeated `Block_Size` times.
363  `Block_Content` consists of a single byte.
364  On the decompression side, this byte must be repeated `Block_Size` times.
365
366- `Compressed_Block` - this is a [Zstandard compressed block](#compressed-blocks),
367  explained later on.
368  `Block_Size` is the length of `Block_Content`, the compressed data.
369  The decompressed size is not known,
370  but its maximum possible value is guaranteed (see below)
371
372- `Reserved` - this is not a block.
373  This value cannot be used with current version of this specification.
374  If such a value is present, it is considered corrupted data.
375
376__`Block_Size`__
377
378The upper 21 bits of `Block_Header` represent the `Block_Size`.
379
380When `Block_Type` is `Compressed_Block` or `Raw_Block`,
381`Block_Size` is the size of `Block_Content` (hence excluding `Block_Header`).
382
383When `Block_Type` is `RLE_Block`, since `Block_Content`’s size is always 1,
384`Block_Size` represents the number of times this byte must be repeated.
385
386`Block_Size` is limited by `Block_Maximum_Size` (see below).
387
388__`Block_Content`__ and __`Block_Maximum_Size`__
389
390The size of `Block_Content` is limited by `Block_Maximum_Size`,
391which is the smallest of:
392-  `Window_Size`
393-  128 KB
394
395`Block_Maximum_Size` is constant for a given frame.
396This maximum is applicable to both the decompressed size
397and the compressed size of any block in the frame.
398
399The reasoning for this limit is that a decoder can read this information
400at the beginning of a frame and use it to allocate buffers.
401The guarantees on the size of blocks ensure that
402the buffers will be large enough for any following block of the valid frame.
403
404
405Compressed Blocks
406-----------------
407To decompress a compressed block, the compressed size must be provided
408from `Block_Size` field within `Block_Header`.
409
410A compressed block consists of 2 sections :
411- [Literals Section](#literals-section)
412- [Sequences Section](#sequences-section)
413
414The results of the two sections are then combined to produce the decompressed
415data in [Sequence Execution](#sequence-execution)
416
417#### Prerequisites
418To decode a compressed block, the following elements are necessary :
419- Previous decoded data, up to a distance of `Window_Size`,
420  or beginning of the Frame, whichever is smaller.
421- List of "recent offsets" from previous `Compressed_Block`.
422- The previous Huffman tree, required by `Treeless_Literals_Block` type
423- Previous FSE decoding tables, required by `Repeat_Mode`
424  for each symbol type (literals lengths, match lengths, offsets)
425
426Note that decoding tables aren't always from the previous `Compressed_Block`.
427
428- Every decoding table can come from a dictionary.
429- The Huffman tree comes from the previous `Compressed_Literals_Block`.
430
431Literals Section
432----------------
433All literals are regrouped in the first part of the block.
434They can be decoded first, and then copied during [Sequence Execution],
435or they can be decoded on the flow during [Sequence Execution].
436
437Literals can be stored uncompressed or compressed using Huffman prefix codes.
438When compressed, an optional tree description can be present,
439followed by 1 or 4 streams.
440
441| `Literals_Section_Header` | [`Huffman_Tree_Description`] | [jumpTable] | Stream1 | [Stream2] | [Stream3] | [Stream4] |
442| ------------------------- | ---------------------------- | ----------- | ------- | --------- | --------- | --------- |
443
444
445### `Literals_Section_Header`
446
447Header is in charge of describing how literals are packed.
448It's a byte-aligned variable-size bitfield, ranging from 1 to 5 bytes,
449using __little-endian__ convention.
450
451| `Literals_Block_Type` | `Size_Format` | `Regenerated_Size` | [`Compressed_Size`] |
452| --------------------- | ------------- | ------------------ | ------------------- |
453|       2 bits          |  1 - 2 bits   |    5 - 20 bits     |     0 - 18 bits     |
454
455In this representation, bits on the left are the lowest bits.
456
457__`Literals_Block_Type`__
458
459This field uses 2 lowest bits of first byte, describing 4 different block types :
460
461| `Literals_Block_Type`       | Value |
462| --------------------------- | ----- |
463| `Raw_Literals_Block`        |   0   |
464| `RLE_Literals_Block`        |   1   |
465| `Compressed_Literals_Block` |   2   |
466| `Treeless_Literals_Block`   |   3   |
467
468- `Raw_Literals_Block` - Literals are stored uncompressed.
469- `RLE_Literals_Block` - Literals consist of a single byte value
470        repeated `Regenerated_Size` times.
471- `Compressed_Literals_Block` - This is a standard Huffman-compressed block,
472        starting with a Huffman tree description.
473        See details below.
474- `Treeless_Literals_Block` - This is a Huffman-compressed block,
475        using Huffman tree _from previous Huffman-compressed literals block_.
476        `Huffman_Tree_Description` will be skipped.
477        Note: If this mode is triggered without any previous Huffman-table in the frame
478        (or [dictionary](#dictionary-format)), this should be treated as data corruption.
479
480__`Size_Format`__
481
482`Size_Format` is divided into 2 families :
483
484- For `Raw_Literals_Block` and `RLE_Literals_Block`,
485  it's only necessary to decode `Regenerated_Size`.
486  There is no `Compressed_Size` field.
487- For `Compressed_Block` and `Treeless_Literals_Block`,
488  it's required to decode both `Compressed_Size`
489  and `Regenerated_Size` (the decompressed size).
490  It's also necessary to decode the number of streams (1 or 4).
491
492For values spanning several bytes, convention is __little-endian__.
493
494__`Size_Format` for `Raw_Literals_Block` and `RLE_Literals_Block`__ :
495
496`Size_Format` uses 1 _or_ 2 bits.
497Its value is : `Size_Format = (Literals_Section_Header[0]>>2) & 3`
498
499- `Size_Format` == 00 or 10 : `Size_Format` uses 1 bit.
500               `Regenerated_Size` uses 5 bits (0-31).
501               `Literals_Section_Header` uses 1 byte.
502               `Regenerated_Size = Literals_Section_Header[0]>>3`
503- `Size_Format` == 01 : `Size_Format` uses 2 bits.
504               `Regenerated_Size` uses 12 bits (0-4095).
505               `Literals_Section_Header` uses 2 bytes.
506               `Regenerated_Size = (Literals_Section_Header[0]>>4) + (Literals_Section_Header[1]<<4)`
507- `Size_Format` == 11 : `Size_Format` uses 2 bits.
508               `Regenerated_Size` uses 20 bits (0-1048575).
509               `Literals_Section_Header` uses 3 bytes.
510               `Regenerated_Size = (Literals_Section_Header[0]>>4) + (Literals_Section_Header[1]<<4) + (Literals_Section_Header[2]<<12)`
511
512Only Stream1 is present for these cases.
513Note : it's allowed to represent a short value (for example `13`)
514using a long format, even if it's less efficient.
515
516__`Size_Format` for `Compressed_Literals_Block` and `Treeless_Literals_Block`__ :
517
518`Size_Format` always uses 2 bits.
519
520- `Size_Format` == 00 : _A single stream_.
521               Both `Regenerated_Size` and `Compressed_Size` use 10 bits (0-1023).
522               `Literals_Section_Header` uses 3 bytes.
523- `Size_Format` == 01 : 4 streams.
524               Both `Regenerated_Size` and `Compressed_Size` use 10 bits (0-1023).
525               `Literals_Section_Header` uses 3 bytes.
526- `Size_Format` == 10 : 4 streams.
527               Both `Regenerated_Size` and `Compressed_Size` use 14 bits (0-16383).
528               `Literals_Section_Header` uses 4 bytes.
529- `Size_Format` == 11 : 4 streams.
530               Both `Regenerated_Size` and `Compressed_Size` use 18 bits (0-262143).
531               `Literals_Section_Header` uses 5 bytes.
532
533Both `Compressed_Size` and `Regenerated_Size` fields follow __little-endian__ convention.
534Note: `Compressed_Size` __includes__ the size of the Huffman Tree description
535_when_ it is present.
536
537#### Raw Literals Block
538The data in Stream1 is `Regenerated_Size` bytes long,
539it contains the raw literals data to be used during [Sequence Execution].
540
541#### RLE Literals Block
542Stream1 consists of a single byte which should be repeated `Regenerated_Size` times
543to generate the decoded literals.
544
545#### Compressed Literals Block and Treeless Literals Block
546Both of these modes contain Huffman encoded data.
547
548For `Treeless_Literals_Block`,
549the Huffman table comes from previously compressed literals block,
550or from a dictionary.
551
552
553### `Huffman_Tree_Description`
554This section is only present when `Literals_Block_Type` type is `Compressed_Literals_Block` (`2`).
555The format of the Huffman tree description can be found at [Huffman Tree description](#huffman-tree-description).
556The size of `Huffman_Tree_Description` is determined during decoding process,
557it must be used to determine where streams begin.
558`Total_Streams_Size = Compressed_Size - Huffman_Tree_Description_Size`.
559
560
561### Jump Table
562The Jump Table is only present when there are 4 Huffman-coded streams.
563
564Reminder : Huffman compressed data consists of either 1 or 4 Huffman-coded streams.
565
566If only one stream is present, it is a single bitstream occupying the entire
567remaining portion of the literals block, encoded as described within
568[Huffman-Coded Streams](#huffman-coded-streams).
569
570If there are four streams, `Literals_Section_Header` only provided
571enough information to know the decompressed and compressed sizes
572of all four streams _combined_.
573The decompressed size of _each_ stream is equal to `(Regenerated_Size+3)/4`,
574except for the last stream which may be up to 3 bytes smaller,
575to reach a total decompressed size as specified in `Regenerated_Size`.
576
577The compressed size of each stream is provided explicitly in the Jump Table.
578Jump Table is 6 bytes long, and consist of three 2-byte __little-endian__ fields,
579describing the compressed sizes of the first three streams.
580`Stream4_Size` is computed from total `Total_Streams_Size` minus sizes of other streams.
581
582`Stream4_Size = Total_Streams_Size - 6 - Stream1_Size - Stream2_Size - Stream3_Size`.
583
584Note: if `Stream1_Size + Stream2_Size + Stream3_Size > Total_Streams_Size`,
585data is considered corrupted.
586
587Each of these 4 bitstreams is then decoded independently as a Huffman-Coded stream,
588as described at [Huffman-Coded Streams](#huffman-coded-streams)
589
590
591Sequences Section
592-----------------
593A compressed block is a succession of _sequences_ .
594A sequence is a literal copy command, followed by a match copy command.
595A literal copy command specifies a length.
596It is the number of bytes to be copied (or extracted) from the Literals Section.
597A match copy command specifies an offset and a length.
598
599When all _sequences_ are decoded,
600if there are literals left in the _literals section_,
601these bytes are added at the end of the block.
602
603This is described in more detail in [Sequence Execution](#sequence-execution).
604
605The `Sequences_Section` regroup all symbols required to decode commands.
606There are 3 symbol types : literals lengths, offsets and match lengths.
607They are encoded together, interleaved, in a single _bitstream_.
608
609The `Sequences_Section` starts by a header,
610followed by optional probability tables for each symbol type,
611followed by the bitstream.
612
613| `Sequences_Section_Header` | [`Literals_Length_Table`] | [`Offset_Table`] | [`Match_Length_Table`] | bitStream |
614| -------------------------- | ------------------------- | ---------------- | ---------------------- | --------- |
615
616To decode the `Sequences_Section`, it's required to know its size.
617Its size is deduced from the size of `Literals_Section`:
618`Sequences_Section_Size = Block_Size - Literals_Section_Size`.
619
620
621#### `Sequences_Section_Header`
622
623Consists of 2 items:
624- `Number_of_Sequences`
625- Symbol compression modes
626
627__`Number_of_Sequences`__
628
629This is a variable size field using between 1 and 3 bytes.
630Let's call its first byte `byte0`.
631- `if (byte0 == 0)` : there are no sequences.
632            The sequence section stops there.
633            Decompressed content is defined entirely as Literals Section content.
634            The FSE tables used in `Repeat_Mode` aren't updated.
635- `if (byte0 < 128)` : `Number_of_Sequences = byte0` . Uses 1 byte.
636- `if (byte0 < 255)` : `Number_of_Sequences = ((byte0-128) << 8) + byte1` . Uses 2 bytes.
637- `if (byte0 == 255)`: `Number_of_Sequences = byte1 + (byte2<<8) + 0x7F00` . Uses 3 bytes.
638
639__Symbol compression modes__
640
641This is a single byte, defining the compression mode of each symbol type.
642
643|Bit number|          7-6            |      5-4       |        3-2           |     1-0    |
644| -------- | ----------------------- | -------------- | -------------------- | ---------- |
645|Field name| `Literals_Lengths_Mode` | `Offsets_Mode` | `Match_Lengths_Mode` | `Reserved` |
646
647The last field, `Reserved`, must be all-zeroes.
648
649`Literals_Lengths_Mode`, `Offsets_Mode` and `Match_Lengths_Mode` define the `Compression_Mode` of
650literals lengths, offsets, and match lengths symbols respectively.
651
652They follow the same enumeration :
653
654|        Value       |         0         |      1     |           2           |       3       |
655| ------------------ | ----------------- | ---------- | --------------------- | ------------- |
656| `Compression_Mode` | `Predefined_Mode` | `RLE_Mode` | `FSE_Compressed_Mode` | `Repeat_Mode` |
657
658- `Predefined_Mode` : A predefined FSE distribution table is used, defined in
659          [default distributions](#default-distributions).
660          No distribution table will be present.
661- `RLE_Mode` : The table description consists of a single byte, which contains the symbol's value.
662          This symbol will be used for all sequences.
663- `FSE_Compressed_Mode` : standard FSE compression.
664          A distribution table will be present.
665          The format of this distribution table is described in [FSE Table Description](#fse-table-description).
666          Note that the maximum allowed accuracy log for literals length and match length tables is 9,
667          and the maximum accuracy log for the offsets table is 8.
668          `FSE_Compressed_Mode` must not be used when only one symbol is present,
669          `RLE_Mode` should be used instead (although any other mode will work).
670- `Repeat_Mode` : The table used in the previous `Compressed_Block` with `Number_of_Sequences > 0` will be used again,
671          or if this is the first block, table in the dictionary will be used.
672          Note that this includes `RLE_mode`, so if `Repeat_Mode` follows `RLE_Mode`, the same symbol will be repeated.
673          It also includes `Predefined_Mode`, in which case `Repeat_Mode` will have same outcome as `Predefined_Mode`.
674          No distribution table will be present.
675          If this mode is used without any previous sequence table in the frame
676          (nor [dictionary](#dictionary-format)) to repeat, this should be treated as corruption.
677
678#### The codes for literals lengths, match lengths, and offsets.
679
680Each symbol is a _code_ in its own context,
681which specifies `Baseline` and `Number_of_Bits` to add.
682_Codes_ are FSE compressed,
683and interleaved with raw additional bits in the same bitstream.
684
685##### Literals length codes
686
687Literals length codes are values ranging from `0` to `35` included.
688They define lengths from 0 to 131071 bytes.
689The literals length is equal to the decoded `Baseline` plus
690the result of reading `Number_of_Bits` bits from the bitstream,
691as a __little-endian__ value.
692
693| `Literals_Length_Code` |         0-15           |
694| ---------------------- | ---------------------- |
695| length                 | `Literals_Length_Code` |
696| `Number_of_Bits`       |          0             |
697
698| `Literals_Length_Code` |  16  |  17  |  18  |  19  |  20  |  21  |  22  |  23  |
699| ---------------------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
700| `Baseline`             |  16  |  18  |  20  |  22  |  24  |  28  |  32  |  40  |
701| `Number_of_Bits`       |   1  |   1  |   1  |   1  |   2  |   2  |   3  |   3  |
702
703| `Literals_Length_Code` |  24  |  25  |  26  |  27  |  28  |  29  |  30  |  31  |
704| ---------------------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
705| `Baseline`             |  48  |  64  |  128 |  256 |  512 | 1024 | 2048 | 4096 |
706| `Number_of_Bits`       |   4  |   6  |   7  |   8  |   9  |  10  |  11  |  12  |
707
708| `Literals_Length_Code` |  32  |  33  |  34  |  35  |
709| ---------------------- | ---- | ---- | ---- | ---- |
710| `Baseline`             | 8192 |16384 |32768 |65536 |
711| `Number_of_Bits`       |  13  |  14  |  15  |  16  |
712
713
714##### Match length codes
715
716Match length codes are values ranging from `0` to `52` included.
717They define lengths from 3 to 131074 bytes.
718The match length is equal to the decoded `Baseline` plus
719the result of reading `Number_of_Bits` bits from the bitstream,
720as a __little-endian__ value.
721
722| `Match_Length_Code` |         0-31            |
723| ------------------- | ----------------------- |
724| value               | `Match_Length_Code` + 3 |
725| `Number_of_Bits`    |          0              |
726
727| `Match_Length_Code` |  32  |  33  |  34  |  35  |  36  |  37  |  38  |  39  |
728| ------------------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
729| `Baseline`          |  35  |  37  |  39  |  41  |  43  |  47  |  51  |  59  |
730| `Number_of_Bits`    |   1  |   1  |   1  |   1  |   2  |   2  |   3  |   3  |
731
732| `Match_Length_Code` |  40  |  41  |  42  |  43  |  44  |  45  |  46  |  47  |
733| ------------------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
734| `Baseline`          |  67  |  83  |  99  |  131 |  259 |  515 | 1027 | 2051 |
735| `Number_of_Bits`    |   4  |   4  |   5  |   7  |   8  |   9  |  10  |  11  |
736
737| `Match_Length_Code` |  48  |  49  |  50  |  51  |  52  |
738| ------------------- | ---- | ---- | ---- | ---- | ---- |
739| `Baseline`          | 4099 | 8195 |16387 |32771 |65539 |
740| `Number_of_Bits`    |  12  |  13  |  14  |  15  |  16  |
741
742##### Offset codes
743
744Offset codes are values ranging from `0` to `N`.
745
746A decoder is free to limit its maximum `N` supported.
747Recommendation is to support at least up to `22`.
748For information, at the time of this writing.
749the reference decoder supports a maximum `N` value of `31`.
750
751An offset code is also the number of additional bits to read in __little-endian__ fashion,
752and can be translated into an `Offset_Value` using the following formulas :
753
754```
755Offset_Value = (1 << offsetCode) + readNBits(offsetCode);
756if (Offset_Value > 3) offset = Offset_Value - 3;
757```
758It means that maximum `Offset_Value` is `(2^(N+1))-1`
759supporting back-reference distances up to `(2^(N+1))-4`,
760but is limited by [maximum back-reference distance](#window_descriptor).
761
762`Offset_Value` from 1 to 3 are special : they define "repeat codes".
763This is described in more detail in [Repeat Offsets](#repeat-offsets).
764
765#### Decoding Sequences
766FSE bitstreams are read in reverse direction than written. In zstd,
767the compressor writes bits forward into a block and the decompressor
768must read the bitstream _backwards_.
769
770To find the start of the bitstream it is therefore necessary to
771know the offset of the last byte of the block which can be found
772by counting `Block_Size` bytes after the block header.
773
774After writing the last bit containing information, the compressor
775writes a single `1`-bit and then fills the byte with 0-7 `0` bits of
776padding. The last byte of the compressed bitstream cannot be `0` for
777that reason.
778
779When decompressing, the last byte containing the padding is the first
780byte to read. The decompressor needs to skip 0-7 initial `0`-bits and
781the first `1`-bit it occurs. Afterwards, the useful part of the bitstream
782begins.
783
784FSE decoding requires a 'state' to be carried from symbol to symbol.
785For more explanation on FSE decoding, see the [FSE section](#fse).
786
787For sequence decoding, a separate state keeps track of each
788literal lengths, offsets, and match lengths symbols.
789Some FSE primitives are also used.
790For more details on the operation of these primitives, see the [FSE section](#fse).
791
792##### Starting states
793The bitstream starts with initial FSE state values,
794each using the required number of bits in their respective _accuracy_,
795decoded previously from their normalized distribution.
796
797It starts by `Literals_Length_State`,
798followed by `Offset_State`,
799and finally `Match_Length_State`.
800
801Reminder : always keep in mind that all values are read _backward_,
802so the 'start' of the bitstream is at the highest position in memory,
803immediately before the last `1`-bit for padding.
804
805After decoding the starting states, a single sequence is decoded
806`Number_Of_Sequences` times.
807These sequences are decoded in order from first to last.
808Since the compressor writes the bitstream in the forward direction,
809this means the compressor must encode the sequences starting with the last
810one and ending with the first.
811
812##### Decoding a sequence
813For each of the symbol types, the FSE state can be used to determine the appropriate code.
814The code then defines the `Baseline` and `Number_of_Bits` to read for each type.
815See the [description of the codes] for how to determine these values.
816
817[description of the codes]: #the-codes-for-literals-lengths-match-lengths-and-offsets
818
819Decoding starts by reading the `Number_of_Bits` required to decode `Offset`.
820It then does the same for `Match_Length`, and then for `Literals_Length`.
821This sequence is then used for [sequence execution](#sequence-execution).
822
823If it is not the last sequence in the block,
824the next operation is to update states.
825Using the rules pre-calculated in the decoding tables,
826`Literals_Length_State` is updated,
827followed by `Match_Length_State`,
828and then `Offset_State`.
829See the [FSE section](#fse) for details on how to update states from the bitstream.
830
831This operation will be repeated `Number_of_Sequences` times.
832At the end, the bitstream shall be entirely consumed,
833otherwise the bitstream is considered corrupted.
834
835#### Default Distributions
836If `Predefined_Mode` is selected for a symbol type,
837its FSE decoding table is generated from a predefined distribution table defined here.
838For details on how to convert this distribution into a decoding table, see the [FSE section].
839
840[FSE section]: #from-normalized-distribution-to-decoding-tables
841
842##### Literals Length
843The decoding table uses an accuracy log of 6 bits (64 states).
844```
845short literalsLength_defaultDistribution[36] =
846        { 4, 3, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1,
847          2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 2, 1, 1, 1, 1, 1,
848         -1,-1,-1,-1 };
849```
850
851##### Match Length
852The decoding table uses an accuracy log of 6 bits (64 states).
853```
854short matchLengths_defaultDistribution[53] =
855        { 1, 4, 3, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1,
856          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
857          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,-1,-1,
858         -1,-1,-1,-1,-1 };
859```
860
861##### Offset Codes
862The decoding table uses an accuracy log of 5 bits (32 states),
863and supports a maximum `N` value of 28, allowing offset values up to 536,870,908 .
864
865If any sequence in the compressed block requires a larger offset than this,
866it's not possible to use the default distribution to represent it.
867```
868short offsetCodes_defaultDistribution[29] =
869        { 1, 1, 1, 1, 1, 1, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1,
870          1, 1, 1, 1, 1, 1, 1, 1,-1,-1,-1,-1,-1 };
871```
872
873
874Sequence Execution
875------------------
876Once literals and sequences have been decoded,
877they are combined to produce the decoded content of a block.
878
879Each sequence consists of a tuple of (`literals_length`, `offset_value`, `match_length`),
880decoded as described in the [Sequences Section](#sequences-section).
881To execute a sequence, first copy `literals_length` bytes
882from the decoded literals to the output.
883
884Then `match_length` bytes are copied from previous decoded data.
885The offset to copy from is determined by `offset_value`:
886if `offset_value > 3`, then the offset is `offset_value - 3`.
887If `offset_value` is from 1-3, the offset is a special repeat offset value.
888See the [repeat offset](#repeat-offsets) section for how the offset is determined
889in this case.
890
891The offset is defined as from the current position, so an offset of 6
892and a match length of 3 means that 3 bytes should be copied from 6 bytes back.
893Note that all offsets leading to previously decoded data
894must be smaller than `Window_Size` defined in `Frame_Header_Descriptor`.
895
896#### Repeat offsets
897As seen in [Sequence Execution](#sequence-execution),
898the first 3 values define a repeated offset and we will call them
899`Repeated_Offset1`, `Repeated_Offset2`, and `Repeated_Offset3`.
900They are sorted in recency order, with `Repeated_Offset1` meaning "most recent one".
901
902If `offset_value == 1`, then the offset used is `Repeated_Offset1`, etc.
903
904There is an exception though, when current sequence's `literals_length = 0`.
905In this case, repeated offsets are shifted by one,
906so an `offset_value` of 1 means `Repeated_Offset2`,
907an `offset_value` of 2 means `Repeated_Offset3`,
908and an `offset_value` of 3 means `Repeated_Offset1 - 1_byte`.
909
910For the first block, the starting offset history is populated with following values :
911`Repeated_Offset1`=1, `Repeated_Offset2`=4, `Repeated_Offset3`=8,
912unless a dictionary is used, in which case they come from the dictionary.
913
914Then each block gets its starting offset history from the ending values of the most recent `Compressed_Block`.
915Note that blocks which are not `Compressed_Block` are skipped, they do not contribute to offset history.
916
917[Offset Codes]: #offset-codes
918
919###### Offset updates rules
920
921During the execution of the sequences of a `Compressed_Block`, the
922`Repeated_Offsets`' values are kept up to date, so that they always represent
923the three most-recently used offsets. In order to achieve that, they are
924updated after executing each sequence in the following way:
925
926When the sequence's `offset_value` does not refer to one of the
927`Repeated_Offsets`--when it has value greater than 3, or when it has value 3
928and the sequence's `literals_length` is zero--the `Repeated_Offsets`' values
929are shifted back one, and `Repeated_Offset1` takes on the value of the
930just-used offset.
931
932Otherwise, when the sequence's `offset_value` refers to one of the
933`Repeated_Offsets`--when it has value 1 or 2, or when it has value 3 and the
934sequence's `literals_length` is non-zero--the `Repeated_Offsets` are re-ordered
935so that `Repeated_Offset1` takes on the value of the used Repeated_Offset, and
936the existing values are pushed back from the first `Repeated_Offset` through to
937the `Repeated_Offset` selected by the `offset_value`. This effectively performs
938a single-stepped wrapping rotation of the values of these offsets, so that
939their order again reflects the recency of their use.
940
941The following table shows the values of the `Repeated_Offsets` as a series of
942sequences are applied to them:
943
944| `offset_value` | `literals_length` | `Repeated_Offset1` | `Repeated_Offset2` | `Repeated_Offset3` | Comment                 |
945|:--------------:|:-----------------:|:------------------:|:------------------:|:------------------:|:-----------------------:|
946|                |                   |                  1 |                  4 |                  8 | starting values         |
947|           1114 |                11 |               1111 |                  1 |                  4 | non-repeat              |
948|              1 |                22 |               1111 |                  1 |                  4 | repeat 1; no change     |
949|           2225 |                22 |               2222 |               1111 |                  1 | non-repeat              |
950|           1114 |               111 |               1111 |               2222 |               1111 | non-repeat              |
951|           3336 |                33 |               3333 |               1111 |               2222 | non-repeat              |
952|              2 |                22 |               1111 |               3333 |               2222 | repeat 2; swap 1 & 2    |
953|              3 |                33 |               2222 |               1111 |               3333 | repeat 3; rotate 3 to 1 |
954|              3 |                 0 |               2221 |               2222 |               1111 | insert resolved offset  |
955|              1 |                 0 |               2222 |               2221 |               3333 | repeat 2                |
956
957
958Skippable Frames
959----------------
960
961| `Magic_Number` | `Frame_Size` | `User_Data` |
962|:--------------:|:------------:|:-----------:|
963|   4 bytes      |  4 bytes     |   n bytes   |
964
965Skippable frames allow the insertion of user-defined metadata
966into a flow of concatenated frames.
967
968Skippable frames defined in this specification are compatible with [LZ4] ones.
969
970[LZ4]:http://www.lz4.org
971
972From a compliant decoder perspective, skippable frames need just be skipped,
973and their content ignored, resuming decoding after the skippable frame.
974
975It can be noted that a skippable frame
976can be used to watermark a stream of concatenated frames
977embedding any kind of tracking information (even just an UUID).
978Users wary of such possibility should scan the stream of concatenated frames
979in an attempt to detect such frame for analysis or removal.
980
981__`Magic_Number`__
982
9834 Bytes, __little-endian__ format.
984Value : 0x184D2A5?, which means any value from 0x184D2A50 to 0x184D2A5F.
985All 16 values are valid to identify a skippable frame.
986This specification doesn't detail any specific tagging for skippable frames.
987
988__`Frame_Size`__
989
990This is the size, in bytes, of the following `User_Data`
991(without including the magic number nor the size field itself).
992This field is represented using 4 Bytes, __little-endian__ format, unsigned 32-bits.
993This means `User_Data` can’t be bigger than (2^32-1) bytes.
994
995__`User_Data`__
996
997The `User_Data` can be anything. Data will just be skipped by the decoder.
998
999
1000
1001Entropy Encoding
1002----------------
1003Two types of entropy encoding are used by the Zstandard format:
1004FSE, and Huffman coding.
1005Huffman is used to compress literals,
1006while FSE is used for all other symbols
1007(`Literals_Length_Code`, `Match_Length_Code`, offset codes)
1008and to compress Huffman headers.
1009
1010
1011FSE
1012---
1013FSE, short for Finite State Entropy, is an entropy codec based on [ANS].
1014FSE encoding/decoding involves a state that is carried over between symbols,
1015so decoding must be done in the opposite direction as encoding.
1016Therefore, all FSE bitstreams are read from end to beginning.
1017Note that the order of the bits in the stream is not reversed,
1018we just read the elements in the reverse order they are written.
1019
1020For additional details on FSE, see [Finite State Entropy].
1021
1022[Finite State Entropy]:https://github.com/Cyan4973/FiniteStateEntropy/
1023
1024FSE decoding involves a decoding table which has a power of 2 size, and contain three elements:
1025`Symbol`, `Num_Bits`, and `Baseline`.
1026The `log2` of the table size is its `Accuracy_Log`.
1027An FSE state value represents an index in this table.
1028
1029To obtain the initial state value, consume `Accuracy_Log` bits from the stream as a __little-endian__ value.
1030The next symbol in the stream is the `Symbol` indicated in the table for that state.
1031To obtain the next state value,
1032the decoder should consume `Num_Bits` bits from the stream as a __little-endian__ value and add it to `Baseline`.
1033
1034[ANS]: https://en.wikipedia.org/wiki/Asymmetric_Numeral_Systems
1035
1036### FSE Table Description
1037To decode FSE streams, it is necessary to construct the decoding table.
1038The Zstandard format encodes FSE table descriptions as follows:
1039
1040An FSE distribution table describes the probabilities of all symbols
1041from `0` to the last present one (included)
1042on a normalized scale of `1 << Accuracy_Log` .
1043Note that there must be two or more symbols with nonzero probability.
1044
1045It's a bitstream which is read forward, in __little-endian__ fashion.
1046It's not necessary to know bitstream exact size,
1047it will be discovered and reported by the decoding process.
1048
1049The bitstream starts by reporting on which scale it operates.
1050Let's `low4Bits` designate the lowest 4 bits of the first byte :
1051`Accuracy_Log = low4bits + 5`.
1052
1053Then follows each symbol value, from `0` to last present one.
1054The number of bits used by each field is variable.
1055It depends on :
1056
1057- Remaining probabilities + 1 :
1058  __example__ :
1059  Presuming an `Accuracy_Log` of 8,
1060  and presuming 100 probabilities points have already been distributed,
1061  the decoder may read any value from `0` to `256 - 100 + 1 == 157` (inclusive).
1062  Therefore, it must read `log2sup(157) == 8` bits.
1063
1064- Value decoded : small values use 1 less bit :
1065  __example__ :
1066  Presuming values from 0 to 157 (inclusive) are possible,
1067  255-157 = 98 values are remaining in an 8-bits field.
1068  They are used this way :
1069  first 98 values (hence from 0 to 97) use only 7 bits,
1070  values from 98 to 157 use 8 bits.
1071  This is achieved through this scheme :
1072
1073  | Value read | Value decoded | Number of bits used |
1074  | ---------- | ------------- | ------------------- |
1075  |   0 -  97  |   0 -  97     |  7                  |
1076  |  98 - 127  |  98 - 127     |  8                  |
1077  | 128 - 225  |   0 -  97     |  7                  |
1078  | 226 - 255  | 128 - 157     |  8                  |
1079
1080Symbols probabilities are read one by one, in order.
1081
1082Probability is obtained from Value decoded by following formula :
1083`Proba = value - 1`
1084
1085It means value `0` becomes negative probability `-1`.
1086`-1` is a special probability, which means "less than 1".
1087Its effect on distribution table is described in the [next section].
1088For the purpose of calculating total allocated probability points, it counts as one.
1089
1090[next section]:#from-normalized-distribution-to-decoding-tables
1091
1092When a symbol has a __probability__ of `zero`,
1093it is followed by a 2-bits repeat flag.
1094This repeat flag tells how many probabilities of zeroes follow the current one.
1095It provides a number ranging from 0 to 3.
1096If it is a 3, another 2-bits repeat flag follows, and so on.
1097
1098When last symbol reaches cumulated total of `1 << Accuracy_Log`,
1099decoding is complete.
1100If the last symbol makes cumulated total go above `1 << Accuracy_Log`,
1101distribution is considered corrupted.
1102
1103Then the decoder can tell how many bytes were used in this process,
1104and how many symbols are present.
1105The bitstream consumes a round number of bytes.
1106Any remaining bit within the last byte is just unused.
1107
1108#### From normalized distribution to decoding tables
1109
1110The distribution of normalized probabilities is enough
1111to create a unique decoding table.
1112
1113It follows the following build rule :
1114
1115The table has a size of `Table_Size = 1 << Accuracy_Log`.
1116Each cell describes the symbol decoded,
1117and instructions to get the next state (`Number_of_Bits` and `Baseline`).
1118
1119Symbols are scanned in their natural order for "less than 1" probabilities.
1120Symbols with this probability are being attributed a single cell,
1121starting from the end of the table and retreating.
1122These symbols define a full state reset, reading `Accuracy_Log` bits.
1123
1124Then, all remaining symbols, sorted in natural order, are allocated cells.
1125Starting from symbol `0` (if it exists), and table position `0`,
1126each symbol gets allocated as many cells as its probability.
1127Cell allocation is spreaded, not linear :
1128each successor position follows this rule :
1129
1130```
1131position += (tableSize>>1) + (tableSize>>3) + 3;
1132position &= tableSize-1;
1133```
1134
1135A position is skipped if already occupied by a "less than 1" probability symbol.
1136`position` does not reset between symbols, it simply iterates through
1137each position in the table, switching to the next symbol when enough
1138states have been allocated to the current one.
1139
1140The process guarantees that the table is entirely filled.
1141Each cell corresponds to a state value, which contains the symbol being decoded.
1142
1143To add the `Number_of_Bits` and `Baseline` required to retrieve next state,
1144it's first necessary to sort all occurrences of each symbol in state order.
1145Lower states will need 1 more bit than higher ones.
1146The process is repeated for each symbol.
1147
1148__Example__ :
1149Presuming a symbol has a probability of 5,
1150it receives 5 cells, corresponding to 5 state values.
1151These state values are then sorted in natural order.
1152
1153Next power of 2 after 5 is 8.
1154Space of probabilities must be divided into 8 equal parts.
1155Presuming the `Accuracy_Log` is 7, it defines a space of 128 states.
1156Divided by 8, each share is 16 large.
1157
1158In order to reach 8 shares, 8-5=3 lowest states will count "double",
1159doubling their shares (32 in width), hence requiring one more bit.
1160
1161Baseline is assigned starting from the higher states using fewer bits,
1162increasing at each state, then resuming at the first state,
1163each state takes its allocated width from Baseline.
1164
1165| state value      |   1   |  39   |   77   |  84  |  122   |
1166| state order      |   0   |   1   |    2   |   3  |    4   |
1167| ---------------- | ----- | ----- | ------ | ---- | ------ |
1168| width            |  32   |  32   |   32   |  16  |   16   |
1169| `Number_of_Bits` |   5   |   5   |    5   |   4  |    4   |
1170| range number     |   2   |   4   |    6   |   0  |    1   |
1171| `Baseline`       |  32   |  64   |   96   |   0  |   16   |
1172| range            | 32-63 | 64-95 | 96-127 | 0-15 | 16-31  |
1173
1174During decoding, the next state value is determined from current state value,
1175by reading the required `Number_of_Bits`, and adding the specified `Baseline`.
1176
1177See [Appendix A] for the results of this process applied to the default distributions.
1178
1179[Appendix A]: #appendix-a---decoding-tables-for-predefined-codes
1180
1181
1182Huffman Coding
1183--------------
1184Zstandard Huffman-coded streams are read backwards,
1185similar to the FSE bitstreams.
1186Therefore, to find the start of the bitstream, it is therefore to
1187know the offset of the last byte of the Huffman-coded stream.
1188
1189After writing the last bit containing information, the compressor
1190writes a single `1`-bit and then fills the byte with 0-7 `0` bits of
1191padding. The last byte of the compressed bitstream cannot be `0` for
1192that reason.
1193
1194When decompressing, the last byte containing the padding is the first
1195byte to read. The decompressor needs to skip 0-7 initial `0`-bits and
1196the first `1`-bit it occurs. Afterwards, the useful part of the bitstream
1197begins.
1198
1199The bitstream contains Huffman-coded symbols in __little-endian__ order,
1200with the codes defined by the method below.
1201
1202### Huffman Tree Description
1203
1204Prefix coding represents symbols from an a priori known alphabet
1205by bit sequences (codewords), one codeword for each symbol,
1206in a manner such that different symbols may be represented
1207by bit sequences of different lengths,
1208but a parser can always parse an encoded string
1209unambiguously symbol-by-symbol.
1210
1211Given an alphabet with known symbol frequencies,
1212the Huffman algorithm allows the construction of an optimal prefix code
1213using the fewest bits of any possible prefix codes for that alphabet.
1214
1215Prefix code must not exceed a maximum code length.
1216More bits improve accuracy but cost more header size,
1217and require more memory or more complex decoding operations.
1218This specification limits maximum code length to 11 bits.
1219
1220#### Representation
1221
1222All literal values from zero (included) to last present one (excluded)
1223are represented by `Weight` with values from `0` to `Max_Number_of_Bits`.
1224Transformation from `Weight` to `Number_of_Bits` follows this formula :
1225```
1226Number_of_Bits = Weight ? (Max_Number_of_Bits + 1 - Weight) : 0
1227```
1228The last symbol's `Weight` is deduced from previously decoded ones,
1229by completing to the nearest power of 2.
1230This power of 2 gives `Max_Number_of_Bits`, the depth of the current tree.
1231`Max_Number_of_Bits` must be <= 11,
1232otherwise the representation is considered corrupted.
1233
1234__Example__ :
1235Let's presume the following Huffman tree must be described :
1236
1237|  literal value   |  0  |  1  |  2  |  3  |  4  |  5  |
1238| ---------------- | --- | --- | --- | --- | --- | --- |
1239| `Number_of_Bits` |  1  |  2  |  3  |  0  |  4  |  4  |
1240
1241The tree depth is 4, since its longest elements uses 4 bits
1242(longest elements are the one with smallest frequency).
1243Value `5` will not be listed, as it can be determined from values for 0-4,
1244nor will values above `5` as they are all 0.
1245Values from `0` to `4` will be listed using `Weight` instead of `Number_of_Bits`.
1246Weight formula is :
1247```
1248Weight = Number_of_Bits ? (Max_Number_of_Bits + 1 - Number_of_Bits) : 0
1249```
1250It gives the following series of weights :
1251
1252| literal value |  0  |  1  |  2  |  3  |  4  |
1253| ------------- | --- | --- | --- | --- | --- |
1254|   `Weight`    |  4  |  3  |  2  |  0  |  1  |
1255
1256The decoder will do the inverse operation :
1257having collected weights of literal symbols from `0` to `4`,
1258it knows the last literal, `5`, is present with a non-zero `Weight`.
1259The `Weight` of `5` can be determined by advancing to the next power of 2.
1260The sum of `2^(Weight-1)` (excluding 0's) is :
1261`8 + 4 + 2 + 0 + 1 = 15`.
1262Nearest larger power of 2 value is 16.
1263Therefore, `Max_Number_of_Bits = 4` and `Weight[5] = 16-15 = 1`.
1264
1265#### Huffman Tree header
1266
1267This is a single byte value (0-255),
1268which describes how the series of weights is encoded.
1269
1270- if `headerByte` < 128 :
1271  the series of weights is compressed using FSE (see below).
1272  The length of the FSE-compressed series is equal to `headerByte` (0-127).
1273
1274- if `headerByte` >= 128 :
1275  + the series of weights uses a direct representation,
1276    where each `Weight` is encoded directly as a 4 bits field (0-15).
1277  + They are encoded forward, 2 weights to a byte,
1278    first weight taking the top four bits and second one taking the bottom four.
1279    * e.g. the following operations could be used to read the weights:
1280      `Weight[0] = (Byte[0] >> 4), Weight[1] = (Byte[0] & 0xf)`, etc.
1281  + The full representation occupies `Ceiling(Number_of_Weights/2)` bytes,
1282    meaning it uses only full bytes even if `Number_of_Weights` is odd.
1283  + `Number_of_Weights = headerByte - 127`.
1284    * Note that maximum `Number_of_Weights` is 255-127 = 128,
1285      therefore, only up to 128 `Weight` can be encoded using direct representation.
1286    * Since the last non-zero `Weight` is _not_ encoded,
1287      this scheme is compatible with alphabet sizes of up to 129 symbols,
1288      hence including literal symbol 128.
1289    * If any literal symbol > 128 has a non-zero `Weight`,
1290      direct representation is not possible.
1291      In such case, it's necessary to use FSE compression.
1292
1293
1294#### Finite State Entropy (FSE) compression of Huffman weights
1295
1296In this case, the series of Huffman weights is compressed using FSE compression.
1297It's a single bitstream with 2 interleaved states,
1298sharing a single distribution table.
1299
1300To decode an FSE bitstream, it is necessary to know its compressed size.
1301Compressed size is provided by `headerByte`.
1302It's also necessary to know its _maximum possible_ decompressed size,
1303which is `255`, since literal values span from `0` to `255`,
1304and last symbol's `Weight` is not represented.
1305
1306An FSE bitstream starts by a header, describing probabilities distribution.
1307It will create a Decoding Table.
1308For a list of Huffman weights, the maximum accuracy log is 6 bits.
1309For more description see the [FSE header description](#fse-table-description)
1310
1311The Huffman header compression uses 2 states,
1312which share the same FSE distribution table.
1313The first state (`State1`) encodes the even indexed symbols,
1314and the second (`State2`) encodes the odd indexed symbols.
1315`State1` is initialized first, and then `State2`, and they take turns
1316decoding a single symbol and updating their state.
1317For more details on these FSE operations, see the [FSE section](#fse).
1318
1319The number of symbols to decode is determined
1320by tracking bitStream overflow condition:
1321If updating state after decoding a symbol would require more bits than
1322remain in the stream, it is assumed that extra bits are 0.  Then,
1323symbols for each of the final states are decoded and the process is complete.
1324
1325#### Conversion from weights to Huffman prefix codes
1326
1327All present symbols shall now have a `Weight` value.
1328It is possible to transform weights into `Number_of_Bits`, using this formula:
1329```
1330Number_of_Bits = (Weight>0) ? Max_Number_of_Bits + 1 - Weight : 0
1331```
1332Symbols are sorted by `Weight`.
1333Within same `Weight`, symbols keep natural sequential order.
1334Symbols with a `Weight` of zero are removed.
1335Then, starting from lowest `Weight`, prefix codes are distributed in sequential order.
1336
1337__Example__ :
1338Let's presume the following list of weights has been decoded :
1339
1340| Literal  |  0  |  1  |  2  |  3  |  4  |  5  |
1341| -------- | --- | --- | --- | --- | --- | --- |
1342| `Weight` |  4  |  3  |  2  |  0  |  1  |  1  |
1343
1344Sorted by weight and then natural sequential order,
1345it gives the following distribution :
1346
1347| Literal          |  3  |  4  |  5  |  2  |  1  |   0  |
1348| ---------------- | --- | --- | --- | --- | --- | ---- |
1349| `Weight`         |  0  |  1  |  1  |  2  |  3  |   4  |
1350| `Number_of_Bits` |  0  |  4  |  4  |  3  |  2  |   1  |
1351| prefix codes     | N/A | 0000| 0001| 001 | 01  |   1  |
1352
1353### Huffman-coded Streams
1354
1355Given a Huffman decoding table,
1356it's possible to decode a Huffman-coded stream.
1357
1358Each bitstream must be read _backward_,
1359that is starting from the end down to the beginning.
1360Therefore it's necessary to know the size of each bitstream.
1361
1362It's also necessary to know exactly which _bit_ is the last one.
1363This is detected by a final bit flag :
1364the highest bit of latest byte is a final-bit-flag.
1365Consequently, a last byte of `0` is not possible.
1366And the final-bit-flag itself is not part of the useful bitstream.
1367Hence, the last byte contains between 0 and 7 useful bits.
1368
1369Starting from the end,
1370it's possible to read the bitstream in a __little-endian__ fashion,
1371keeping track of already used bits. Since the bitstream is encoded in reverse
1372order, starting from the end read symbols in forward order.
1373
1374For example, if the literal sequence "0145" was encoded using above prefix code,
1375it would be encoded (in reverse order) as:
1376
1377|Symbol  |   5  |   4  |  1 | 0 | Padding |
1378|--------|------|------|----|---|---------|
1379|Encoding|`0000`|`0001`|`01`|`1`| `00001` |
1380
1381Resulting in following 2-bytes bitstream :
1382```
138300010000 00001101
1384```
1385
1386Here is an alternative representation with the symbol codes separated by underscore:
1387```
13880001_0000 00001_1_01
1389```
1390
1391Reading highest `Max_Number_of_Bits` bits,
1392it's possible to compare extracted value to decoding table,
1393determining the symbol to decode and number of bits to discard.
1394
1395The process continues up to reading the required number of symbols per stream.
1396If a bitstream is not entirely and exactly consumed,
1397hence reaching exactly its beginning position with _all_ bits consumed,
1398the decoding process is considered faulty.
1399
1400
1401Dictionary Format
1402-----------------
1403
1404Zstandard is compatible with "raw content" dictionaries,
1405free of any format restriction, except that they must be at least 8 bytes.
1406These dictionaries function as if they were just the `Content` part
1407of a formatted dictionary.
1408
1409But dictionaries created by `zstd --train` follow a format, described here.
1410
1411__Pre-requisites__ : a dictionary has a size,
1412                     defined either by a buffer limit, or a file size.
1413
1414| `Magic_Number` | `Dictionary_ID` | `Entropy_Tables` | `Content` |
1415| -------------- | --------------- | ---------------- | --------- |
1416
1417__`Magic_Number`__ : 4 bytes ID, value 0xEC30A437, __little-endian__ format
1418
1419__`Dictionary_ID`__ : 4 bytes, stored in __little-endian__ format.
1420              `Dictionary_ID` can be any value, except 0 (which means no `Dictionary_ID`).
1421              It's used by decoders to check if they use the correct dictionary.
1422
1423_Reserved ranges :_
1424If the dictionary is going to be distributed in a public environment,
1425the following ranges of `Dictionary_ID` are reserved for some future registrar
1426and shall not be used :
1427
1428    - low range  : <= 32767
1429    - high range : >= (2^31)
1430
1431Outside of these ranges, any value of `Dictionary_ID`
1432which is both `>= 32768` and `< (1<<31)` can be used freely,
1433even in public environment.
1434
1435
1436__`Entropy_Tables`__ : follow the same format as tables in [compressed blocks].
1437              See the relevant [FSE](#fse-table-description)
1438              and [Huffman](#huffman-tree-description) sections for how to decode these tables.
1439              They are stored in following order :
1440              Huffman tables for literals, FSE table for offsets,
1441              FSE table for match lengths, and FSE table for literals lengths.
1442              These tables populate the Repeat Stats literals mode and
1443              Repeat distribution mode for sequence decoding.
1444              It's finally followed by 3 offset values, populating recent offsets (instead of using `{1,4,8}`),
1445              stored in order, 4-bytes __little-endian__ each, for a total of 12 bytes.
1446              Each recent offset must have a value <= dictionary content size, and cannot equal 0.
1447
1448__`Content`__ : The rest of the dictionary is its content.
1449              The content act as a "past" in front of data to compress or decompress,
1450              so it can be referenced in sequence commands.
1451              As long as the amount of data decoded from this frame is less than or
1452              equal to `Window_Size`, sequence commands may specify offsets longer
1453              than the total length of decoded output so far to reference back to the
1454              dictionary, even parts of the dictionary with offsets larger than `Window_Size`.
1455              After the total output has surpassed `Window_Size` however,
1456              this is no longer allowed and the dictionary is no longer accessible.
1457
1458[compressed blocks]: #the-format-of-compressed_block
1459
1460If a dictionary is provided by an external source,
1461it should be loaded with great care, its content considered untrusted.
1462
1463
1464
1465Appendix A - Decoding tables for predefined codes
1466-------------------------------------------------
1467
1468This appendix contains FSE decoding tables
1469for the predefined literal length, match length, and offset codes.
1470The tables have been constructed using the algorithm as given above in chapter
1471"from normalized distribution to decoding tables".
1472The tables here can be used as examples
1473to crosscheck that an implementation build its decoding tables correctly.
1474
1475#### Literal Length Code:
1476
1477| State | Symbol | Number_Of_Bits | Base |
1478| ----- | ------ | -------------- | ---- |
1479|     0 |      0 |              4 |    0 |
1480|     1 |      0 |              4 |   16 |
1481|     2 |      1 |              5 |   32 |
1482|     3 |      3 |              5 |    0 |
1483|     4 |      4 |              5 |    0 |
1484|     5 |      6 |              5 |    0 |
1485|     6 |      7 |              5 |    0 |
1486|     7 |      9 |              5 |    0 |
1487|     8 |     10 |              5 |    0 |
1488|     9 |     12 |              5 |    0 |
1489|    10 |     14 |              6 |    0 |
1490|    11 |     16 |              5 |    0 |
1491|    12 |     18 |              5 |    0 |
1492|    13 |     19 |              5 |    0 |
1493|    14 |     21 |              5 |    0 |
1494|    15 |     22 |              5 |    0 |
1495|    16 |     24 |              5 |    0 |
1496|    17 |     25 |              5 |   32 |
1497|    18 |     26 |              5 |    0 |
1498|    19 |     27 |              6 |    0 |
1499|    20 |     29 |              6 |    0 |
1500|    21 |     31 |              6 |    0 |
1501|    22 |      0 |              4 |   32 |
1502|    23 |      1 |              4 |    0 |
1503|    24 |      2 |              5 |    0 |
1504|    25 |      4 |              5 |   32 |
1505|    26 |      5 |              5 |    0 |
1506|    27 |      7 |              5 |   32 |
1507|    28 |      8 |              5 |    0 |
1508|    29 |     10 |              5 |   32 |
1509|    30 |     11 |              5 |    0 |
1510|    31 |     13 |              6 |    0 |
1511|    32 |     16 |              5 |   32 |
1512|    33 |     17 |              5 |    0 |
1513|    34 |     19 |              5 |   32 |
1514|    35 |     20 |              5 |    0 |
1515|    36 |     22 |              5 |   32 |
1516|    37 |     23 |              5 |    0 |
1517|    38 |     25 |              4 |    0 |
1518|    39 |     25 |              4 |   16 |
1519|    40 |     26 |              5 |   32 |
1520|    41 |     28 |              6 |    0 |
1521|    42 |     30 |              6 |    0 |
1522|    43 |      0 |              4 |   48 |
1523|    44 |      1 |              4 |   16 |
1524|    45 |      2 |              5 |   32 |
1525|    46 |      3 |              5 |   32 |
1526|    47 |      5 |              5 |   32 |
1527|    48 |      6 |              5 |   32 |
1528|    49 |      8 |              5 |   32 |
1529|    50 |      9 |              5 |   32 |
1530|    51 |     11 |              5 |   32 |
1531|    52 |     12 |              5 |   32 |
1532|    53 |     15 |              6 |    0 |
1533|    54 |     17 |              5 |   32 |
1534|    55 |     18 |              5 |   32 |
1535|    56 |     20 |              5 |   32 |
1536|    57 |     21 |              5 |   32 |
1537|    58 |     23 |              5 |   32 |
1538|    59 |     24 |              5 |   32 |
1539|    60 |     35 |              6 |    0 |
1540|    61 |     34 |              6 |    0 |
1541|    62 |     33 |              6 |    0 |
1542|    63 |     32 |              6 |    0 |
1543
1544#### Match Length Code:
1545
1546| State | Symbol | Number_Of_Bits | Base |
1547| ----- | ------ | -------------- | ---- |
1548|     0 |      0 |              6 |    0 |
1549|     1 |      1 |              4 |    0 |
1550|     2 |      2 |              5 |   32 |
1551|     3 |      3 |              5 |    0 |
1552|     4 |      5 |              5 |    0 |
1553|     5 |      6 |              5 |    0 |
1554|     6 |      8 |              5 |    0 |
1555|     7 |     10 |              6 |    0 |
1556|     8 |     13 |              6 |    0 |
1557|     9 |     16 |              6 |    0 |
1558|    10 |     19 |              6 |    0 |
1559|    11 |     22 |              6 |    0 |
1560|    12 |     25 |              6 |    0 |
1561|    13 |     28 |              6 |    0 |
1562|    14 |     31 |              6 |    0 |
1563|    15 |     33 |              6 |    0 |
1564|    16 |     35 |              6 |    0 |
1565|    17 |     37 |              6 |    0 |
1566|    18 |     39 |              6 |    0 |
1567|    19 |     41 |              6 |    0 |
1568|    20 |     43 |              6 |    0 |
1569|    21 |     45 |              6 |    0 |
1570|    22 |      1 |              4 |   16 |
1571|    23 |      2 |              4 |    0 |
1572|    24 |      3 |              5 |   32 |
1573|    25 |      4 |              5 |    0 |
1574|    26 |      6 |              5 |   32 |
1575|    27 |      7 |              5 |    0 |
1576|    28 |      9 |              6 |    0 |
1577|    29 |     12 |              6 |    0 |
1578|    30 |     15 |              6 |    0 |
1579|    31 |     18 |              6 |    0 |
1580|    32 |     21 |              6 |    0 |
1581|    33 |     24 |              6 |    0 |
1582|    34 |     27 |              6 |    0 |
1583|    35 |     30 |              6 |    0 |
1584|    36 |     32 |              6 |    0 |
1585|    37 |     34 |              6 |    0 |
1586|    38 |     36 |              6 |    0 |
1587|    39 |     38 |              6 |    0 |
1588|    40 |     40 |              6 |    0 |
1589|    41 |     42 |              6 |    0 |
1590|    42 |     44 |              6 |    0 |
1591|    43 |      1 |              4 |   32 |
1592|    44 |      1 |              4 |   48 |
1593|    45 |      2 |              4 |   16 |
1594|    46 |      4 |              5 |   32 |
1595|    47 |      5 |              5 |   32 |
1596|    48 |      7 |              5 |   32 |
1597|    49 |      8 |              5 |   32 |
1598|    50 |     11 |              6 |    0 |
1599|    51 |     14 |              6 |    0 |
1600|    52 |     17 |              6 |    0 |
1601|    53 |     20 |              6 |    0 |
1602|    54 |     23 |              6 |    0 |
1603|    55 |     26 |              6 |    0 |
1604|    56 |     29 |              6 |    0 |
1605|    57 |     52 |              6 |    0 |
1606|    58 |     51 |              6 |    0 |
1607|    59 |     50 |              6 |    0 |
1608|    60 |     49 |              6 |    0 |
1609|    61 |     48 |              6 |    0 |
1610|    62 |     47 |              6 |    0 |
1611|    63 |     46 |              6 |    0 |
1612
1613#### Offset Code:
1614
1615| State | Symbol | Number_Of_Bits | Base |
1616| ----- | ------ | -------------- | ---- |
1617|     0 |      0 |              5 |    0 |
1618|     1 |      6 |              4 |    0 |
1619|     2 |      9 |              5 |    0 |
1620|     3 |     15 |              5 |    0 |
1621|     4 |     21 |              5 |    0 |
1622|     5 |      3 |              5 |    0 |
1623|     6 |      7 |              4 |    0 |
1624|     7 |     12 |              5 |    0 |
1625|     8 |     18 |              5 |    0 |
1626|     9 |     23 |              5 |    0 |
1627|    10 |      5 |              5 |    0 |
1628|    11 |      8 |              4 |    0 |
1629|    12 |     14 |              5 |    0 |
1630|    13 |     20 |              5 |    0 |
1631|    14 |      2 |              5 |    0 |
1632|    15 |      7 |              4 |   16 |
1633|    16 |     11 |              5 |    0 |
1634|    17 |     17 |              5 |    0 |
1635|    18 |     22 |              5 |    0 |
1636|    19 |      4 |              5 |    0 |
1637|    20 |      8 |              4 |   16 |
1638|    21 |     13 |              5 |    0 |
1639|    22 |     19 |              5 |    0 |
1640|    23 |      1 |              5 |    0 |
1641|    24 |      6 |              4 |   16 |
1642|    25 |     10 |              5 |    0 |
1643|    26 |     16 |              5 |    0 |
1644|    27 |     28 |              5 |    0 |
1645|    28 |     27 |              5 |    0 |
1646|    29 |     26 |              5 |    0 |
1647|    30 |     25 |              5 |    0 |
1648|    31 |     24 |              5 |    0 |
1649
1650
1651
1652Appendix B - Resources for implementers
1653-------------------------------------------------
1654
1655An open source reference implementation is available on :
1656https://github.com/facebook/zstd
1657
1658The project contains a frame generator, called [decodeCorpus],
1659which can be used by any 3rd-party implementation
1660to verify that a tested decoder is compliant with the specification.
1661
1662[decodeCorpus]: https://github.com/facebook/zstd/tree/v1.3.4/tests#decodecorpus---tool-to-generate-zstandard-frames-for-decoder-testing
1663
1664`decodeCorpus` generates random valid frames.
1665A compliant decoder should be able to decode them all,
1666or at least provide a meaningful error code explaining for which reason it cannot
1667(memory limit restrictions for example).
1668
1669
1670Version changes
1671---------------
1672- 0.3.7 : clarifications for Repeat_Offsets
1673- 0.3.6 : clarifications for Dictionary_ID
1674- 0.3.5 : clarifications for Block_Maximum_Size
1675- 0.3.4 : clarifications for FSE decoding table
1676- 0.3.3 : clarifications for field Block_Size
1677- 0.3.2 : remove additional block size restriction on compressed blocks
1678- 0.3.1 : minor clarification regarding offset history update rules
1679- 0.3.0 : minor edits to match RFC8478
1680- 0.2.9 : clarifications for huffman weights direct representation, by Ulrich Kunitz
1681- 0.2.8 : clarifications for IETF RFC discuss
1682- 0.2.7 : clarifications from IETF RFC review, by Vijay Gurbani and Nick Terrell
1683- 0.2.6 : fixed an error in huffman example, by Ulrich Kunitz
1684- 0.2.5 : minor typos and clarifications
1685- 0.2.4 : section restructuring, by Sean Purcell
1686- 0.2.3 : clarified several details, by Sean Purcell
1687- 0.2.2 : added predefined codes, by Johannes Rudolph
1688- 0.2.1 : clarify field names, by Przemyslaw Skibinski
1689- 0.2.0 : numerous format adjustments for zstd v0.8+
1690- 0.1.2 : limit Huffman tree depth to 11 bits
1691- 0.1.1 : reserved dictID ranges
1692- 0.1.0 : initial release
1693