README.md
1## zlib-ng
2*zlib data compression library for the next generation systems*
3
4Maintained by Hans Kristian Rosbach
5 aka Dead2 (zlib-ng àt circlestorm dót org)
6
7|CI|Status|
8|:-|-|
9|GitHub Actions|[](https://github.com/zlib-ng/zlib-ng/actions) [](https://github.com/zlib-ng/zlib-ng/actions) [](https://github.com/zlib-ng/zlib-ng/actions)|
10|Buildkite|[](https://buildkite.com/circlestorm-productions/zlib-ng)|
11|CodeFactor|[](https://www.codefactor.io/repository/github/zlib-ng/zlib-ng)|
12|OSS-Fuzz|[](https://bugs.chromium.org/p/oss-fuzz/issues/list?sort=-opened&can=1&q=proj:zlib-ng)
13|Codecov|[](https://codecov.io/github/zlib-ng/zlib-ng/)|
14
15
16Features
17--------
18
19* Zlib compatible API with support for dual-linking
20* Modernized native API based on zlib API for ease of porting
21* Modern C99 syntax and a clean code layout
22* Deflate medium and quick algorithms based on Intels zlib fork
23* Support for CPU intrinsics when available
24 * Adler32 implementation using SSSE3, AVX2, Neon & VSX
25 * CRC32-B implementation using PCLMULQDQ & ACLE
26 * Hash table implementation using CRC32-C intrinsics on x86 and ARM
27 * Slide hash implementations using SSE2, AVX2, Neon & VSX
28 * Compare256/258 implementations using SSE4.2 & AVX2
29 * Inflate chunk copying using SSE2, AVX2 & Neon
30 * Support for hardware-accelerated deflate using IBM Z DFLTCC
31* Unaligned memory read/writes and large bit buffer improvements
32* Includes improvements from Cloudflare and Intel forks
33* Configure, CMake, and NMake build system support
34* Comprehensive set of CMake unit tests
35* Code sanitizers, fuzzing, and coverage
36* GitHub Actions continuous integration on Windows, macOS, and Linux
37 * Emulated CI for ARM, AARCH64, PPC, PPC64, SPARC64, S390x using qemu
38
39
40History
41-------
42
43The motivation for this fork came after seeing several 3rd party
44contributions containing new optimizations not getting implemented
45into the official zlib repository.
46
47Mark Adler has been maintaining zlib for a very long time, and he has
48done a great job and hopefully he will continue for a long time yet.
49The idea of zlib-ng is not to replace zlib, but to co-exist as a
50drop-in replacement with a lower threshold for code change.
51
52zlib has a long history and is incredibly portable, even supporting
53lots of systems that predate the Internet. This is great, but it does
54complicate further development and maintainability.
55The zlib code has numerous workarounds for old compilers that do not
56understand ANSI-C or to accommodate systems with limitations such as
57operating in a 16-bit environment.
58
59Many of these workarounds are only maintenance burdens, some of them
60are pretty huge code-wise. For example, the [v]s[n]printf workaround
61code has a whopping 8 different implementations just to cater to
62various old compilers. With this many workarounds cluttered throughout
63the code, new programmers with an idea/interest for zlib will need
64to take some time to figure out why all of these seemingly strange
65things are used, and how to work within those confines.
66
67So I decided to make a fork, merge all the Intel optimizations, merge
68the Cloudflare optimizations that did not conflict, plus a couple
69of other smaller patches. Then I started cleaning out workarounds,
70various dead code, all contrib and example code as there is little
71point in having those in this fork for various reasons.
72
73A lot of improvements have gone into zlib-ng since its start, and
74numerous people and companies have contributed both small and big
75improvements, or valuable testing.
76
77Please read LICENSE.md, it is very simple and very liberal.
78
79
80Build
81-----
82
83There are two ways to build zlib-ng:
84
85### Cmake
86
87To build zlib-ng using the cross-platform makefile generator cmake.
88
89```
90cmake .
91cmake --build . --config Release
92ctest --verbose -C Release
93```
94
95Alternatively, you can use the cmake configuration GUI tool ccmake:
96
97```
98ccmake .
99```
100
101### Configure
102
103To build zlib-ng using the bash configure script:
104
105```
106./configure
107make
108make test
109```
110
111Build Options
112-------------
113
114| CMake | configure | Description | Default |
115|:-------------------------|:-------------------------|:--------------------------------------------------------------------------------------|---------|
116| ZLIB_COMPAT | --zlib-compat | Compile with zlib compatible API | OFF |
117| ZLIB_ENABLE_TESTS | | Build test binaries | ON |
118| WITH_GZFILEOP | --without-gzfileops | Compile with support for gzFile related functions | ON |
119| WITH_OPTIM | --without-optimizations | Build with optimisations | ON |
120| WITH_NEW_STRATEGIES | --without-new-strategies | Use new strategies | ON |
121| WITH_NATIVE_INSTRUCTIONS | --native | Compiles with full instruction set supported on this host (gcc/clang -march=native) | OFF |
122| WITH_SANITIZER | --with-sanitizer | Build with sanitizer (memory, address, undefined) | OFF |
123| WITH_FUZZERS | --with-fuzzers | Build test/fuzz | OFF |
124| WITH_MAINTAINER_WARNINGS | | Build with project maintainer warnings | OFF |
125| WITH_CODE_COVERAGE | | Enable code coverage reporting | OFF |
126
127
128Install
129-------
130
131WARNING: We do not recommend manually installing unless you really
132know what you are doing, because this can potentially override the system
133default zlib library, and any incompatibility or wrong configuration of
134zlib-ng can make the whole system unusable, requiring recovery or reinstall.
135If you still want a manual install, we recommend using the /opt/ path prefix.
136
137For Linux distros, an alternative way to use zlib-ng (if compiled in
138zlib-compat mode) instead of zlib, is through the use of the
139_LD_PRELOAD_ environment variable. If the program is dynamically linked
140with zlib, then zlib-ng will temporarily be used instead by the program,
141without risking system-wide instability.
142
143```
144LD_PRELOAD=/opt/zlib-ng/libz.so.1.2.11.zlib-ng /usr/bin/program
145```
146
147### Cmake
148
149To install zlib-ng system-wide using cmake:
150
151```
152cmake --build . --target install
153```
154
155### Configure
156
157To install zlib-ng system-wide using the configure script:
158
159```
160make install
161```
162
163Contributing
164------------
165
166Zlib-ng is a aiming to be open to contributions, and we would be delighted to
167receive pull requests on github.
168Just remember that any code you submit must be your own and it must be zlib licensed.
169Help with testing and reviewing of pull requests etc is also very much appreciated.
170
171If you are interested in contributing, please consider joining our
172IRC channel #zlib-ng on the Freenode IRC network.
173
174
175Acknowledgments
176----------------
177
178Thanks to Servebolt.com for sponsoring my maintainership of zlib-ng.
179
180Thanks go out to all the people and companies who have taken the time to contribute
181code reviews, testing and/or patches. Zlib-ng would not have been nearly as good without you.
182
183The deflate format used by zlib was defined by Phil Katz.
184The deflate and zlib specifications were written by L. Peter Deutsch.
185
186zlib was originally created by Jean-loup Gailly (compression)
187and Mark Adler (decompression).
188
189
190Advanced Build Options
191----------------------
192
193| CMake | configure | Description | Default |
194|:--------------------------------|:----------------------|:--------------------------------------------------------------------|------------------------|
195| ZLIB_DUAL_LINK | | Dual link tests with system zlib | OFF |
196| UNALIGNED_OK | | Allow unaligned reads | ON (x86, arm) |
197| | --force-sse2 | Skip runtime check for SSE2 instructions (Always on for x86_64) | OFF (x86) |
198| WITH_AVX2 | | Build with AVX2 intrinsics | ON |
199| WITH_SSE2 | | Build with SSE2 intrinsics | ON |
200| WITH_SSE4 | | Build with SSE4 intrinsics | ON |
201| WITH_PCLMULQDQ | | Build with PCLMULQDQ intrinsics | ON |
202| WITH_ACLE | --without-acle | Build with ACLE intrinsics | ON |
203| WITH_NEON | --without-neon | Build with NEON intrinsics | ON |
204| WITH_POWER8 | | Build with POWER8 optimisations | ON |
205| WITH_DFLTCC_DEFLATE | --with-dfltcc-deflate | Build with DFLTCC intrinsics for compression on IBM Z | OFF |
206| WITH_DFLTCC_INFLATE | --with-dfltcc-inflate | Build with DFLTCC intrinsics for decompression on IBM Z | OFF |
207| WITH_UNALIGNED | | Allow optimizations that use unaligned reads if safe on current arch| ON |
208| WITH_INFLATE_STRICT | | Build with strict inflate distance checking | OFF |
209| WITH_INFLATE_ALLOW_INVALID_DIST | | Build with zero fill for inflate invalid distances | OFF |
210| INSTALL_UTILS | | Copy minigzip and minideflate during install | OFF |
211
212
213Related Projects
214----------------
215
216* Fork of the popular minigzip https://github.com/zlib-ng/minizip-ng
217* Python tool to benchmark minigzip/minideflate https://github.com/zlib-ng/deflatebench
218* Python tool to benchmark pigz https://github.com/zlib-ng/pigzbench
219* 3rd party patches for zlib-ng compatibility https://github.com/zlib-ng/patches
220
221
){kind=link}
){kind=link}
){kind=link}
){kind=link}
){kind=link}
){kind=link}