1 2XZ Utils Release Notes 3====================== 4 55.2.5 (2020-03-17) 6 7 * liblzma: 8 9 - Fixed several C99/C11 conformance bugs. Now the code is clean 10 under gcc/clang -fsanitize=undefined. Some of these changes 11 might have a negative effect on performance with old GCC 12 versions or compilers other than GCC and Clang. The configure 13 option --enable-unsafe-type-punning can be used to (mostly) 14 restore the old behavior but it shouldn't normally be used. 15 16 - Improved API documentation of lzma_properties_decode(). 17 18 - Added a very minor encoder speed optimization. 19 20 * xz: 21 22 - Fixed a crash in "xz -dcfv not_an_xz_file". All four options 23 were required to trigger it. The crash occurred in the 24 progress indicator code when xz was in passthru mode where 25 xz works like "cat". 26 27 - Fixed an integer overflow with 32-bit off_t. It could happen 28 when decompressing a file that has a long run of zero bytes 29 which xz would try to write as a sparse file. Since the build 30 system enables large file support by default, off_t is 31 normally 64-bit even on 32-bit systems. 32 33 - Fixes for --flush-timeout: 34 * Fix semi-busy-waiting. 35 * Avoid unneeded flushes when no new input has arrived 36 since the previous flush was completed. 37 38 - Added a special case for 32-bit xz: If --memlimit-compress is 39 used to specify a limit that exceeds 4020 MiB, the limit will 40 be set to 4020 MiB. The values "0" and "max" aren't affected 41 by this and neither is decompression. This hack can be 42 helpful when a 32-bit xz has access to 4 GiB address space 43 but the specified memlimit exceeds 4 GiB. This can happen 44 e.g. with some scripts. 45 46 - Capsicum sandbox is now enabled by default where available 47 (FreeBSD >= 10). The sandbox debug messages (xz -vv) were 48 removed since they seemed to be more annoying than useful. 49 50 - DOS build now requires DJGPP 2.05 instead of 2.04beta. 51 A workaround for a locale problem with DJGPP 2.05 was added. 52 53 * xzgrep and other scripts: 54 55 - Added a configure option --enable-path-for-scripts=PREFIX. 56 It is disabled by default except on Solaris where the default 57 is /usr/xpg4/bin. See INSTALL for details. 58 59 - Added a workaround for a POSIX shell detection problem on 60 Solaris. 61 62 * Build systems: 63 64 - Added preliminary build instructions for z/OS. See INSTALL 65 section 1.2.9. 66 67 - Experimental CMake support was added. It should work to build 68 static liblzma on a few operating systems. It may or may not 69 work to build shared liblzma. On some platforms it can build 70 xz and xzdec too but those are only for testing. See the 71 comment in the beginning of CMakeLists.txt for details. 72 73 - Visual Studio project files were updated. 74 WindowsTargetPlatformVersion was removed from VS2017 files 75 and set to "10.0" in the added VS2019 files. In the future 76 the VS project files will be removed when CMake support is 77 good enough. 78 79 - New #defines in config.h: HAVE___BUILTIN_ASSUME_ALIGNED, 80 HAVE___BUILTIN_BSWAPXX, and TUKLIB_USE_UNSAFE_TYPE_PUNNING. 81 82 - autogen.sh has a new optional dependency on po4a and a new 83 option --no-po4a to skip that step. This matters only if one 84 wants to remake the build files. po4a is used to update the 85 translated man pages but as long as the man pages haven't 86 been modified, there's nothing to update and one can use 87 --no-po4a to avoid the dependency on po4a. 88 89 * Translations: 90 91 - XZ Utils translations are now handled by the Translation 92 Project: https://translationproject.org/domain/xz.html 93 94 - All man pages are now included in German too. 95 96 - New xz translations: Brazilian Portuguese, Finnish, 97 Hungarian, Chinese (simplified), Chinese (traditional), 98 and Danish (partial translation) 99 100 - Updated xz translations: French, German, Italian, and Polish 101 102 - Unfortunately a few new xz translations weren't included due 103 to technical problems like too long lines in --help output or 104 misaligned column headings in tables. In the future, many of 105 these strings will be split and e.g. the table column 106 alignment will be handled in software. This should make the 107 strings easier to translate. 108 109 1105.2.4 (2018-04-29) 111 112 * liblzma: 113 114 - Allow 0 as memory usage limit instead of returning 115 LZMA_PROG_ERROR. Now 0 is treated as if 1 byte was specified, 116 which effectively is the same as 0. 117 118 - Use "noexcept" keyword instead of "throw()" in the public 119 headers when a C++11 (or newer standard) compiler is used. 120 121 - Added a portability fix for recent Intel C Compilers. 122 123 - Microsoft Visual Studio build files have been moved under 124 windows/vs2013 and windows/vs2017. 125 126 * xz: 127 128 - Fix "xz --list --robot missing_or_bad_file.xz" which would 129 try to print an uninitialized string and thus produce garbage 130 output. Since the exit status is non-zero, most uses of such 131 a command won't try to interpret the garbage output. 132 133 - "xz --list foo.xz" could print "Internal error (bug)" in a 134 corner case where a specific memory usage limit had been set. 135 136 1375.2.3 (2016-12-30) 138 139 * xz: 140 141 - Always close a file before trying to delete it to avoid 142 problems on some operating system and file system combinations. 143 144 - Fixed copying of file timestamps on Windows. 145 146 - Added experimental (disabled by default) sandbox support using 147 Capsicum (FreeBSD >= 10). See --enable-sandbox in INSTALL. 148 149 * C99/C11 conformance fixes to liblzma. The issues affected at least 150 some builds using link-time optimizations. 151 152 * Fixed bugs in the rarely-used function lzma_index_dup(). 153 154 * Use of external SHA-256 code is now disabled by default. 155 It can still be enabled by passing --enable-external-sha256 156 to configure. The reasons to disable it by default (see INSTALL 157 for more details): 158 159 - Some OS-specific SHA-256 implementations conflict with 160 OpenSSL and cause problems in programs that link against both 161 liblzma and libcrypto. At least FreeBSD 10 and MINIX 3.3.0 162 are affected. 163 164 - The internal SHA-256 is faster than the SHA-256 code in 165 some operating systems. 166 167 * Changed CPU core count detection to use sched_getaffinity() on 168 GNU/Linux and GNU/kFreeBSD. 169 170 * Fixes to the build-system and xz to make xz buildable even when 171 encoders, decoders, or threading have been disabled from libilzma 172 using configure options. These fixes added two new #defines to 173 config.h: HAVE_ENCODERS and HAVE_DECODERS. 174 175 1765.2.2 (2015-09-29) 177 178 * Fixed bugs in QNX-specific code. 179 180 * Omitted the use of pipe2() even if it is available to avoid 181 portability issues with some old Linux and glibc combinations. 182 183 * Updated German translation. 184 185 * Added project files to build static and shared liblzma (not the 186 whole XZ Utils) with Visual Studio 2013 update 2 or later. 187 188 * Documented that threaded decompression hasn't been implemented 189 yet. A 5.2.0 NEWS entry describing multi-threading support had 190 incorrectly said "decompression" when it should have said 191 "compression". 192 193 1945.2.1 (2015-02-26) 195 196 * Fixed a compression-ratio regression in fast mode of LZMA1 and 197 LZMA2. The bug is present in 5.1.4beta and 5.2.0 releases. 198 199 * Fixed a portability problem in xz that affected at least OpenBSD. 200 201 * Fixed xzdiff to be compatible with FreeBSD's mktemp which differs 202 from most other mktemp implementations. 203 204 * Changed CPU core count detection to use cpuset_getaffinity() on 205 FreeBSD. 206 207 2085.2.0 (2014-12-21) 209 210 Since 5.1.4beta: 211 212 * All fixes from 5.0.8 213 214 * liblzma: Fixed lzma_stream_encoder_mt_memusage() when a preset 215 was used. 216 217 * xzdiff: If mktemp isn't installed, mkdir will be used as 218 a fallback to create a temporary directory. Installing mktemp 219 is still recommended. 220 221 * Updated French, German, Italian, Polish, and Vietnamese 222 translations. 223 224 Summary of fixes and new features added in the 5.1.x development 225 releases: 226 227 * liblzma: 228 229 - Added support for multi-threaded compression. See the 230 lzma_mt structure, lzma_stream_encoder_mt(), and 231 lzma_stream_encoder_mt_memusage() in <lzma/container.h>, 232 lzma_get_progress() in <lzma/base.h>, and lzma_cputhreads() 233 in <lzma/hardware.h> for details. 234 235 - Made the uses of lzma_allocator const correct. 236 237 - Added lzma_block_uncomp_encode() to create uncompressed 238 .xz Blocks using LZMA2 uncompressed chunks. 239 240 - Added support for LZMA_IGNORE_CHECK. 241 242 - A few speed optimizations were made. 243 244 - Added support for symbol versioning. It is enabled by default 245 on GNU/Linux, other GNU-based systems, and FreeBSD. 246 247 - liblzma (not the whole XZ Utils) should now be buildable 248 with MSVC 2013 update 2 or later using windows/config.h. 249 250 * xz: 251 252 - Fixed a race condition in the signal handling. It was 253 possible that e.g. the first SIGINT didn't make xz exit 254 if reading or writing blocked and one had bad luck. The fix 255 is non-trivial, so as of writing it is unknown if it will be 256 backported to the v5.0 branch. 257 258 - Multi-threaded compression can be enabled with the 259 --threads (-T) option. 260 [Fixed: This originally said "decompression".] 261 262 - New command line options in xz: --single-stream, 263 --block-size=SIZE, --block-list=SIZES, 264 --flush-timeout=TIMEOUT, and --ignore-check. 265 266 - xz -lvv now shows the minimum xz version that is required to 267 decompress the file. Currently it is 5.0.0 for all supported 268 .xz files except files with empty LZMA2 streams require 5.0.2. 269 270 * xzdiff and xzgrep now support .lzo files if lzop is installed. 271 The .tzo suffix is also recognized as a shorthand for .tar.lzo. 272 273 2745.1.4beta (2014-09-14) 275 276 * All fixes from 5.0.6 277 278 * liblzma: Fixed the use of presets in threaded encoder 279 initialization. 280 281 * xz --block-list and --block-size can now be used together 282 in single-threaded mode. Previously the combination only 283 worked in multi-threaded mode. 284 285 * Added support for LZMA_IGNORE_CHECK to liblzma and made it 286 available in xz as --ignore-check. 287 288 * liblzma speed optimizations: 289 290 - Initialization of a new LZMA1 or LZMA2 encoder has been 291 optimized. (The speed of reinitializing an already-allocated 292 encoder isn't affected.) This helps when compressing many 293 small buffers with lzma_stream_buffer_encode() and other 294 similar situations where an already-allocated encoder state 295 isn't reused. This speed-up is visible in xz too if one 296 compresses many small files one at a time instead running xz 297 once and giving all files as command-line arguments. 298 299 - Buffer comparisons are now much faster when unaligned access 300 is allowed (configured with --enable-unaligned-access). This 301 speeds up encoding significantly. There is arch-specific code 302 for 32-bit and 64-bit x86 (32-bit needs SSE2 for the best 303 results and there's no run-time CPU detection for now). 304 For other archs there is only generic code which probably 305 isn't as optimal as arch-specific solutions could be. 306 307 - A few speed optimizations were made to the SHA-256 code. 308 (Note that the builtin SHA-256 code isn't used on all 309 operating systems.) 310 311 * liblzma can now be built with MSVC 2013 update 2 or later 312 using windows/config.h. 313 314 * Vietnamese translation was added. 315 316 3175.1.3alpha (2013-10-26) 318 319 * All fixes from 5.0.5 320 321 * liblzma: 322 323 - Fixed a deadlock in the threaded encoder. 324 325 - Made the uses of lzma_allocator const correct. 326 327 - Added lzma_block_uncomp_encode() to create uncompressed 328 .xz Blocks using LZMA2 uncompressed chunks. 329 330 - Added support for native threads on Windows and the ability 331 to detect the number of CPU cores. 332 333 * xz: 334 335 - Fixed a race condition in the signal handling. It was 336 possible that e.g. the first SIGINT didn't make xz exit 337 if reading or writing blocked and one had bad luck. The fix 338 is non-trivial, so as of writing it is unknown if it will be 339 backported to the v5.0 branch. 340 341 - Made the progress indicator work correctly in threaded mode. 342 343 - Threaded encoder now works together with --block-list=SIZES. 344 345 - Added preliminary support for --flush-timeout=TIMEOUT. 346 It can be useful for (somewhat) real-time streaming. For 347 now the decompression side has to be done with something 348 else than the xz tool due to how xz does buffering, but this 349 should be fixed. 350 351 3525.1.2alpha (2012-07-04) 353 354 * All fixes from 5.0.3 and 5.0.4 355 356 * liblzma: 357 358 - Fixed a deadlock and an invalid free() in the threaded encoder. 359 360 - Added support for symbol versioning. It is enabled by default 361 on GNU/Linux, other GNU-based systems, and FreeBSD. 362 363 - Use SHA-256 implementation from the operating system if one is 364 available in libc, libmd, or libutil. liblzma won't use e.g. 365 OpenSSL or libgcrypt to avoid introducing new dependencies. 366 367 - Fixed liblzma.pc for static linking. 368 369 - Fixed a few portability bugs. 370 371 * xz --decompress --single-stream now fixes the input position after 372 successful decompression. Now the following works: 373 374 echo foo | xz > foo.xz 375 echo bar | xz >> foo.xz 376 ( xz -dc --single-stream ; xz -dc --single-stream ) < foo.xz 377 378 Note that it doesn't work if the input is not seekable 379 or if there is Stream Padding between the concatenated 380 .xz Streams. 381 382 * xz -lvv now shows the minimum xz version that is required to 383 decompress the file. Currently it is 5.0.0 for all supported .xz 384 files except files with empty LZMA2 streams require 5.0.2. 385 386 * Added an *incomplete* implementation of --block-list=SIZES to xz. 387 It only works correctly in single-threaded mode and when 388 --block-size isn't used at the same time. --block-list allows 389 specifying the sizes of Blocks which can be useful e.g. when 390 creating files for random-access reading. 391 392 3935.1.1alpha (2011-04-12) 394 395 * All fixes from 5.0.2 396 397 * liblzma fixes that will also be included in 5.0.3: 398 399 - A memory leak was fixed. 400 401 - lzma_stream_buffer_encode() no longer creates an empty .xz 402 Block if encoding an empty buffer. Such an empty Block with 403 LZMA2 data would trigger a bug in 5.0.1 and older (see the 404 first bullet point in 5.0.2 notes). When releasing 5.0.2, 405 I thought that no encoder creates this kind of files but 406 I was wrong. 407 408 - Validate function arguments better in a few functions. Most 409 importantly, specifying an unsupported integrity check to 410 lzma_stream_buffer_encode() no longer creates a corrupt .xz 411 file. Probably no application tries to do that, so this 412 shouldn't be a big problem in practice. 413 414 - Document that lzma_block_buffer_encode(), 415 lzma_easy_buffer_encode(), lzma_stream_encoder(), and 416 lzma_stream_buffer_encode() may return LZMA_UNSUPPORTED_CHECK. 417 418 - The return values of the _memusage() functions are now 419 documented better. 420 421 * Support for multithreaded compression was added using the simplest 422 method, which splits the input data into blocks and compresses 423 them independently. Other methods will be added in the future. 424 The current method has room for improvement, e.g. it is possible 425 to reduce the memory usage. 426 427 * Added the options --single-stream and --block-size=SIZE to xz. 428 429 * xzdiff and xzgrep now support .lzo files if lzop is installed. 430 The .tzo suffix is also recognized as a shorthand for .tar.lzo. 431 432 * Support for short 8.3 filenames under DOS was added to xz. It is 433 experimental and may change before it gets into a stable release. 434 435 4365.0.8 (2014-12-21) 437 438 * Fixed an old bug in xzgrep that affected OpenBSD and probably 439 a few other operating systems too. 440 441 * Updated French and German translations. 442 443 * Added support for detecting the amount of RAM on AmigaOS/AROS. 444 445 * Minor build system updates. 446 447 4485.0.7 (2014-09-20) 449 450 * Fix regressions introduced in 5.0.6: 451 452 - Fix building with non-GNU make. 453 454 - Fix invalid Libs.private value in liblzma.pc which broke 455 static linking against liblzma if the linker flags were 456 taken from pkg-config. 457 458 4595.0.6 (2014-09-14) 460 461 * xzgrep now exits with status 0 if at least one file matched. 462 463 * A few minor portability and build system fixes 464 465 4665.0.5 (2013-06-30) 467 468 * lzmadec and liblzma's lzma_alone_decoder(): Support decompressing 469 .lzma files that have less common settings in the headers 470 (dictionary size other than 2^n or 2^n + 2^(n-1), or uncompressed 471 size greater than 256 GiB). The limitations existed to avoid false 472 positives when detecting .lzma files. The lc + lp <= 4 limitation 473 still remains since liblzma's LZMA decoder has that limitation. 474 475 NOTE: xz's .lzma support or liblzma's lzma_auto_decoder() are NOT 476 affected by this change. They still consider uncommon .lzma headers 477 as not being in the .lzma format. Changing this would give way too 478 many false positives. 479 480 * xz: 481 482 - Interaction of preset and custom filter chain options was 483 made less illogical. This affects only certain less typical 484 uses cases so few people are expected to notice this change. 485 486 Now when a custom filter chain option (e.g. --lzma2) is 487 specified, all preset options (-0 ... -9, -e) earlier are on 488 the command line are completely forgotten. Similarly, when 489 a preset option is specified, all custom filter chain options 490 earlier on the command line are completely forgotten. 491 492 Example 1: "xz -9 --lzma2=preset=5 -e" is equivalent to "xz -e" 493 which is equivalent to "xz -6e". Earlier -e didn't put xz back 494 into preset mode and thus the example command was equivalent 495 to "xz --lzma2=preset=5". 496 497 Example 2: "xz -9e --lzma2=preset=5 -7" is equivalent to 498 "xz -7". Earlier a custom filter chain option didn't make 499 xz forget the -e option so the example was equivalent to 500 "xz -7e". 501 502 - Fixes and improvements to error handling. 503 504 - Various fixes to the man page. 505 506 * xzless: Fixed to work with "less" versions 448 and later. 507 508 * xzgrep: Made -h an alias for --no-filename. 509 510 * Include the previously missing debug/translation.bash which can 511 be useful for translators. 512 513 * Include a build script for Mac OS X. This has been in the Git 514 repository since 2010 but due to a mistake in Makefile.am the 515 script hasn't been included in a release tarball before. 516 517 5185.0.4 (2012-06-22) 519 520 * liblzma: 521 522 - Fix lzma_index_init(). It could crash if memory allocation 523 failed. 524 525 - Fix the possibility of an incorrect LZMA_BUF_ERROR when a BCJ 526 filter is used and the application only provides exactly as 527 much output space as is the uncompressed size of the file. 528 529 - Fix a bug in doc/examples_old/xz_pipe_decompress.c. It didn't 530 check if the last call to lzma_code() really returned 531 LZMA_STREAM_END, which made the program think that truncated 532 files are valid. 533 534 - New example programs in doc/examples (old programs are now in 535 doc/examples_old). These have more comments and more detailed 536 error handling. 537 538 * Fix "xz -lvv foo.xz". It could crash on some corrupted files. 539 540 * Fix output of "xz --robot -lv" and "xz --robot -lvv" which 541 incorrectly printed the filename also in the "foo (x/x)" format. 542 543 * Fix exit status of "xzdiff foo.xz bar.xz". 544 545 * Fix exit status of "xzgrep foo binary_file". 546 547 * Fix portability to EBCDIC systems. 548 549 * Fix a configure issue on AIX with the XL C compiler. See INSTALL 550 for details. 551 552 * Update French, German, Italian, and Polish translations. 553 554 5555.0.3 (2011-05-21) 556 557 * liblzma fixes: 558 559 - A memory leak was fixed. 560 561 - lzma_stream_buffer_encode() no longer creates an empty .xz 562 Block if encoding an empty buffer. Such an empty Block with 563 LZMA2 data would trigger a bug in 5.0.1 and older (see the 564 first bullet point in 5.0.2 notes). When releasing 5.0.2, 565 I thought that no encoder creates this kind of files but 566 I was wrong. 567 568 - Validate function arguments better in a few functions. Most 569 importantly, specifying an unsupported integrity check to 570 lzma_stream_buffer_encode() no longer creates a corrupt .xz 571 file. Probably no application tries to do that, so this 572 shouldn't be a big problem in practice. 573 574 - Document that lzma_block_buffer_encode(), 575 lzma_easy_buffer_encode(), lzma_stream_encoder(), and 576 lzma_stream_buffer_encode() may return LZMA_UNSUPPORTED_CHECK. 577 578 - The return values of the _memusage() functions are now 579 documented better. 580 581 * Fix command name detection in xzgrep. xzegrep and xzfgrep now 582 correctly use egrep and fgrep instead of grep. 583 584 * French translation was added. 585 586 5875.0.2 (2011-04-01) 588 589 * LZMA2 decompressor now correctly accepts LZMA2 streams with no 590 uncompressed data. Previously it considered them corrupt. The 591 bug can affect applications that use raw LZMA2 streams. It is 592 very unlikely to affect .xz files because no compressor creates 593 .xz files with empty LZMA2 streams. (Empty .xz files are a 594 different thing than empty LZMA2 streams.) 595 596 * "xz --suffix=.foo filename.foo" now refuses to compress the 597 file due to it already having the suffix .foo. It was already 598 documented on the man page, but the code lacked the test. 599 600 * "xzgrep -l foo bar.xz" works now. 601 602 * Polish translation was added. 603 604 6055.0.1 (2011-01-29) 606 607 * xz --force now (de)compresses files that have setuid, setgid, 608 or sticky bit set and files that have multiple hard links. 609 The man page had it documented this way already, but the code 610 had a bug. 611 612 * gzip and bzip2 support in xzdiff was fixed. 613 614 * Portability fixes 615 616 * Minor fix to Czech translation 617 618 6195.0.0 (2010-10-23) 620 621 Only the most important changes compared to 4.999.9beta are listed 622 here. One change is especially important: 623 624 * The memory usage limit is now disabled by default. Some scripts 625 written before this change may have used --memory=max on xz command 626 line or in XZ_OPT. THESE USES OF --memory=max SHOULD BE REMOVED 627 NOW, because they interfere with user's ability to set the memory 628 usage limit himself. If user-specified limit causes problems to 629 your script, blame the user. 630 631 Other significant changes: 632 633 * Added support for XZ_DEFAULTS environment variable. This variable 634 allows users to set default options for xz, e.g. default memory 635 usage limit or default compression level. Scripts that use xz 636 must never set or unset XZ_DEFAULTS. Scripts should use XZ_OPT 637 instead if they need a way to pass options to xz via an 638 environment variable. 639 640 * The compression settings associated with the preset levels 641 -0 ... -9 have been changed. --extreme was changed a little too. 642 It is now less likely to make compression worse, but with some 643 files the new --extreme may compress slightly worse than the old 644 --extreme. 645 646 * If a preset level (-0 ... -9) is specified after a custom filter 647 chain options have been used (e.g. --lzma2), the custom filter 648 chain will be forgotten. Earlier the preset options were 649 completely ignored after custom filter chain options had been 650 seen. 651 652 * xz will create sparse files when decompressing if the uncompressed 653 data contains long sequences of binary zeros. This is done even 654 when writing to standard output that is connected to a regular 655 file and certain additional conditions are met to make it safe. 656 657 * Support for "xz --list" was added. Combine with --verbose or 658 --verbose --verbose (-vv) for detailed output. 659 660 * I had hoped that liblzma API would have been stable after 661 4.999.9beta, but there have been a couple of changes in the 662 advanced features, which don't affect most applications: 663 664 - Index handling code was revised. If you were using the old 665 API, you will get a compiler error (so it's easy to notice). 666 667 - A subtle but important change was made to the Block handling 668 API. lzma_block.version has to be initialized even for 669 lzma_block_header_decode(). Code that doesn't do it will work 670 for now, but might break in the future, which makes this API 671 change easy to miss. 672 673 * The major soname has been bumped to 5.0.0. liblzma API and ABI 674 are now stable, so the need to recompile programs linking against 675 liblzma shouldn't arise soon. 676 677