Re: [PATCH] xz: Update to version 5.4.0

["Peter Müller" <peter.mueller@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 15844 bytes --]

Reviewed-by: Peter Müller <peter.mueller(a)ipfire.org>

> - Update from version 5.2.8 to 5.4.0
> - Update of rootfile
> - Changelog
>     5.4.0 (2022-12-13)
> 	    This bumps the minor version of liblzma because new features were
> 	    added. The API and ABI are still backward compatible with liblzma
> 	    5.2.x and 5.0.x.
> 	    Since 5.3.5beta:
> 	    * All fixes from 5.2.10.
> 	    * The ARM64 filter is now stable. The xz option is now --arm64.
> 	      Decompression requires XZ Utils 5.4.0. In the future the ARM64
> 	      filter will be supported by XZ for Java, XZ Embedded (including
> 	      the version in Linux), LZMA SDK, and 7-Zip.
> 	    * Translations:
> 	        - Updated Catalan, Croatian, German, Romanian, and Turkish
> 	          translations.
> 	        - Updated German man page translations.
> 	        - Added Romanian man page translations.
> 	    Summary of new features added in the 5.3.x development releases:
> 	    * liblzma:
> 	        - Added threaded .xz decompressor lzma_stream_decoder_mt().
> 	          It can use multiple threads with .xz files that have multiple
> 	          Blocks with size information in Block Headers. The threaded
> 	          encoder in xz has always created such files.
> 	          Single-threaded encoder cannot store the size information in
> 	          Block Headers even if one used LZMA_FULL_FLUSH to create
> 	          multiple Blocks, so this threaded decoder cannot use multiple
> 	          threads with such files.
> 	          If there are multiple Streams (concatenated .xz files), one
> 	          Stream will be decompressed completely before starting the
> 	          next Stream.
> 	        - A new decoder flag LZMA_FAIL_FAST was added. It makes the
> 	          threaded decompressor report errors soon instead of first
> 	          flushing all pending data before the error location.
> 	        - New Filter IDs:
> 	            * LZMA_FILTER_ARM64 is for ARM64 binaries.
> 	            * LZMA_FILTER_LZMA1EXT is for raw LZMA1 streams that don't
> 	              necessarily use the end marker.
> 	        - Added lzma_str_to_filters(), lzma_str_from_filters(), and
> 	          lzma_str_list_filters() to convert a preset or a filter chain
> 	          string to a lzma_filter[] and vice versa. These should make
> 	          it easier to write applications that allow users to specify
> 	          custom compression options.
> 	        - Added lzma_filters_free() which can be convenient for freeing
> 	          the filter options in a filter chain (an array of lzma_filter
> 	          structures).
> 	        - lzma_file_info_decoder() to makes it a little easier to get
> 	          the Index field from .xz files. This helps in getting the
> 	          uncompressed file size but an easy-to-use random access
> 	          API is still missing which has existed in XZ for Java for
> 	          a long time.
> 	        - Added lzma_microlzma_encoder() and lzma_microlzma_decoder().
> 	          It is used by erofs-utils and may be used by others too.
> 	          The MicroLZMA format is a raw LZMA stream (without end marker)
> 	          whose first byte (always 0x00) has been replaced with
> 	          bitwise-negation of the LZMA properties (lc/lp/pb). It was
> 	          created for use in EROFS but may be used in other contexts
> 	          as well where it is important to avoid wasting bytes for
> 	          stream headers or footers. The format is also supported by
> 	          XZ Embedded (the XZ Embedded version in Linux got MicroLZMA
> 	          support in Linux 5.16).
> 	          The MicroLZMA encoder API in liblzma can compress into a
> 	          fixed-sized output buffer so that as much data is compressed
> 	          as can be fit into the buffer while still creating a valid
> 	          MicroLZMA stream. This is needed for EROFS.
> 	        - Added lzma_lzip_decoder() to decompress the .lz (lzip) file
> 	          format version 0 and the original unextended version 1 files.
> 	          Also lzma_auto_decoder() supports .lz files.
> 	        - lzma_filters_update() can now be used with the multi-threaded
> 	          encoder (lzma_stream_encoder_mt()) to change the filter chain
> 	          after LZMA_FULL_BARRIER or LZMA_FULL_FLUSH.
> 	        - In lzma_options_lzma, allow nice_len = 2 and 3 with the match
> 	          finders that require at least 3 or 4. Now it is internally
> 	          rounded up if needed.
> 	        - CLMUL-based CRC64 on x86-64 and E2K with runtime processor
> 	          detection. On 32-bit x86 it currently isn't available unless
> 	          --disable-assembler is used which can make the non-CLMUL
> 	          CRC64 slower; this might be fixed in the future.
> 	        - Building with --disable-threads --enable-small
> 	          is now thread-safe if the compiler supports
> 	          __attribute__((__constructor__)).
> 	    * xz:
> 	        - Using -T0 (--threads=0) will now use multi-threaded encoder
> 	          even on a single-core system. This is to ensure that output
> 	          from the same xz binary is identical on both single-core and
> 	          multi-core systems.
> 	        - --threads=+1 or -T+1 is now a way to put xz into
> 	          multi-threaded mode while using only one worker thread.
> 	          The + is ignored if the number is not 1.
> 	        - A default soft memory usage limit is now used for compression
> 	          when -T0 is used and no explicit limit has been specified.
> 	          This soft limit is used to restrict the number of threads
> 	          but if the limit is exceeded with even one thread then xz
> 	          will continue with one thread using the multi-threaded
> 	          encoder and this limit is ignored. If the number of threads
> 	          is specified manually then no default limit will be used;
> 	          this affects only -T0.
> 	          This change helps on systems that have very many cores and
> 	          using all of them for xz makes no sense. Previously xz -T0
> 	          could run out of memory on such systems because it attempted
> 	          to reserve memory for too many threads.
> 	          This also helps with 32-bit builds which don't have a large
> 	          amount of address space that would be required for many
> 	          threads. The default soft limit for -T0 is at most 1400 MiB
> 	          on all 32-bit platforms.
> 	        - Previously a low value in --memlimit-compress wouldn't cause
> 	          xz to switch from multi-threaded mode to single-threaded mode
> 	          if the limit cannot otherwise be met; xz failed instead. Now
> 	          xz can switch to single-threaded mode and then, if needed,
> 	          scale down the LZMA2 dictionary size too just like it already
> 	          did when it was started in single-threaded mode.
> 	        - The option --no-adjust no longer prevents xz from scaling down
> 	          the number of threads as that doesn't affect the compressed
> 	          output (only performance). Now --no-adjust only prevents
> 	          adjustments that affect compressed output, that is, with
> 	          --no-adjust xz won't switch from multi-threaded mode to
> 	          single-threaded mode and won't scale down the LZMA2
> 	          dictionary size.
> 	        - Added a new option --memlimit-mt-decompress=LIMIT. This is
> 	          used to limit the number of decompressor threads (possibly
> 	          falling back to single-threaded mode) but it will never make
> 	          xz refuse to decompress a file. This has a system-specific
> 	          default value because without any limit xz could end up
> 	          allocating memory for the whole compressed input file, the
> 	          whole uncompressed output file, multiple thread-specific
> 	          decompressor instances and so on. Basically xz could
> 	          attempt to use an insane amount of memory even with fairly
> 	          common files. The system-specific default value is currently
> 	          the same as the one used for compression with -T0.
> 	          The new option works together with the existing option
> 	          --memlimit-decompress=LIMIT. The old option sets a hard limit
> 	          that must not be exceeded (xz will refuse to decompress)
> 	          while the new option only restricts the number of threads.
> 	          If the limit set with --memlimit-mt-decompress is greater
> 	          than the limit set with --memlimit-compress, then the latter
> 	          value is used also for --memlimit-mt-decompress.
> 	        - Added new information to the output of xz --info-memory and
> 	          new fields to the output of xz --robot --info-memory.
> 	        - In --lzma2=nice=NUMBER allow 2 and 3 with all match finders
> 	          now that liblzma handles it.
> 	        - Don't mention endianness for ARM and ARM-Thumb filters in
> 	          --long-help. The filters only work for little endian
> 	          instruction encoding but modern ARM processors using
> 	          big endian data access still use little endian
> 	          instruction encoding. So the help text was misleading.
> 	          In contrast, the PowerPC filter is only for big endian
> 	          32/64-bit PowerPC code. Little endian PowerPC would need
> 	          a separate filter.
> 	        - Added decompression support for the .lz (lzip) file format
> 	          version 0 and the original unextended version 1. It is
> 	          autodetected by default. See also the option --format on
> 	          the xz man page.
> 	        - Sandboxing enabled by default:
> 	            * Capsicum (FreeBSD)
> 	            * pledge(2) (OpenBSD)
> 	    * Scripts now support the .lz format using xz.
> 	    * A few new tests were added.
> 	    * The liblzma-specific tests are now supported in CMake-based
> 	      builds too ("make test").
>     5.2.10 (2022-12-13)
> 	    * xz: Don't modify argv[] when parsing the --memlimit* and
> 	      --block-list command line options. This fixes confusing
> 	      arguments in process listing (like "ps auxf").
> 	    * GNU/Linux only: Use __has_attribute(__symver__) to detect if
> 	      that attribute is supported. This fixes build on Mandriva where
> 	      Clang is patched to define __GNUC__ to 11 by default (instead
> 	      of 4 as used by Clang upstream).
>     5.2.9 (2022-11-30)
> 	    * liblzma:
> 	        - Fixed an infinite loop in LZMA encoder initialization
> 	          if dict_size >= 2 GiB. (The encoder only supports up
> 	          to 1536 MiB.)
> 	        - Fixed two cases of invalid free() that can happen if
> 	          a tiny allocation fails in encoder re-initialization
> 	          or in lzma_filters_update(). These bugs had some
> 	          similarities with the bug fixed in 5.2.7.
> 	        - Fixed lzma_block_encoder() not allowing the use of
> 	          LZMA_SYNC_FLUSH with lzma_code() even though it was
> 	          documented to be supported. The sync-flush code in
> 	          the Block encoder was already used internally via
> 	          lzma_stream_encoder(), so this was just a missing flag
> 	          in the lzma_block_encoder() API function.
> 	        - GNU/Linux only: Don't put symbol versions into static
> 	          liblzma as it breaks things in some cases (and even if
> 	          it didn't break anything, symbol versions in static
> 	          libraries are useless anyway). The downside of the fix
> 	          is that if the configure options --with-pic or --without-pic
> 	          are used then it's not possible to build both shared and
> 	          static liblzma at the same time on GNU/Linux anymore;
> 	          with those options --disable-static or --disable-shared
> 	          must be used too.
> 	    * New email address for bug reports is <xz(a)tukaani.org> which
> 	      forwards messages to Lasse Collin and Jia Tan.
> 
> Signed-off-by: Adolf Belka <adolf.belka(a)ipfire.org>
> ---
>  config/rootfiles/common/xz | 32 +++++++++++++++++++++++++++++++-
>  lfs/xz                     |  6 +++---
>  2 files changed, 34 insertions(+), 4 deletions(-)
> 
> diff --git a/config/rootfiles/common/xz b/config/rootfiles/common/xz
> index d2f1d44cc..1e4a43729 100644
> --- a/config/rootfiles/common/xz
> +++ b/config/rootfiles/common/xz
> @@ -41,7 +41,7 @@ usr/bin/xzmore
>  #usr/lib/liblzma.la
>  usr/lib/liblzma.so
>  usr/lib/liblzma.so.5
> -usr/lib/liblzma.so.5.2.8
> +usr/lib/liblzma.so.5.4.0
>  #usr/lib/pkgconfig/liblzma.pc
>  #usr/share/doc/xz
>  #usr/share/doc/xz/AUTHORS
> @@ -93,6 +93,9 @@ usr/lib/liblzma.so.5.2.8
>  #usr/share/man/de/man1/lzcat.1
>  #usr/share/man/de/man1/lzcmp.1
>  #usr/share/man/de/man1/lzdiff.1
> +#usr/share/man/de/man1/lzegrep.1
> +#usr/share/man/de/man1/lzfgrep.1
> +#usr/share/man/de/man1/lzgrep.1
>  #usr/share/man/de/man1/lzless.1
>  #usr/share/man/de/man1/lzma.1
>  #usr/share/man/de/man1/lzmadec.1
> @@ -104,6 +107,9 @@ usr/lib/liblzma.so.5.2.8
>  #usr/share/man/de/man1/xzcmp.1
>  #usr/share/man/de/man1/xzdec.1
>  #usr/share/man/de/man1/xzdiff.1
> +#usr/share/man/de/man1/xzegrep.1
> +#usr/share/man/de/man1/xzfgrep.1
> +#usr/share/man/de/man1/xzgrep.1
>  #usr/share/man/de/man1/xzless.1
>  #usr/share/man/de/man1/xzmore.1
>  #usr/share/man/fr
> @@ -147,3 +153,27 @@ usr/lib/liblzma.so.5.2.8
>  #usr/share/man/man1/xzgrep.1
>  #usr/share/man/man1/xzless.1
>  #usr/share/man/man1/xzmore.1
> +#usr/share/man/ro
> +#usr/share/man/ro/man1
> +#usr/share/man/ro/man1/lzcat.1
> +#usr/share/man/ro/man1/lzcmp.1
> +#usr/share/man/ro/man1/lzdiff.1
> +#usr/share/man/ro/man1/lzegrep.1
> +#usr/share/man/ro/man1/lzfgrep.1
> +#usr/share/man/ro/man1/lzgrep.1
> +#usr/share/man/ro/man1/lzless.1
> +#usr/share/man/ro/man1/lzma.1
> +#usr/share/man/ro/man1/lzmadec.1
> +#usr/share/man/ro/man1/lzmore.1
> +#usr/share/man/ro/man1/unlzma.1
> +#usr/share/man/ro/man1/unxz.1
> +#usr/share/man/ro/man1/xz.1
> +#usr/share/man/ro/man1/xzcat.1
> +#usr/share/man/ro/man1/xzcmp.1
> +#usr/share/man/ro/man1/xzdec.1
> +#usr/share/man/ro/man1/xzdiff.1
> +#usr/share/man/ro/man1/xzegrep.1
> +#usr/share/man/ro/man1/xzfgrep.1
> +#usr/share/man/ro/man1/xzgrep.1
> +#usr/share/man/ro/man1/xzless.1
> +#usr/share/man/ro/man1/xzmore.1
> diff --git a/lfs/xz b/lfs/xz
> index 83a724e1a..1e4020dde 100644
> --- a/lfs/xz
> +++ b/lfs/xz
> @@ -1,7 +1,7 @@
>  ###############################################################################
>  #                                                                             #
>  # IPFire.org - A linux based firewall                                         #
> -# Copyright (C) 2007-2020  IPFire Team  <info(a)ipfire.org>                     #
> +# Copyright (C) 2007-2022  IPFire Team  <info(a)ipfire.org>                     #
>  #                                                                             #
>  # This program is free software: you can redistribute it and/or modify        #
>  # it under the terms of the GNU General Public License as published by        #
> @@ -24,7 +24,7 @@
>  
>  include Config
>  
> -VER        = 5.2.8
> +VER        = 5.4.0
>  
>  THISAPP    = xz-$(VER)
>  DL_FILE    = $(THISAPP).tar.xz
> @@ -45,7 +45,7 @@ objects = $(DL_FILE)
>  
>  $(DL_FILE) = $(DL_FROM)/$(DL_FILE)
>  
> -$(DL_FILE)_BLAKE2 = 44d1ddd783b2527f3b17481fc277b671808eb5639c10d31bfaca9fd29ac4413628654ecb9e207955a9477c83eb30f61cf5607cd9a49dd71732707731e4444ace
> +$(DL_FILE)_BLAKE2 = 8896044ae2761561161f4dcfc1e8e0bc6a58a2a7784567156b23e589cdb80028b37655af65802a20d2631ef05c209e406424f061c0293458a41b16dbf75a4f29
>  
>  install : $(TARGET)
>