Re: [PATCH] xz: Update to version 5.2.8

["Peter Müller" <peter.mueller@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 21251 bytes --]

Reviewed-by: Peter Müller <peter.mueller(a)ipfire.org>

> - Update from version 5.2.5 to 5.2.8
> - Update of rootfile
> - Remove xzgrep-ZDI-CAN-16587 patch as the contents are now integrated into the source
>    tarball and with an improved quicker method - see changelog below.
> - Changelog
> 	5.2.8 (2022-11-13)
> 	    * xz:
> 	        - If xz cannot remove an input file when it should, this
> 	          is now treated as a warning (exit status 2) instead of
> 	          an error (exit status 1). This matches GNU gzip and it
> 	          is more logical as at that point the output file has
> 	          already been successfully closed.
> 	        - Fix handling of .xz files with an unsupported check type.
> 	          Previously such printed a warning message but then xz
> 	          behaved as if an error had occurred (didn't decompress,
> 	          exit status 1). Now a warning is printed, decompression
> 	          is done anyway, and exit status is 2. This used to work
> 	          slightly before 5.0.0. In practice this bug matters only
> 	          if xz has been built with some check types disabled. As
> 	          instructed in PACKAGERS, such builds should be done in
> 	          special situations only.
> 	        - Fix "xz -dc --single-stream tests/files/good-0-empty.xz"
> 	          which failed with "Internal error (bug)". That is,
> 	          --single-stream was broken if the first .xz stream in
> 	          the input file didn't contain any uncompressed data.
> 	        - Fix displaying file sizes in the progress indicator when
> 	          working in passthru mode and there are multiple input files.
> 	          Just like "gzip -cdf", "xz -cdf" works like "cat" when the
> 	          input file isn't a supported compressed file format. In
> 	          this case the file size counters weren't reset between
> 	          files so with multiple input files the progress indicator
> 	          displayed an incorrect (too large) value.
> 	    * liblzma:
> 	        - API docs in lzma/container.h:
> 	            * Update the list of decoder flags in the decoder
> 	              function docs.
> 	            * Explain LZMA_CONCATENATED behavior with .lzma files
> 	              in lzma_auto_decoder() docs.
> 	        - OpenBSD: Use HW_NCPUONLINE to detect the number of
> 	          available hardware threads in lzma_physmem().
> 	        - Fix use of wrong macro to detect x86 SSE2 support.
> 	          __SSE2_MATH__ was used with GCC/Clang but the correct
> 	          one is __SSE2__. The first one means that SSE2 is used
> 	          for floating point math which is irrelevant here.
> 	          The affected SSE2 code isn't used on x86-64 so this affects
> 	          only 32-bit x86 builds that use -msse2 without -mfpmath=sse
> 	          (there is no runtime detection for SSE2). It improves LZMA
> 	          compression speed (not decompression).
> 	        - Fix the build with Intel C compiler 2021 (ICC, not ICX)
> 	          on Linux. It defines __GNUC__ to 10 but doesn't support
> 	          the __symver__ attribute introduced in GCC 10.
> 	    * Scripts: Ignore warnings from xz by using --quiet --no-warn.
> 	      This is needed if the input .xz files use an unsupported
> 	      check type.
> 	    * Translations:
> 	        - Updated Croatian and Turkish translations.
> 	        - One new translations wasn't included because it needed
> 	          technical fixes. It will be in upcoming 5.4.0. No new
> 	          translations will be added to the 5.2.x branch anymore.
> 	        - Renamed the French man page translation file from
> 	          fr_FR.po to fr.po and thus also its install directory
> 	          (like /usr/share/man/fr_FR -> .../fr).
> 	        - Man page translations for upcoming 5.4.0 are now handled
> 	          in the Translation Project.
> 	    * Update doc/faq.txt a little so it's less out-of-date.
> 	5.2.7 (2022-09-30)
> 	    * liblzma:
> 	        - Made lzma_filters_copy() to never modify the destination
> 	          array if an error occurs. lzma_stream_encoder() and
> 	          lzma_stream_encoder_mt() already assumed this. Before this
> 	          change, if a tiny memory allocation in lzma_filters_copy()
> 	          failed it would lead to a crash (invalid free() or invalid
> 	          memory reads) in the cleanup paths of these two encoder
> 	          initialization functions.
> 	        - Added missing integer overflow check to lzma_index_append().
> 	          This affects xz --list and other applications that decode
> 	          the Index field from .xz files using lzma_index_decoder().
> 	          Normal decompression of .xz files doesn't call this code
> 	          and thus most applications using liblzma aren't affected
> 	          by this bug.
> 	        - Single-threaded .xz decoder (lzma_stream_decoder()): If
> 	          lzma_code() returns LZMA_MEMLIMIT_ERROR it is now possible
> 	          to use lzma_memlimit_set() to increase the limit and continue
> 	          decoding. This was supposed to work from the beginning
> 	          but there was a bug. With other decoders (.lzma or
> 	          threaded .xz decoder) this already worked correctly.
> 	        - Fixed accumulation of integrity check type statistics in
> 	          lzma_index_cat(). This bug made lzma_index_checks() return
> 	          only the type of the integrity check of the last Stream
> 	          when multiple lzma_indexes were concatenated. Most
> 	          applications don't use these APIs but in xz it made
> 	          xz --list not list all check types from concatenated .xz
> 	          files. In xz --list --verbose only the per-file "Check:"
> 	          lines were affected and in xz --robot --list only the "file"
> 	          line was affected.
> 	        - Added ABI compatibility with executables that were linked
> 	          against liblzma in RHEL/CentOS 7 or other liblzma builds
> 	          that had copied the problematic patch from RHEL/CentOS 7
> 	          (xz-5.2.2-compat-libs.patch). For the details, see the
> 	          comment at the top of src/liblzma/validate_map.sh.
> 	          WARNING: This uses __symver__ attribute with GCC >= 10.
> 	          In other cases the traditional __asm__(".symver ...")
> 	          is used. Using link-time optimization (LTO, -flto) with
> 	          GCC versions older than 10 can silently result in
> 	          broken liblzma.so.5 (incorrect symbol versions)! If you
> 	          want to use -flto with GCC, you must use GCC >= 10.
> 	          LTO with Clang seems to work even with the traditional
> 	          __asm__(".symver ...") method.
> 	    * xzgrep: Fixed compatibility with old shells that break if
> 	      comments inside command substitutions have apostrophes (').
> 	      This problem was introduced in 5.2.6.
> 	    * Build systems:
> 	        - New #define in config.h: HAVE_SYMBOL_VERSIONS_LINUX
> 	        - Windows: Fixed liblzma.dll build with Visual Studio project
> 	          files. It broke in 5.2.6 due to a change that was made to
> 	          improve CMake support.
> 	        - Windows: Building liblzma with UNICODE defined should now
> 	          work.
> 	        - CMake files are now actually included in the release tarball.
> 	          They should have been in 5.2.5 already.
> 	        - Minor CMake fixes and improvements.
> 	    * Added a new translation: Turkish
> 	5.2.6 (2022-08-12)
> 	    * xz:
> 	        - The --keep option now accepts symlinks, hardlinks, and
> 	          setuid, setgid, and sticky files. Previously this required
> 	          using --force.
> 	        - When copying metadata from the source file to the destination
> 	          file, don't try to set the group (GID) if it is already set
> 	          correctly. This avoids a failure on OpenBSD (and possibly on
> 	          a few other OSes) where files may get created so that their
> 	          group doesn't belong to the user, and fchown(2) can fail even
> 	          if it needs to do nothing.
> 	        - Cap --memlimit-compress to 2000 MiB instead of 4020 MiB on
> 	          MIPS32 because on MIPS32 userspace processes are limited
> 	          to 2 GiB of address space.
> 	    * liblzma:
> 	        - Fixed a missing error-check in the threaded encoder. If a
> 	          small memory allocation fails, a .xz file with an invalid
> 	          Index field would be created. Decompressing such a file would
> 	          produce the correct output but result in an error at the end.
> 	          Thus this is a "mild" data corruption bug. Note that while
> 	          a failed memory allocation can trigger the bug, it cannot
> 	          cause invalid memory access.
> 	        - The decoder for .lzma files now supports files that have
> 	          uncompressed size stored in the header and still use the
> 	          end of payload marker (end of stream marker) at the end
> 	          of the LZMA stream. Such files are rare but, according to
> 	          the documentation in LZMA SDK, they are valid.
> 	          doc/lzma-file-format.txt was updated too.
> 	        - Improved 32-bit x86 assembly files:
> 	            * Support Intel Control-flow Enforcement Technology (CET)
> 	            * Use non-executable stack on FreeBSD.
> 	        - Visual Studio: Use non-standard _MSVC_LANG to detect C++
> 	          standard version in the lzma.h API header. It's used to
> 	          detect when "noexcept" can be used.
> 	    * xzgrep:
> 	        - Fixed arbitrary command injection via a malicious filename
> 	          (CVE-2022-1271, ZDI-CAN-16587). A standalone patch for
> 	          this was released to the public on 2022-04-07. A slight
> 	          robustness improvement has been made since then and, if
> 	          using GNU or *BSD grep, a new faster method is now used
> 	          that doesn't use the old sed-based construct at all. This
> 	          also fixes bad output with GNU grep >= 3.5 (2020-09-27)
> 	          when xzgrepping binary files.
> 	          This vulnerability was discovered by:
> 	          cleemy desu wayo working with Trend Micro Zero Day Initiative
> 	        - Fixed detection of corrupt .bz2 files.
> 	        - Improved error handling to fix exit status in some situations
> 	          and to fix handling of signals: in some situations a signal
> 	          didn't make xzgrep exit when it clearly should have. It's
> 	          possible that the signal handling still isn't quite perfect
> 	          but hopefully it's good enough.
> 	        - Documented exit statuses on the man page.
> 	        - xzegrep and xzfgrep now use "grep -E" and "grep -F" instead
> 	          of the deprecated egrep and fgrep commands.
> 	        - Fixed parsing of the options -E, -F, -G, -P, and -X. The
> 	          problem occurred when multiple options were specied in
> 	          a single argument, for example,
> 	              echo foo | xzgrep -Fe foo
> 	          treated foo as a filename because -Fe wasn't correctly
> 	          split into -F -e.
> 	        - Added zstd support.
> 	    * xzdiff/xzcmp:
> 	        - Fixed wrong exit status. Exit status could be 2 when the
> 	          correct value is 1.
> 	        - Documented on the man page that exit status of 2 is used
> 	          for decompression errors.
> 	        - Added zstd support.
> 	    * xzless:
> 	        - Fix less(1) version detection. It failed if the version number
> 	          from "less -V" contained a dot.
> 	    * Translations:
> 	        - Added new translations: Catalan, Croatian, Esperanto,
> 	          Korean, Portuguese, Romanian, Serbian, Spanish, Swedish,
> 	          and Ukrainian
> 	        - Updated the Brazilian Portuguese translation.
> 	        - Added French man page translation. This and the existing
> 	          German translation aren't complete anymore because the
> 	          English man pages got a few updates and the translators
> 	          weren't reached so that they could update their work.
> 	    * Build systems:
> 	        - Windows: Fix building of resource files when config.h isn't
> 	          used. CMake + Visual Studio can now build liblzma.dll.
> 	        - Various fixes to the CMake support. Building static or shared
> 	          liblzma should work fine in most cases. In contrast, building
> 	          the command line tools with CMake is still clearly incomplete
> 	          and experimental and should be used for testing only.
> 
> Signed-off-by: Adolf Belka <adolf.belka(a)ipfire.org>
> ---
>  config/rootfiles/common/xz             | 37 ++++++++--
>  lfs/xz                                 |  5 +-
>  src/patches/xzgrep-ZDI-CAN-16587.patch | 94 --------------------------
>  3 files changed, 32 insertions(+), 104 deletions(-)
>  delete mode 100644 src/patches/xzgrep-ZDI-CAN-16587.patch
> 
> diff --git a/config/rootfiles/common/xz b/config/rootfiles/common/xz
> index c7bd0b302..d2f1d44cc 100644
> --- a/config/rootfiles/common/xz
> +++ b/config/rootfiles/common/xz
> @@ -41,7 +41,7 @@ usr/bin/xzmore
>  #usr/lib/liblzma.la
>  usr/lib/liblzma.so
>  usr/lib/liblzma.so.5
> -usr/lib/liblzma.so.5.2.5
> +usr/lib/liblzma.so.5.2.8
>  #usr/lib/pkgconfig/liblzma.pc
>  #usr/share/doc/xz
>  #usr/share/doc/xz/AUTHORS
> @@ -65,15 +65,26 @@ usr/lib/liblzma.so.5.2.5
>  #usr/share/doc/xz/history.txt
>  #usr/share/doc/xz/lzma-file-format.txt
>  #usr/share/doc/xz/xz-file-format.txt
> +#usr/share/locale/ca/LC_MESSAGES/xz.mo
>  #usr/share/locale/cs/LC_MESSAGES/xz.mo
>  #usr/share/locale/da/LC_MESSAGES/xz.mo
>  #usr/share/locale/de/LC_MESSAGES/xz.mo
> +#usr/share/locale/eo/LC_MESSAGES/xz.mo
> +#usr/share/locale/es/LC_MESSAGES/xz.mo
>  #usr/share/locale/fi/LC_MESSAGES/xz.mo
>  #usr/share/locale/fr/LC_MESSAGES/xz.mo
> +#usr/share/locale/hr/LC_MESSAGES/xz.mo
>  #usr/share/locale/hu/LC_MESSAGES/xz.mo
>  #usr/share/locale/it/LC_MESSAGES/xz.mo
> +#usr/share/locale/ko/LC_MESSAGES/xz.mo
>  #usr/share/locale/pl/LC_MESSAGES/xz.mo
> +#usr/share/locale/pt/LC_MESSAGES/xz.mo
>  #usr/share/locale/pt_BR/LC_MESSAGES/xz.mo
> +#usr/share/locale/ro/LC_MESSAGES/xz.mo
> +#usr/share/locale/sr/LC_MESSAGES/xz.mo
> +#usr/share/locale/sv/LC_MESSAGES/xz.mo
> +#usr/share/locale/tr/LC_MESSAGES/xz.mo
> +#usr/share/locale/uk/LC_MESSAGES/xz.mo
>  #usr/share/locale/vi/LC_MESSAGES/xz.mo
>  #usr/share/locale/zh_CN/LC_MESSAGES/xz.mo
>  #usr/share/locale/zh_TW/LC_MESSAGES/xz.mo
> @@ -82,9 +93,6 @@ usr/lib/liblzma.so.5.2.5
>  #usr/share/man/de/man1/lzcat.1
>  #usr/share/man/de/man1/lzcmp.1
>  #usr/share/man/de/man1/lzdiff.1
> -#usr/share/man/de/man1/lzegrep.1
> -#usr/share/man/de/man1/lzfgrep.1
> -#usr/share/man/de/man1/lzgrep.1
>  #usr/share/man/de/man1/lzless.1
>  #usr/share/man/de/man1/lzma.1
>  #usr/share/man/de/man1/lzmadec.1
> @@ -96,11 +104,26 @@ usr/lib/liblzma.so.5.2.5
>  #usr/share/man/de/man1/xzcmp.1
>  #usr/share/man/de/man1/xzdec.1
>  #usr/share/man/de/man1/xzdiff.1
> -#usr/share/man/de/man1/xzegrep.1
> -#usr/share/man/de/man1/xzfgrep.1
> -#usr/share/man/de/man1/xzgrep.1
>  #usr/share/man/de/man1/xzless.1
>  #usr/share/man/de/man1/xzmore.1
> +#usr/share/man/fr
> +#usr/share/man/fr/man1
> +#usr/share/man/fr/man1/lzcat.1
> +#usr/share/man/fr/man1/lzcmp.1
> +#usr/share/man/fr/man1/lzdiff.1
> +#usr/share/man/fr/man1/lzless.1
> +#usr/share/man/fr/man1/lzma.1
> +#usr/share/man/fr/man1/lzmadec.1
> +#usr/share/man/fr/man1/lzmore.1
> +#usr/share/man/fr/man1/unlzma.1
> +#usr/share/man/fr/man1/unxz.1
> +#usr/share/man/fr/man1/xz.1
> +#usr/share/man/fr/man1/xzcat.1
> +#usr/share/man/fr/man1/xzcmp.1
> +#usr/share/man/fr/man1/xzdec.1
> +#usr/share/man/fr/man1/xzdiff.1
> +#usr/share/man/fr/man1/xzless.1
> +#usr/share/man/fr/man1/xzmore.1
>  #usr/share/man/man1/lzcat.1
>  #usr/share/man/man1/lzcmp.1
>  #usr/share/man/man1/lzdiff.1
> diff --git a/lfs/xz b/lfs/xz
> index 9345df954..83a724e1a 100644
> --- a/lfs/xz
> +++ b/lfs/xz
> @@ -24,7 +24,7 @@
>  
>  include Config
>  
> -VER        = 5.2.5
> +VER        = 5.2.8
>  
>  THISAPP    = xz-$(VER)
>  DL_FILE    = $(THISAPP).tar.xz
> @@ -45,7 +45,7 @@ objects = $(DL_FILE)
>  
>  $(DL_FILE) = $(DL_FROM)/$(DL_FILE)
>  
> -$(DL_FILE)_BLAKE2 = 9b9b58e33722ecfe799bb50f3ffe4e86386f734ab4468eb54ff92771ddf899302d1ffa4d88bdb0de351fc3eab8a6ea103f00d7e79f33db879fe22b2e1a7e62db
> +$(DL_FILE)_BLAKE2 = 44d1ddd783b2527f3b17481fc277b671808eb5639c10d31bfaca9fd29ac4413628654ecb9e207955a9477c83eb30f61cf5607cd9a49dd71732707731e4444ace
>  
>  install : $(TARGET)
>  
> @@ -75,7 +75,6 @@ $(subst %,%_BLAKE2,$(objects)) :
>  $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
>  	@$(PREBUILD)
>  	@rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar axf $(DIR_DL)/$(DL_FILE)
> -	cd $(DIR_APP) && patch -p1 -i $(DIR_SRC)/src/patches/xzgrep-ZDI-CAN-16587.patch
>  	cd $(DIR_APP) && ./configure --prefix=$(PREFIX)
>  	cd $(DIR_APP) && make $(MAKETUNING)
>  	cd $(DIR_APP) && make install
> diff --git a/src/patches/xzgrep-ZDI-CAN-16587.patch b/src/patches/xzgrep-ZDI-CAN-16587.patch
> deleted file mode 100644
> index 406ded590..000000000
> --- a/src/patches/xzgrep-ZDI-CAN-16587.patch
> +++ /dev/null
> @@ -1,94 +0,0 @@
> -From 69d1b3fc29677af8ade8dc15dba83f0589cb63d6 Mon Sep 17 00:00:00 2001
> -From: Lasse Collin <lasse.collin(a)tukaani.org>
> -Date: Tue, 29 Mar 2022 19:19:12 +0300
> -Subject: [PATCH] xzgrep: Fix escaping of malicious filenames (ZDI-CAN-16587).
> -
> -Malicious filenames can make xzgrep to write to arbitrary files
> -or (with a GNU sed extension) lead to arbitrary code execution.
> -
> -xzgrep from XZ Utils versions up to and including 5.2.5 are
> -affected. 5.3.1alpha and 5.3.2alpha are affected as well.
> -This patch works for all of them.
> -
> -This bug was inherited from gzip's zgrep. gzip 1.12 includes
> -a fix for zgrep.
> -
> -The issue with the old sed script is that with multiple newlines,
> -the N-command will read the second line of input, then the
> -s-commands will be skipped because it's not the end of the
> -file yet, then a new sed cycle starts and the pattern space
> -is printed and emptied. So only the last line or two get escaped.
> -
> -One way to fix this would be to read all lines into the pattern
> -space first. However, the included fix is even simpler: All lines
> -except the last line get a backslash appended at the end. To ensure
> -that shell command substitution doesn't eat a possible trailing
> -newline, a colon is appended to the filename before escaping.
> -The colon is later used to separate the filename from the grep
> -output so it is fine to add it here instead of a few lines later.
> -
> -The old code also wasn't POSIX compliant as it used \n in the
> -replacement section of the s-command. Using \<newline> is the
> -POSIX compatible method.
> -
> -LC_ALL=C was added to the two critical sed commands. POSIX sed
> -manual recommends it when using sed to manipulate pathnames
> -because in other locales invalid multibyte sequences might
> -cause issues with some sed implementations. In case of GNU sed,
> -these particular sed scripts wouldn't have such problems but some
> -other scripts could have, see:
> -
> -    info '(sed)Locale Considerations'
> -
> -This vulnerability was discovered by:
> -cleemy desu wayo working with Trend Micro Zero Day Initiative
> -
> -Thanks to Jim Meyering and Paul Eggert discussing the different
> -ways to fix this and for coordinating the patch release schedule
> -with gzip.
> ----
> - src/scripts/xzgrep.in | 20 ++++++++++++--------
> - 1 file changed, 12 insertions(+), 8 deletions(-)
> -
> -diff --git a/src/scripts/xzgrep.in b/src/scripts/xzgrep.in
> -index b180936..e5186ba 100644
> ---- a/src/scripts/xzgrep.in
> -+++ b/src/scripts/xzgrep.in
> -@@ -180,22 +180,26 @@ for i; do
> -          { test $# -eq 1 || test $no_filename -eq 1; }; then
> -       eval "$grep"
> -     else
> -+      # Append a colon so that the last character will never be a newline
> -+      # which would otherwise get lost in shell command substitution.
> -+      i="$i:"
> -+
> -+      # Escape & \ | and newlines only if such characters are present
> -+      # (speed optimization).
> -       case $i in
> -       (*'
> - '* | *'&'* | *'\'* | *'|'*)
> --        i=$(printf '%s\n' "$i" |
> --            sed '
> --              $!N
> --              $s/[&\|]/\\&/g
> --              $s/\n/\\n/g
> --            ');;
> -+        i=$(printf '%s\n' "$i" | LC_ALL=C sed 's/[&\|]/\\&/g; $!s/$/\\/');;
> -       esac
> --      sed_script="s|^|$i:|"
> -+
> -+      # $i already ends with a colon so don't add it here.
> -+      sed_script="s|^|$i|"
> - 
> -       # Fail if grep or sed fails.
> -       r=$(
> -         exec 4>&1
> --        (eval "$grep" 4>&-; echo $? >&4) 3>&- | sed "$sed_script" >&3 4>&-
> -+        (eval "$grep" 4>&-; echo $? >&4) 3>&- |
> -+            LC_ALL=C sed "$sed_script" >&3 4>&-
> -       ) || r=2
> -       exit $r
> -     fi >&3 5>&-
> --- 
> -2.35.1
> -