Re: [PATCH] xz: Update to version 5.2.8

[Michael Tremer <michael.tremer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 21361 bytes --]

Reviewed-by: Michael Tremer <michael.tremer(a)ipfire.org>

> On 21 Nov 2022, at 08:28, Peter Müller <peter.mueller(a)ipfire.org> wrote:
> 
> Reviewed-by: Peter Müller <peter.mueller(a)ipfire.org>
> 
>> - Update from version 5.2.5 to 5.2.8
>> - Update of rootfile
>> - Remove xzgrep-ZDI-CAN-16587 patch as the contents are now integrated into the source
>>   tarball and with an improved quicker method - see changelog below.
>> - Changelog
>> 5.2.8 (2022-11-13)
>>    * xz:
>>        - If xz cannot remove an input file when it should, this
>>          is now treated as a warning (exit status 2) instead of
>>          an error (exit status 1). This matches GNU gzip and it
>>          is more logical as at that point the output file has
>>          already been successfully closed.
>>        - Fix handling of .xz files with an unsupported check type.
>>          Previously such printed a warning message but then xz
>>          behaved as if an error had occurred (didn't decompress,
>>          exit status 1). Now a warning is printed, decompression
>>          is done anyway, and exit status is 2. This used to work
>>          slightly before 5.0.0. In practice this bug matters only
>>          if xz has been built with some check types disabled. As
>>          instructed in PACKAGERS, such builds should be done in
>>          special situations only.
>>        - Fix "xz -dc --single-stream tests/files/good-0-empty.xz"
>>          which failed with "Internal error (bug)". That is,
>>          --single-stream was broken if the first .xz stream in
>>          the input file didn't contain any uncompressed data.
>>        - Fix displaying file sizes in the progress indicator when
>>          working in passthru mode and there are multiple input files.
>>          Just like "gzip -cdf", "xz -cdf" works like "cat" when the
>>          input file isn't a supported compressed file format. In
>>          this case the file size counters weren't reset between
>>          files so with multiple input files the progress indicator
>>          displayed an incorrect (too large) value.
>>    * liblzma:
>>        - API docs in lzma/container.h:
>>            * Update the list of decoder flags in the decoder
>>              function docs.
>>            * Explain LZMA_CONCATENATED behavior with .lzma files
>>              in lzma_auto_decoder() docs.
>>        - OpenBSD: Use HW_NCPUONLINE to detect the number of
>>          available hardware threads in lzma_physmem().
>>        - Fix use of wrong macro to detect x86 SSE2 support.
>>          __SSE2_MATH__ was used with GCC/Clang but the correct
>>          one is __SSE2__. The first one means that SSE2 is used
>>          for floating point math which is irrelevant here.
>>          The affected SSE2 code isn't used on x86-64 so this affects
>>          only 32-bit x86 builds that use -msse2 without -mfpmath=sse
>>          (there is no runtime detection for SSE2). It improves LZMA
>>          compression speed (not decompression).
>>        - Fix the build with Intel C compiler 2021 (ICC, not ICX)
>>          on Linux. It defines __GNUC__ to 10 but doesn't support
>>          the __symver__ attribute introduced in GCC 10.
>>    * Scripts: Ignore warnings from xz by using --quiet --no-warn.
>>      This is needed if the input .xz files use an unsupported
>>      check type.
>>    * Translations:
>>        - Updated Croatian and Turkish translations.
>>        - One new translations wasn't included because it needed
>>          technical fixes. It will be in upcoming 5.4.0. No new
>>          translations will be added to the 5.2.x branch anymore.
>>        - Renamed the French man page translation file from
>>          fr_FR.po to fr.po and thus also its install directory
>>          (like /usr/share/man/fr_FR -> .../fr).
>>        - Man page translations for upcoming 5.4.0 are now handled
>>          in the Translation Project.
>>    * Update doc/faq.txt a little so it's less out-of-date.
>> 5.2.7 (2022-09-30)
>>    * liblzma:
>>        - Made lzma_filters_copy() to never modify the destination
>>          array if an error occurs. lzma_stream_encoder() and
>>          lzma_stream_encoder_mt() already assumed this. Before this
>>          change, if a tiny memory allocation in lzma_filters_copy()
>>          failed it would lead to a crash (invalid free() or invalid
>>          memory reads) in the cleanup paths of these two encoder
>>          initialization functions.
>>        - Added missing integer overflow check to lzma_index_append().
>>          This affects xz --list and other applications that decode
>>          the Index field from .xz files using lzma_index_decoder().
>>          Normal decompression of .xz files doesn't call this code
>>          and thus most applications using liblzma aren't affected
>>          by this bug.
>>        - Single-threaded .xz decoder (lzma_stream_decoder()): If
>>          lzma_code() returns LZMA_MEMLIMIT_ERROR it is now possible
>>          to use lzma_memlimit_set() to increase the limit and continue
>>          decoding. This was supposed to work from the beginning
>>          but there was a bug. With other decoders (.lzma or
>>          threaded .xz decoder) this already worked correctly.
>>        - Fixed accumulation of integrity check type statistics in
>>          lzma_index_cat(). This bug made lzma_index_checks() return
>>          only the type of the integrity check of the last Stream
>>          when multiple lzma_indexes were concatenated. Most
>>          applications don't use these APIs but in xz it made
>>          xz --list not list all check types from concatenated .xz
>>          files. In xz --list --verbose only the per-file "Check:"
>>          lines were affected and in xz --robot --list only the "file"
>>          line was affected.
>>        - Added ABI compatibility with executables that were linked
>>          against liblzma in RHEL/CentOS 7 or other liblzma builds
>>          that had copied the problematic patch from RHEL/CentOS 7
>>          (xz-5.2.2-compat-libs.patch). For the details, see the
>>          comment at the top of src/liblzma/validate_map.sh.
>>          WARNING: This uses __symver__ attribute with GCC >= 10.
>>          In other cases the traditional __asm__(".symver ...")
>>          is used. Using link-time optimization (LTO, -flto) with
>>          GCC versions older than 10 can silently result in
>>          broken liblzma.so.5 (incorrect symbol versions)! If you
>>          want to use -flto with GCC, you must use GCC >= 10.
>>          LTO with Clang seems to work even with the traditional
>>          __asm__(".symver ...") method.
>>    * xzgrep: Fixed compatibility with old shells that break if
>>      comments inside command substitutions have apostrophes (').
>>      This problem was introduced in 5.2.6.
>>    * Build systems:
>>        - New #define in config.h: HAVE_SYMBOL_VERSIONS_LINUX
>>        - Windows: Fixed liblzma.dll build with Visual Studio project
>>          files. It broke in 5.2.6 due to a change that was made to
>>          improve CMake support.
>>        - Windows: Building liblzma with UNICODE defined should now
>>          work.
>>        - CMake files are now actually included in the release tarball.
>>          They should have been in 5.2.5 already.
>>        - Minor CMake fixes and improvements.
>>    * Added a new translation: Turkish
>> 5.2.6 (2022-08-12)
>>    * xz:
>>        - The --keep option now accepts symlinks, hardlinks, and
>>          setuid, setgid, and sticky files. Previously this required
>>          using --force.
>>        - When copying metadata from the source file to the destination
>>          file, don't try to set the group (GID) if it is already set
>>          correctly. This avoids a failure on OpenBSD (and possibly on
>>          a few other OSes) where files may get created so that their
>>          group doesn't belong to the user, and fchown(2) can fail even
>>          if it needs to do nothing.
>>        - Cap --memlimit-compress to 2000 MiB instead of 4020 MiB on
>>          MIPS32 because on MIPS32 userspace processes are limited
>>          to 2 GiB of address space.
>>    * liblzma:
>>        - Fixed a missing error-check in the threaded encoder. If a
>>          small memory allocation fails, a .xz file with an invalid
>>          Index field would be created. Decompressing such a file would
>>          produce the correct output but result in an error at the end.
>>          Thus this is a "mild" data corruption bug. Note that while
>>          a failed memory allocation can trigger the bug, it cannot
>>          cause invalid memory access.
>>        - The decoder for .lzma files now supports files that have
>>          uncompressed size stored in the header and still use the
>>          end of payload marker (end of stream marker) at the end
>>          of the LZMA stream. Such files are rare but, according to
>>          the documentation in LZMA SDK, they are valid.
>>          doc/lzma-file-format.txt was updated too.
>>        - Improved 32-bit x86 assembly files:
>>            * Support Intel Control-flow Enforcement Technology (CET)
>>            * Use non-executable stack on FreeBSD.
>>        - Visual Studio: Use non-standard _MSVC_LANG to detect C++
>>          standard version in the lzma.h API header. It's used to
>>          detect when "noexcept" can be used.
>>    * xzgrep:
>>        - Fixed arbitrary command injection via a malicious filename
>>          (CVE-2022-1271, ZDI-CAN-16587). A standalone patch for
>>          this was released to the public on 2022-04-07. A slight
>>          robustness improvement has been made since then and, if
>>          using GNU or *BSD grep, a new faster method is now used
>>          that doesn't use the old sed-based construct at all. This
>>          also fixes bad output with GNU grep >= 3.5 (2020-09-27)
>>          when xzgrepping binary files.
>>          This vulnerability was discovered by:
>>          cleemy desu wayo working with Trend Micro Zero Day Initiative
>>        - Fixed detection of corrupt .bz2 files.
>>        - Improved error handling to fix exit status in some situations
>>          and to fix handling of signals: in some situations a signal
>>          didn't make xzgrep exit when it clearly should have. It's
>>          possible that the signal handling still isn't quite perfect
>>          but hopefully it's good enough.
>>        - Documented exit statuses on the man page.
>>        - xzegrep and xzfgrep now use "grep -E" and "grep -F" instead
>>          of the deprecated egrep and fgrep commands.
>>        - Fixed parsing of the options -E, -F, -G, -P, and -X. The
>>          problem occurred when multiple options were specied in
>>          a single argument, for example,
>>              echo foo | xzgrep -Fe foo
>>          treated foo as a filename because -Fe wasn't correctly
>>          split into -F -e.
>>        - Added zstd support.
>>    * xzdiff/xzcmp:
>>        - Fixed wrong exit status. Exit status could be 2 when the
>>          correct value is 1.
>>        - Documented on the man page that exit status of 2 is used
>>          for decompression errors.
>>        - Added zstd support.
>>    * xzless:
>>        - Fix less(1) version detection. It failed if the version number
>>          from "less -V" contained a dot.
>>    * Translations:
>>        - Added new translations: Catalan, Croatian, Esperanto,
>>          Korean, Portuguese, Romanian, Serbian, Spanish, Swedish,
>>          and Ukrainian
>>        - Updated the Brazilian Portuguese translation.
>>        - Added French man page translation. This and the existing
>>          German translation aren't complete anymore because the
>>          English man pages got a few updates and the translators
>>          weren't reached so that they could update their work.
>>    * Build systems:
>>        - Windows: Fix building of resource files when config.h isn't
>>          used. CMake + Visual Studio can now build liblzma.dll.
>>        - Various fixes to the CMake support. Building static or shared
>>          liblzma should work fine in most cases. In contrast, building
>>          the command line tools with CMake is still clearly incomplete
>>          and experimental and should be used for testing only.
>> 
>> Signed-off-by: Adolf Belka <adolf.belka(a)ipfire.org>
>> ---
>> config/rootfiles/common/xz             | 37 ++++++++--
>> lfs/xz                                 |  5 +-
>> src/patches/xzgrep-ZDI-CAN-16587.patch | 94 --------------------------
>> 3 files changed, 32 insertions(+), 104 deletions(-)
>> delete mode 100644 src/patches/xzgrep-ZDI-CAN-16587.patch
>> 
>> diff --git a/config/rootfiles/common/xz b/config/rootfiles/common/xz
>> index c7bd0b302..d2f1d44cc 100644
>> --- a/config/rootfiles/common/xz
>> +++ b/config/rootfiles/common/xz
>> @@ -41,7 +41,7 @@ usr/bin/xzmore
>> #usr/lib/liblzma.la
>> usr/lib/liblzma.so
>> usr/lib/liblzma.so.5
>> -usr/lib/liblzma.so.5.2.5
>> +usr/lib/liblzma.so.5.2.8
>> #usr/lib/pkgconfig/liblzma.pc
>> #usr/share/doc/xz
>> #usr/share/doc/xz/AUTHORS
>> @@ -65,15 +65,26 @@ usr/lib/liblzma.so.5.2.5
>> #usr/share/doc/xz/history.txt
>> #usr/share/doc/xz/lzma-file-format.txt
>> #usr/share/doc/xz/xz-file-format.txt
>> +#usr/share/locale/ca/LC_MESSAGES/xz.mo
>> #usr/share/locale/cs/LC_MESSAGES/xz.mo
>> #usr/share/locale/da/LC_MESSAGES/xz.mo
>> #usr/share/locale/de/LC_MESSAGES/xz.mo
>> +#usr/share/locale/eo/LC_MESSAGES/xz.mo
>> +#usr/share/locale/es/LC_MESSAGES/xz.mo
>> #usr/share/locale/fi/LC_MESSAGES/xz.mo
>> #usr/share/locale/fr/LC_MESSAGES/xz.mo
>> +#usr/share/locale/hr/LC_MESSAGES/xz.mo
>> #usr/share/locale/hu/LC_MESSAGES/xz.mo
>> #usr/share/locale/it/LC_MESSAGES/xz.mo
>> +#usr/share/locale/ko/LC_MESSAGES/xz.mo
>> #usr/share/locale/pl/LC_MESSAGES/xz.mo
>> +#usr/share/locale/pt/LC_MESSAGES/xz.mo
>> #usr/share/locale/pt_BR/LC_MESSAGES/xz.mo
>> +#usr/share/locale/ro/LC_MESSAGES/xz.mo
>> +#usr/share/locale/sr/LC_MESSAGES/xz.mo
>> +#usr/share/locale/sv/LC_MESSAGES/xz.mo
>> +#usr/share/locale/tr/LC_MESSAGES/xz.mo
>> +#usr/share/locale/uk/LC_MESSAGES/xz.mo
>> #usr/share/locale/vi/LC_MESSAGES/xz.mo
>> #usr/share/locale/zh_CN/LC_MESSAGES/xz.mo
>> #usr/share/locale/zh_TW/LC_MESSAGES/xz.mo
>> @@ -82,9 +93,6 @@ usr/lib/liblzma.so.5.2.5
>> #usr/share/man/de/man1/lzcat.1
>> #usr/share/man/de/man1/lzcmp.1
>> #usr/share/man/de/man1/lzdiff.1
>> -#usr/share/man/de/man1/lzegrep.1
>> -#usr/share/man/de/man1/lzfgrep.1
>> -#usr/share/man/de/man1/lzgrep.1
>> #usr/share/man/de/man1/lzless.1
>> #usr/share/man/de/man1/lzma.1
>> #usr/share/man/de/man1/lzmadec.1
>> @@ -96,11 +104,26 @@ usr/lib/liblzma.so.5.2.5
>> #usr/share/man/de/man1/xzcmp.1
>> #usr/share/man/de/man1/xzdec.1
>> #usr/share/man/de/man1/xzdiff.1
>> -#usr/share/man/de/man1/xzegrep.1
>> -#usr/share/man/de/man1/xzfgrep.1
>> -#usr/share/man/de/man1/xzgrep.1
>> #usr/share/man/de/man1/xzless.1
>> #usr/share/man/de/man1/xzmore.1
>> +#usr/share/man/fr
>> +#usr/share/man/fr/man1
>> +#usr/share/man/fr/man1/lzcat.1
>> +#usr/share/man/fr/man1/lzcmp.1
>> +#usr/share/man/fr/man1/lzdiff.1
>> +#usr/share/man/fr/man1/lzless.1
>> +#usr/share/man/fr/man1/lzma.1
>> +#usr/share/man/fr/man1/lzmadec.1
>> +#usr/share/man/fr/man1/lzmore.1
>> +#usr/share/man/fr/man1/unlzma.1
>> +#usr/share/man/fr/man1/unxz.1
>> +#usr/share/man/fr/man1/xz.1
>> +#usr/share/man/fr/man1/xzcat.1
>> +#usr/share/man/fr/man1/xzcmp.1
>> +#usr/share/man/fr/man1/xzdec.1
>> +#usr/share/man/fr/man1/xzdiff.1
>> +#usr/share/man/fr/man1/xzless.1
>> +#usr/share/man/fr/man1/xzmore.1
>> #usr/share/man/man1/lzcat.1
>> #usr/share/man/man1/lzcmp.1
>> #usr/share/man/man1/lzdiff.1
>> diff --git a/lfs/xz b/lfs/xz
>> index 9345df954..83a724e1a 100644
>> --- a/lfs/xz
>> +++ b/lfs/xz
>> @@ -24,7 +24,7 @@
>> 
>> include Config
>> 
>> -VER        = 5.2.5
>> +VER        = 5.2.8
>> 
>> THISAPP    = xz-$(VER)
>> DL_FILE    = $(THISAPP).tar.xz
>> @@ -45,7 +45,7 @@ objects = $(DL_FILE)
>> 
>> $(DL_FILE) = $(DL_FROM)/$(DL_FILE)
>> 
>> -$(DL_FILE)_BLAKE2 = 9b9b58e33722ecfe799bb50f3ffe4e86386f734ab4468eb54ff92771ddf899302d1ffa4d88bdb0de351fc3eab8a6ea103f00d7e79f33db879fe22b2e1a7e62db
>> +$(DL_FILE)_BLAKE2 = 44d1ddd783b2527f3b17481fc277b671808eb5639c10d31bfaca9fd29ac4413628654ecb9e207955a9477c83eb30f61cf5607cd9a49dd71732707731e4444ace
>> 
>> install : $(TARGET)
>> 
>> @@ -75,7 +75,6 @@ $(subst %,%_BLAKE2,$(objects)) :
>> $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
>> @$(PREBUILD)
>> @rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar axf $(DIR_DL)/$(DL_FILE)
>> - cd $(DIR_APP) && patch -p1 -i $(DIR_SRC)/src/patches/xzgrep-ZDI-CAN-16587.patch
>> cd $(DIR_APP) && ./configure --prefix=$(PREFIX)
>> cd $(DIR_APP) && make $(MAKETUNING)
>> cd $(DIR_APP) && make install
>> diff --git a/src/patches/xzgrep-ZDI-CAN-16587.patch b/src/patches/xzgrep-ZDI-CAN-16587.patch
>> deleted file mode 100644
>> index 406ded590..000000000
>> --- a/src/patches/xzgrep-ZDI-CAN-16587.patch
>> +++ /dev/null
>> @@ -1,94 +0,0 @@
>> -From 69d1b3fc29677af8ade8dc15dba83f0589cb63d6 Mon Sep 17 00:00:00 2001
>> -From: Lasse Collin <lasse.collin(a)tukaani.org>
>> -Date: Tue, 29 Mar 2022 19:19:12 +0300
>> -Subject: [PATCH] xzgrep: Fix escaping of malicious filenames (ZDI-CAN-16587).
>> -
>> -Malicious filenames can make xzgrep to write to arbitrary files
>> -or (with a GNU sed extension) lead to arbitrary code execution.
>> -
>> -xzgrep from XZ Utils versions up to and including 5.2.5 are
>> -affected. 5.3.1alpha and 5.3.2alpha are affected as well.
>> -This patch works for all of them.
>> -
>> -This bug was inherited from gzip's zgrep. gzip 1.12 includes
>> -a fix for zgrep.
>> -
>> -The issue with the old sed script is that with multiple newlines,
>> -the N-command will read the second line of input, then the
>> -s-commands will be skipped because it's not the end of the
>> -file yet, then a new sed cycle starts and the pattern space
>> -is printed and emptied. So only the last line or two get escaped.
>> -
>> -One way to fix this would be to read all lines into the pattern
>> -space first. However, the included fix is even simpler: All lines
>> -except the last line get a backslash appended at the end. To ensure
>> -that shell command substitution doesn't eat a possible trailing
>> -newline, a colon is appended to the filename before escaping.
>> -The colon is later used to separate the filename from the grep
>> -output so it is fine to add it here instead of a few lines later.
>> -
>> -The old code also wasn't POSIX compliant as it used \n in the
>> -replacement section of the s-command. Using \<newline> is the
>> -POSIX compatible method.
>> -
>> -LC_ALL=C was added to the two critical sed commands. POSIX sed
>> -manual recommends it when using sed to manipulate pathnames
>> -because in other locales invalid multibyte sequences might
>> -cause issues with some sed implementations. In case of GNU sed,
>> -these particular sed scripts wouldn't have such problems but some
>> -other scripts could have, see:
>> -
>> -    info '(sed)Locale Considerations'
>> -
>> -This vulnerability was discovered by:
>> -cleemy desu wayo working with Trend Micro Zero Day Initiative
>> -
>> -Thanks to Jim Meyering and Paul Eggert discussing the different
>> -ways to fix this and for coordinating the patch release schedule
>> -with gzip.
>> ----
>> - src/scripts/xzgrep.in | 20 ++++++++++++--------
>> - 1 file changed, 12 insertions(+), 8 deletions(-)
>> -
>> -diff --git a/src/scripts/xzgrep.in b/src/scripts/xzgrep.in
>> -index b180936..e5186ba 100644
>> ---- a/src/scripts/xzgrep.in
>> -+++ b/src/scripts/xzgrep.in
>> -@@ -180,22 +180,26 @@ for i; do
>> -          { test $# -eq 1 || test $no_filename -eq 1; }; then
>> -       eval "$grep"
>> -     else
>> -+      # Append a colon so that the last character will never be a newline
>> -+      # which would otherwise get lost in shell command substitution.
>> -+      i="$i:"
>> -+
>> -+      # Escape & \ | and newlines only if such characters are present
>> -+      # (speed optimization).
>> -       case $i in
>> -       (*'
>> - '* | *'&'* | *'\'* | *'|'*)
>> --        i=$(printf '%s\n' "$i" |
>> --            sed '
>> --              $!N
>> --              $s/[&\|]/\\&/g
>> --              $s/\n/\\n/g
>> --            ');;
>> -+        i=$(printf '%s\n' "$i" | LC_ALL=C sed 's/[&\|]/\\&/g; $!s/$/\\/');;
>> -       esac
>> --      sed_script="s|^|$i:|"
>> -+
>> -+      # $i already ends with a colon so don't add it here.
>> -+      sed_script="s|^|$i|"
>> - 
>> -       # Fail if grep or sed fails.
>> -       r=$(
>> -         exec 4>&1
>> --        (eval "$grep" 4>&-; echo $? >&4) 3>&- | sed "$sed_script" >&3 4>&-
>> -+        (eval "$grep" 4>&-; echo $? >&4) 3>&- |
>> -+            LC_ALL=C sed "$sed_script" >&3 4>&-
>> -       ) || r=2
>> -       exit $r
>> -     fi >&3 5>&-
>> --- 
>> -2.35.1
>> -