From mboxrd@z Thu Jan 1 00:00:00 1970 From: Peter =?utf-8?q?M=C3=BCller?= To: development@lists.ipfire.org Subject: Re: [PATCH] xz: Update to version 5.2.8 Date: Mon, 21 Nov 2022 08:28:30 +0000 Message-ID: <31a223fd-bb60-03dc-3dd0-ed4d2bcfa26f@ipfire.org> In-Reply-To: <20221119175236.4019257-1-adolf.belka@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============3337328784134683612==" List-Id: --===============3337328784134683612== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Reviewed-by: Peter M=C3=BCller > - Update from version 5.2.5 to 5.2.8 > - Update of rootfile > - Remove xzgrep-ZDI-CAN-16587 patch as the contents are now integrated into= the source > tarball and with an improved quicker method - see changelog below. > - Changelog > 5.2.8 (2022-11-13) > * xz: > - If xz cannot remove an input file when it should, this > is now treated as a warning (exit status 2) instead of > an error (exit status 1). This matches GNU gzip and it > is more logical as at that point the output file has > already been successfully closed. > - Fix handling of .xz files with an unsupported check type. > Previously such printed a warning message but then xz > behaved as if an error had occurred (didn't decompress, > exit status 1). Now a warning is printed, decompression > is done anyway, and exit status is 2. This used to work > slightly before 5.0.0. In practice this bug matters only > if xz has been built with some check types disabled. As > instructed in PACKAGERS, such builds should be done in > special situations only. > - Fix "xz -dc --single-stream tests/files/good-0-empty.xz" > which failed with "Internal error (bug)". That is, > --single-stream was broken if the first .xz stream in > the input file didn't contain any uncompressed data. > - Fix displaying file sizes in the progress indicator when > working in passthru mode and there are multiple input files. > Just like "gzip -cdf", "xz -cdf" works like "cat" when the > input file isn't a supported compressed file format. In > this case the file size counters weren't reset between > files so with multiple input files the progress indicator > displayed an incorrect (too large) value. > * liblzma: > - API docs in lzma/container.h: > * Update the list of decoder flags in the decoder > function docs. > * Explain LZMA_CONCATENATED behavior with .lzma files > in lzma_auto_decoder() docs. > - OpenBSD: Use HW_NCPUONLINE to detect the number of > available hardware threads in lzma_physmem(). > - Fix use of wrong macro to detect x86 SSE2 support. > __SSE2_MATH__ was used with GCC/Clang but the correct > one is __SSE2__. The first one means that SSE2 is used > for floating point math which is irrelevant here. > The affected SSE2 code isn't used on x86-64 so this affects > only 32-bit x86 builds that use -msse2 without -mfpmath=3Dsse > (there is no runtime detection for SSE2). It improves LZMA > compression speed (not decompression). > - Fix the build with Intel C compiler 2021 (ICC, not ICX) > on Linux. It defines __GNUC__ to 10 but doesn't support > the __symver__ attribute introduced in GCC 10. > * Scripts: Ignore warnings from xz by using --quiet --no-warn. > This is needed if the input .xz files use an unsupported > check type. > * Translations: > - Updated Croatian and Turkish translations. > - One new translations wasn't included because it needed > technical fixes. It will be in upcoming 5.4.0. No new > translations will be added to the 5.2.x branch anymore. > - Renamed the French man page translation file from > fr_FR.po to fr.po and thus also its install directory > (like /usr/share/man/fr_FR -> .../fr). > - Man page translations for upcoming 5.4.0 are now handled > in the Translation Project. > * Update doc/faq.txt a little so it's less out-of-date. > 5.2.7 (2022-09-30) > * liblzma: > - Made lzma_filters_copy() to never modify the destination > array if an error occurs. lzma_stream_encoder() and > lzma_stream_encoder_mt() already assumed this. Before this > change, if a tiny memory allocation in lzma_filters_copy() > failed it would lead to a crash (invalid free() or invalid > memory reads) in the cleanup paths of these two encoder > initialization functions. > - Added missing integer overflow check to lzma_index_append(). > This affects xz --list and other applications that decode > the Index field from .xz files using lzma_index_decoder(). > Normal decompression of .xz files doesn't call this code > and thus most applications using liblzma aren't affected > by this bug. > - Single-threaded .xz decoder (lzma_stream_decoder()): If > lzma_code() returns LZMA_MEMLIMIT_ERROR it is now possible > to use lzma_memlimit_set() to increase the limit and continue > decoding. This was supposed to work from the beginning > but there was a bug. With other decoders (.lzma or > threaded .xz decoder) this already worked correctly. > - Fixed accumulation of integrity check type statistics in > lzma_index_cat(). This bug made lzma_index_checks() return > only the type of the integrity check of the last Stream > when multiple lzma_indexes were concatenated. Most > applications don't use these APIs but in xz it made > xz --list not list all check types from concatenated .xz > files. In xz --list --verbose only the per-file "Check:" > lines were affected and in xz --robot --list only the "file" > line was affected. > - Added ABI compatibility with executables that were linked > against liblzma in RHEL/CentOS 7 or other liblzma builds > that had copied the problematic patch from RHEL/CentOS 7 > (xz-5.2.2-compat-libs.patch). For the details, see the > comment at the top of src/liblzma/validate_map.sh. > WARNING: This uses __symver__ attribute with GCC >=3D 10. > In other cases the traditional __asm__(".symver ...") > is used. Using link-time optimization (LTO, -flto) with > GCC versions older than 10 can silently result in > broken liblzma.so.5 (incorrect symbol versions)! If you > want to use -flto with GCC, you must use GCC >=3D 10. > LTO with Clang seems to work even with the traditional > __asm__(".symver ...") method. > * xzgrep: Fixed compatibility with old shells that break if > comments inside command substitutions have apostrophes ('). > This problem was introduced in 5.2.6. > * Build systems: > - New #define in config.h: HAVE_SYMBOL_VERSIONS_LINUX > - Windows: Fixed liblzma.dll build with Visual Studio project > files. It broke in 5.2.6 due to a change that was made to > improve CMake support. > - Windows: Building liblzma with UNICODE defined should now > work. > - CMake files are now actually included in the release tarball. > They should have been in 5.2.5 already. > - Minor CMake fixes and improvements. > * Added a new translation: Turkish > 5.2.6 (2022-08-12) > * xz: > - The --keep option now accepts symlinks, hardlinks, and > setuid, setgid, and sticky files. Previously this required > using --force. > - When copying metadata from the source file to the destination > file, don't try to set the group (GID) if it is already set > correctly. This avoids a failure on OpenBSD (and possibly on > a few other OSes) where files may get created so that their > group doesn't belong to the user, and fchown(2) can fail even > if it needs to do nothing. > - Cap --memlimit-compress to 2000 MiB instead of 4020 MiB on > MIPS32 because on MIPS32 userspace processes are limited > to 2 GiB of address space. > * liblzma: > - Fixed a missing error-check in the threaded encoder. If a > small memory allocation fails, a .xz file with an invalid > Index field would be created. Decompressing such a file would > produce the correct output but result in an error at the end. > Thus this is a "mild" data corruption bug. Note that while > a failed memory allocation can trigger the bug, it cannot > cause invalid memory access. > - The decoder for .lzma files now supports files that have > uncompressed size stored in the header and still use the > end of payload marker (end of stream marker) at the end > of the LZMA stream. Such files are rare but, according to > the documentation in LZMA SDK, they are valid. > doc/lzma-file-format.txt was updated too. > - Improved 32-bit x86 assembly files: > * Support Intel Control-flow Enforcement Technology (CET) > * Use non-executable stack on FreeBSD. > - Visual Studio: Use non-standard _MSVC_LANG to detect C++ > standard version in the lzma.h API header. It's used to > detect when "noexcept" can be used. > * xzgrep: > - Fixed arbitrary command injection via a malicious filename > (CVE-2022-1271, ZDI-CAN-16587). A standalone patch for > this was released to the public on 2022-04-07. A slight > robustness improvement has been made since then and, if > using GNU or *BSD grep, a new faster method is now used > that doesn't use the old sed-based construct at all. This > also fixes bad output with GNU grep >=3D 3.5 (2020-09-27) > when xzgrepping binary files. > This vulnerability was discovered by: > cleemy desu wayo working with Trend Micro Zero Day Initiative > - Fixed detection of corrupt .bz2 files. > - Improved error handling to fix exit status in some situations > and to fix handling of signals: in some situations a signal > didn't make xzgrep exit when it clearly should have. It's > possible that the signal handling still isn't quite perfect > but hopefully it's good enough. > - Documented exit statuses on the man page. > - xzegrep and xzfgrep now use "grep -E" and "grep -F" instead > of the deprecated egrep and fgrep commands. > - Fixed parsing of the options -E, -F, -G, -P, and -X. The > problem occurred when multiple options were specied in > a single argument, for example, > echo foo | xzgrep -Fe foo > treated foo as a filename because -Fe wasn't correctly > split into -F -e. > - Added zstd support. > * xzdiff/xzcmp: > - Fixed wrong exit status. Exit status could be 2 when the > correct value is 1. > - Documented on the man page that exit status of 2 is used > for decompression errors. > - Added zstd support. > * xzless: > - Fix less(1) version detection. It failed if the version number > from "less -V" contained a dot. > * Translations: > - Added new translations: Catalan, Croatian, Esperanto, > Korean, Portuguese, Romanian, Serbian, Spanish, Swedish, > and Ukrainian > - Updated the Brazilian Portuguese translation. > - Added French man page translation. This and the existing > German translation aren't complete anymore because the > English man pages got a few updates and the translators > weren't reached so that they could update their work. > * Build systems: > - Windows: Fix building of resource files when config.h isn't > used. CMake + Visual Studio can now build liblzma.dll. > - Various fixes to the CMake support. Building static or shared > liblzma should work fine in most cases. In contrast, building > the command line tools with CMake is still clearly incomplete > and experimental and should be used for testing only. >=20 > Signed-off-by: Adolf Belka > --- > config/rootfiles/common/xz | 37 ++++++++-- > lfs/xz | 5 +- > src/patches/xzgrep-ZDI-CAN-16587.patch | 94 -------------------------- > 3 files changed, 32 insertions(+), 104 deletions(-) > delete mode 100644 src/patches/xzgrep-ZDI-CAN-16587.patch >=20 > diff --git a/config/rootfiles/common/xz b/config/rootfiles/common/xz > index c7bd0b302..d2f1d44cc 100644 > --- a/config/rootfiles/common/xz > +++ b/config/rootfiles/common/xz > @@ -41,7 +41,7 @@ usr/bin/xzmore > #usr/lib/liblzma.la > usr/lib/liblzma.so > usr/lib/liblzma.so.5 > -usr/lib/liblzma.so.5.2.5 > +usr/lib/liblzma.so.5.2.8 > #usr/lib/pkgconfig/liblzma.pc > #usr/share/doc/xz > #usr/share/doc/xz/AUTHORS > @@ -65,15 +65,26 @@ usr/lib/liblzma.so.5.2.5 > #usr/share/doc/xz/history.txt > #usr/share/doc/xz/lzma-file-format.txt > #usr/share/doc/xz/xz-file-format.txt > +#usr/share/locale/ca/LC_MESSAGES/xz.mo > #usr/share/locale/cs/LC_MESSAGES/xz.mo > #usr/share/locale/da/LC_MESSAGES/xz.mo > #usr/share/locale/de/LC_MESSAGES/xz.mo > +#usr/share/locale/eo/LC_MESSAGES/xz.mo > +#usr/share/locale/es/LC_MESSAGES/xz.mo > #usr/share/locale/fi/LC_MESSAGES/xz.mo > #usr/share/locale/fr/LC_MESSAGES/xz.mo > +#usr/share/locale/hr/LC_MESSAGES/xz.mo > #usr/share/locale/hu/LC_MESSAGES/xz.mo > #usr/share/locale/it/LC_MESSAGES/xz.mo > +#usr/share/locale/ko/LC_MESSAGES/xz.mo > #usr/share/locale/pl/LC_MESSAGES/xz.mo > +#usr/share/locale/pt/LC_MESSAGES/xz.mo > #usr/share/locale/pt_BR/LC_MESSAGES/xz.mo > +#usr/share/locale/ro/LC_MESSAGES/xz.mo > +#usr/share/locale/sr/LC_MESSAGES/xz.mo > +#usr/share/locale/sv/LC_MESSAGES/xz.mo > +#usr/share/locale/tr/LC_MESSAGES/xz.mo > +#usr/share/locale/uk/LC_MESSAGES/xz.mo > #usr/share/locale/vi/LC_MESSAGES/xz.mo > #usr/share/locale/zh_CN/LC_MESSAGES/xz.mo > #usr/share/locale/zh_TW/LC_MESSAGES/xz.mo > @@ -82,9 +93,6 @@ usr/lib/liblzma.so.5.2.5 > #usr/share/man/de/man1/lzcat.1 > #usr/share/man/de/man1/lzcmp.1 > #usr/share/man/de/man1/lzdiff.1 > -#usr/share/man/de/man1/lzegrep.1 > -#usr/share/man/de/man1/lzfgrep.1 > -#usr/share/man/de/man1/lzgrep.1 > #usr/share/man/de/man1/lzless.1 > #usr/share/man/de/man1/lzma.1 > #usr/share/man/de/man1/lzmadec.1 > @@ -96,11 +104,26 @@ usr/lib/liblzma.so.5.2.5 > #usr/share/man/de/man1/xzcmp.1 > #usr/share/man/de/man1/xzdec.1 > #usr/share/man/de/man1/xzdiff.1 > -#usr/share/man/de/man1/xzegrep.1 > -#usr/share/man/de/man1/xzfgrep.1 > -#usr/share/man/de/man1/xzgrep.1 > #usr/share/man/de/man1/xzless.1 > #usr/share/man/de/man1/xzmore.1 > +#usr/share/man/fr > +#usr/share/man/fr/man1 > +#usr/share/man/fr/man1/lzcat.1 > +#usr/share/man/fr/man1/lzcmp.1 > +#usr/share/man/fr/man1/lzdiff.1 > +#usr/share/man/fr/man1/lzless.1 > +#usr/share/man/fr/man1/lzma.1 > +#usr/share/man/fr/man1/lzmadec.1 > +#usr/share/man/fr/man1/lzmore.1 > +#usr/share/man/fr/man1/unlzma.1 > +#usr/share/man/fr/man1/unxz.1 > +#usr/share/man/fr/man1/xz.1 > +#usr/share/man/fr/man1/xzcat.1 > +#usr/share/man/fr/man1/xzcmp.1 > +#usr/share/man/fr/man1/xzdec.1 > +#usr/share/man/fr/man1/xzdiff.1 > +#usr/share/man/fr/man1/xzless.1 > +#usr/share/man/fr/man1/xzmore.1 > #usr/share/man/man1/lzcat.1 > #usr/share/man/man1/lzcmp.1 > #usr/share/man/man1/lzdiff.1 > diff --git a/lfs/xz b/lfs/xz > index 9345df954..83a724e1a 100644 > --- a/lfs/xz > +++ b/lfs/xz > @@ -24,7 +24,7 @@ > =20 > include Config > =20 > -VER =3D 5.2.5 > +VER =3D 5.2.8 > =20 > THISAPP =3D xz-$(VER) > DL_FILE =3D $(THISAPP).tar.xz > @@ -45,7 +45,7 @@ objects =3D $(DL_FILE) > =20 > $(DL_FILE) =3D $(DL_FROM)/$(DL_FILE) > =20 > -$(DL_FILE)_BLAKE2 =3D 9b9b58e33722ecfe799bb50f3ffe4e86386f734ab4468eb54ff9= 2771ddf899302d1ffa4d88bdb0de351fc3eab8a6ea103f00d7e79f33db879fe22b2e1a7e62db > +$(DL_FILE)_BLAKE2 =3D 44d1ddd783b2527f3b17481fc277b671808eb5639c10d31bfaca= 9fd29ac4413628654ecb9e207955a9477c83eb30f61cf5607cd9a49dd71732707731e4444ace > =20 > install : $(TARGET) > =20 > @@ -75,7 +75,6 @@ $(subst %,%_BLAKE2,$(objects)) : > $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) > @$(PREBUILD) > @rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar axf $(DIR_DL)/$(DL_FILE) > - cd $(DIR_APP) && patch -p1 -i $(DIR_SRC)/src/patches/xzgrep-ZDI-CAN-16587= .patch > cd $(DIR_APP) && ./configure --prefix=3D$(PREFIX) > cd $(DIR_APP) && make $(MAKETUNING) > cd $(DIR_APP) && make install > diff --git a/src/patches/xzgrep-ZDI-CAN-16587.patch b/src/patches/xzgrep-ZD= I-CAN-16587.patch > deleted file mode 100644 > index 406ded590..000000000 > --- a/src/patches/xzgrep-ZDI-CAN-16587.patch > +++ /dev/null > @@ -1,94 +0,0 @@ > -From 69d1b3fc29677af8ade8dc15dba83f0589cb63d6 Mon Sep 17 00:00:00 2001 > -From: Lasse Collin > -Date: Tue, 29 Mar 2022 19:19:12 +0300 > -Subject: [PATCH] xzgrep: Fix escaping of malicious filenames (ZDI-CAN-1658= 7). > - > -Malicious filenames can make xzgrep to write to arbitrary files > -or (with a GNU sed extension) lead to arbitrary code execution. > - > -xzgrep from XZ Utils versions up to and including 5.2.5 are > -affected. 5.3.1alpha and 5.3.2alpha are affected as well. > -This patch works for all of them. > - > -This bug was inherited from gzip's zgrep. gzip 1.12 includes > -a fix for zgrep. > - > -The issue with the old sed script is that with multiple newlines, > -the N-command will read the second line of input, then the > -s-commands will be skipped because it's not the end of the > -file yet, then a new sed cycle starts and the pattern space > -is printed and emptied. So only the last line or two get escaped. > - > -One way to fix this would be to read all lines into the pattern > -space first. However, the included fix is even simpler: All lines > -except the last line get a backslash appended at the end. To ensure > -that shell command substitution doesn't eat a possible trailing > -newline, a colon is appended to the filename before escaping. > -The colon is later used to separate the filename from the grep > -output so it is fine to add it here instead of a few lines later. > - > -The old code also wasn't POSIX compliant as it used \n in the > -replacement section of the s-command. Using \ is the > -POSIX compatible method. > - > -LC_ALL=3DC was added to the two critical sed commands. POSIX sed > -manual recommends it when using sed to manipulate pathnames > -because in other locales invalid multibyte sequences might > -cause issues with some sed implementations. In case of GNU sed, > -these particular sed scripts wouldn't have such problems but some > -other scripts could have, see: > - > - info '(sed)Locale Considerations' > - > -This vulnerability was discovered by: > -cleemy desu wayo working with Trend Micro Zero Day Initiative > - > -Thanks to Jim Meyering and Paul Eggert discussing the different > -ways to fix this and for coordinating the patch release schedule > -with gzip. > ---- > - src/scripts/xzgrep.in | 20 ++++++++++++-------- > - 1 file changed, 12 insertions(+), 8 deletions(-) > - > -diff --git a/src/scripts/xzgrep.in b/src/scripts/xzgrep.in > -index b180936..e5186ba 100644 > ---- a/src/scripts/xzgrep.in > -+++ b/src/scripts/xzgrep.in > -@@ -180,22 +180,26 @@ for i; do > - { test $# -eq 1 || test $no_filename -eq 1; }; then > - eval "$grep" > - else > -+ # Append a colon so that the last character will never be a newline > -+ # which would otherwise get lost in shell command substitution. > -+ i=3D"$i:" > -+ > -+ # Escape & \ | and newlines only if such characters are present > -+ # (speed optimization). > - case $i in > - (*' > - '* | *'&'* | *'\'* | *'|'*) > -- i=3D$(printf '%s\n' "$i" | > -- sed ' > -- $!N > -- $s/[&\|]/\\&/g > -- $s/\n/\\n/g > -- ');; > -+ i=3D$(printf '%s\n' "$i" | LC_ALL=3DC sed 's/[&\|]/\\&/g; $!s/$/\= \/');; > - esac > -- sed_script=3D"s|^|$i:|" > -+ > -+ # $i already ends with a colon so don't add it here. > -+ sed_script=3D"s|^|$i|" > -=20 > - # Fail if grep or sed fails. > - r=3D$( > - exec 4>&1 > -- (eval "$grep" 4>&-; echo $? >&4) 3>&- | sed "$sed_script" >&3 4>&- > -+ (eval "$grep" 4>&-; echo $? >&4) 3>&- | > -+ LC_ALL=3DC sed "$sed_script" >&3 4>&- > - ) || r=3D2 > - exit $r > - fi >&3 5>&- > ---=20 > -2.35.1 > - --===============3337328784134683612==--