From mboxrd@z Thu Jan 1 00:00:00 1970 From: Adolf Belka To: development@lists.ipfire.org Subject: Re: clamav 0.105.1-3 needs rust >1.61 Date: Tue, 22 Nov 2022 16:39:37 +0100 Message-ID: <5a798895-d240-f2b7-4736-ad1a86b3bf96@ipfire.org> In-Reply-To: <493b25ac-47b1-a7a4-6896-bcb3f16d7adb@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============6133671041901225108==" List-Id: --===============6133671041901225108== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi Matthias, You are experiencing Rust Hell. That is what I have had especially with=20 the python3-cryptography module. Often when it has been updated a new=20 rust module is required which requires another new module etc, etc, etc. On 21/11/2022 20:05, Matthias Fischer wrote: > On 21.11.2022 11:44, Michael Tremer wrote: >> Hello Matthias, > Hi Michael, > > updated cipher to '0.4.3'. > > Clean build, result: > > ***SNIP*** > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Installing cipher-0.4.3 ... > Install started; saving file list to /usr/src/lsalr ... > cd /usr/src/cipher-0.4.3 && mkdir -p > /usr/src/cipher-0.4.3/.cargo && echo "${CARGO_CONFIG}" > > /usr/src/cipher-0.4.3/.cargo/config && rm -f Cargo.lock > cd /usr/src/cipher-0.4.3 && CARGOPATH=3D/usr/src/cipher-0.4.3/.cargo > RUSTC_BOOTSTRAP=3D1 cargo --offline build --release -Z avoid-dev-deps -j8 > error: no matching package named `crypto-common` found > location searched: registry `crates-io` > required by package `cipher v0.4.3 (/usr/src/cipher-0.4.3)` > As a reminder, you're using offline mode (--offline) which can > sometimes cause surprising resolution failures, if this error is too > confusing you may wish to retry without the offline flag. > make: *** [rust-cipher:77: /usr/src/log/cipher-0.4.3] Error 101 > ***SNAP*** > > Hm. > > Just guessing =3D> updated 'lfs/rust-crypto-common' from '0.1.1' to > '0.1.6' through the helper script. > > =3D> Identical error: "no matching package found". > > Hmmm! > > To follow the "reminder" would mean to delete the '--offline' option in > line 209 in 'lfs'/config', but that would be only further guessing. And > this would affect all other files. Doesn't feel good. You can't do that as the IPFire build is run in a chroot that has no=20 internet access. > I'm not familiar with this rust thing - sorry: any ideas about the best What I have done so far is just to add each new rust module that has=20 been highlighted and then re-run the build. I have had the situation of=20 12 additional rust modules being required with an update. After some discussions with my son in the future I intend to run the=20 build of just a specific package in a directory on my machine, the same=20 as I do when building any package from source on my computer (not in the=20 IPFire build shell). This build would then run with internet access and=20 it should then download and install all required rust modules. Then after that build (of clamav in your case) has been successful you=20 can look at the log details for that build to see which rust modules=20 have been downloaded and installed and then you can use the script that=20 Michael mentioned to add all of those modules to the build in one go. Warning, I have not tried this approach myself yet but it seemed to make=20 sense to me when my son suggested it. You would need to test it out and=20 see if it helped or not. You will need to check if any rust modules have been required to have a=20 certain version n umber of range. That can be the case that a rust=20 module needs to be updated but not to the latest version. If that is the=20 case then you can use the rust script to download and install into an=20 lfs a specific version. It can also be the case that some rust modules or packages require a not=20 latest version and other modules require the same module but at a=20 different version number. In this case you have to relabel the lfs file=20 to include the version=C2=A0 number. If you look at the list of rust lfs=20 files you will see that some have a specific version specified as well=20 as the plain named module. For example rust-indoc (version 1.0.3) and rust-indoc-0.3.6 (version 0.3.6). What would help would be if for every package that used rust modules=20 that there was a dependency file that listed all the ones required. It=20 doesn't exist. By the way, you also need to watch out that some rust packages will by=20 default install the versions for multiple architectures such as windows=20 or mac etc, which is obviously not needed for IPFire. The ones that I=20 have found like that are rust-chrono and rust-iana-time-zone which then=20 needed patches to comment out the meta-data for building the windows etc=20 versions. If this happens with a package you will find rust modules that=20 have win or some other OS name in the title so that usually flags up to=20 me that I need to go back to an earlier module and stop the win based=20 builds. See rust-chrono-0.4.22-fix-metadata.patch in the /src/patches directory=20 as an example. Sorry that my input is probably not what you were hoping for.=C2=A0 It can be= =20 worked though, it just can take some time. Regards, Adolf. > way to proceed? > >>> On 19 Nov 2022, at 15:56, Matthias Fischer wrote: >>> >>> Hi, >>> >>> ...I'd like to have a small problem... ;-) >>> >>> A few days ago, 'clamav 0.105.1' was updated, again: >>> >>> =3D> >>> https://blog.clamav.net/2022/11/second-clamav-100-release-candidate-and.h= tml >>> >>> "...[it] was intended to also include bug fixes for the jpeg and tiff >>> Rust-based libraries that are bundled with the source code tarball. >>> Unfortunately, those fixes were not all release-ready in time for the >>> 0.105.1-2 packages." >>> >>> So far, so [oh, forget it!]. >> This is *really* bad that they bundle so many libraries and make it very d= ifficult for us to keep track of what vulnerabilities might be in clamav alth= ough they are part of a third-party library. >> >> We should try to remove all of them and always build against the system li= braries. >> >>> Unfortunately, building the third version of 'clamav 0.105.1' with >>> current 'next' failed: >>> >>> ***SNIP*** >>> ... >>> error: package `tiff v0.8.0` cannot be built because it requires >>> rustc 1.61.0 or newer, while the currently active rustc version is >>> 1.60.0-nightly. >>> >>> [193/379] Building C object >>> libclamav/CMakeFiles/lzma_sdk.dir/7z/7zIn.c.o >>> [194/379] Building C object >>> libclamav/CMakeFiles/bytecode_runtime.dir/bytecode_nojit.c.o >>> [195/379] Building C object >>> libclamav/CMakeFiles/yara.dir/yara_grammar.c.o >>> [196/379] Building C object libclamav/CMakeFiles/yara.dir/yara_lexer.= c.o >>> yara_lexer.c:2571:24: warning: 'yy_fatal_error' defined but not used >>> [-Wunused-function] >>> yara_lexer.c: In function 'yara_yylex': >>> yara_lexer.l:263:16: warning: '%s' directive output may be truncated >>> writing up to 1023 bytes into a region of size 999 [-Wformat-truncation= =3D] >>> In file included from /usr/include/stdio.h:906, >>> from yara_lexer.c:32: >>> /usr/include/bits/stdio2.h:54:10: note: '__builtin___snprintf_chk' >>> output between 26 and 1049 bytes into a destination of size 1024 >>> 54 | return __builtin___snprintf_chk (__s, __n, >>> __USE_FORTIFY_LEVEL - 1, >>> | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~= ~~ >>> 55 | __glibc_objsize (__s), __fmt, >>> | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> 56 | __va_arg_pack ()); >>> | ~~~~~~~~~~~~~~~~~ >>> ninja: build stopped: subcommand failed. >>> make: *** [clamav:89: /usr/src/log/clamav-0.105.1] Error 1 >>> ***SNAP*** >> Great code quality. This is however not the reason why the build stopped. = This is only a warning. >> >>> Hm. Great. >>> >>> So I tried the current 'rust 1.65' version. >>> >>> This time, the building failed because of a rust component: >>> >>> ***SNIP*** >>> ... >>> Finished release [optimized] target(s) in 1.92s >>> cd /usr/src/cipher-0.3.0 && mkdir -pv >>> "/usr/share/cargo/registry/cipher-0.3.0" && if >>> CARGOPATH=3D/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=3D1 cargo --offl= ine >>> metadata --format-version 1 --no-deps | jq -e >>> ".packages[].targets[].kind | any(. =3D=3D \"lib\")" | grep -q "true" || >>> CARGOPATH=3D/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=3D1 cargo --offl= ine >>> metadata --format-version 1 --no-deps | jq -e >>> ".packages[].targets[].kind | any(. =3D=3D \"rlib\")" | grep -q "true" || >>> CARGOPATH=3D/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=3D1 cargo --offl= ine >>> metadata --format-version 1 --no-deps | jq -e >>> ".packages[].targets[].kind | any(. =3D=3D \"proc-macro\")" | grep -q >>> "true"; then awk >>> '/^\\\[((.+\\\.)?((dev|build)-)?dependencies|features)/{f=3D1;next} >>> /^\\\[/{f=3D0}; !f' < Cargo.toml > Cargo.toml.deps && >>> CARGOPATH=3D/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=3D1 cargo --offl= ine >>> package -l | grep -wEv "Cargo.(lock|toml.orig)" | xargs -d "\n" cp -v >>> --parents -a -t /usr/share/cargo/registry/cipher-0.3.0 && install -v -m >>> 644 Cargo.toml.deps /usr/share/cargo/registry/cipher-0.3.0/Cargo.toml && >>> echo "{\"files\":{},\"package\":\"\"}" > >>> /usr/share/cargo/registry/cipher-0.3.0/.cargo-checksum.json; fi && if >>> true && CARGOPATH=3D/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=3D1 cargo >>> --offline metadata --format-version 1 --no-deps | jq -e >>> ".packages[].targets[].kind | any(. =3D=3D \"bin\")" | grep -q "true"; th= en >>> CARGOPATH=3D/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=3D1 cargo --offl= ine >>> install -Z avoid-dev-deps -j8 --no-track --path .; fi >>> mkdir: created directory '/usr/share/cargo/registry/cipher-0.3.0' >>> warning: No (git) VCS found for `/usr/src/cipher-0.3.0` >>> error: invalid inclusion of reserved file name Cargo.toml.orig in >>> package source >>> cp: missing file operand >>> Try 'cp --help' for more information. >>> make: *** [rust-cipher:78: /usr/src/log/cipher-0.3.0] Error 123 >>> ***SNAP*** >> Rust is an absolute dependency hell. Ask Adolf and look at his latest patc= hset :) >> >>> Ok, even greater. >>> >>> Does anyone have an idea to solve this? I can't even find an updated >>> package for , e.g., 'cipher-0.3.0tar.gz', although apparently I found at >>> least an updated version (0.4.3) here: >>> >>> =3D> https://docs.rs/cipher/latest/cipher/# >>> >>> But no download links... Hm! Where on earth did 'cipher-0.3.0.tar.gz' >>> came from? >> There is a little helper script in tools/ which you can use to automatical= ly download the source and even generate an LFS file, because they all look t= he same: >> >> https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dblob;f=3Dtools/download-rus= t-crate;h=3Df6a0fe035d30fdbddaa843ccac45251b0049088a;hb=3DHEAD >> >> You can just run this as =E2=80=9Ctools/download-rust-crate cipher=E2=80= =9D and it should create everything you need. Just add it to make.sh and it s= hould build. >> >>> What makes me a bit nervous though is the fact that if clamav really can >>> only be made to work with a major rust update, the other rust components >>> might have to be updated as well. And I found 103 rust*-lfs files... >> Yes. And every time we change one of those packages, we will have to ship = *everything* that is related to Rust. >> >> Such a great language. Stop using Rust, people. >> >> -Michael >> >>> Any thoughts and hints welcome! >>> >>> Best, >>> Matthias --=20 Sent from my laptop --===============6133671041901225108==--