Hi Matthias,
On 22/11/2022 17:11, Matthias Fischer wrote:
On 22.11.2022 16:39, Adolf Belka wrote:
Hi Matthias,
Hi Adolf,
You are experiencing Rust Hell.
Yep. I can feel it... ;-)
Many thanks for your tips. I'll try.
But what makes me wondering: why do I get the error "no matching package named `crypto-common` found" although this package definitely exists!? Why is this package not found at all? I can't get a grip on this.
rust-cipher is now requiring rust-crypto-common but rust-crypto-common is later in the build sequence that rust-crypto so it is not existing when rust-cipher is looking for it.
You need to move rust-crypto-common to just before rust-cipher.
Regards,
Adolf.
Best, Matthias
That is what I have had especially with the python3-cryptography module. Often when it has been updated a new rust module is required which requires another new module etc, etc, etc.
On 21/11/2022 20:05, Matthias Fischer wrote:
On 21.11.2022 11:44, Michael Tremer wrote:
Hello Matthias,
Hi Michael,
updated cipher to '0.4.3'.
Clean build, result:
***SNIP*** ====================================== Installing cipher-0.4.3 ... Install started; saving file list to /usr/src/lsalr ... cd /usr/src/cipher-0.4.3 && mkdir -p /usr/src/cipher-0.4.3/.cargo && echo "${CARGO_CONFIG}" > /usr/src/cipher-0.4.3/.cargo/config && rm -f Cargo.lock cd /usr/src/cipher-0.4.3 && CARGOPATH=/usr/src/cipher-0.4.3/.cargo RUSTC_BOOTSTRAP=1 cargo --offline build --release -Z avoid-dev-deps -j8 error: no matching package named `crypto-common` found location searched: registry `crates-io` required by package `cipher v0.4.3 (/usr/src/cipher-0.4.3)` As a reminder, you're using offline mode (--offline) which can sometimes cause surprising resolution failures, if this error is too confusing you may wish to retry without the offline flag. make: *** [rust-cipher:77: /usr/src/log/cipher-0.4.3] Error 101 ***SNAP***
Hm.
Just guessing => updated 'lfs/rust-crypto-common' from '0.1.1' to '0.1.6' through the helper script.
=> Identical error: "no matching package found".
Hmmm!
To follow the "reminder" would mean to delete the '--offline' option in line 209 in 'lfs'/config', but that would be only further guessing. And this would affect all other files. Doesn't feel good.
You can't do that as the IPFire build is run in a chroot that has no internet access.
I'm not familiar with this rust thing - sorry: any ideas about the best
What I have done so far is just to add each new rust module that has been highlighted and then re-run the build. I have had the situation of 12 additional rust modules being required with an update.
After some discussions with my son in the future I intend to run the build of just a specific package in a directory on my machine, the same as I do when building any package from source on my computer (not in the IPFire build shell). This build would then run with internet access and it should then download and install all required rust modules.
Then after that build (of clamav in your case) has been successful you can look at the log details for that build to see which rust modules have been downloaded and installed and then you can use the script that Michael mentioned to add all of those modules to the build in one go. Warning, I have not tried this approach myself yet but it seemed to make sense to me when my son suggested it. You would need to test it out and see if it helped or not.
You will need to check if any rust modules have been required to have a certain version n umber of range. That can be the case that a rust module needs to be updated but not to the latest version. If that is the case then you can use the rust script to download and install into an lfs a specific version.
It can also be the case that some rust modules or packages require a not latest version and other modules require the same module but at a different version number. In this case you have to relabel the lfs file to include the version number. If you look at the list of rust lfs files you will see that some have a specific version specified as well as the plain named module. For example rust-indoc (version 1.0.3) and rust-indoc-0.3.6 (version 0.3.6).
What would help would be if for every package that used rust modules that there was a dependency file that listed all the ones required. It doesn't exist.
By the way, you also need to watch out that some rust packages will by default install the versions for multiple architectures such as windows or mac etc, which is obviously not needed for IPFire. The ones that I have found like that are rust-chrono and rust-iana-time-zone which then needed patches to comment out the meta-data for building the windows etc versions. If this happens with a package you will find rust modules that have win or some other OS name in the title so that usually flags up to me that I need to go back to an earlier module and stop the win based builds. See rust-chrono-0.4.22-fix-metadata.patch in the /src/patches directory as an example.
Sorry that my input is probably not what you were hoping for. It can be worked though, it just can take some time.
Regards, Adolf.
way to proceed?
On 19 Nov 2022, at 15:56, Matthias Fischer matthias.fischer@ipfire.org wrote:
Hi,
...I'd like to have a small problem... ;-)
A few days ago, 'clamav 0.105.1' was updated, again:
=> https://blog.clamav.net/2022/11/second-clamav-100-release-candidate-and.html
"...[it] was intended to also include bug fixes for the jpeg and tiff Rust-based libraries that are bundled with the source code tarball. Unfortunately, those fixes were not all release-ready in time for the 0.105.1-2 packages."
So far, so [oh, forget it!].
This is *really* bad that they bundle so many libraries and make it very difficult for us to keep track of what vulnerabilities might be in clamav although they are part of a third-party library.
We should try to remove all of them and always build against the system libraries.
Unfortunately, building the third version of 'clamav 0.105.1' with current 'next' failed:
***SNIP*** ... error: package `tiff v0.8.0` cannot be built because it requires rustc 1.61.0 or newer, while the currently active rustc version is 1.60.0-nightly.
[193/379] Building C objectlibclamav/CMakeFiles/lzma_sdk.dir/7z/7zIn.c.o [194/379] Building C object libclamav/CMakeFiles/bytecode_runtime.dir/bytecode_nojit.c.o [195/379] Building C object libclamav/CMakeFiles/yara.dir/yara_grammar.c.o [196/379] Building C object libclamav/CMakeFiles/yara.dir/yara_lexer.c.o yara_lexer.c:2571:24: warning: 'yy_fatal_error' defined but not used [-Wunused-function] yara_lexer.c: In function 'yara_yylex': yara_lexer.l:263:16: warning: '%s' directive output may be truncated writing up to 1023 bytes into a region of size 999 [-Wformat-truncation=] In file included from /usr/include/stdio.h:906, from yara_lexer.c:32: /usr/include/bits/stdio2.h:54:10: note: '__builtin___snprintf_chk' output between 26 and 1049 bytes into a destination of size 1024 54 | return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1, | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 55 | __glibc_objsize (__s), __fmt, | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 56 | __va_arg_pack ()); | ~~~~~~~~~~~~~~~~~ ninja: build stopped: subcommand failed. make: *** [clamav:89: /usr/src/log/clamav-0.105.1] Error 1 ***SNAP***
Great code quality. This is however not the reason why the build stopped. This is only a warning.
Hm. Great.
So I tried the current 'rust 1.65' version.
This time, the building failed because of a rust component:
***SNIP*** ... Finished release [optimized] target(s) in 1.92s cd /usr/src/cipher-0.3.0 && mkdir -pv "/usr/share/cargo/registry/cipher-0.3.0" && if CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "lib")" | grep -q "true" || CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "rlib")" | grep -q "true" || CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "proc-macro")" | grep -q "true"; then awk '/^\[((.+\.)?((dev|build)-)?dependencies|features)/{f=1;next} /^\[/{f=0}; !f' < Cargo.toml > Cargo.toml.deps && CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline package -l | grep -wEv "Cargo.(lock|toml.orig)" | xargs -d "\n" cp -v --parents -a -t /usr/share/cargo/registry/cipher-0.3.0 && install -v -m 644 Cargo.toml.deps /usr/share/cargo/registry/cipher-0.3.0/Cargo.toml && echo "{"files":{},"package":""}" > /usr/share/cargo/registry/cipher-0.3.0/.cargo-checksum.json; fi && if true && CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "bin")" | grep -q "true"; then CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline install -Z avoid-dev-deps -j8 --no-track --path .; fi mkdir: created directory '/usr/share/cargo/registry/cipher-0.3.0' warning: No (git) VCS found for `/usr/src/cipher-0.3.0` error: invalid inclusion of reserved file name Cargo.toml.orig in package source cp: missing file operand Try 'cp --help' for more information. make: *** [rust-cipher:78: /usr/src/log/cipher-0.3.0] Error 123 ***SNAP***
Rust is an absolute dependency hell. Ask Adolf and look at his latest patchset :)
Ok, even greater.
Does anyone have an idea to solve this? I can't even find an updated package for , e.g., 'cipher-0.3.0tar.gz', although apparently I found at least an updated version (0.4.3) here:
=> https://docs.rs/cipher/latest/cipher/#
But no download links... Hm! Where on earth did 'cipher-0.3.0.tar.gz' came from?
There is a little helper script in tools/ which you can use to automatically download the source and even generate an LFS file, because they all look the same:
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=tools/download-rust-crate;...
You can just run this as “tools/download-rust-crate cipher” and it should create everything you need. Just add it to make.sh and it should build.
What makes me a bit nervous though is the fact that if clamav really can only be made to work with a major rust update, the other rust components might have to be updated as well. And I found 103 rust*-lfs files...
Yes. And every time we change one of those packages, we will have to ship *everything* that is related to Rust.
Such a great language. Stop using Rust, people.
-Michael
Any thoughts and hints welcome!
Best, Matthias