On 22.11.2022 16:39, Adolf Belka wrote: > Hi Matthias, Hi Adolf, > You are experiencing Rust Hell. Yep. I can feel it... ;-) Many thanks for your tips. I'll try. But what makes me wondering: why do I get the error "no matching package named `crypto-common` found" although this package definitely exists!? Why is this package not found at all? I can't get a grip on this. Best, Matthias > That is what I have had especially with > the python3-cryptography module. Often when it has been updated a new > rust module is required which requires another new module etc, etc, etc. > > On 21/11/2022 20:05, Matthias Fischer wrote: >> On 21.11.2022 11:44, Michael Tremer wrote: >>> Hello Matthias, >> Hi Michael, >> >> updated cipher to '0.4.3'. >> >> Clean build, result: >> >> ***SNIP*** >> ====================================== Installing cipher-0.4.3 ... >> Install started; saving file list to /usr/src/lsalr ... >> cd /usr/src/cipher-0.4.3 && mkdir -p >> /usr/src/cipher-0.4.3/.cargo && echo "${CARGO_CONFIG}" > >> /usr/src/cipher-0.4.3/.cargo/config && rm -f Cargo.lock >> cd /usr/src/cipher-0.4.3 && CARGOPATH=/usr/src/cipher-0.4.3/.cargo >> RUSTC_BOOTSTRAP=1 cargo --offline build --release -Z avoid-dev-deps -j8 >> error: no matching package named `crypto-common` found >> location searched: registry `crates-io` >> required by package `cipher v0.4.3 (/usr/src/cipher-0.4.3)` >> As a reminder, you're using offline mode (--offline) which can >> sometimes cause surprising resolution failures, if this error is too >> confusing you may wish to retry without the offline flag. >> make: *** [rust-cipher:77: /usr/src/log/cipher-0.4.3] Error 101 >> ***SNAP*** >> >> Hm. >> >> Just guessing => updated 'lfs/rust-crypto-common' from '0.1.1' to >> '0.1.6' through the helper script. >> >> => Identical error: "no matching package found". >> >> Hmmm! >> >> To follow the "reminder" would mean to delete the '--offline' option in >> line 209 in 'lfs'/config', but that would be only further guessing. And >> this would affect all other files. Doesn't feel good. > You can't do that as the IPFire build is run in a chroot that has no > internet access. >> I'm not familiar with this rust thing - sorry: any ideas about the best > What I have done so far is just to add each new rust module that has > been highlighted and then re-run the build. I have had the situation of > 12 additional rust modules being required with an update. > > After some discussions with my son in the future I intend to run the > build of just a specific package in a directory on my machine, the same > as I do when building any package from source on my computer (not in the > IPFire build shell). This build would then run with internet access and > it should then download and install all required rust modules. > > Then after that build (of clamav in your case) has been successful you > can look at the log details for that build to see which rust modules > have been downloaded and installed and then you can use the script that > Michael mentioned to add all of those modules to the build in one go. > Warning, I have not tried this approach myself yet but it seemed to make > sense to me when my son suggested it. You would need to test it out and > see if it helped or not. > > You will need to check if any rust modules have been required to have a > certain version n umber of range. That can be the case that a rust > module needs to be updated but not to the latest version. If that is the > case then you can use the rust script to download and install into an > lfs a specific version. > > It can also be the case that some rust modules or packages require a not > latest version and other modules require the same module but at a > different version number. In this case you have to relabel the lfs file > to include the version  number. If you look at the list of rust lfs > files you will see that some have a specific version specified as well > as the plain named module. > For example rust-indoc (version 1.0.3) and rust-indoc-0.3.6 (version 0.3.6). > > What would help would be if for every package that used rust modules > that there was a dependency file that listed all the ones required. It > doesn't exist. > > By the way, you also need to watch out that some rust packages will by > default install the versions for multiple architectures such as windows > or mac etc, which is obviously not needed for IPFire. The ones that I > have found like that are rust-chrono and rust-iana-time-zone which then > needed patches to comment out the meta-data for building the windows etc > versions. If this happens with a package you will find rust modules that > have win or some other OS name in the title so that usually flags up to > me that I need to go back to an earlier module and stop the win based > builds. > See rust-chrono-0.4.22-fix-metadata.patch in the /src/patches directory > as an example. > > Sorry that my input is probably not what you were hoping for.  It can be > worked though, it just can take some time. > > Regards, > Adolf. >> way to proceed? >> >>>> On 19 Nov 2022, at 15:56, Matthias Fischer wrote: >>>> >>>> Hi, >>>> >>>> ...I'd like to have a small problem... ;-) >>>> >>>> A few days ago, 'clamav 0.105.1' was updated, again: >>>> >>>> => >>>> https://blog.clamav.net/2022/11/second-clamav-100-release-candidate-and.html >>>> >>>> "...[it] was intended to also include bug fixes for the jpeg and tiff >>>> Rust-based libraries that are bundled with the source code tarball. >>>> Unfortunately, those fixes were not all release-ready in time for the >>>> 0.105.1-2 packages." >>>> >>>> So far, so [oh, forget it!]. >>> This is *really* bad that they bundle so many libraries and make it very difficult for us to keep track of what vulnerabilities might be in clamav although they are part of a third-party library. >>> >>> We should try to remove all of them and always build against the system libraries. >>> >>>> Unfortunately, building the third version of 'clamav 0.105.1' with >>>> current 'next' failed: >>>> >>>> ***SNIP*** >>>> ... >>>> error: package `tiff v0.8.0` cannot be built because it requires >>>> rustc 1.61.0 or newer, while the currently active rustc version is >>>> 1.60.0-nightly. >>>> >>>> [193/379] Building C object >>>> libclamav/CMakeFiles/lzma_sdk.dir/7z/7zIn.c.o >>>> [194/379] Building C object >>>> libclamav/CMakeFiles/bytecode_runtime.dir/bytecode_nojit.c.o >>>> [195/379] Building C object >>>> libclamav/CMakeFiles/yara.dir/yara_grammar.c.o >>>> [196/379] Building C object libclamav/CMakeFiles/yara.dir/yara_lexer.c.o >>>> yara_lexer.c:2571:24: warning: 'yy_fatal_error' defined but not used >>>> [-Wunused-function] >>>> yara_lexer.c: In function 'yara_yylex': >>>> yara_lexer.l:263:16: warning: '%s' directive output may be truncated >>>> writing up to 1023 bytes into a region of size 999 [-Wformat-truncation=] >>>> In file included from /usr/include/stdio.h:906, >>>> from yara_lexer.c:32: >>>> /usr/include/bits/stdio2.h:54:10: note: '__builtin___snprintf_chk' >>>> output between 26 and 1049 bytes into a destination of size 1024 >>>> 54 | return __builtin___snprintf_chk (__s, __n, >>>> __USE_FORTIFY_LEVEL - 1, >>>> | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>> 55 | __glibc_objsize (__s), __fmt, >>>> | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>> 56 | __va_arg_pack ()); >>>> | ~~~~~~~~~~~~~~~~~ >>>> ninja: build stopped: subcommand failed. >>>> make: *** [clamav:89: /usr/src/log/clamav-0.105.1] Error 1 >>>> ***SNAP*** >>> Great code quality. This is however not the reason why the build stopped. This is only a warning. >>> >>>> Hm. Great. >>>> >>>> So I tried the current 'rust 1.65' version. >>>> >>>> This time, the building failed because of a rust component: >>>> >>>> ***SNIP*** >>>> ... >>>> Finished release [optimized] target(s) in 1.92s >>>> cd /usr/src/cipher-0.3.0 && mkdir -pv >>>> "/usr/share/cargo/registry/cipher-0.3.0" && if >>>> CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline >>>> metadata --format-version 1 --no-deps | jq -e >>>> ".packages[].targets[].kind | any(. == \"lib\")" | grep -q "true" || >>>> CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline >>>> metadata --format-version 1 --no-deps | jq -e >>>> ".packages[].targets[].kind | any(. == \"rlib\")" | grep -q "true" || >>>> CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline >>>> metadata --format-version 1 --no-deps | jq -e >>>> ".packages[].targets[].kind | any(. == \"proc-macro\")" | grep -q >>>> "true"; then awk >>>> '/^\\\[((.+\\\.)?((dev|build)-)?dependencies|features)/{f=1;next} >>>> /^\\\[/{f=0}; !f' < Cargo.toml > Cargo.toml.deps && >>>> CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline >>>> package -l | grep -wEv "Cargo.(lock|toml.orig)" | xargs -d "\n" cp -v >>>> --parents -a -t /usr/share/cargo/registry/cipher-0.3.0 && install -v -m >>>> 644 Cargo.toml.deps /usr/share/cargo/registry/cipher-0.3.0/Cargo.toml && >>>> echo "{\"files\":{},\"package\":\"\"}" > >>>> /usr/share/cargo/registry/cipher-0.3.0/.cargo-checksum.json; fi && if >>>> true && CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo >>>> --offline metadata --format-version 1 --no-deps | jq -e >>>> ".packages[].targets[].kind | any(. == \"bin\")" | grep -q "true"; then >>>> CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline >>>> install -Z avoid-dev-deps -j8 --no-track --path .; fi >>>> mkdir: created directory '/usr/share/cargo/registry/cipher-0.3.0' >>>> warning: No (git) VCS found for `/usr/src/cipher-0.3.0` >>>> error: invalid inclusion of reserved file name Cargo.toml.orig in >>>> package source >>>> cp: missing file operand >>>> Try 'cp --help' for more information. >>>> make: *** [rust-cipher:78: /usr/src/log/cipher-0.3.0] Error 123 >>>> ***SNAP*** >>> Rust is an absolute dependency hell. Ask Adolf and look at his latest patchset :) >>> >>>> Ok, even greater. >>>> >>>> Does anyone have an idea to solve this? I can't even find an updated >>>> package for , e.g., 'cipher-0.3.0tar.gz', although apparently I found at >>>> least an updated version (0.4.3) here: >>>> >>>> => https://docs.rs/cipher/latest/cipher/# >>>> >>>> But no download links... Hm! Where on earth did 'cipher-0.3.0.tar.gz' >>>> came from? >>> There is a little helper script in tools/ which you can use to automatically download the source and even generate an LFS file, because they all look the same: >>> >>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=tools/download-rust-crate;h=f6a0fe035d30fdbddaa843ccac45251b0049088a;hb=HEAD >>> >>> You can just run this as “tools/download-rust-crate cipher” and it should create everything you need. Just add it to make.sh and it should build. >>> >>>> What makes me a bit nervous though is the fact that if clamav really can >>>> only be made to work with a major rust update, the other rust components >>>> might have to be updated as well. And I found 103 rust*-lfs files... >>> Yes. And every time we change one of those packages, we will have to ship *everything* that is related to Rust. >>> >>> Such a great language. Stop using Rust, people. >>> >>> -Michael >>> >>>> Any thoughts and hints welcome! >>>> >>>> Best, >>>> Matthias >