Hi,
...I'd like to have a small problem... ;-)
A few days ago, 'clamav 0.105.1' was updated, again:
=> https://blog.clamav.net/2022/11/second-clamav-100-release-candidate-and.html
"...[it] was intended to also include bug fixes for the jpeg and tiff Rust-based libraries that are bundled with the source code tarball. Unfortunately, those fixes were not all release-ready in time for the 0.105.1-2 packages."
So far, so [oh, forget it!].
Unfortunately, building the third version of 'clamav 0.105.1' with current 'next' failed:
***SNIP*** ... error: package `tiff v0.8.0` cannot be built because it requires rustc 1.61.0 or newer, while the currently active rustc version is 1.60.0-nightly.
[193/379] Building C object libclamav/CMakeFiles/lzma_sdk.dir/7z/7zIn.c.o [194/379] Building C object libclamav/CMakeFiles/bytecode_runtime.dir/bytecode_nojit.c.o [195/379] Building C object libclamav/CMakeFiles/yara.dir/yara_grammar.c.o [196/379] Building C object libclamav/CMakeFiles/yara.dir/yara_lexer.c.o yara_lexer.c:2571:24: warning: 'yy_fatal_error' defined but not used [-Wunused-function] yara_lexer.c: In function 'yara_yylex': yara_lexer.l:263:16: warning: '%s' directive output may be truncated writing up to 1023 bytes into a region of size 999 [-Wformat-truncation=] In file included from /usr/include/stdio.h:906, from yara_lexer.c:32: /usr/include/bits/stdio2.h:54:10: note: '__builtin___snprintf_chk' output between 26 and 1049 bytes into a destination of size 1024 54 | return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1, | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 55 | __glibc_objsize (__s), __fmt, | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 56 | __va_arg_pack ()); | ~~~~~~~~~~~~~~~~~ ninja: build stopped: subcommand failed. make: *** [clamav:89: /usr/src/log/clamav-0.105.1] Error 1 ***SNAP***
Hm. Great.
So I tried the current 'rust 1.65' version.
This time, the building failed because of a rust component:
***SNIP*** ... Finished release [optimized] target(s) in 1.92s cd /usr/src/cipher-0.3.0 && mkdir -pv "/usr/share/cargo/registry/cipher-0.3.0" && if CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "lib")" | grep -q "true" || CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "rlib")" | grep -q "true" || CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "proc-macro")" | grep -q "true"; then awk '/^\[((.+\.)?((dev|build)-)?dependencies|features)/{f=1;next} /^\[/{f=0}; !f' < Cargo.toml > Cargo.toml.deps && CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline package -l | grep -wEv "Cargo.(lock|toml.orig)" | xargs -d "\n" cp -v --parents -a -t /usr/share/cargo/registry/cipher-0.3.0 && install -v -m 644 Cargo.toml.deps /usr/share/cargo/registry/cipher-0.3.0/Cargo.toml && echo "{"files":{},"package":""}" > /usr/share/cargo/registry/cipher-0.3.0/.cargo-checksum.json; fi && if true && CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "bin")" | grep -q "true"; then CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline install -Z avoid-dev-deps -j8 --no-track --path .; fi mkdir: created directory '/usr/share/cargo/registry/cipher-0.3.0' warning: No (git) VCS found for `/usr/src/cipher-0.3.0` error: invalid inclusion of reserved file name Cargo.toml.orig in package source cp: missing file operand Try 'cp --help' for more information. make: *** [rust-cipher:78: /usr/src/log/cipher-0.3.0] Error 123 ***SNAP***
Ok, even greater.
Does anyone have an idea to solve this? I can't even find an updated package for , e.g., 'cipher-0.3.0tar.gz', although apparently I found at least an updated version (0.4.3) here:
=> https://docs.rs/cipher/latest/cipher/#
But no download links... Hm! Where on earth did 'cipher-0.3.0.tar.gz' came from?
What makes me a bit nervous though is the fact that if clamav really can only be made to work with a major rust update, the other rust components might have to be updated as well. And I found 103 rust*-lfs files...
Any thoughts and hints welcome!
Best, Matthias
Hello Matthias,
On 19 Nov 2022, at 15:56, Matthias Fischer matthias.fischer@ipfire.org wrote:
Hi,
...I'd like to have a small problem... ;-)
A few days ago, 'clamav 0.105.1' was updated, again:
=> https://blog.clamav.net/2022/11/second-clamav-100-release-candidate-and.html
"...[it] was intended to also include bug fixes for the jpeg and tiff Rust-based libraries that are bundled with the source code tarball. Unfortunately, those fixes were not all release-ready in time for the 0.105.1-2 packages."
So far, so [oh, forget it!].
This is *really* bad that they bundle so many libraries and make it very difficult for us to keep track of what vulnerabilities might be in clamav although they are part of a third-party library.
We should try to remove all of them and always build against the system libraries.
Unfortunately, building the third version of 'clamav 0.105.1' with current 'next' failed:
***SNIP*** ... error: package `tiff v0.8.0` cannot be built because it requires rustc 1.61.0 or newer, while the currently active rustc version is 1.60.0-nightly.
[193/379] Building C object libclamav/CMakeFiles/lzma_sdk.dir/7z/7zIn.c.o [194/379] Building C object libclamav/CMakeFiles/bytecode_runtime.dir/bytecode_nojit.c.o [195/379] Building C object libclamav/CMakeFiles/yara.dir/yara_grammar.c.o [196/379] Building C object libclamav/CMakeFiles/yara.dir/yara_lexer.c.o yara_lexer.c:2571:24: warning: 'yy_fatal_error' defined but not used [-Wunused-function] yara_lexer.c: In function 'yara_yylex': yara_lexer.l:263:16: warning: '%s' directive output may be truncated writing up to 1023 bytes into a region of size 999 [-Wformat-truncation=] In file included from /usr/include/stdio.h:906, from yara_lexer.c:32: /usr/include/bits/stdio2.h:54:10: note: '__builtin___snprintf_chk' output between 26 and 1049 bytes into a destination of size 1024 54 | return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1, | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 55 | __glibc_objsize (__s), __fmt, | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 56 | __va_arg_pack ()); | ~~~~~~~~~~~~~~~~~ ninja: build stopped: subcommand failed. make: *** [clamav:89: /usr/src/log/clamav-0.105.1] Error 1 ***SNAP***
Great code quality. This is however not the reason why the build stopped. This is only a warning.
Hm. Great.
So I tried the current 'rust 1.65' version.
This time, the building failed because of a rust component:
***SNIP*** ... Finished release [optimized] target(s) in 1.92s cd /usr/src/cipher-0.3.0 && mkdir -pv "/usr/share/cargo/registry/cipher-0.3.0" && if CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "lib")" | grep -q "true" || CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "rlib")" | grep -q "true" || CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "proc-macro")" | grep -q "true"; then awk '/^\[((.+\.)?((dev|build)-)?dependencies|features)/{f=1;next} /^\[/{f=0}; !f' < Cargo.toml > Cargo.toml.deps && CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline package -l | grep -wEv "Cargo.(lock|toml.orig)" | xargs -d "\n" cp -v --parents -a -t /usr/share/cargo/registry/cipher-0.3.0 && install -v -m 644 Cargo.toml.deps /usr/share/cargo/registry/cipher-0.3.0/Cargo.toml && echo "{"files":{},"package":""}" > /usr/share/cargo/registry/cipher-0.3.0/.cargo-checksum.json; fi && if true && CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "bin")" | grep -q "true"; then CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline install -Z avoid-dev-deps -j8 --no-track --path .; fi mkdir: created directory '/usr/share/cargo/registry/cipher-0.3.0' warning: No (git) VCS found for `/usr/src/cipher-0.3.0` error: invalid inclusion of reserved file name Cargo.toml.orig in package source cp: missing file operand Try 'cp --help' for more information. make: *** [rust-cipher:78: /usr/src/log/cipher-0.3.0] Error 123 ***SNAP***
Rust is an absolute dependency hell. Ask Adolf and look at his latest patchset :)
Ok, even greater.
Does anyone have an idea to solve this? I can't even find an updated package for , e.g., 'cipher-0.3.0tar.gz', although apparently I found at least an updated version (0.4.3) here:
=> https://docs.rs/cipher/latest/cipher/#
But no download links... Hm! Where on earth did 'cipher-0.3.0.tar.gz' came from?
There is a little helper script in tools/ which you can use to automatically download the source and even generate an LFS file, because they all look the same:
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=tools/download-rust-crate;...
You can just run this as “tools/download-rust-crate cipher” and it should create everything you need. Just add it to make.sh and it should build.
What makes me a bit nervous though is the fact that if clamav really can only be made to work with a major rust update, the other rust components might have to be updated as well. And I found 103 rust*-lfs files...
Yes. And every time we change one of those packages, we will have to ship *everything* that is related to Rust.
Such a great language. Stop using Rust, people.
-Michael
Any thoughts and hints welcome!
Best, Matthias
On 21.11.2022 11:44, Michael Tremer wrote:
Hello Matthias,
Hi Michael,
please see comments below...
On 19 Nov 2022, at 15:56, Matthias Fischer matthias.fischer@ipfire.org wrote:
Hi,
...I'd like to have a small problem... ;-)
A few days ago, 'clamav 0.105.1' was updated, again: ...
This is *really* bad that they bundle so many libraries and make it very difficult for us to keep track of what vulnerabilities might be in clamav although they are part of a third-party library.
Yep.
We should try to remove all of them and always build against the system libraries.
Puh. Sounds difficult. For now, I'll be happy if I get 'clamav' and 'rust' building at all.
Unfortunately, building the third version of 'clamav 0.105.1' with current 'next' failed: .... ***SNIP*** ... error: package `tiff v0.8.0` cannot be built because it requires rustc 1.61.0 or newer, while the currently active rustc version is 1.60.0-nightly. ... ninja: build stopped: subcommand failed. make: *** [clamav:89: /usr/src/log/clamav-0.105.1] Error 1 ***SNAP***
Great code quality. This is however not the reason why the build stopped. This is only a warning.
Hm. Great.
So I tried the current 'rust 1.65' version.
This time, the building failed because of a rust component:
***SNIP*** ... Finished release [optimized] target(s) in 1.92s cd /usr/src/cipher-0.3.0 && mkdir -pv ... install -Z avoid-dev-deps -j8 --no-track --path .; fi mkdir: created directory '/usr/share/cargo/registry/cipher-0.3.0' warning: No (git) VCS found for `/usr/src/cipher-0.3.0` error: invalid inclusion of reserved file name Cargo.toml.orig in package source cp: missing file operand Try 'cp --help' for more information. make: *** [rust-cipher:78: /usr/src/log/cipher-0.3.0] Error 123 ***SNAP***
Rust is an absolute dependency hell. Ask Adolf and look at his latest patchset :)
Yes. I saw that. Too much for me...
Ok, even greater.
Does anyone have an idea to solve this? I can't even find an updated package for , e.g., 'cipher-0.3.0tar.gz', although apparently I found at least an updated version (0.4.3) here:
=> https://docs.rs/cipher/latest/cipher/#
But no download links... Hm! Where on earth did 'cipher-0.3.0.tar.gz' came from?
There is a little helper script in tools/ which you can use to automatically download the source and even generate an LFS file, because they all look the same:
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=tools/download-rust-crate;...
I didn't saw this one. Thanks!
You can just run this as “tools/download-rust-crate cipher” and it should create everything you need. Just add it to make.sh and it should build.
The funny part: I hadn't 'jq' on my Devel - never heard of it or needed it until now - but I got the build running now. After an 'apt install jq' everything seems to be ok. ;-)
Devel is running, I looking forward how far I will get. I'm curious what 'suricata' thinks of 'rust 1.65'...
What makes me a bit nervous though is the fact that if clamav really can only be made to work with a major rust update, the other rust components might have to be updated as well. And I found 103 rust*-lfs files...
Yes. And every time we change one of those packages, we will have to ship *everything* that is related to Rust.
Should I check the other rust-* packages (the remaining 102...) for possible updates?
Best, Matthias
Such a great language. Stop using Rust, people.
-Michael
Any thoughts and hints welcome!
Best, Matthias
On 21.11.2022 11:44, Michael Tremer wrote:
Hello Matthias,
Hi Michael,
updated cipher to '0.4.3'.
Clean build, result:
***SNIP*** ====================================== Installing cipher-0.4.3 ... Install started; saving file list to /usr/src/lsalr ... cd /usr/src/cipher-0.4.3 && mkdir -p /usr/src/cipher-0.4.3/.cargo && echo "${CARGO_CONFIG}" > /usr/src/cipher-0.4.3/.cargo/config && rm -f Cargo.lock cd /usr/src/cipher-0.4.3 && CARGOPATH=/usr/src/cipher-0.4.3/.cargo RUSTC_BOOTSTRAP=1 cargo --offline build --release -Z avoid-dev-deps -j8 error: no matching package named `crypto-common` found location searched: registry `crates-io` required by package `cipher v0.4.3 (/usr/src/cipher-0.4.3)` As a reminder, you're using offline mode (--offline) which can sometimes cause surprising resolution failures, if this error is too confusing you may wish to retry without the offline flag. make: *** [rust-cipher:77: /usr/src/log/cipher-0.4.3] Error 101 ***SNAP***
Hm.
Just guessing => updated 'lfs/rust-crypto-common' from '0.1.1' to '0.1.6' through the helper script.
=> Identical error: "no matching package found".
Hmmm!
To follow the "reminder" would mean to delete the '--offline' option in line 209 in 'lfs'/config', but that would be only further guessing. And this would affect all other files. Doesn't feel good.
I'm not familiar with this rust thing - sorry: any ideas about the best way to proceed?
On 19 Nov 2022, at 15:56, Matthias Fischer matthias.fischer@ipfire.org wrote:
Hi,
...I'd like to have a small problem... ;-)
A few days ago, 'clamav 0.105.1' was updated, again:
=> https://blog.clamav.net/2022/11/second-clamav-100-release-candidate-and.html
"...[it] was intended to also include bug fixes for the jpeg and tiff Rust-based libraries that are bundled with the source code tarball. Unfortunately, those fixes were not all release-ready in time for the 0.105.1-2 packages."
So far, so [oh, forget it!].
This is *really* bad that they bundle so many libraries and make it very difficult for us to keep track of what vulnerabilities might be in clamav although they are part of a third-party library.
We should try to remove all of them and always build against the system libraries.
Unfortunately, building the third version of 'clamav 0.105.1' with current 'next' failed:
***SNIP*** ... error: package `tiff v0.8.0` cannot be built because it requires rustc 1.61.0 or newer, while the currently active rustc version is 1.60.0-nightly.
[193/379] Building C object libclamav/CMakeFiles/lzma_sdk.dir/7z/7zIn.c.o [194/379] Building C object libclamav/CMakeFiles/bytecode_runtime.dir/bytecode_nojit.c.o [195/379] Building C object libclamav/CMakeFiles/yara.dir/yara_grammar.c.o [196/379] Building C object libclamav/CMakeFiles/yara.dir/yara_lexer.c.o yara_lexer.c:2571:24: warning: 'yy_fatal_error' defined but not used [-Wunused-function] yara_lexer.c: In function 'yara_yylex': yara_lexer.l:263:16: warning: '%s' directive output may be truncated writing up to 1023 bytes into a region of size 999 [-Wformat-truncation=] In file included from /usr/include/stdio.h:906, from yara_lexer.c:32: /usr/include/bits/stdio2.h:54:10: note: '__builtin___snprintf_chk' output between 26 and 1049 bytes into a destination of size 1024 54 | return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1, | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 55 | __glibc_objsize (__s), __fmt, | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 56 | __va_arg_pack ()); | ~~~~~~~~~~~~~~~~~ ninja: build stopped: subcommand failed. make: *** [clamav:89: /usr/src/log/clamav-0.105.1] Error 1 ***SNAP***
Great code quality. This is however not the reason why the build stopped. This is only a warning.
Hm. Great.
So I tried the current 'rust 1.65' version.
This time, the building failed because of a rust component:
***SNIP*** ... Finished release [optimized] target(s) in 1.92s cd /usr/src/cipher-0.3.0 && mkdir -pv "/usr/share/cargo/registry/cipher-0.3.0" && if CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "lib")" | grep -q "true" || CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "rlib")" | grep -q "true" || CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "proc-macro")" | grep -q "true"; then awk '/^\[((.+\.)?((dev|build)-)?dependencies|features)/{f=1;next} /^\[/{f=0}; !f' < Cargo.toml > Cargo.toml.deps && CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline package -l | grep -wEv "Cargo.(lock|toml.orig)" | xargs -d "\n" cp -v --parents -a -t /usr/share/cargo/registry/cipher-0.3.0 && install -v -m 644 Cargo.toml.deps /usr/share/cargo/registry/cipher-0.3.0/Cargo.toml && echo "{"files":{},"package":""}" > /usr/share/cargo/registry/cipher-0.3.0/.cargo-checksum.json; fi && if true && CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "bin")" | grep -q "true"; then CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline install -Z avoid-dev-deps -j8 --no-track --path .; fi mkdir: created directory '/usr/share/cargo/registry/cipher-0.3.0' warning: No (git) VCS found for `/usr/src/cipher-0.3.0` error: invalid inclusion of reserved file name Cargo.toml.orig in package source cp: missing file operand Try 'cp --help' for more information. make: *** [rust-cipher:78: /usr/src/log/cipher-0.3.0] Error 123 ***SNAP***
Rust is an absolute dependency hell. Ask Adolf and look at his latest patchset :)
Ok, even greater.
Does anyone have an idea to solve this? I can't even find an updated package for , e.g., 'cipher-0.3.0tar.gz', although apparently I found at least an updated version (0.4.3) here:
=> https://docs.rs/cipher/latest/cipher/#
But no download links... Hm! Where on earth did 'cipher-0.3.0.tar.gz' came from?
There is a little helper script in tools/ which you can use to automatically download the source and even generate an LFS file, because they all look the same:
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=tools/download-rust-crate;...
You can just run this as “tools/download-rust-crate cipher” and it should create everything you need. Just add it to make.sh and it should build.
What makes me a bit nervous though is the fact that if clamav really can only be made to work with a major rust update, the other rust components might have to be updated as well. And I found 103 rust*-lfs files...
Yes. And every time we change one of those packages, we will have to ship *everything* that is related to Rust.
Such a great language. Stop using Rust, people.
-Michael
Any thoughts and hints welcome!
Best, Matthias
Hi Matthias,
You are experiencing Rust Hell. That is what I have had especially with the python3-cryptography module. Often when it has been updated a new rust module is required which requires another new module etc, etc, etc.
On 21/11/2022 20:05, Matthias Fischer wrote:
On 21.11.2022 11:44, Michael Tremer wrote:
Hello Matthias,
Hi Michael,
updated cipher to '0.4.3'.
Clean build, result:
***SNIP*** ====================================== Installing cipher-0.4.3 ... Install started; saving file list to /usr/src/lsalr ... cd /usr/src/cipher-0.4.3 && mkdir -p /usr/src/cipher-0.4.3/.cargo && echo "${CARGO_CONFIG}" > /usr/src/cipher-0.4.3/.cargo/config && rm -f Cargo.lock cd /usr/src/cipher-0.4.3 && CARGOPATH=/usr/src/cipher-0.4.3/.cargo RUSTC_BOOTSTRAP=1 cargo --offline build --release -Z avoid-dev-deps -j8 error: no matching package named `crypto-common` found location searched: registry `crates-io` required by package `cipher v0.4.3 (/usr/src/cipher-0.4.3)` As a reminder, you're using offline mode (--offline) which can sometimes cause surprising resolution failures, if this error is too confusing you may wish to retry without the offline flag. make: *** [rust-cipher:77: /usr/src/log/cipher-0.4.3] Error 101 ***SNAP***
Hm.
Just guessing => updated 'lfs/rust-crypto-common' from '0.1.1' to '0.1.6' through the helper script.
=> Identical error: "no matching package found".
Hmmm!
To follow the "reminder" would mean to delete the '--offline' option in line 209 in 'lfs'/config', but that would be only further guessing. And this would affect all other files. Doesn't feel good.
You can't do that as the IPFire build is run in a chroot that has no internet access.
I'm not familiar with this rust thing - sorry: any ideas about the best
What I have done so far is just to add each new rust module that has been highlighted and then re-run the build. I have had the situation of 12 additional rust modules being required with an update.
After some discussions with my son in the future I intend to run the build of just a specific package in a directory on my machine, the same as I do when building any package from source on my computer (not in the IPFire build shell). This build would then run with internet access and it should then download and install all required rust modules.
Then after that build (of clamav in your case) has been successful you can look at the log details for that build to see which rust modules have been downloaded and installed and then you can use the script that Michael mentioned to add all of those modules to the build in one go. Warning, I have not tried this approach myself yet but it seemed to make sense to me when my son suggested it. You would need to test it out and see if it helped or not.
You will need to check if any rust modules have been required to have a certain version n umber of range. That can be the case that a rust module needs to be updated but not to the latest version. If that is the case then you can use the rust script to download and install into an lfs a specific version.
It can also be the case that some rust modules or packages require a not latest version and other modules require the same module but at a different version number. In this case you have to relabel the lfs file to include the version number. If you look at the list of rust lfs files you will see that some have a specific version specified as well as the plain named module. For example rust-indoc (version 1.0.3) and rust-indoc-0.3.6 (version 0.3.6).
What would help would be if for every package that used rust modules that there was a dependency file that listed all the ones required. It doesn't exist.
By the way, you also need to watch out that some rust packages will by default install the versions for multiple architectures such as windows or mac etc, which is obviously not needed for IPFire. The ones that I have found like that are rust-chrono and rust-iana-time-zone which then needed patches to comment out the meta-data for building the windows etc versions. If this happens with a package you will find rust modules that have win or some other OS name in the title so that usually flags up to me that I need to go back to an earlier module and stop the win based builds. See rust-chrono-0.4.22-fix-metadata.patch in the /src/patches directory as an example.
Sorry that my input is probably not what you were hoping for. It can be worked though, it just can take some time.
Regards, Adolf.
way to proceed?
On 19 Nov 2022, at 15:56, Matthias Fischer matthias.fischer@ipfire.org wrote:
Hi,
...I'd like to have a small problem... ;-)
A few days ago, 'clamav 0.105.1' was updated, again:
=> https://blog.clamav.net/2022/11/second-clamav-100-release-candidate-and.html
"...[it] was intended to also include bug fixes for the jpeg and tiff Rust-based libraries that are bundled with the source code tarball. Unfortunately, those fixes were not all release-ready in time for the 0.105.1-2 packages."
So far, so [oh, forget it!].
This is *really* bad that they bundle so many libraries and make it very difficult for us to keep track of what vulnerabilities might be in clamav although they are part of a third-party library.
We should try to remove all of them and always build against the system libraries.
Unfortunately, building the third version of 'clamav 0.105.1' with current 'next' failed:
***SNIP*** ... error: package `tiff v0.8.0` cannot be built because it requires rustc 1.61.0 or newer, while the currently active rustc version is 1.60.0-nightly.
[193/379] Building C objectlibclamav/CMakeFiles/lzma_sdk.dir/7z/7zIn.c.o [194/379] Building C object libclamav/CMakeFiles/bytecode_runtime.dir/bytecode_nojit.c.o [195/379] Building C object libclamav/CMakeFiles/yara.dir/yara_grammar.c.o [196/379] Building C object libclamav/CMakeFiles/yara.dir/yara_lexer.c.o yara_lexer.c:2571:24: warning: 'yy_fatal_error' defined but not used [-Wunused-function] yara_lexer.c: In function 'yara_yylex': yara_lexer.l:263:16: warning: '%s' directive output may be truncated writing up to 1023 bytes into a region of size 999 [-Wformat-truncation=] In file included from /usr/include/stdio.h:906, from yara_lexer.c:32: /usr/include/bits/stdio2.h:54:10: note: '__builtin___snprintf_chk' output between 26 and 1049 bytes into a destination of size 1024 54 | return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1, | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 55 | __glibc_objsize (__s), __fmt, | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 56 | __va_arg_pack ()); | ~~~~~~~~~~~~~~~~~ ninja: build stopped: subcommand failed. make: *** [clamav:89: /usr/src/log/clamav-0.105.1] Error 1 ***SNAP***
Great code quality. This is however not the reason why the build stopped. This is only a warning.
Hm. Great.
So I tried the current 'rust 1.65' version.
This time, the building failed because of a rust component:
***SNIP*** ... Finished release [optimized] target(s) in 1.92s cd /usr/src/cipher-0.3.0 && mkdir -pv "/usr/share/cargo/registry/cipher-0.3.0" && if CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "lib")" | grep -q "true" || CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "rlib")" | grep -q "true" || CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "proc-macro")" | grep -q "true"; then awk '/^\[((.+\.)?((dev|build)-)?dependencies|features)/{f=1;next} /^\[/{f=0}; !f' < Cargo.toml > Cargo.toml.deps && CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline package -l | grep -wEv "Cargo.(lock|toml.orig)" | xargs -d "\n" cp -v --parents -a -t /usr/share/cargo/registry/cipher-0.3.0 && install -v -m 644 Cargo.toml.deps /usr/share/cargo/registry/cipher-0.3.0/Cargo.toml && echo "{"files":{},"package":""}" > /usr/share/cargo/registry/cipher-0.3.0/.cargo-checksum.json; fi && if true && CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "bin")" | grep -q "true"; then CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline install -Z avoid-dev-deps -j8 --no-track --path .; fi mkdir: created directory '/usr/share/cargo/registry/cipher-0.3.0' warning: No (git) VCS found for `/usr/src/cipher-0.3.0` error: invalid inclusion of reserved file name Cargo.toml.orig in package source cp: missing file operand Try 'cp --help' for more information. make: *** [rust-cipher:78: /usr/src/log/cipher-0.3.0] Error 123 ***SNAP***
Rust is an absolute dependency hell. Ask Adolf and look at his latest patchset :)
Ok, even greater.
Does anyone have an idea to solve this? I can't even find an updated package for , e.g., 'cipher-0.3.0tar.gz', although apparently I found at least an updated version (0.4.3) here:
=> https://docs.rs/cipher/latest/cipher/#
But no download links... Hm! Where on earth did 'cipher-0.3.0.tar.gz' came from?
There is a little helper script in tools/ which you can use to automatically download the source and even generate an LFS file, because they all look the same:
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=tools/download-rust-crate;...
You can just run this as “tools/download-rust-crate cipher” and it should create everything you need. Just add it to make.sh and it should build.
What makes me a bit nervous though is the fact that if clamav really can only be made to work with a major rust update, the other rust components might have to be updated as well. And I found 103 rust*-lfs files...
Yes. And every time we change one of those packages, we will have to ship *everything* that is related to Rust.
Such a great language. Stop using Rust, people.
-Michael
Any thoughts and hints welcome!
Best, Matthias
On 22.11.2022 16:39, Adolf Belka wrote:
Hi Matthias,
Hi Adolf,
You are experiencing Rust Hell.
Yep. I can feel it... ;-)
Many thanks for your tips. I'll try.
But what makes me wondering: why do I get the error "no matching package named `crypto-common` found" although this package definitely exists!? Why is this package not found at all? I can't get a grip on this.
Best, Matthias
That is what I have had especially with the python3-cryptography module. Often when it has been updated a new rust module is required which requires another new module etc, etc, etc.
On 21/11/2022 20:05, Matthias Fischer wrote:
On 21.11.2022 11:44, Michael Tremer wrote:
Hello Matthias,
Hi Michael,
updated cipher to '0.4.3'.
Clean build, result:
***SNIP*** ====================================== Installing cipher-0.4.3 ... Install started; saving file list to /usr/src/lsalr ... cd /usr/src/cipher-0.4.3 && mkdir -p /usr/src/cipher-0.4.3/.cargo && echo "${CARGO_CONFIG}" > /usr/src/cipher-0.4.3/.cargo/config && rm -f Cargo.lock cd /usr/src/cipher-0.4.3 && CARGOPATH=/usr/src/cipher-0.4.3/.cargo RUSTC_BOOTSTRAP=1 cargo --offline build --release -Z avoid-dev-deps -j8 error: no matching package named `crypto-common` found location searched: registry `crates-io` required by package `cipher v0.4.3 (/usr/src/cipher-0.4.3)` As a reminder, you're using offline mode (--offline) which can sometimes cause surprising resolution failures, if this error is too confusing you may wish to retry without the offline flag. make: *** [rust-cipher:77: /usr/src/log/cipher-0.4.3] Error 101 ***SNAP***
Hm.
Just guessing => updated 'lfs/rust-crypto-common' from '0.1.1' to '0.1.6' through the helper script.
=> Identical error: "no matching package found".
Hmmm!
To follow the "reminder" would mean to delete the '--offline' option in line 209 in 'lfs'/config', but that would be only further guessing. And this would affect all other files. Doesn't feel good.
You can't do that as the IPFire build is run in a chroot that has no internet access.
I'm not familiar with this rust thing - sorry: any ideas about the best
What I have done so far is just to add each new rust module that has been highlighted and then re-run the build. I have had the situation of 12 additional rust modules being required with an update.
After some discussions with my son in the future I intend to run the build of just a specific package in a directory on my machine, the same as I do when building any package from source on my computer (not in the IPFire build shell). This build would then run with internet access and it should then download and install all required rust modules.
Then after that build (of clamav in your case) has been successful you can look at the log details for that build to see which rust modules have been downloaded and installed and then you can use the script that Michael mentioned to add all of those modules to the build in one go. Warning, I have not tried this approach myself yet but it seemed to make sense to me when my son suggested it. You would need to test it out and see if it helped or not.
You will need to check if any rust modules have been required to have a certain version n umber of range. That can be the case that a rust module needs to be updated but not to the latest version. If that is the case then you can use the rust script to download and install into an lfs a specific version.
It can also be the case that some rust modules or packages require a not latest version and other modules require the same module but at a different version number. In this case you have to relabel the lfs file to include the version number. If you look at the list of rust lfs files you will see that some have a specific version specified as well as the plain named module. For example rust-indoc (version 1.0.3) and rust-indoc-0.3.6 (version 0.3.6).
What would help would be if for every package that used rust modules that there was a dependency file that listed all the ones required. It doesn't exist.
By the way, you also need to watch out that some rust packages will by default install the versions for multiple architectures such as windows or mac etc, which is obviously not needed for IPFire. The ones that I have found like that are rust-chrono and rust-iana-time-zone which then needed patches to comment out the meta-data for building the windows etc versions. If this happens with a package you will find rust modules that have win or some other OS name in the title so that usually flags up to me that I need to go back to an earlier module and stop the win based builds. See rust-chrono-0.4.22-fix-metadata.patch in the /src/patches directory as an example.
Sorry that my input is probably not what you were hoping for. It can be worked though, it just can take some time.
Regards, Adolf.
way to proceed?
On 19 Nov 2022, at 15:56, Matthias Fischer matthias.fischer@ipfire.org wrote:
Hi,
...I'd like to have a small problem... ;-)
A few days ago, 'clamav 0.105.1' was updated, again:
=> https://blog.clamav.net/2022/11/second-clamav-100-release-candidate-and.html
"...[it] was intended to also include bug fixes for the jpeg and tiff Rust-based libraries that are bundled with the source code tarball. Unfortunately, those fixes were not all release-ready in time for the 0.105.1-2 packages."
So far, so [oh, forget it!].
This is *really* bad that they bundle so many libraries and make it very difficult for us to keep track of what vulnerabilities might be in clamav although they are part of a third-party library.
We should try to remove all of them and always build against the system libraries.
Unfortunately, building the third version of 'clamav 0.105.1' with current 'next' failed:
***SNIP*** ... error: package `tiff v0.8.0` cannot be built because it requires rustc 1.61.0 or newer, while the currently active rustc version is 1.60.0-nightly.
[193/379] Building C objectlibclamav/CMakeFiles/lzma_sdk.dir/7z/7zIn.c.o [194/379] Building C object libclamav/CMakeFiles/bytecode_runtime.dir/bytecode_nojit.c.o [195/379] Building C object libclamav/CMakeFiles/yara.dir/yara_grammar.c.o [196/379] Building C object libclamav/CMakeFiles/yara.dir/yara_lexer.c.o yara_lexer.c:2571:24: warning: 'yy_fatal_error' defined but not used [-Wunused-function] yara_lexer.c: In function 'yara_yylex': yara_lexer.l:263:16: warning: '%s' directive output may be truncated writing up to 1023 bytes into a region of size 999 [-Wformat-truncation=] In file included from /usr/include/stdio.h:906, from yara_lexer.c:32: /usr/include/bits/stdio2.h:54:10: note: '__builtin___snprintf_chk' output between 26 and 1049 bytes into a destination of size 1024 54 | return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1, | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 55 | __glibc_objsize (__s), __fmt, | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 56 | __va_arg_pack ()); | ~~~~~~~~~~~~~~~~~ ninja: build stopped: subcommand failed. make: *** [clamav:89: /usr/src/log/clamav-0.105.1] Error 1 ***SNAP***
Great code quality. This is however not the reason why the build stopped. This is only a warning.
Hm. Great.
So I tried the current 'rust 1.65' version.
This time, the building failed because of a rust component:
***SNIP*** ... Finished release [optimized] target(s) in 1.92s cd /usr/src/cipher-0.3.0 && mkdir -pv "/usr/share/cargo/registry/cipher-0.3.0" && if CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "lib")" | grep -q "true" || CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "rlib")" | grep -q "true" || CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "proc-macro")" | grep -q "true"; then awk '/^\[((.+\.)?((dev|build)-)?dependencies|features)/{f=1;next} /^\[/{f=0}; !f' < Cargo.toml > Cargo.toml.deps && CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline package -l | grep -wEv "Cargo.(lock|toml.orig)" | xargs -d "\n" cp -v --parents -a -t /usr/share/cargo/registry/cipher-0.3.0 && install -v -m 644 Cargo.toml.deps /usr/share/cargo/registry/cipher-0.3.0/Cargo.toml && echo "{"files":{},"package":""}" > /usr/share/cargo/registry/cipher-0.3.0/.cargo-checksum.json; fi && if true && CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "bin")" | grep -q "true"; then CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline install -Z avoid-dev-deps -j8 --no-track --path .; fi mkdir: created directory '/usr/share/cargo/registry/cipher-0.3.0' warning: No (git) VCS found for `/usr/src/cipher-0.3.0` error: invalid inclusion of reserved file name Cargo.toml.orig in package source cp: missing file operand Try 'cp --help' for more information. make: *** [rust-cipher:78: /usr/src/log/cipher-0.3.0] Error 123 ***SNAP***
Rust is an absolute dependency hell. Ask Adolf and look at his latest patchset :)
Ok, even greater.
Does anyone have an idea to solve this? I can't even find an updated package for , e.g., 'cipher-0.3.0tar.gz', although apparently I found at least an updated version (0.4.3) here:
=> https://docs.rs/cipher/latest/cipher/#
But no download links... Hm! Where on earth did 'cipher-0.3.0.tar.gz' came from?
There is a little helper script in tools/ which you can use to automatically download the source and even generate an LFS file, because they all look the same:
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=tools/download-rust-crate;...
You can just run this as “tools/download-rust-crate cipher” and it should create everything you need. Just add it to make.sh and it should build.
What makes me a bit nervous though is the fact that if clamav really can only be made to work with a major rust update, the other rust components might have to be updated as well. And I found 103 rust*-lfs files...
Yes. And every time we change one of those packages, we will have to ship *everything* that is related to Rust.
Such a great language. Stop using Rust, people.
-Michael
Any thoughts and hints welcome!
Best, Matthias
Hi Matthias,
On 22/11/2022 17:11, Matthias Fischer wrote:
On 22.11.2022 16:39, Adolf Belka wrote:
Hi Matthias,
Hi Adolf,
You are experiencing Rust Hell.
Yep. I can feel it... ;-)
Many thanks for your tips. I'll try.
But what makes me wondering: why do I get the error "no matching package named `crypto-common` found" although this package definitely exists!? Why is this package not found at all? I can't get a grip on this.
rust-cipher is now requiring rust-crypto-common but rust-crypto-common is later in the build sequence that rust-crypto so it is not existing when rust-cipher is looking for it.
You need to move rust-crypto-common to just before rust-cipher.
Regards,
Adolf.
Best, Matthias
That is what I have had especially with the python3-cryptography module. Often when it has been updated a new rust module is required which requires another new module etc, etc, etc.
On 21/11/2022 20:05, Matthias Fischer wrote:
On 21.11.2022 11:44, Michael Tremer wrote:
Hello Matthias,
Hi Michael,
updated cipher to '0.4.3'.
Clean build, result:
***SNIP*** ====================================== Installing cipher-0.4.3 ... Install started; saving file list to /usr/src/lsalr ... cd /usr/src/cipher-0.4.3 && mkdir -p /usr/src/cipher-0.4.3/.cargo && echo "${CARGO_CONFIG}" > /usr/src/cipher-0.4.3/.cargo/config && rm -f Cargo.lock cd /usr/src/cipher-0.4.3 && CARGOPATH=/usr/src/cipher-0.4.3/.cargo RUSTC_BOOTSTRAP=1 cargo --offline build --release -Z avoid-dev-deps -j8 error: no matching package named `crypto-common` found location searched: registry `crates-io` required by package `cipher v0.4.3 (/usr/src/cipher-0.4.3)` As a reminder, you're using offline mode (--offline) which can sometimes cause surprising resolution failures, if this error is too confusing you may wish to retry without the offline flag. make: *** [rust-cipher:77: /usr/src/log/cipher-0.4.3] Error 101 ***SNAP***
Hm.
Just guessing => updated 'lfs/rust-crypto-common' from '0.1.1' to '0.1.6' through the helper script.
=> Identical error: "no matching package found".
Hmmm!
To follow the "reminder" would mean to delete the '--offline' option in line 209 in 'lfs'/config', but that would be only further guessing. And this would affect all other files. Doesn't feel good.
You can't do that as the IPFire build is run in a chroot that has no internet access.
I'm not familiar with this rust thing - sorry: any ideas about the best
What I have done so far is just to add each new rust module that has been highlighted and then re-run the build. I have had the situation of 12 additional rust modules being required with an update.
After some discussions with my son in the future I intend to run the build of just a specific package in a directory on my machine, the same as I do when building any package from source on my computer (not in the IPFire build shell). This build would then run with internet access and it should then download and install all required rust modules.
Then after that build (of clamav in your case) has been successful you can look at the log details for that build to see which rust modules have been downloaded and installed and then you can use the script that Michael mentioned to add all of those modules to the build in one go. Warning, I have not tried this approach myself yet but it seemed to make sense to me when my son suggested it. You would need to test it out and see if it helped or not.
You will need to check if any rust modules have been required to have a certain version n umber of range. That can be the case that a rust module needs to be updated but not to the latest version. If that is the case then you can use the rust script to download and install into an lfs a specific version.
It can also be the case that some rust modules or packages require a not latest version and other modules require the same module but at a different version number. In this case you have to relabel the lfs file to include the version number. If you look at the list of rust lfs files you will see that some have a specific version specified as well as the plain named module. For example rust-indoc (version 1.0.3) and rust-indoc-0.3.6 (version 0.3.6).
What would help would be if for every package that used rust modules that there was a dependency file that listed all the ones required. It doesn't exist.
By the way, you also need to watch out that some rust packages will by default install the versions for multiple architectures such as windows or mac etc, which is obviously not needed for IPFire. The ones that I have found like that are rust-chrono and rust-iana-time-zone which then needed patches to comment out the meta-data for building the windows etc versions. If this happens with a package you will find rust modules that have win or some other OS name in the title so that usually flags up to me that I need to go back to an earlier module and stop the win based builds. See rust-chrono-0.4.22-fix-metadata.patch in the /src/patches directory as an example.
Sorry that my input is probably not what you were hoping for. It can be worked though, it just can take some time.
Regards, Adolf.
way to proceed?
On 19 Nov 2022, at 15:56, Matthias Fischer matthias.fischer@ipfire.org wrote:
Hi,
...I'd like to have a small problem... ;-)
A few days ago, 'clamav 0.105.1' was updated, again:
=> https://blog.clamav.net/2022/11/second-clamav-100-release-candidate-and.html
"...[it] was intended to also include bug fixes for the jpeg and tiff Rust-based libraries that are bundled with the source code tarball. Unfortunately, those fixes were not all release-ready in time for the 0.105.1-2 packages."
So far, so [oh, forget it!].
This is *really* bad that they bundle so many libraries and make it very difficult for us to keep track of what vulnerabilities might be in clamav although they are part of a third-party library.
We should try to remove all of them and always build against the system libraries.
Unfortunately, building the third version of 'clamav 0.105.1' with current 'next' failed:
***SNIP*** ... error: package `tiff v0.8.0` cannot be built because it requires rustc 1.61.0 or newer, while the currently active rustc version is 1.60.0-nightly.
[193/379] Building C objectlibclamav/CMakeFiles/lzma_sdk.dir/7z/7zIn.c.o [194/379] Building C object libclamav/CMakeFiles/bytecode_runtime.dir/bytecode_nojit.c.o [195/379] Building C object libclamav/CMakeFiles/yara.dir/yara_grammar.c.o [196/379] Building C object libclamav/CMakeFiles/yara.dir/yara_lexer.c.o yara_lexer.c:2571:24: warning: 'yy_fatal_error' defined but not used [-Wunused-function] yara_lexer.c: In function 'yara_yylex': yara_lexer.l:263:16: warning: '%s' directive output may be truncated writing up to 1023 bytes into a region of size 999 [-Wformat-truncation=] In file included from /usr/include/stdio.h:906, from yara_lexer.c:32: /usr/include/bits/stdio2.h:54:10: note: '__builtin___snprintf_chk' output between 26 and 1049 bytes into a destination of size 1024 54 | return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1, | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 55 | __glibc_objsize (__s), __fmt, | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 56 | __va_arg_pack ()); | ~~~~~~~~~~~~~~~~~ ninja: build stopped: subcommand failed. make: *** [clamav:89: /usr/src/log/clamav-0.105.1] Error 1 ***SNAP***
Great code quality. This is however not the reason why the build stopped. This is only a warning.
Hm. Great.
So I tried the current 'rust 1.65' version.
This time, the building failed because of a rust component:
***SNIP*** ... Finished release [optimized] target(s) in 1.92s cd /usr/src/cipher-0.3.0 && mkdir -pv "/usr/share/cargo/registry/cipher-0.3.0" && if CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "lib")" | grep -q "true" || CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "rlib")" | grep -q "true" || CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "proc-macro")" | grep -q "true"; then awk '/^\[((.+\.)?((dev|build)-)?dependencies|features)/{f=1;next} /^\[/{f=0}; !f' < Cargo.toml > Cargo.toml.deps && CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline package -l | grep -wEv "Cargo.(lock|toml.orig)" | xargs -d "\n" cp -v --parents -a -t /usr/share/cargo/registry/cipher-0.3.0 && install -v -m 644 Cargo.toml.deps /usr/share/cargo/registry/cipher-0.3.0/Cargo.toml && echo "{"files":{},"package":""}" > /usr/share/cargo/registry/cipher-0.3.0/.cargo-checksum.json; fi && if true && CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "bin")" | grep -q "true"; then CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline install -Z avoid-dev-deps -j8 --no-track --path .; fi mkdir: created directory '/usr/share/cargo/registry/cipher-0.3.0' warning: No (git) VCS found for `/usr/src/cipher-0.3.0` error: invalid inclusion of reserved file name Cargo.toml.orig in package source cp: missing file operand Try 'cp --help' for more information. make: *** [rust-cipher:78: /usr/src/log/cipher-0.3.0] Error 123 ***SNAP***
Rust is an absolute dependency hell. Ask Adolf and look at his latest patchset :)
Ok, even greater.
Does anyone have an idea to solve this? I can't even find an updated package for , e.g., 'cipher-0.3.0tar.gz', although apparently I found at least an updated version (0.4.3) here:
=> https://docs.rs/cipher/latest/cipher/#
But no download links... Hm! Where on earth did 'cipher-0.3.0.tar.gz' came from?
There is a little helper script in tools/ which you can use to automatically download the source and even generate an LFS file, because they all look the same:
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=tools/download-rust-crate;...
You can just run this as “tools/download-rust-crate cipher” and it should create everything you need. Just add it to make.sh and it should build.
What makes me a bit nervous though is the fact that if clamav really can only be made to work with a major rust update, the other rust components might have to be updated as well. And I found 103 rust*-lfs files...
Yes. And every time we change one of those packages, we will have to ship *everything* that is related to Rust.
Such a great language. Stop using Rust, people.
-Michael
Any thoughts and hints welcome!
Best, Matthias
Hi,
On 22.11.2022 17:11, Matthias Fischer wrote:
On 22.11.2022 16:39, Adolf Belka wrote:
Hi Matthias,
Hi Adolf,
You are experiencing Rust Hell.
... Many thanks for your tips. I'll try.
Ok, I tried. And I'm in the midst of giving up. I'll try to summarize the situation...
My main goals - current:
1. I gave up updating clamav to '0.105.1-3', I'm now looking at '1.0.0 LTS', see: => https://blog.clamav.net/2022/11/clamav-100-lts-released.html
2. 'clamav 1.0.0 LTS' "...requires, at a minimum, Rust compiler version 1.56, as it relies on features introduced in the Rust 2021 Edition."
3. Ok. So far, so bad.
4. The current stable version of rust is '1.65'. Ok, I'll try to update 'rust' to '1.65'. Let's see.
[5. Thanks to Adolf! I moved rust-crypto-common to just before rust-cipher.]
6. But no matter what I tried, the build always stops with:
... warning: No (git) VCS found for `/usr/src/crypto-common-0.1.1` error: invalid inclusion of reserved file name Cargo.toml.orig in package source ...
7. Updated crypto-common to '0.1.6'. Stops with the same warning.
8. Why? No clue. No idea. No visible reason. No hints on the net.
9. I tried a few things. Changed order, looked for dependencies, updated several modules. Nothing helped. I have not noted all error messages but as far as I can remember it was always the VCS error quoted above. As said, I'm in the midst of giving up.
What I have done so far is just to add each new rust module that has been highlighted and then re-run the build. I have had the situation of 12 additional rust modules being required with an update.
That would be no problem. When it comes down to it, I would update *50* modules if only there was *some* indication of *which* module(s) are in need of an update.
After some discussions with my son in the future I intend to run the build of just a specific package in a directory on my machine ... This build would then run with internet access and it should then download and install all required rust modules.
If I get you right, then this would mean to set up some kind of "secondary build directory" - with internet access - away from my standard GIT directory where I usually start './make.sh build'. And the whole thing only so that 'rust' can do whatever it wants and/or download what it needs. How do I set this up!? This can't be true... ;-)
Then after that build (of clamav in your case) has been successful you can look at the log details for that build to see which rust modules have been downloaded and installed and then you can use the script that Michael mentioned to add all of those modules to the build in one go.
And after this I need to compare these two build environments and integrate the needed changes into GIT. And pray. Right?
I even tried to update *all* modules - one by one. No chance.
... You would need to test it out and see if it helped or not.
Terrific... ;-)
You will need to check if any rust modules have been required to have a certain version number of range. That can be the case that a rust module needs to be updated but not to the latest version. If that is the case then you can use the rust script to download and install into an lfs a specific version.
I'm far away from seeing if a module needs an update or if a module is missing. I just see: "No, this won't build - but I don't tell you the (final) reason why!" And when searching for the error message or error code, I usually get an *empty* - or *nearly* empty - hit list.
It can also be the case that some rust modules or packages require a not latest version and other modules require the same module but at a different version number. In this case you have to relabel the lfs file to include the version number. If you look at the list of rust lfs files you will see that some have a specific version specified as well as the plain named module. For example rust-indoc (version 1.0.3) and rust-indoc-0.3.6 (version 0.3.6).
What would help would be if for every package that used rust modules that there was a dependency file that listed all the ones required. It doesn't exist.
Again: terrific. That's the way it shouldn't be.
By the way, you also need to watch out that some rust packages will by default install the versions for multiple architectures such as windows or mac etc, which is obviously not needed for IPFire ... See rust-chrono-0.4.22-fix-metadata.patch in the /src/patches directory as an example.
Sorry that my input is probably not what you were hoping for. It can be worked though, it just can take some time.
No problem, Adolf - I *really* appreciate that you share your experiences with me - many thanks! - but I can't see how to solve this. I'm NOT getting a grip on this, right now...
I shutting down the 'Devel' now. Perhaps I have some new ideas during the next days.
Best, Matthias
...
On 19 Nov 2022, at 15:56, Matthias Fischer matthias.fischer@ipfire.org wrote:
Hi,
...I'd like to have a small problem... ;-)
A few days ago, 'clamav 0.105.1' was updated, again:
=> https://blog.clamav.net/2022/11/second-clamav-100-release-candidate-and.html
"...[it] was intended to also include bug fixes for the jpeg and tiff Rust-based libraries that are bundled with the source code tarball. Unfortunately, those fixes were not all release-ready in time for the 0.105.1-2 packages."
So far, so [oh, forget it!].
This is *really* bad that they bundle so many libraries and make it very difficult for us to keep track of what vulnerabilities might be in clamav although they are part of a third-party library.
We should try to remove all of them and always build against the system libraries.
Unfortunately, building the third version of 'clamav 0.105.1' with current 'next' failed:
***SNIP*** ... error: package `tiff v0.8.0` cannot be built because it requires rustc 1.61.0 or newer, while the currently active rustc version is 1.60.0-nightly.
[193/379] Building C objectlibclamav/CMakeFiles/lzma_sdk.dir/7z/7zIn.c.o [194/379] Building C object libclamav/CMakeFiles/bytecode_runtime.dir/bytecode_nojit.c.o [195/379] Building C object libclamav/CMakeFiles/yara.dir/yara_grammar.c.o [196/379] Building C object libclamav/CMakeFiles/yara.dir/yara_lexer.c.o yara_lexer.c:2571:24: warning: 'yy_fatal_error' defined but not used [-Wunused-function] yara_lexer.c: In function 'yara_yylex': yara_lexer.l:263:16: warning: '%s' directive output may be truncated writing up to 1023 bytes into a region of size 999 [-Wformat-truncation=] In file included from /usr/include/stdio.h:906, from yara_lexer.c:32: /usr/include/bits/stdio2.h:54:10: note: '__builtin___snprintf_chk' output between 26 and 1049 bytes into a destination of size 1024 54 | return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1, | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 55 | __glibc_objsize (__s), __fmt, | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 56 | __va_arg_pack ()); | ~~~~~~~~~~~~~~~~~ ninja: build stopped: subcommand failed. make: *** [clamav:89: /usr/src/log/clamav-0.105.1] Error 1 ***SNAP***
Great code quality. This is however not the reason why the build stopped. This is only a warning.
Hm. Great.
So I tried the current 'rust 1.65' version.
This time, the building failed because of a rust component:
***SNIP*** ... Finished release [optimized] target(s) in 1.92s cd /usr/src/cipher-0.3.0 && mkdir -pv "/usr/share/cargo/registry/cipher-0.3.0" && if CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "lib")" | grep -q "true" || CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "rlib")" | grep -q "true" || CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "proc-macro")" | grep -q "true"; then awk '/^\[((.+\.)?((dev|build)-)?dependencies|features)/{f=1;next} /^\[/{f=0}; !f' < Cargo.toml > Cargo.toml.deps && CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline package -l | grep -wEv "Cargo.(lock|toml.orig)" | xargs -d "\n" cp -v --parents -a -t /usr/share/cargo/registry/cipher-0.3.0 && install -v -m 644 Cargo.toml.deps /usr/share/cargo/registry/cipher-0.3.0/Cargo.toml && echo "{"files":{},"package":""}" > /usr/share/cargo/registry/cipher-0.3.0/.cargo-checksum.json; fi && if true && CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "bin")" | grep -q "true"; then CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline install -Z avoid-dev-deps -j8 --no-track --path .; fi mkdir: created directory '/usr/share/cargo/registry/cipher-0.3.0' warning: No (git) VCS found for `/usr/src/cipher-0.3.0` error: invalid inclusion of reserved file name Cargo.toml.orig in package source cp: missing file operand Try 'cp --help' for more information. make: *** [rust-cipher:78: /usr/src/log/cipher-0.3.0] Error 123 ***SNAP***
Rust is an absolute dependency hell. Ask Adolf and look at his latest patchset :)
Ok, even greater.
Does anyone have an idea to solve this? I can't even find an updated package for , e.g., 'cipher-0.3.0tar.gz', although apparently I found at least an updated version (0.4.3) here:
=> https://docs.rs/cipher/latest/cipher/#
But no download links... Hm! Where on earth did 'cipher-0.3.0.tar.gz' came from?
There is a little helper script in tools/ which you can use to automatically download the source and even generate an LFS file, because they all look the same:
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=tools/download-rust-crate;...
You can just run this as “tools/download-rust-crate cipher” and it should create everything you need. Just add it to make.sh and it should build.
What makes me a bit nervous though is the fact that if clamav really can only be made to work with a major rust update, the other rust components might have to be updated as well. And I found 103 rust*-lfs files...
Yes. And every time we change one of those packages, we will have to ship *everything* that is related to Rust.
Such a great language. Stop using Rust, people.
-Michael
Any thoughts and hints welcome!
Best, Matthias
Hi,
On 21.11.2022 11:44, Michael Tremer wrote:
...
[Shortened to the important part]
... Such a great language. Stop using Rust, people. ...
By the way...
*This* thread eventually caught my eye while updating 'mc':
=> https://lists.midnight-commander.org/pipermail/mc-devel/2021-August/011173.h...
Vitold wants to "rewrite mc in rust"!? He didn't find friends.
Clear answer: => https://lists.midnight-commander.org/pipermail/mc-devel/2021-August/011173.h...
"+1" from me.
Matthias
-
Adolf Belka -
Matthias Fischer -
Michael Tremer