Hi,
On 22.11.2022 17:11, Matthias Fischer wrote:
On 22.11.2022 16:39, Adolf Belka wrote:
Hi Matthias,
Hi Adolf,
You are experiencing Rust Hell.
... Many thanks for your tips. I'll try.
Ok, I tried. And I'm in the midst of giving up. I'll try to summarize the situation...
My main goals - current:
1. I gave up updating clamav to '0.105.1-3', I'm now looking at '1.0.0 LTS', see: => https://blog.clamav.net/2022/11/clamav-100-lts-released.html
2. 'clamav 1.0.0 LTS' "...requires, at a minimum, Rust compiler version 1.56, as it relies on features introduced in the Rust 2021 Edition."
3. Ok. So far, so bad.
4. The current stable version of rust is '1.65'. Ok, I'll try to update 'rust' to '1.65'. Let's see.
[5. Thanks to Adolf! I moved rust-crypto-common to just before rust-cipher.]
6. But no matter what I tried, the build always stops with:
... warning: No (git) VCS found for `/usr/src/crypto-common-0.1.1` error: invalid inclusion of reserved file name Cargo.toml.orig in package source ...
7. Updated crypto-common to '0.1.6'. Stops with the same warning.
8. Why? No clue. No idea. No visible reason. No hints on the net.
9. I tried a few things. Changed order, looked for dependencies, updated several modules. Nothing helped. I have not noted all error messages but as far as I can remember it was always the VCS error quoted above. As said, I'm in the midst of giving up.
What I have done so far is just to add each new rust module that has been highlighted and then re-run the build. I have had the situation of 12 additional rust modules being required with an update.
That would be no problem. When it comes down to it, I would update *50* modules if only there was *some* indication of *which* module(s) are in need of an update.
After some discussions with my son in the future I intend to run the build of just a specific package in a directory on my machine ... This build would then run with internet access and it should then download and install all required rust modules.
If I get you right, then this would mean to set up some kind of "secondary build directory" - with internet access - away from my standard GIT directory where I usually start './make.sh build'. And the whole thing only so that 'rust' can do whatever it wants and/or download what it needs. How do I set this up!? This can't be true... ;-)
Then after that build (of clamav in your case) has been successful you can look at the log details for that build to see which rust modules have been downloaded and installed and then you can use the script that Michael mentioned to add all of those modules to the build in one go.
And after this I need to compare these two build environments and integrate the needed changes into GIT. And pray. Right?
I even tried to update *all* modules - one by one. No chance.
... You would need to test it out and see if it helped or not.
Terrific... ;-)
You will need to check if any rust modules have been required to have a certain version number of range. That can be the case that a rust module needs to be updated but not to the latest version. If that is the case then you can use the rust script to download and install into an lfs a specific version.
I'm far away from seeing if a module needs an update or if a module is missing. I just see: "No, this won't build - but I don't tell you the (final) reason why!" And when searching for the error message or error code, I usually get an *empty* - or *nearly* empty - hit list.
It can also be the case that some rust modules or packages require a not latest version and other modules require the same module but at a different version number. In this case you have to relabel the lfs file to include the version number. If you look at the list of rust lfs files you will see that some have a specific version specified as well as the plain named module. For example rust-indoc (version 1.0.3) and rust-indoc-0.3.6 (version 0.3.6).
What would help would be if for every package that used rust modules that there was a dependency file that listed all the ones required. It doesn't exist.
Again: terrific. That's the way it shouldn't be.
By the way, you also need to watch out that some rust packages will by default install the versions for multiple architectures such as windows or mac etc, which is obviously not needed for IPFire ... See rust-chrono-0.4.22-fix-metadata.patch in the /src/patches directory as an example.
Sorry that my input is probably not what you were hoping for. It can be worked though, it just can take some time.
No problem, Adolf - I *really* appreciate that you share your experiences with me - many thanks! - but I can't see how to solve this. I'm NOT getting a grip on this, right now...
I shutting down the 'Devel' now. Perhaps I have some new ideas during the next days.
Best, Matthias
...
On 19 Nov 2022, at 15:56, Matthias Fischer matthias.fischer@ipfire.org wrote:
Hi,
...I'd like to have a small problem... ;-)
A few days ago, 'clamav 0.105.1' was updated, again:
=> https://blog.clamav.net/2022/11/second-clamav-100-release-candidate-and.html
"...[it] was intended to also include bug fixes for the jpeg and tiff Rust-based libraries that are bundled with the source code tarball. Unfortunately, those fixes were not all release-ready in time for the 0.105.1-2 packages."
So far, so [oh, forget it!].
This is *really* bad that they bundle so many libraries and make it very difficult for us to keep track of what vulnerabilities might be in clamav although they are part of a third-party library.
We should try to remove all of them and always build against the system libraries.
Unfortunately, building the third version of 'clamav 0.105.1' with current 'next' failed:
***SNIP*** ... error: package `tiff v0.8.0` cannot be built because it requires rustc 1.61.0 or newer, while the currently active rustc version is 1.60.0-nightly.
[193/379] Building C objectlibclamav/CMakeFiles/lzma_sdk.dir/7z/7zIn.c.o [194/379] Building C object libclamav/CMakeFiles/bytecode_runtime.dir/bytecode_nojit.c.o [195/379] Building C object libclamav/CMakeFiles/yara.dir/yara_grammar.c.o [196/379] Building C object libclamav/CMakeFiles/yara.dir/yara_lexer.c.o yara_lexer.c:2571:24: warning: 'yy_fatal_error' defined but not used [-Wunused-function] yara_lexer.c: In function 'yara_yylex': yara_lexer.l:263:16: warning: '%s' directive output may be truncated writing up to 1023 bytes into a region of size 999 [-Wformat-truncation=] In file included from /usr/include/stdio.h:906, from yara_lexer.c:32: /usr/include/bits/stdio2.h:54:10: note: '__builtin___snprintf_chk' output between 26 and 1049 bytes into a destination of size 1024 54 | return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1, | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 55 | __glibc_objsize (__s), __fmt, | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 56 | __va_arg_pack ()); | ~~~~~~~~~~~~~~~~~ ninja: build stopped: subcommand failed. make: *** [clamav:89: /usr/src/log/clamav-0.105.1] Error 1 ***SNAP***
Great code quality. This is however not the reason why the build stopped. This is only a warning.
Hm. Great.
So I tried the current 'rust 1.65' version.
This time, the building failed because of a rust component:
***SNIP*** ... Finished release [optimized] target(s) in 1.92s cd /usr/src/cipher-0.3.0 && mkdir -pv "/usr/share/cargo/registry/cipher-0.3.0" && if CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "lib")" | grep -q "true" || CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "rlib")" | grep -q "true" || CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "proc-macro")" | grep -q "true"; then awk '/^\[((.+\.)?((dev|build)-)?dependencies|features)/{f=1;next} /^\[/{f=0}; !f' < Cargo.toml > Cargo.toml.deps && CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline package -l | grep -wEv "Cargo.(lock|toml.orig)" | xargs -d "\n" cp -v --parents -a -t /usr/share/cargo/registry/cipher-0.3.0 && install -v -m 644 Cargo.toml.deps /usr/share/cargo/registry/cipher-0.3.0/Cargo.toml && echo "{"files":{},"package":""}" > /usr/share/cargo/registry/cipher-0.3.0/.cargo-checksum.json; fi && if true && CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline metadata --format-version 1 --no-deps | jq -e ".packages[].targets[].kind | any(. == "bin")" | grep -q "true"; then CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline install -Z avoid-dev-deps -j8 --no-track --path .; fi mkdir: created directory '/usr/share/cargo/registry/cipher-0.3.0' warning: No (git) VCS found for `/usr/src/cipher-0.3.0` error: invalid inclusion of reserved file name Cargo.toml.orig in package source cp: missing file operand Try 'cp --help' for more information. make: *** [rust-cipher:78: /usr/src/log/cipher-0.3.0] Error 123 ***SNAP***
Rust is an absolute dependency hell. Ask Adolf and look at his latest patchset :)
Ok, even greater.
Does anyone have an idea to solve this? I can't even find an updated package for , e.g., 'cipher-0.3.0tar.gz', although apparently I found at least an updated version (0.4.3) here:
=> https://docs.rs/cipher/latest/cipher/#
But no download links... Hm! Where on earth did 'cipher-0.3.0.tar.gz' came from?
There is a little helper script in tools/ which you can use to automatically download the source and even generate an LFS file, because they all look the same:
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=tools/download-rust-crate;...
You can just run this as “tools/download-rust-crate cipher” and it should create everything you need. Just add it to make.sh and it should build.
What makes me a bit nervous though is the fact that if clamav really can only be made to work with a major rust update, the other rust components might have to be updated as well. And I found 103 rust*-lfs files...
Yes. And every time we change one of those packages, we will have to ship *everything* that is related to Rust.
Such a great language. Stop using Rust, people.
-Michael
Any thoughts and hints welcome!
Best, Matthias