Re: clamav 0.105.1-3 needs rust >1.61

[Adolf Belka <adolf.belka@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 11281 bytes --]

Hi Matthias,

You are experiencing Rust Hell. That is what I have had especially with 
the python3-cryptography module. Often when it has been updated a new 
rust module is required which requires another new module etc, etc, etc.

On 21/11/2022 20:05, Matthias Fischer wrote:
> On 21.11.2022 11:44, Michael Tremer wrote:
>> Hello Matthias,
> Hi Michael,
>
> updated cipher to '0.4.3'.
>
> Clean build, result:
>
> ***SNIP***
>   ====================================== Installing cipher-0.4.3 ...
>      Install started; saving file list to /usr/src/lsalr ...
>      cd /usr/src/cipher-0.4.3 &&         mkdir -p
> /usr/src/cipher-0.4.3/.cargo && echo "${CARGO_CONFIG}" >
> /usr/src/cipher-0.4.3/.cargo/config && rm -f Cargo.lock
>      cd /usr/src/cipher-0.4.3 && CARGOPATH=/usr/src/cipher-0.4.3/.cargo
> RUSTC_BOOTSTRAP=1 cargo --offline build --release -Z avoid-dev-deps -j8
>      error: no matching package named `crypto-common` found
>      location searched: registry `crates-io`
>      required by package `cipher v0.4.3 (/usr/src/cipher-0.4.3)`
>      As a reminder, you're using offline mode (--offline) which can
> sometimes cause surprising resolution failures, if this error is too
> confusing you may wish to retry without the offline flag.
>      make: *** [rust-cipher:77: /usr/src/log/cipher-0.4.3] Error 101
> ***SNAP***
>
> Hm.
>
> Just guessing => updated 'lfs/rust-crypto-common' from '0.1.1' to
> '0.1.6' through the helper script.
>
> => Identical error: "no matching package found".
>
> Hmmm!
>
> To follow the "reminder" would mean to delete the '--offline' option in
> line 209 in 'lfs'/config', but that would be only further guessing. And
> this would affect all other files. Doesn't feel good.
You can't do that as the IPFire build is run in a chroot that has no 
internet access.
> I'm not familiar with this rust thing - sorry: any ideas about the best
What I have done so far is just to add each new rust module that has 
been highlighted and then re-run the build. I have had the situation of 
12 additional rust modules being required with an update.

After some discussions with my son in the future I intend to run the 
build of just a specific package in a directory on my machine, the same 
as I do when building any package from source on my computer (not in the 
IPFire build shell). This build would then run with internet access and 
it should then download and install all required rust modules.

Then after that build (of clamav in your case) has been successful you 
can look at the log details for that build to see which rust modules 
have been downloaded and installed and then you can use the script that 
Michael mentioned to add all of those modules to the build in one go.
Warning, I have not tried this approach myself yet but it seemed to make 
sense to me when my son suggested it. You would need to test it out and 
see if it helped or not.

You will need to check if any rust modules have been required to have a 
certain version n umber of range. That can be the case that a rust 
module needs to be updated but not to the latest version. If that is the 
case then you can use the rust script to download and install into an 
lfs a specific version.

It can also be the case that some rust modules or packages require a not 
latest version and other modules require the same module but at a 
different version number. In this case you have to relabel the lfs file 
to include the version  number. If you look at the list of rust lfs 
files you will see that some have a specific version specified as well 
as the plain named module.
For example rust-indoc (version 1.0.3) and rust-indoc-0.3.6 (version 0.3.6).

What would help would be if for every package that used rust modules 
that there was a dependency file that listed all the ones required. It 
doesn't exist.

By the way, you also need to watch out that some rust packages will by 
default install the versions for multiple architectures such as windows 
or mac etc, which is obviously not needed for IPFire. The ones that I 
have found like that are rust-chrono and rust-iana-time-zone which then 
needed patches to comment out the meta-data for building the windows etc 
versions. If this happens with a package you will find rust modules that 
have win or some other OS name in the title so that usually flags up to 
me that I need to go back to an earlier module and stop the win based 
builds.
See rust-chrono-0.4.22-fix-metadata.patch in the /src/patches directory 
as an example.

Sorry that my input is probably not what you were hoping for.  It can be 
worked though, it just can take some time.

Regards,
Adolf.
> way to proceed?
>
>>> On 19 Nov 2022, at 15:56, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
>>>
>>> Hi,
>>>
>>> ...I'd like to have a small problem... ;-)
>>>
>>> A few days ago, 'clamav 0.105.1' was updated, again:
>>>
>>> =>
>>> https://blog.clamav.net/2022/11/second-clamav-100-release-candidate-and.html
>>>
>>> "...[it] was intended to also include bug fixes for the jpeg and tiff
>>> Rust-based libraries that are bundled with the source code tarball.
>>> Unfortunately, those fixes were not all release-ready in time for the
>>> 0.105.1-2 packages."
>>>
>>> So far, so [oh, forget it!].
>> This is *really* bad that they bundle so many libraries and make it very difficult for us to keep track of what vulnerabilities might be in clamav although they are part of a third-party library.
>>
>> We should try to remove all of them and always build against the system libraries.
>>
>>> Unfortunately, building the third version of 'clamav 0.105.1' with
>>> current 'next' failed:
>>>
>>> ***SNIP***
>>> ...
>>>     error: package `tiff v0.8.0` cannot be built because it requires
>>> rustc 1.61.0 or newer, while the currently active rustc version is
>>> 1.60.0-nightly.
>>>
>>>     [193/379] Building C object
>>> libclamav/CMakeFiles/lzma_sdk.dir/7z/7zIn.c.o
>>>     [194/379] Building C object
>>> libclamav/CMakeFiles/bytecode_runtime.dir/bytecode_nojit.c.o
>>>     [195/379] Building C object
>>> libclamav/CMakeFiles/yara.dir/yara_grammar.c.o
>>>     [196/379] Building C object libclamav/CMakeFiles/yara.dir/yara_lexer.c.o
>>>     yara_lexer.c:2571:24: warning: 'yy_fatal_error' defined but not used
>>> [-Wunused-function]
>>>     yara_lexer.c: In function 'yara_yylex':
>>>     yara_lexer.l:263:16: warning: '%s' directive output may be truncated
>>> writing up to 1023 bytes into a region of size 999 [-Wformat-truncation=]
>>>     In file included from /usr/include/stdio.h:906,
>>>     from yara_lexer.c:32:
>>>     /usr/include/bits/stdio2.h:54:10: note: '__builtin___snprintf_chk'
>>> output between 26 and 1049 bytes into a destination of size 1024
>>>     54 |   return __builtin___snprintf_chk (__s, __n,
>>> __USE_FORTIFY_LEVEL - 1,
>>>     |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>     55 |                                    __glibc_objsize (__s), __fmt,
>>>     |                                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>     56 |                                    __va_arg_pack ());
>>>     |                                    ~~~~~~~~~~~~~~~~~
>>>     ninja: build stopped: subcommand failed.
>>>     make: *** [clamav:89: /usr/src/log/clamav-0.105.1] Error 1
>>> ***SNAP***
>> Great code quality. This is however not the reason why the build stopped. This is only a warning.
>>
>>> Hm. Great.
>>>
>>> So I tried the current 'rust 1.65' version.
>>>
>>> This time, the building failed because of a rust component:
>>>
>>> ***SNIP***
>>> ...
>>> Finished release [optimized] target(s) in 1.92s
>>>     cd /usr/src/cipher-0.3.0 &&         mkdir -pv
>>> "/usr/share/cargo/registry/cipher-0.3.0" && if
>>> CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline
>>> metadata --format-version 1 --no-deps | jq -e
>>> ".packages[].targets[].kind | any(. == \"lib\")" | grep -q "true" ||
>>> CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline
>>> metadata --format-version 1 --no-deps | jq -e
>>> ".packages[].targets[].kind | any(. == \"rlib\")" | grep -q "true" ||
>>> CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline
>>> metadata --format-version 1 --no-deps | jq -e
>>> ".packages[].targets[].kind | any(. == \"proc-macro\")" | grep -q
>>> "true"; then awk
>>> '/^\\\[((.+\\\.)?((dev|build)-)?dependencies|features)/{f=1;next}
>>> /^\\\[/{f=0}; !f' < Cargo.toml > Cargo.toml.deps &&
>>> CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline
>>> package -l | grep -wEv "Cargo.(lock|toml.orig)" | xargs -d "\n" cp -v
>>> --parents -a -t /usr/share/cargo/registry/cipher-0.3.0 && install -v -m
>>> 644 Cargo.toml.deps /usr/share/cargo/registry/cipher-0.3.0/Cargo.toml &&
>>> echo "{\"files\":{},\"package\":\"\"}" >
>>> /usr/share/cargo/registry/cipher-0.3.0/.cargo-checksum.json; fi && if
>>> true && CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo
>>> --offline metadata --format-version 1 --no-deps | jq -e
>>> ".packages[].targets[].kind | any(. == \"bin\")" | grep -q "true"; then
>>> CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline
>>> install -Z avoid-dev-deps -j8 --no-track --path .; fi
>>>     mkdir: created directory '/usr/share/cargo/registry/cipher-0.3.0'
>>>     warning: No (git) VCS found for `/usr/src/cipher-0.3.0`
>>>     error: invalid inclusion of reserved file name Cargo.toml.orig in
>>> package source
>>>     cp: missing file operand
>>>     Try 'cp --help' for more information.
>>>     make: *** [rust-cipher:78: /usr/src/log/cipher-0.3.0] Error 123
>>> ***SNAP***
>> Rust is an absolute dependency hell. Ask Adolf and look at his latest patchset :)
>>
>>> Ok, even greater.
>>>
>>> Does anyone have an idea to solve this? I can't even find an updated
>>> package for , e.g., 'cipher-0.3.0tar.gz', although apparently I found at
>>> least an updated version (0.4.3) here:
>>>
>>> => https://docs.rs/cipher/latest/cipher/#
>>>
>>> But no download links... Hm! Where on earth did 'cipher-0.3.0.tar.gz'
>>> came from?
>> There is a little helper script in tools/ which you can use to automatically download the source and even generate an LFS file, because they all look the same:
>>
>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=tools/download-rust-crate;h=f6a0fe035d30fdbddaa843ccac45251b0049088a;hb=HEAD
>>
>> You can just run this as “tools/download-rust-crate cipher” and it should create everything you need. Just add it to make.sh and it should build.
>>
>>> What makes me a bit nervous though is the fact that if clamav really can
>>> only be made to work with a major rust update, the other rust components
>>> might have to be updated as well. And I found 103 rust*-lfs files...
>> Yes. And every time we change one of those packages, we will have to ship *everything* that is related to Rust.
>>
>> Such a great language. Stop using Rust, people.
>>
>> -Michael
>>
>>> Any thoughts and hints welcome!
>>>
>>> Best,
>>> Matthias

-- 
Sent from my laptop