Re: clamav 0.105.1-3 needs rust >1.61

[Adolf Belka <adolf.belka@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 12537 bytes --]

Hi Matthias,

On 22/11/2022 17:11, Matthias Fischer wrote:
> On 22.11.2022 16:39, Adolf Belka wrote:
>> Hi Matthias,
> 
> Hi Adolf,
> 
>> You are experiencing Rust Hell.
> 
> Yep. I can feel it... ;-)
> 
> Many thanks for your tips. I'll try.
> 
> But what makes me wondering: why do I get the error "no matching package
> named `crypto-common` found" although this package definitely exists!?
> Why is this package not found at all? I can't get a grip on this.
rust-cipher is now requiring rust-crypto-common but rust-crypto-common 
is later in the build sequence that rust-crypto so it is not existing 
when rust-cipher is looking for it.

You need to move rust-crypto-common to just before rust-cipher.


Regards,

Adolf.
> 
> Best,
> Matthias
> 
> 
>> That is what I have had especially with
>> the python3-cryptography module. Often when it has been updated a new
>> rust module is required which requires another new module etc, etc, etc.
>>
>> On 21/11/2022 20:05, Matthias Fischer wrote:
>>> On 21.11.2022 11:44, Michael Tremer wrote:
>>>> Hello Matthias,
>>> Hi Michael,
>>>
>>> updated cipher to '0.4.3'.
>>>
>>> Clean build, result:
>>>
>>> ***SNIP***
>>>    ====================================== Installing cipher-0.4.3 ...
>>>       Install started; saving file list to /usr/src/lsalr ...
>>>       cd /usr/src/cipher-0.4.3 &&         mkdir -p
>>> /usr/src/cipher-0.4.3/.cargo && echo "${CARGO_CONFIG}" >
>>> /usr/src/cipher-0.4.3/.cargo/config && rm -f Cargo.lock
>>>       cd /usr/src/cipher-0.4.3 && CARGOPATH=/usr/src/cipher-0.4.3/.cargo
>>> RUSTC_BOOTSTRAP=1 cargo --offline build --release -Z avoid-dev-deps -j8
>>>       error: no matching package named `crypto-common` found
>>>       location searched: registry `crates-io`
>>>       required by package `cipher v0.4.3 (/usr/src/cipher-0.4.3)`
>>>       As a reminder, you're using offline mode (--offline) which can
>>> sometimes cause surprising resolution failures, if this error is too
>>> confusing you may wish to retry without the offline flag.
>>>       make: *** [rust-cipher:77: /usr/src/log/cipher-0.4.3] Error 101
>>> ***SNAP***
>>>
>>> Hm.
>>>
>>> Just guessing => updated 'lfs/rust-crypto-common' from '0.1.1' to
>>> '0.1.6' through the helper script.
>>>
>>> => Identical error: "no matching package found".
>>>
>>> Hmmm!
>>>
>>> To follow the "reminder" would mean to delete the '--offline' option in
>>> line 209 in 'lfs'/config', but that would be only further guessing. And
>>> this would affect all other files. Doesn't feel good.
>> You can't do that as the IPFire build is run in a chroot that has no
>> internet access.
>>> I'm not familiar with this rust thing - sorry: any ideas about the best
>> What I have done so far is just to add each new rust module that has
>> been highlighted and then re-run the build. I have had the situation of
>> 12 additional rust modules being required with an update.
>>
>> After some discussions with my son in the future I intend to run the
>> build of just a specific package in a directory on my machine, the same
>> as I do when building any package from source on my computer (not in the
>> IPFire build shell). This build would then run with internet access and
>> it should then download and install all required rust modules.
>>
>> Then after that build (of clamav in your case) has been successful you
>> can look at the log details for that build to see which rust modules
>> have been downloaded and installed and then you can use the script that
>> Michael mentioned to add all of those modules to the build in one go.
>> Warning, I have not tried this approach myself yet but it seemed to make
>> sense to me when my son suggested it. You would need to test it out and
>> see if it helped or not.
>>
>> You will need to check if any rust modules have been required to have a
>> certain version n umber of range. That can be the case that a rust
>> module needs to be updated but not to the latest version. If that is the
>> case then you can use the rust script to download and install into an
>> lfs a specific version.
>>
>> It can also be the case that some rust modules or packages require a not
>> latest version and other modules require the same module but at a
>> different version number. In this case you have to relabel the lfs file
>> to include the version  number. If you look at the list of rust lfs
>> files you will see that some have a specific version specified as well
>> as the plain named module.
>> For example rust-indoc (version 1.0.3) and rust-indoc-0.3.6 (version 0.3.6).
>>
>> What would help would be if for every package that used rust modules
>> that there was a dependency file that listed all the ones required. It
>> doesn't exist.
>>
>> By the way, you also need to watch out that some rust packages will by
>> default install the versions for multiple architectures such as windows
>> or mac etc, which is obviously not needed for IPFire. The ones that I
>> have found like that are rust-chrono and rust-iana-time-zone which then
>> needed patches to comment out the meta-data for building the windows etc
>> versions. If this happens with a package you will find rust modules that
>> have win or some other OS name in the title so that usually flags up to
>> me that I need to go back to an earlier module and stop the win based
>> builds.
>> See rust-chrono-0.4.22-fix-metadata.patch in the /src/patches directory
>> as an example.
>>
>> Sorry that my input is probably not what you were hoping for.  It can be
>> worked though, it just can take some time.
>>
>> Regards,
>> Adolf.
>>> way to proceed?
>>>
>>>>> On 19 Nov 2022, at 15:56, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> ...I'd like to have a small problem... ;-)
>>>>>
>>>>> A few days ago, 'clamav 0.105.1' was updated, again:
>>>>>
>>>>> =>
>>>>> https://blog.clamav.net/2022/11/second-clamav-100-release-candidate-and.html
>>>>>
>>>>> "...[it] was intended to also include bug fixes for the jpeg and tiff
>>>>> Rust-based libraries that are bundled with the source code tarball.
>>>>> Unfortunately, those fixes were not all release-ready in time for the
>>>>> 0.105.1-2 packages."
>>>>>
>>>>> So far, so [oh, forget it!].
>>>> This is *really* bad that they bundle so many libraries and make it very difficult for us to keep track of what vulnerabilities might be in clamav although they are part of a third-party library.
>>>>
>>>> We should try to remove all of them and always build against the system libraries.
>>>>
>>>>> Unfortunately, building the third version of 'clamav 0.105.1' with
>>>>> current 'next' failed:
>>>>>
>>>>> ***SNIP***
>>>>> ...
>>>>>      error: package `tiff v0.8.0` cannot be built because it requires
>>>>> rustc 1.61.0 or newer, while the currently active rustc version is
>>>>> 1.60.0-nightly.
>>>>>
>>>>>      [193/379] Building C object
>>>>> libclamav/CMakeFiles/lzma_sdk.dir/7z/7zIn.c.o
>>>>>      [194/379] Building C object
>>>>> libclamav/CMakeFiles/bytecode_runtime.dir/bytecode_nojit.c.o
>>>>>      [195/379] Building C object
>>>>> libclamav/CMakeFiles/yara.dir/yara_grammar.c.o
>>>>>      [196/379] Building C object libclamav/CMakeFiles/yara.dir/yara_lexer.c.o
>>>>>      yara_lexer.c:2571:24: warning: 'yy_fatal_error' defined but not used
>>>>> [-Wunused-function]
>>>>>      yara_lexer.c: In function 'yara_yylex':
>>>>>      yara_lexer.l:263:16: warning: '%s' directive output may be truncated
>>>>> writing up to 1023 bytes into a region of size 999 [-Wformat-truncation=]
>>>>>      In file included from /usr/include/stdio.h:906,
>>>>>      from yara_lexer.c:32:
>>>>>      /usr/include/bits/stdio2.h:54:10: note: '__builtin___snprintf_chk'
>>>>> output between 26 and 1049 bytes into a destination of size 1024
>>>>>      54 |   return __builtin___snprintf_chk (__s, __n,
>>>>> __USE_FORTIFY_LEVEL - 1,
>>>>>      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>      55 |                                    __glibc_objsize (__s), __fmt,
>>>>>      |                                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>      56 |                                    __va_arg_pack ());
>>>>>      |                                    ~~~~~~~~~~~~~~~~~
>>>>>      ninja: build stopped: subcommand failed.
>>>>>      make: *** [clamav:89: /usr/src/log/clamav-0.105.1] Error 1
>>>>> ***SNAP***
>>>> Great code quality. This is however not the reason why the build stopped. This is only a warning.
>>>>
>>>>> Hm. Great.
>>>>>
>>>>> So I tried the current 'rust 1.65' version.
>>>>>
>>>>> This time, the building failed because of a rust component:
>>>>>
>>>>> ***SNIP***
>>>>> ...
>>>>> Finished release [optimized] target(s) in 1.92s
>>>>>      cd /usr/src/cipher-0.3.0 &&         mkdir -pv
>>>>> "/usr/share/cargo/registry/cipher-0.3.0" && if
>>>>> CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline
>>>>> metadata --format-version 1 --no-deps | jq -e
>>>>> ".packages[].targets[].kind | any(. == \"lib\")" | grep -q "true" ||
>>>>> CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline
>>>>> metadata --format-version 1 --no-deps | jq -e
>>>>> ".packages[].targets[].kind | any(. == \"rlib\")" | grep -q "true" ||
>>>>> CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline
>>>>> metadata --format-version 1 --no-deps | jq -e
>>>>> ".packages[].targets[].kind | any(. == \"proc-macro\")" | grep -q
>>>>> "true"; then awk
>>>>> '/^\\\[((.+\\\.)?((dev|build)-)?dependencies|features)/{f=1;next}
>>>>> /^\\\[/{f=0}; !f' < Cargo.toml > Cargo.toml.deps &&
>>>>> CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline
>>>>> package -l | grep -wEv "Cargo.(lock|toml.orig)" | xargs -d "\n" cp -v
>>>>> --parents -a -t /usr/share/cargo/registry/cipher-0.3.0 && install -v -m
>>>>> 644 Cargo.toml.deps /usr/share/cargo/registry/cipher-0.3.0/Cargo.toml &&
>>>>> echo "{\"files\":{},\"package\":\"\"}" >
>>>>> /usr/share/cargo/registry/cipher-0.3.0/.cargo-checksum.json; fi && if
>>>>> true && CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo
>>>>> --offline metadata --format-version 1 --no-deps | jq -e
>>>>> ".packages[].targets[].kind | any(. == \"bin\")" | grep -q "true"; then
>>>>> CARGOPATH=/usr/src/cipher-0.3.0/.cargo RUSTC_BOOTSTRAP=1 cargo --offline
>>>>> install -Z avoid-dev-deps -j8 --no-track --path .; fi
>>>>>      mkdir: created directory '/usr/share/cargo/registry/cipher-0.3.0'
>>>>>      warning: No (git) VCS found for `/usr/src/cipher-0.3.0`
>>>>>      error: invalid inclusion of reserved file name Cargo.toml.orig in
>>>>> package source
>>>>>      cp: missing file operand
>>>>>      Try 'cp --help' for more information.
>>>>>      make: *** [rust-cipher:78: /usr/src/log/cipher-0.3.0] Error 123
>>>>> ***SNAP***
>>>> Rust is an absolute dependency hell. Ask Adolf and look at his latest patchset :)
>>>>
>>>>> Ok, even greater.
>>>>>
>>>>> Does anyone have an idea to solve this? I can't even find an updated
>>>>> package for , e.g., 'cipher-0.3.0tar.gz', although apparently I found at
>>>>> least an updated version (0.4.3) here:
>>>>>
>>>>> => https://docs.rs/cipher/latest/cipher/#
>>>>>
>>>>> But no download links... Hm! Where on earth did 'cipher-0.3.0.tar.gz'
>>>>> came from?
>>>> There is a little helper script in tools/ which you can use to automatically download the source and even generate an LFS file, because they all look the same:
>>>>
>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=tools/download-rust-crate;h=f6a0fe035d30fdbddaa843ccac45251b0049088a;hb=HEAD
>>>>
>>>> You can just run this as “tools/download-rust-crate cipher” and it should create everything you need. Just add it to make.sh and it should build.
>>>>
>>>>> What makes me a bit nervous though is the fact that if clamav really can
>>>>> only be made to work with a major rust update, the other rust components
>>>>> might have to be updated as well. And I found 103 rust*-lfs files...
>>>> Yes. And every time we change one of those packages, we will have to ship *everything* that is related to Rust.
>>>>
>>>> Such a great language. Stop using Rust, people.
>>>>
>>>> -Michael
>>>>
>>>>> Any thoughts and hints welcome!
>>>>>
>>>>> Best,
>>>>> Matthias
>>
> 

-- 
Sent from my laptop