Re: Updating rust and eco system

[Adolf Belka <adolf.belka@ipfire.org>]

Hi Michael,

On 27/01/2026 11:36, Michael Tremer wrote:
> Hello Adolf,
> 
>> On 26 Jan 2026, at 19:07, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>
>> Hi Michael & Stefan,
>>
>> On 26/01/2026 18:23, Michael Tremer wrote:
>>> Hello Stefan,
>>> Thanks for looking into this.
>>> I would suggest that for the cleanup project, it would be best to keep two versions of rust-syn. Obviously this will inflate the number of packages for now, but actually starting to update and therefore potentially add more dependencies does not sounds wise to me. We would mix up too many changes into one which is never good.
>>
>> I am pretty certain from my work on the python update, which required rust updates, that you will find that you will need two different versions of rust-syn as one rust module will require an older version and another rust module will require a newer version and the two won't overlap. In my work there were even some rust modules where I ended up with the latest version plus two older versions being required.
> 
> I hope that at some point someone will be able to explain to me what the benefit is to ship an older version of a crate that will have some bugs that have been fixed in the newer version. And if this is regarding LTS or breaking changes, other libraries totally manage this...

I believe this is due to some packages that use another rust module not having been updated for some time (1 or 2 years) and the module build is linked to the older version of the additional required module.

It might be as simple as a rust module just being updated to use the new other modules or there might be some work to make them work but the original module developer needs to do the update.

I just did a search for option-ext and crates.io came up with 5 results. Th oldest last updated is almost 5 years (works with a 2018 edition of Rust) and the youngest updated is over 1 year ago (works with a 2021 edition of Rust, which is not particularly recent).

Being tied to an older edition of Rust doesn't mean it won't build but it increases the chances of the module needing to be re-written to be updated and work with the latest Rust edition.

Searching for rust modules with the search text of option found 46425 results and on the first page there is one module updated 13 days ago and another that was updated almost 8 years ago.

 From my work on adding required rust modules for the python build it is not clear to me that a module developer ensures they are using other up to date rust modules. I think they find they need a rust module and that might be up to date but it might be using other modules that are quite old and that large chain of dependencies can easily end up with requiring different versions of some other modules than another module that also requires the same dependency.

Regards,

Adolf.

> 
>>> Please start a list with all those packages that we would have to have a look at once the cleanup has been completed and we will start with updating them. That way we should be able keep this cleaner and hopefully won’t introduce too many new dependencies.
>>> In the end, I guess we will have to run this cleanup more than just this time, because we never know which package has dropped any dependencies. This is however rather unlikely.
>>> Best,
>>> -Michael
>>>> On 26 Jan 2026, at 15:31, Stefan Schantl <stefan.schantl@ipfire.org> wrote:
>>>>
>>>> Hello list it's me again,
>>>>
>>>> the build process now reached python-cryptography which requires rust-
>>>> asn1, which requires rust-ans1_derive, which did not build because of a
>>>> to new version of rust-syn.
>>>>
>>>> rust-asn1_derive (0.12.2)
>>>> [        1 ][ FAIL ]
>>>>
>>>>     make: Nothing to be done for 'download'.
>>>>     make: Leaving directory '/home/ipfire-2.x/lfs'
>>>>     make: Entering directory '/usr/src/lfs'
>>>>     toml-0.8.19.tar.gz checksum OK
>>>>     make: Nothing to be done for 'install'.
>>>>     make: Leaving directory '/usr/src/lfs'
>>>>     Jän 26 15:18:59: Building rust-asn1_derive make: Entering directory
>>>> '/home/ipfire-2.x/lfs'
>>>>     make: Nothing to be done for 'download'.
>>>>     make: Leaving directory '/home/ipfire-2.x/lfs'
>>>>     make: Entering directory '/usr/src/lfs'
>>>>     asn1_derive-0.12.2.tar.gz checksum OK
>>>>     ====================================== Installing asn1_derive-
>>>> 0.12.2 ...
>>>>     Install started; saving file list to /usr/src/lsalr ...
>>>>     cd /usr/src/asn1_derive-0.12.2 && if [ -f Cargo.toml.orig ]; then \
>>>>     rm -f Cargo.toml.orig; \
>>>>     fi; \
>>>>
>>>>     cd /usr/src/asn1_derive-0.12.2 && mkdir -p /usr/src/asn1_derive-
>>>> 0.12.2/.cargo && echo "${CARGO_CONFIG}" > /usr/src/asn1_derive-
>>>> 0.12.2/.cargo/config && rm -f Cargo.lock
>>>>     cd /usr/src/asn1_derive-0.12.2 && CARGOPATH=/usr/src/asn1_derive-
>>>> 0.12.2/.cargo RUSTC_BOOTSTRAP=1 cargo --offline build --release -Z
>>>> avoid-dev-deps -j12
>>>>     warning: `/usr/src/asn1_derive-0.12.2/.cargo/config` is deprecated
>>>> in favor of `config.toml`
>>>>     |
>>>>     = help: if you need to support cargo 1.38 or earlier, you can
>>>> symlink `config` to `config.toml`
>>>>     error: failed to select a version for the requirement `syn =
>>>> "^1.0.58"`
>>>>     candidate versions found which didn't match: 2.0.114
>>>>     location searched: directory source `/usr/share/cargo/registry`
>>>> (which is replacing registry `crates-io`)
>>>>     required by package `asn1_derive v0.12.2 (/usr/src/asn1_derive-
>>>> 0.12.2)`
>>>>     perhaps a crate was updated and forgotten to be re-vendored?
>>>>     As a reminder, you're using offline mode (--offline) which can
>>>> sometimes cause surprising resolution failures, if this error is too
>>>> confusing you may wish to retry without `--offline`.
>>>>     make: *** [rust-asn1_derive:78: /usr/src/log/asn1_derive-0.12.2]
>>>> Error 101
>>>>     make: Leaving directory '/usr/src/lfs'
>>>>
>>>> ERROR: Building rust-asn1_derive
>>>> [ FAIL ]
>>>>     Check /home/ipfire-2.x/log_x86_64/_build.ipfire.log for errors if
>>>> applicable
>>>> [ FAIL ]
>>>> root@localhost:/home/ipfire-2.x#
>>>>
>>>> Currently there is an older version of the rust-syn packaged, which
>>>> would allow me to bypass this issue, but would violence the goal of
>>>> getting rid of unneccessary rust modules.
>>>>
>>>> Theoretically I also could update the asn1_derive crate to the latest
>>>> version but this may break building the next modules.
>>>>
>>>> May this could act as starting point for the python update, where all
>>>> the rust stuff also needs to be touched.....
>>>>
>>>> @Adolf, @Michael what do you think about that?
>>>>
>>>> Thanks in advance,
>>>>
>>>> -Stefan
>>>>
>>>>
>>>>
>>>>> Hello list,
>>>>>
>>>>> currently I'm working on cleaning up the rust packages.
>>>>>
>>>>> For these I disabled all rust modules in the make.sh file and perform
>>>>> a
>>>>> clean build as Michael suggested.
>>>>>
>>>>> At the moment I'm past the stage where "cbindgen" successfully has
>>>>> been
>>>>> build and have 103 rust modules (inlcluding there sub-dependencies)
>>>>> only for this one tool.
>>>>>
>>>>> An additional rust module is required to build suricata. This is
>>>>> because of patching the source code the required rust module is not
>>>>> part of their source tarball.
>>>>>
>>>>> This currently summs to 104 rust modules for the moment.
>>>>>
>>>>> I'm looking forward when python-cryptography kicks in its module
>>>>> whishes....
>>>>>
>>>>> Best regards,
>>>>>
>>>>> -Stefan
>>>>>> Hello Adolf,
>>>>>> Hello Michael,
>>>>>>
>>>>>> I would give the rust cleanup a try in the next few days.
>>>>>>
>>>>>> Adolf may I can ask you to put your current state of the python
>>>>>> update
>>>>>> into a git repositry?
>>
>> As my last work from November last year was not completed and is based on an older status now and also on python3.13 vs the current 3.14, I think it makes more sense that once Stefan has completed the clean up of the rust modules, I then take that as the starting point and go through all my changes as done before until I get to the problem I experience with python-cryptography not being able to find any of the rust modules required by python-maturin. That has always been the point where I got stuck.
> 
> Did you try to update python-cryptography first without touching Python and after that try the Python update?
> 
>> At that time I will then put that git branch I am working on into my personal IPFire git repo so that the two of you can look at it to see what I am doing wrong at that stage.
>>
>> That way, I can still contribute with all the update steps that I can do but hand it over when it gets to the step that has consistently beat me.
>>
>> Regards,
>>
>> Adolf.
>>
>>
>>>>>>
>>>>>> Thanks in advance,
>>>>>>
>>>>>> -Stefan
>>>>>>
>>>>>>> Hello Adolf,
>>>>>>>
>>>>>>>> On 23 Jan 2026, at 11:06, Adolf Belka <adolf.belka@ipfire.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi Michael,
>>>>>>>>
>>>>>>>> On 23/01/2026 11:31, Michael Tremer wrote:
>>>>>>>>> Hello Stefan,
>>>>>>>>> Hello list,
>>>>>>>>> Thank you for looking at this. Of course it is very important
>>>>>>>>> that we are able to stay on the latest version of Suricata.
>>>>>>>>> I have merged your monster of a patch so that we can move on
>>>>>>>>> for
>>>>>>>>> now, but I have a couple of bigger questions that we all
>>>>>>>>> should
>>>>>>>>> have a look at:
>>>>>>>>> Adolf has in the past spent a lot of time on updating Rust.
>>>>>>>>> This
>>>>>>>>> is all tapping into Python - or rather python-cryptography -
>>>>>>>>> having some Rust code that has further dependencies. In
>>>>>>>>> essence,
>>>>>>>>> it has been a huge headache to update this. Maybe Adolf even
>>>>>>>>> has
>>>>>>>>> some other words for this all.
>>>>>>>>
>>>>>>>> My words on this are that I have now tried multiple times to
>>>>>>>> get
>>>>>>>> a
>>>>>>>> new python update built. Each time I have done it a bit
>>>>>>>> different
>>>>>>>> but the end result has been the same and that is that python-
>>>>>>>> cryptography (which requires rust modules to be built) ends up
>>>>>>>> requiring python-maturin that requires more rust modules but at
>>>>>>>> the
>>>>>>>> end of this the python-cryptography fails to find the built
>>>>>>>> rust
>>>>>>>> modules.
>>>>>>>>
>>>>>>>> I have been stuck at this last point so many times that I have
>>>>>>>> realised that I am finding lots of reasons not to go and work
>>>>>>>> on
>>>>>>>> the python update.
>>>>>>>> That is not a good position and also python has now moved from
>>>>>>>> 3.13
>>>>>>>> to 3.14 so things are moving away from me.
>>>>>>>>
>>>>>>>> I have come to the conclusion that someone else, more capable
>>>>>>>> than
>>>>>>>> me needs to have a go at the python update, so I am giving up
>>>>>>>> on
>>>>>>>> it
>>>>>>>> but will continue working on other things.
>>>>>>>
>>>>>>> Hmm okay, you sound like you are giving up on this :) I know how
>>>>>>> many
>>>>>>> hours (we probably need to measure those in days or even weeks)
>>>>>>> you
>>>>>>> have spent on this though.
>>>>>>>
>>>>>>> Let’s pool resources together and finally get this done.
>>>>>>> Hopefully
>>>>>>> this will be a smoother ride as a combined effort.
>>>>>>>
>>>>>>>>> Just building cbindgen has required a further ~98 Rust crates
>>>>>>>>> to
>>>>>>>>> be packaged. Often we have the same crate in different
>>>>>>>>> versions
>>>>>>>>> because other crates have pinned a specific version. In
>>>>>>>>> total,
>>>>>>>>> we
>>>>>>>>> currently have ~790 packages in IPFire. Out of those, there
>>>>>>>>> are
>>>>>>>>> 202 packages in the rust-* namespace. That is pretty much a
>>>>>>>>> quarter of the distribution. Although not a lot in size, this
>>>>>>>>> is
>>>>>>>>> a considerable maintenance burden.
>>>>>>>>> ClamAV and Suricata have (recently?) started to bundle all
>>>>>>>>> their
>>>>>>>>> Rust dependencies with their release tarballs. Although this
>>>>>>>>> is
>>>>>>>>> not a good thing for many other reasons, it will move the
>>>>>>>>> onus
>>>>>>>>> onto the upstream projects to provide whatever they need. If
>>>>>>>>> their dependencies (and the dependencies of their
>>>>>>>>> dependencies)
>>>>>>>>> explode, this is not really our problem any more as well as
>>>>>>>>> any
>>>>>>>>> supply chain problems. Great - within reason.
>>>>>>>>> That leaves us with only very few packages that would
>>>>>>>>> actually
>>>>>>>>> require any external Rust crates (Suricata is even configured
>>>>>>>>> to
>>>>>>>>> *exclusively* use their bundled crates): cbindgen as a new
>>>>>>>>> thing,
>>>>>>>>> python-cryptography, anything else? We might actually only
>>>>>>>>> need
>>>>>>>>> a
>>>>>>>>> fraction of the Rust crates that we currently have as the
>>>>>>>>> only
>>>>>>>>> packages that may actually tap into our locally built
>>>>>>>>> repository
>>>>>>>>> are only those two.
>>>>>>>>
>>>>>>>> Unfortunately there is the addon oci-python-sdk that uses
>>>>>>>> python-
>>>>>>>> cryptography.
>>>>>>>
>>>>>>> python-cryptography was on my list. oci-python-sdk only uses Rust
>>>>>>> indirectly through python-cryptography, right?
>>>>>>>
>>>>>>>>> Is anyone happy to give this all a try and cleanup any old
>>>>>>>>> Rust
>>>>>>>>> deps? That way, I hope we will have a much smoother ride
>>>>>>>>> moving
>>>>>>>>> forward with a Python update.
>>>>>>>>
>>>>>>>> I can take the current status, before Stefan's patches, and see
>>>>>>>> how
>>>>>>>> many existing rust modules can be removed. Anything that can be
>>>>>>>> removed is a step forward.
>>>>>>>
>>>>>>> Yes, I think we should try to shrink what we have now if that is
>>>>>>> possible at all. As most packages are bundling all Rust deps,
>>>>>>> there
>>>>>>> should be some we won’t need any more in the system.
>>>>>>>
>>>>>>> Then, we hopefully have much less to update/worry about in any
>>>>>>> other
>>>>>>> way when we start touching python-cryptography.
>>>>>>>
>>>>>>> So who is volunteering to do this? Commenting out all Rust
>>>>>>> packages,
>>>>>>> then build python-cryptography which will fail as it requires
>>>>>>> some
>>>>>>> Rust crates. Those will be there so they will only have to be
>>>>>>> commented in again. Once the package builds, we should then have
>>>>>>> a
>>>>>>> couple of packages still commented that we can drop.
>>>>>>>
>>>>>>>> I think a problem moving forward is that more python modules
>>>>>>>> are
>>>>>>>> ending up being a combination of python and rust as the
>>>>>>>> cryptography and maturin modules have already done. I have also
>>>>>>>> seen a lot of rust modules covering the same stuff as covered
>>>>>>>> by
>>>>>>>> python modules. So the future I think looks like it will
>>>>>>>> continue
>>>>>>>> to be very frustrating.
>>>>>>>
>>>>>>> Yes it does, but we will have to find a way whether we want it or
>>>>>>> not.
>>>>>>>
>>>>>>> -Michael
>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Adolf.
>>>>>>>>
>>>>>>>>
>>>>>>>>> All the best,
>>>>>>>>> -Michael
>>>>>>>>>> On 22 Jan 2026, at 17:38, Stefan Schantl
>>>>>>>>>> <stefan.schantl@ipfire.org> wrote:
>>>>>>>>>>
>>>>>>>>>> Hello list followers,
>>>>>>>>>>
>>>>>>>>>> I'm currently updating rust and affected modules.
>>>>>>>>>>
>>>>>>>>>> This happends mainly because I'm trying to fix the
>>>>>>>>>> "suricata
>>>>>>>>>> cache
>>>>>>>>>> grows infinite" problem, which a lot of people are
>>>>>>>>>> affected.
>>>>>>>>>>
>>>>>>>>>> To archive this, I ported the patches from suricata main
>>>>>>>>>> development
>>>>>>>>>> branch to our used suricata version (8.0.3).
>>>>>>>>>>
>>>>>>>>>> To perform a full build, a new tool called cbindgen - which
>>>>>>>>>> is
>>>>>>>>>> a rust
>>>>>>>>>> to c bindings generator, is required.
>>>>>>>>>>
>>>>>>>>>> Sadly this tool is also written in rust and requires some
>>>>>>>>>> new
>>>>>>>>>> dependencies and a more up to date rust compiler.
>>>>>>>>>>
>>>>>>>>>> I hope to send a patchset for all this very soon to the
>>>>>>>>>> mailing
>>>>>>>>>> list.
>>>>>>>>>>
>>>>>>>>>> Best regards,
>>>>>>>>>>
>>>>>>>>>> -Stefan
> 
>