From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail02.haj.ipfire.org (localhost [IPv6:::1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4f0hfT4P3gz3320 for ; Tue, 27 Jan 2026 10:36:49 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [IPv6:2001:678:b28::25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature ECDSA (secp384r1 raw public key) server-digest SHA384 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mail01.haj.ipfire.org", Issuer "R12" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4f0hfP71LZz2xdr for ; Tue, 27 Jan 2026 10:36:45 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4f0hfM6jlBz5hw; Tue, 27 Jan 2026 10:36:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1769510204; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=f5iuVj8TZYuPZ3YI2BV2ZuRogC7aZ6AnU57BG0/7GVo=; b=eF81eEhze+3qjn/I+hQ6Ysq8Hs9gW07fWuSru3omz21cvTiw8V/qxz8Fmz1Sgo8D+I9x+p zFMQzv38MGo3BqTZAFpv9yRtmHEdw1zmB9N38hmlM3ecjsX6k4CCNJ0q1t/fspB022M237 5qvz2ciHS0lQ1Xiw3aQXKKbTc8eDEMudy8cOGg+exsSfHqorCuXQ17jq6QSYFkXwV0XjFu +DZXC99wWAvoqPH3wsDiEq7Sbjjp+BPGxJeJ2r1rM+VBvInUKECSa6c9LFiT8KHQpB2opb JmKJ4D2T+B6gUlDxI1iOwwu/yPF5tqjsS4vznGs+iWSqqxF/5O/UeecnnUMmaw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1769510204; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=f5iuVj8TZYuPZ3YI2BV2ZuRogC7aZ6AnU57BG0/7GVo=; b=0/fGloknTgKb1UkXv06+0Xd9TQ3ZTjyKOUxlrDwl3zrt1LCJ1vMgVcTCG+12ZrZ0yjjJIT 2E487ZvFIrNiFaDg== Content-Type: text/plain; charset=utf-8 Precedence: list List-Id: List-Subscribe: , List-Unsubscribe: , List-Post: List-Help: Sender: Mail-Followup-To: Mime-Version: 1.0 Subject: Re: Updating rust and eco system From: Michael Tremer In-Reply-To: Date: Tue, 27 Jan 2026 10:36:42 +0000 Cc: Stefan Schantl , "IPFire: Development-List" Content-Transfer-Encoding: quoted-printable Message-Id: <69700B87-C07C-455C-8E94-765E628CE79D@ipfire.org> References: <02AF1D50-1E51-48DE-A5EE-D89C89B3B34E@ipfire.org> <0772cd37-21e8-45c0-9543-957c4688b56d@ipfire.org> <2F324FA8-89B4-4EDC-A9F4-95DBB0E11CF6@ipfire.org> <49cc50bd757ca86f2f744086705935ffd04a6604.camel@ipfire.org> <767ed58307c23d5d391bd243d5681f19f695b596.camel@ipfire.org> To: Adolf Belka Hello Adolf, > On 26 Jan 2026, at 19:07, Adolf Belka wrote: >=20 > Hi Michael & Stefan, >=20 > On 26/01/2026 18:23, Michael Tremer wrote: >> Hello Stefan, >> Thanks for looking into this. >> I would suggest that for the cleanup project, it would be best to = keep two versions of rust-syn. Obviously this will inflate the number of = packages for now, but actually starting to update and therefore = potentially add more dependencies does not sounds wise to me. We would = mix up too many changes into one which is never good. >=20 > I am pretty certain from my work on the python update, which required = rust updates, that you will find that you will need two different = versions of rust-syn as one rust module will require an older version = and another rust module will require a newer version and the two won't = overlap. In my work there were even some rust modules where I ended up = with the latest version plus two older versions being required. I hope that at some point someone will be able to explain to me what the = benefit is to ship an older version of a crate that will have some bugs = that have been fixed in the newer version. And if this is regarding LTS = or breaking changes, other libraries totally manage this... >> Please start a list with all those packages that we would have to = have a look at once the cleanup has been completed and we will start = with updating them. That way we should be able keep this cleaner and = hopefully won=E2=80=99t introduce too many new dependencies. >> In the end, I guess we will have to run this cleanup more than just = this time, because we never know which package has dropped any = dependencies. This is however rather unlikely. >> Best, >> -Michael >>> On 26 Jan 2026, at 15:31, Stefan Schantl = wrote: >>>=20 >>> Hello list it's me again, >>>=20 >>> the build process now reached python-cryptography which requires = rust- >>> asn1, which requires rust-ans1_derive, which did not build because = of a >>> to new version of rust-syn. >>>=20 >>> rust-asn1_derive (0.12.2) >>> [ 1 ][ FAIL ] >>>=20 >>> make: Nothing to be done for 'download'. >>> make: Leaving directory '/home/ipfire-2.x/lfs' >>> make: Entering directory '/usr/src/lfs' >>> toml-0.8.19.tar.gz checksum OK >>> make: Nothing to be done for 'install'. >>> make: Leaving directory '/usr/src/lfs' >>> J=C3=A4n 26 15:18:59: Building rust-asn1_derive make: Entering = directory >>> '/home/ipfire-2.x/lfs' >>> make: Nothing to be done for 'download'. >>> make: Leaving directory '/home/ipfire-2.x/lfs' >>> make: Entering directory '/usr/src/lfs' >>> asn1_derive-0.12.2.tar.gz checksum OK >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Installing asn1_derive- >>> 0.12.2 ... >>> Install started; saving file list to /usr/src/lsalr ... >>> cd /usr/src/asn1_derive-0.12.2 && if [ -f Cargo.toml.orig ]; then = \ >>> rm -f Cargo.toml.orig; \ >>> fi; \ >>>=20 >>> cd /usr/src/asn1_derive-0.12.2 && mkdir -p /usr/src/asn1_derive- >>> 0.12.2/.cargo && echo "${CARGO_CONFIG}" > /usr/src/asn1_derive- >>> 0.12.2/.cargo/config && rm -f Cargo.lock >>> cd /usr/src/asn1_derive-0.12.2 && CARGOPATH=3D/usr/src/asn1_derive-= >>> 0.12.2/.cargo RUSTC_BOOTSTRAP=3D1 cargo --offline build --release -Z >>> avoid-dev-deps -j12 >>> warning: `/usr/src/asn1_derive-0.12.2/.cargo/config` is = deprecated >>> in favor of `config.toml` >>> | >>> =3D help: if you need to support cargo 1.38 or earlier, you can >>> symlink `config` to `config.toml` >>> error: failed to select a version for the requirement `syn =3D >>> "^1.0.58"` >>> candidate versions found which didn't match: 2.0.114 >>> location searched: directory source `/usr/share/cargo/registry` >>> (which is replacing registry `crates-io`) >>> required by package `asn1_derive v0.12.2 (/usr/src/asn1_derive- >>> 0.12.2)` >>> perhaps a crate was updated and forgotten to be re-vendored? >>> As a reminder, you're using offline mode (--offline) which can >>> sometimes cause surprising resolution failures, if this error is too >>> confusing you may wish to retry without `--offline`. >>> make: *** [rust-asn1_derive:78: /usr/src/log/asn1_derive-0.12.2] >>> Error 101 >>> make: Leaving directory '/usr/src/lfs' >>>=20 >>> ERROR: Building rust-asn1_derive >>> [ FAIL ] >>> Check /home/ipfire-2.x/log_x86_64/_build.ipfire.log for errors if >>> applicable >>> [ FAIL ] >>> root@localhost:/home/ipfire-2.x# >>>=20 >>> Currently there is an older version of the rust-syn packaged, which >>> would allow me to bypass this issue, but would violence the goal of >>> getting rid of unneccessary rust modules. >>>=20 >>> Theoretically I also could update the asn1_derive crate to the = latest >>> version but this may break building the next modules. >>>=20 >>> May this could act as starting point for the python update, where = all >>> the rust stuff also needs to be touched..... >>>=20 >>> @Adolf, @Michael what do you think about that? >>>=20 >>> Thanks in advance, >>>=20 >>> -Stefan >>>=20 >>>=20 >>>=20 >>>> Hello list, >>>>=20 >>>> currently I'm working on cleaning up the rust packages. >>>>=20 >>>> For these I disabled all rust modules in the make.sh file and = perform >>>> a >>>> clean build as Michael suggested. >>>>=20 >>>> At the moment I'm past the stage where "cbindgen" successfully has >>>> been >>>> build and have 103 rust modules (inlcluding there sub-dependencies) >>>> only for this one tool. >>>>=20 >>>> An additional rust module is required to build suricata. This is >>>> because of patching the source code the required rust module is not >>>> part of their source tarball. >>>>=20 >>>> This currently summs to 104 rust modules for the moment. >>>>=20 >>>> I'm looking forward when python-cryptography kicks in its module >>>> whishes.... >>>>=20 >>>> Best regards, >>>>=20 >>>> -Stefan >>>>> Hello Adolf, >>>>> Hello Michael, >>>>>=20 >>>>> I would give the rust cleanup a try in the next few days. >>>>>=20 >>>>> Adolf may I can ask you to put your current state of the python >>>>> update >>>>> into a git repositry? >=20 > As my last work from November last year was not completed and is based = on an older status now and also on python3.13 vs the current 3.14, I = think it makes more sense that once Stefan has completed the clean up of = the rust modules, I then take that as the starting point and go through = all my changes as done before until I get to the problem I experience = with python-cryptography not being able to find any of the rust modules = required by python-maturin. That has always been the point where I got = stuck. Did you try to update python-cryptography first without touching Python = and after that try the Python update? > At that time I will then put that git branch I am working on into my = personal IPFire git repo so that the two of you can look at it to see = what I am doing wrong at that stage. >=20 > That way, I can still contribute with all the update steps that I can = do but hand it over when it gets to the step that has consistently beat = me. >=20 > Regards, >=20 > Adolf. >=20 >=20 >>>>>=20 >>>>> Thanks in advance, >>>>>=20 >>>>> -Stefan >>>>>=20 >>>>>> Hello Adolf, >>>>>>=20 >>>>>>> On 23 Jan 2026, at 11:06, Adolf Belka >>>>>>> wrote: >>>>>>>=20 >>>>>>> Hi Michael, >>>>>>>=20 >>>>>>> On 23/01/2026 11:31, Michael Tremer wrote: >>>>>>>> Hello Stefan, >>>>>>>> Hello list, >>>>>>>> Thank you for looking at this. Of course it is very important >>>>>>>> that we are able to stay on the latest version of Suricata. >>>>>>>> I have merged your monster of a patch so that we can move on >>>>>>>> for >>>>>>>> now, but I have a couple of bigger questions that we all >>>>>>>> should >>>>>>>> have a look at: >>>>>>>> Adolf has in the past spent a lot of time on updating Rust. >>>>>>>> This >>>>>>>> is all tapping into Python - or rather python-cryptography - >>>>>>>> having some Rust code that has further dependencies. In >>>>>>>> essence, >>>>>>>> it has been a huge headache to update this. Maybe Adolf even >>>>>>>> has >>>>>>>> some other words for this all. >>>>>>>=20 >>>>>>> My words on this are that I have now tried multiple times to >>>>>>> get >>>>>>> a >>>>>>> new python update built. Each time I have done it a bit >>>>>>> different >>>>>>> but the end result has been the same and that is that python- >>>>>>> cryptography (which requires rust modules to be built) ends up >>>>>>> requiring python-maturin that requires more rust modules but at >>>>>>> the >>>>>>> end of this the python-cryptography fails to find the built >>>>>>> rust >>>>>>> modules. >>>>>>>=20 >>>>>>> I have been stuck at this last point so many times that I have >>>>>>> realised that I am finding lots of reasons not to go and work >>>>>>> on >>>>>>> the python update. >>>>>>> That is not a good position and also python has now moved from >>>>>>> 3.13 >>>>>>> to 3.14 so things are moving away from me. >>>>>>>=20 >>>>>>> I have come to the conclusion that someone else, more capable >>>>>>> than >>>>>>> me needs to have a go at the python update, so I am giving up >>>>>>> on >>>>>>> it >>>>>>> but will continue working on other things. >>>>>>=20 >>>>>> Hmm okay, you sound like you are giving up on this :) I know how >>>>>> many >>>>>> hours (we probably need to measure those in days or even weeks) >>>>>> you >>>>>> have spent on this though. >>>>>>=20 >>>>>> Let=E2=80=99s pool resources together and finally get this done. >>>>>> Hopefully >>>>>> this will be a smoother ride as a combined effort. >>>>>>=20 >>>>>>>> Just building cbindgen has required a further ~98 Rust crates >>>>>>>> to >>>>>>>> be packaged. Often we have the same crate in different >>>>>>>> versions >>>>>>>> because other crates have pinned a specific version. In >>>>>>>> total, >>>>>>>> we >>>>>>>> currently have ~790 packages in IPFire. Out of those, there >>>>>>>> are >>>>>>>> 202 packages in the rust-* namespace. That is pretty much a >>>>>>>> quarter of the distribution. Although not a lot in size, this >>>>>>>> is >>>>>>>> a considerable maintenance burden. >>>>>>>> ClamAV and Suricata have (recently?) started to bundle all >>>>>>>> their >>>>>>>> Rust dependencies with their release tarballs. Although this >>>>>>>> is >>>>>>>> not a good thing for many other reasons, it will move the >>>>>>>> onus >>>>>>>> onto the upstream projects to provide whatever they need. If >>>>>>>> their dependencies (and the dependencies of their >>>>>>>> dependencies) >>>>>>>> explode, this is not really our problem any more as well as >>>>>>>> any >>>>>>>> supply chain problems. Great - within reason. >>>>>>>> That leaves us with only very few packages that would >>>>>>>> actually >>>>>>>> require any external Rust crates (Suricata is even configured >>>>>>>> to >>>>>>>> *exclusively* use their bundled crates): cbindgen as a new >>>>>>>> thing, >>>>>>>> python-cryptography, anything else? We might actually only >>>>>>>> need >>>>>>>> a >>>>>>>> fraction of the Rust crates that we currently have as the >>>>>>>> only >>>>>>>> packages that may actually tap into our locally built >>>>>>>> repository >>>>>>>> are only those two. >>>>>>>=20 >>>>>>> Unfortunately there is the addon oci-python-sdk that uses >>>>>>> python- >>>>>>> cryptography. >>>>>>=20 >>>>>> python-cryptography was on my list. oci-python-sdk only uses Rust >>>>>> indirectly through python-cryptography, right? >>>>>>=20 >>>>>>>> Is anyone happy to give this all a try and cleanup any old >>>>>>>> Rust >>>>>>>> deps? That way, I hope we will have a much smoother ride >>>>>>>> moving >>>>>>>> forward with a Python update. >>>>>>>=20 >>>>>>> I can take the current status, before Stefan's patches, and see >>>>>>> how >>>>>>> many existing rust modules can be removed. Anything that can be >>>>>>> removed is a step forward. >>>>>>=20 >>>>>> Yes, I think we should try to shrink what we have now if that is >>>>>> possible at all. As most packages are bundling all Rust deps, >>>>>> there >>>>>> should be some we won=E2=80=99t need any more in the system. >>>>>>=20 >>>>>> Then, we hopefully have much less to update/worry about in any >>>>>> other >>>>>> way when we start touching python-cryptography. >>>>>>=20 >>>>>> So who is volunteering to do this? Commenting out all Rust >>>>>> packages, >>>>>> then build python-cryptography which will fail as it requires >>>>>> some >>>>>> Rust crates. Those will be there so they will only have to be >>>>>> commented in again. Once the package builds, we should then have >>>>>> a >>>>>> couple of packages still commented that we can drop. >>>>>>=20 >>>>>>> I think a problem moving forward is that more python modules >>>>>>> are >>>>>>> ending up being a combination of python and rust as the >>>>>>> cryptography and maturin modules have already done. I have also >>>>>>> seen a lot of rust modules covering the same stuff as covered >>>>>>> by >>>>>>> python modules. So the future I think looks like it will >>>>>>> continue >>>>>>> to be very frustrating. >>>>>>=20 >>>>>> Yes it does, but we will have to find a way whether we want it or >>>>>> not. >>>>>>=20 >>>>>> -Michael >>>>>>=20 >>>>>>> Regards, >>>>>>>=20 >>>>>>> Adolf. >>>>>>>=20 >>>>>>>=20 >>>>>>>> All the best, >>>>>>>> -Michael >>>>>>>>> On 22 Jan 2026, at 17:38, Stefan Schantl >>>>>>>>> wrote: >>>>>>>>>=20 >>>>>>>>> Hello list followers, >>>>>>>>>=20 >>>>>>>>> I'm currently updating rust and affected modules. >>>>>>>>>=20 >>>>>>>>> This happends mainly because I'm trying to fix the >>>>>>>>> "suricata >>>>>>>>> cache >>>>>>>>> grows infinite" problem, which a lot of people are >>>>>>>>> affected. >>>>>>>>>=20 >>>>>>>>> To archive this, I ported the patches from suricata main >>>>>>>>> development >>>>>>>>> branch to our used suricata version (8.0.3). >>>>>>>>>=20 >>>>>>>>> To perform a full build, a new tool called cbindgen - which >>>>>>>>> is >>>>>>>>> a rust >>>>>>>>> to c bindings generator, is required. >>>>>>>>>=20 >>>>>>>>> Sadly this tool is also written in rust and requires some >>>>>>>>> new >>>>>>>>> dependencies and a more up to date rust compiler. >>>>>>>>>=20 >>>>>>>>> I hope to send a patchset for all this very soon to the >>>>>>>>> mailing >>>>>>>>> list. >>>>>>>>>=20 >>>>>>>>> Best regards, >>>>>>>>>=20 >>>>>>>>> -Stefan