From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail02.haj.ipfire.org (localhost [IPv6:::1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4f0Fkj5fWZz3320 for ; Mon, 26 Jan 2026 17:23:57 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [IPv6:2001:678:b28::25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (Client CN "mail01.haj.ipfire.org", Issuer "R12" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4f0Fkf21hyz2xSM for ; Mon, 26 Jan 2026 17:23:54 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4f0Fkd4DZtz5c2; Mon, 26 Jan 2026 17:23:53 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1769448233; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JqqicTBnxEc4JZVDvJNtN7pLo3t3fwn2sFUk/hhEZHQ=; b=oik+TsspRwLnvJxWUvGm8qTYsBmQKl6oN154S6C6kW/vhwiYbwJPt3OK2gFg1jZB7p+Ccr VfBKqikNrmP7SCBg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1769448233; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JqqicTBnxEc4JZVDvJNtN7pLo3t3fwn2sFUk/hhEZHQ=; b=vvyKTe8fLls9mcXN0mNU61k+j6lu/zkDBSWmIKv6uq6HKYt612fueSPBbZV0Ew7aggRqGy R6lOtIzx5C9tvvQw2dYuVWcKdBJS6fhRo5c/obs41sgrgJEA7zVN4weV44r5Cdab5LrpQt JRA9Q5TzCn4qu9SmAOtA8ULdFlIQ2wT5ghIi9djhcVb4hw9U7k70Jse4W8Rc7jSN9cfsFa CSujspZM9rkV7Oa8jt8XSLfEO7dKY95jP+wHYBTUnB4ptmc+yhHpoBlYFHlkqCKgJcdKMT Pk17r5yHzxP77hl2hMF6xkxPEtv/Sl/AixhtbifbQZDHBoWUQ2HTw2Oun6arLA== Content-Type: text/plain; charset=utf-8 Precedence: list List-Id: List-Subscribe: , List-Unsubscribe: , List-Post: List-Help: Sender: Mail-Followup-To: Mime-Version: 1.0 Subject: Re: Updating rust and eco system From: Michael Tremer In-Reply-To: <767ed58307c23d5d391bd243d5681f19f695b596.camel@ipfire.org> Date: Mon, 26 Jan 2026 17:23:53 +0000 Cc: development@lists.ipfire.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <02AF1D50-1E51-48DE-A5EE-D89C89B3B34E@ipfire.org> <0772cd37-21e8-45c0-9543-957c4688b56d@ipfire.org> <2F324FA8-89B4-4EDC-A9F4-95DBB0E11CF6@ipfire.org> <49cc50bd757ca86f2f744086705935ffd04a6604.camel@ipfire.org> <767ed58307c23d5d391bd243d5681f19f695b596.camel@ipfire.org> To: Stefan Schantl Hello Stefan, Thanks for looking into this. I would suggest that for the cleanup project, it would be best to keep = two versions of rust-syn. Obviously this will inflate the number of = packages for now, but actually starting to update and therefore = potentially add more dependencies does not sounds wise to me. We would = mix up too many changes into one which is never good. Please start a list with all those packages that we would have to have a = look at once the cleanup has been completed and we will start with = updating them. That way we should be able keep this cleaner and = hopefully won=E2=80=99t introduce too many new dependencies. In the end, I guess we will have to run this cleanup more than just this = time, because we never know which package has dropped any dependencies. = This is however rather unlikely. Best, -Michael > On 26 Jan 2026, at 15:31, Stefan Schantl = wrote: >=20 > Hello list it's me again, >=20 > the build process now reached python-cryptography which requires rust- > asn1, which requires rust-ans1_derive, which did not build because of = a > to new version of rust-syn. >=20 > rust-asn1_derive (0.12.2) =20= > [ 1 ][ FAIL ] >=20 > make: Nothing to be done for 'download'. > make: Leaving directory '/home/ipfire-2.x/lfs' > make: Entering directory '/usr/src/lfs' > toml-0.8.19.tar.gz checksum OK > make: Nothing to be done for 'install'. > make: Leaving directory '/usr/src/lfs' > J=C3=A4n 26 15:18:59: Building rust-asn1_derive make: Entering = directory > '/home/ipfire-2.x/lfs' > make: Nothing to be done for 'download'. > make: Leaving directory '/home/ipfire-2.x/lfs' > make: Entering directory '/usr/src/lfs' > asn1_derive-0.12.2.tar.gz checksum OK > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Installing asn1_derive- > 0.12.2 ... > Install started; saving file list to /usr/src/lsalr ... > cd /usr/src/asn1_derive-0.12.2 && if [ -f Cargo.toml.orig ]; then \ > rm -f Cargo.toml.orig; \ > fi; \ >=20 > cd /usr/src/asn1_derive-0.12.2 && mkdir -p /usr/src/asn1_derive- > 0.12.2/.cargo && echo "${CARGO_CONFIG}" > /usr/src/asn1_derive- > 0.12.2/.cargo/config && rm -f Cargo.lock > cd /usr/src/asn1_derive-0.12.2 && CARGOPATH=3D/usr/src/asn1_derive- > 0.12.2/.cargo RUSTC_BOOTSTRAP=3D1 cargo --offline build --release -Z > avoid-dev-deps -j12 > warning: `/usr/src/asn1_derive-0.12.2/.cargo/config` is deprecated > in favor of `config.toml` > | > =3D help: if you need to support cargo 1.38 or earlier, you can > symlink `config` to `config.toml` > error: failed to select a version for the requirement `syn =3D > "^1.0.58"` > candidate versions found which didn't match: 2.0.114 > location searched: directory source `/usr/share/cargo/registry` > (which is replacing registry `crates-io`) > required by package `asn1_derive v0.12.2 (/usr/src/asn1_derive- > 0.12.2)` > perhaps a crate was updated and forgotten to be re-vendored? > As a reminder, you're using offline mode (--offline) which can > sometimes cause surprising resolution failures, if this error is too > confusing you may wish to retry without `--offline`. > make: *** [rust-asn1_derive:78: /usr/src/log/asn1_derive-0.12.2] > Error 101 > make: Leaving directory '/usr/src/lfs' >=20 > ERROR: Building rust-asn1_derive =20= > [ FAIL ] > Check /home/ipfire-2.x/log_x86_64/_build.ipfire.log for errors if > applicable =20= > [ FAIL ] > root@localhost:/home/ipfire-2.x# >=20 > Currently there is an older version of the rust-syn packaged, which > would allow me to bypass this issue, but would violence the goal of > getting rid of unneccessary rust modules. >=20 > Theoretically I also could update the asn1_derive crate to the latest > version but this may break building the next modules. >=20 > May this could act as starting point for the python update, where all > the rust stuff also needs to be touched..... >=20 > @Adolf, @Michael what do you think about that? >=20 > Thanks in advance, >=20 > -Stefan >=20 >=20 >=20 >> Hello list, >>=20 >> currently I'm working on cleaning up the rust packages. >>=20 >> For these I disabled all rust modules in the make.sh file and perform >> a >> clean build as Michael suggested. >>=20 >> At the moment I'm past the stage where "cbindgen" successfully has >> been >> build and have 103 rust modules (inlcluding there sub-dependencies) >> only for this one tool. >>=20 >> An additional rust module is required to build suricata. This is >> because of patching the source code the required rust module is not >> part of their source tarball. >>=20 >> This currently summs to 104 rust modules for the moment. >>=20 >> I'm looking forward when python-cryptography kicks in its module >> whishes.... >>=20 >> Best regards, >>=20 >> -Stefan >>> Hello Adolf, >>> Hello Michael, >>>=20 >>> I would give the rust cleanup a try in the next few days. >>>=20 >>> Adolf may I can ask you to put your current state of the python >>> update >>> into a git repositry? >>>=20 >>> Thanks in advance, >>>=20 >>> -Stefan >>>=20 >>>> Hello Adolf, >>>>=20 >>>>> On 23 Jan 2026, at 11:06, Adolf Belka >>>>> wrote: >>>>>=20 >>>>> Hi Michael, >>>>>=20 >>>>> On 23/01/2026 11:31, Michael Tremer wrote: >>>>>> Hello Stefan, >>>>>> Hello list, >>>>>> Thank you for looking at this. Of course it is very important >>>>>> that we are able to stay on the latest version of Suricata. >>>>>> I have merged your monster of a patch so that we can move on >>>>>> for >>>>>> now, but I have a couple of bigger questions that we all >>>>>> should >>>>>> have a look at: >>>>>> Adolf has in the past spent a lot of time on updating Rust. >>>>>> This >>>>>> is all tapping into Python - or rather python-cryptography - >>>>>> having some Rust code that has further dependencies. In >>>>>> essence, >>>>>> it has been a huge headache to update this. Maybe Adolf even >>>>>> has >>>>>> some other words for this all. >>>>>=20 >>>>> My words on this are that I have now tried multiple times to >>>>> get >>>>> a >>>>> new python update built. Each time I have done it a bit >>>>> different >>>>> but the end result has been the same and that is that python- >>>>> cryptography (which requires rust modules to be built) ends up >>>>> requiring python-maturin that requires more rust modules but at >>>>> the >>>>> end of this the python-cryptography fails to find the built >>>>> rust >>>>> modules. >>>>>=20 >>>>> I have been stuck at this last point so many times that I have >>>>> realised that I am finding lots of reasons not to go and work >>>>> on >>>>> the python update. >>>>> That is not a good position and also python has now moved from >>>>> 3.13 >>>>> to 3.14 so things are moving away from me. >>>>>=20 >>>>> I have come to the conclusion that someone else, more capable >>>>> than >>>>> me needs to have a go at the python update, so I am giving up >>>>> on >>>>> it >>>>> but will continue working on other things. >>>>=20 >>>> Hmm okay, you sound like you are giving up on this :) I know how >>>> many >>>> hours (we probably need to measure those in days or even weeks) >>>> you >>>> have spent on this though. >>>>=20 >>>> Let=E2=80=99s pool resources together and finally get this done. >>>> Hopefully >>>> this will be a smoother ride as a combined effort. >>>>=20 >>>>>> Just building cbindgen has required a further ~98 Rust crates >>>>>> to >>>>>> be packaged. Often we have the same crate in different >>>>>> versions >>>>>> because other crates have pinned a specific version. In >>>>>> total, >>>>>> we >>>>>> currently have ~790 packages in IPFire. Out of those, there >>>>>> are >>>>>> 202 packages in the rust-* namespace. That is pretty much a >>>>>> quarter of the distribution. Although not a lot in size, this >>>>>> is >>>>>> a considerable maintenance burden. >>>>>> ClamAV and Suricata have (recently?) started to bundle all >>>>>> their >>>>>> Rust dependencies with their release tarballs. Although this >>>>>> is >>>>>> not a good thing for many other reasons, it will move the >>>>>> onus >>>>>> onto the upstream projects to provide whatever they need. If >>>>>> their dependencies (and the dependencies of their >>>>>> dependencies) >>>>>> explode, this is not really our problem any more as well as >>>>>> any >>>>>> supply chain problems. Great - within reason. >>>>>> That leaves us with only very few packages that would >>>>>> actually >>>>>> require any external Rust crates (Suricata is even configured >>>>>> to >>>>>> *exclusively* use their bundled crates): cbindgen as a new >>>>>> thing, >>>>>> python-cryptography, anything else? We might actually only >>>>>> need >>>>>> a >>>>>> fraction of the Rust crates that we currently have as the >>>>>> only >>>>>> packages that may actually tap into our locally built >>>>>> repository >>>>>> are only those two. >>>>>=20 >>>>> Unfortunately there is the addon oci-python-sdk that uses >>>>> python- >>>>> cryptography. >>>>=20 >>>> python-cryptography was on my list. oci-python-sdk only uses Rust >>>> indirectly through python-cryptography, right? >>>>=20 >>>>>> Is anyone happy to give this all a try and cleanup any old >>>>>> Rust >>>>>> deps? That way, I hope we will have a much smoother ride >>>>>> moving >>>>>> forward with a Python update. >>>>>=20 >>>>> I can take the current status, before Stefan's patches, and see >>>>> how >>>>> many existing rust modules can be removed. Anything that can be >>>>> removed is a step forward. >>>>=20 >>>> Yes, I think we should try to shrink what we have now if that is >>>> possible at all. As most packages are bundling all Rust deps, >>>> there >>>> should be some we won=E2=80=99t need any more in the system. >>>>=20 >>>> Then, we hopefully have much less to update/worry about in any >>>> other >>>> way when we start touching python-cryptography. >>>>=20 >>>> So who is volunteering to do this? Commenting out all Rust >>>> packages, >>>> then build python-cryptography which will fail as it requires >>>> some >>>> Rust crates. Those will be there so they will only have to be >>>> commented in again. Once the package builds, we should then have >>>> a >>>> couple of packages still commented that we can drop. >>>>=20 >>>>> I think a problem moving forward is that more python modules >>>>> are >>>>> ending up being a combination of python and rust as the >>>>> cryptography and maturin modules have already done. I have also >>>>> seen a lot of rust modules covering the same stuff as covered >>>>> by >>>>> python modules. So the future I think looks like it will >>>>> continue >>>>> to be very frustrating. >>>>=20 >>>> Yes it does, but we will have to find a way whether we want it or >>>> not. >>>>=20 >>>> -Michael >>>>=20 >>>>> Regards, >>>>>=20 >>>>> Adolf. >>>>>=20 >>>>>=20 >>>>>> All the best, >>>>>> -Michael >>>>>>> On 22 Jan 2026, at 17:38, Stefan Schantl >>>>>>> wrote: >>>>>>>=20 >>>>>>> Hello list followers, >>>>>>>=20 >>>>>>> I'm currently updating rust and affected modules. >>>>>>>=20 >>>>>>> This happends mainly because I'm trying to fix the >>>>>>> "suricata >>>>>>> cache >>>>>>> grows infinite" problem, which a lot of people are >>>>>>> affected. >>>>>>>=20 >>>>>>> To archive this, I ported the patches from suricata main >>>>>>> development >>>>>>> branch to our used suricata version (8.0.3). >>>>>>>=20 >>>>>>> To perform a full build, a new tool called cbindgen - which >>>>>>> is >>>>>>> a rust >>>>>>> to c bindings generator, is required. >>>>>>>=20 >>>>>>> Sadly this tool is also written in rust and requires some >>>>>>> new >>>>>>> dependencies and a more up to date rust compiler. >>>>>>>=20 >>>>>>> I hope to send a patchset for all this very soon to the >>>>>>> mailing >>>>>>> list. >>>>>>>=20 >>>>>>> Best regards, >>>>>>>=20 >>>>>>> -Stefan >>>>=20 >>>>=20 >=20