From mboxrd@z Thu Jan 1 00:00:00 1970 From: Adolf Belka To: development@lists.ipfire.org Subject: Re: First results from running build without python2 Date: Wed, 18 Aug 2021 23:08:01 +0200 Message-ID: In-Reply-To: <7b0802c3-3d17-6133-8326-1082460fc9ae@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0184025171560781449==" List-Id: --===============0184025171560781449== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi All, I have successfully built ca-certificates with the new p11-kit approach. The = only difference is that the certificate data such as Version, Serial Number, = Signature Algorithm etc is no longer included in the created bundle files. Th= ere is only the actual certificate itself. This is also the case for the bundles that Arch Linux has created on my syste= ms, that also uses the p11-kit approach. I did a diff on the current ca-bundle.crt/ca-bundle.trust.crt files and the o= nes created by the new approach as used by Fedora and all the same certificat= es are present, it is only the certificate data that is missed. There is a --comment option in the trust extract command that looks like it s= hould add the comments that are in the .p11-kit file into the extracted bundl= e files but nothing changes if it is used or not. I will ask into the p11-kit= git repository whether this should work or not. I did a build just with the new ca-certificate files into Core Update 160 and= then installed the created iso into a vm on my testbed system. Everything worked that I tested out but I am not 100% sure what needs to be e= xercised to test the ca-bundle.crt and ca-bundle.trust.crt files. If someone = can give me some guidance on this then I can do that final test and submit a = patch for this change. I also then did a test build with all python2 programs removed, including pyt= hon2 itself and with the python 3 fixes for spice, frr and ca-certificates in= stalled. The build for every program in the ipfire build section completed su= ccessfully but then I had a stop in the cdrom stage for collectd. It has a py= thon2 plugin module defined which was of course missing. The module could be = disabled in the rootfile or in the build config if the python plugin is not r= equired for IPFire - I could not find any python modules defined. I have been able to successfully define the path to the python interpreter in= the ./configure statement to point to the python3 program and collectd succe= ssfully built with this so if the python plugin is required then a python3 ve= rsion can be created. I have not yet tested this in my vm but will do so. I also found that freeradius has a python2 module defined. This can be disabl= ed. There is a python3 module available since 2.0.20. To use it with 3.0.21 (= current version) you have to have --with-experimental-modules defined. I am l= ooking at latest version 2.0.23 to see if that also needs the experimental mo= dules enabled or if it has become part of the normal modules now and will tes= t the build to see if it works successfully or not. Regards, Adolf. On 12/08/2021 21:30, Adolf Belka wrote: > Hi Michael, > > On 12/08/2021 21:23, Adolf Belka wrote: >> Hi Michael, >> >> On 12/08/2021 15:38, Michael Tremer wrote: >>> Hello, >>> >>> Yes, this is the way to go :) >>> >>> Reach out here if you need any help. >> I am trying to build p11-kit and it has flagged up libtasn1. It is an opti= onal dependency and I can disable it but I would like to check if that is the= correct thing to do or not. >> >> libtasn1 is said to be required to allow the trust policy module to be bui= lt as well as other code that interacts with certificates. >> Is this something that would be needed for IPFire or can I disable it in t= he p11-kit build. >> > Ignore this question. I just did a quick test of disabling libtasn1 and p11= -kit built ok but then my ca-certificates build failed due to the lack of the= trust tool, so that makes it clear that it is required. I will add it in to = the build. > > Slowly getting there. > > Regards, > > Adolf. > >> Regards, >> Adolf. >>> >>> -Michael >>> >>>> On 12 Aug 2021, at 13:10, Adolf Belka wrote: >>>> >>>> Hi Michael, >>>> >>>> On 12/08/2021 13:36, Adolf Belka wrote: >>>>> Hi Michael, >>>>> >>>>> On 12/08/2021 11:17, Michael Tremer wrote: >>>>>> Hello, >>>>>> >>>>>>> On 11 Aug 2021, at 15:03, Adolf Belka wrot= e: >>>>>>> >>>>>>> Hi Michael, >>>>>>> >>>>>>> On 11/08/2021 12:43, Michael Tremer wrote: >>>>>>>> Hello, >>>>>>>> Is this the one with the broken sed command? >>>>>>>> https://src.fedoraproject.org/rpms/ca-certificates/blob/rawhide/f/ce= rtdata2pem.py >>>>>>> Yes, this is that one. Confirmed with a diff. >>>>>>>> This should run if you execute it in the right directory: >>>>>>>> pushd %{name}/certs >>>>>>>> =C2=A0=C2=A0 pwd >>>>>>>> =C2=A0=C2=A0 cp certdata.txt . >>>>>>>> =C2=A0=C2=A0 python3 certdata2pem.py >>>>>>>> popd >>>>>>> I have just learnt about the pushd and popd commands. by doing a quic= k search. Never heard of them before. >>>>>> >>>>>> It is just a version of =E2=80=9Ccd=E2=80=9D that remembers where it h= as been. >>>>>> >>>>>> So if you call =E2=80=9Cpushd some-directory=E2=80=9D, then =E2=80=9Cp= opd=E2=80=9D will bring you back to where you have been before. >>>>>> >>>>>> =E2=80=9Ccd -=E2=80=9C does the same as popd now. >>>>>> >>>>>>>> The fedora version no longer has the build.sh script. >>>>>>> That was the bit I didn't realise. >>>>>> >>>>>> No problem. >>>>> >>>>> Not as simple as I hoped. >>>>> >>>>> The new certdata2pem.py script no longer creates .crt files but .tmp-p1= 1-kit files but that is as far as that script goes. >>>>> >>>>> There is an update-ca-trust file in fedora which splits the various cer= ts to their respective locations, with the openssl ones ending up in ca-bundl= e.trust.crt but this uses p11-kit. I suspect that it uses ca-bundle.trust.p11= -kit generated later on in the %build section of the ca-certificate.spec file= from the tmp-p11-kit files generated by the certdata2pem.py script. So it lo= oks like p11-kit needs to be installed to make this work. >>>>> >>>>> Alternatively I have found the following in LFS. >>>>> https://www.linuxfromscratch.org/blfs/view/svn/postlfs/make-ca.html >>>>> which seems to also create the bundled cert file but also requiring p11= -kit but then talking about different certdata.txt versions that have various= tunings. The one from LFS comes from the mozilla release branch but is modif= ied to something called the Mercurial revision, or the different ones shipped= by RedHat or OpenSUSE which use the version that comes with NSS. >>>>> >>>>> All in all I am not sure which approach to use or how to actually build= it in IPFire. >>>>> >>>>> Definitely help required to know the correct way to go further with thi= s. >>>>> >>>> Having said I needed help, I thought I would have another go and copied = the lines from the %build section of the fedora ca-certificate.spec that crea= ted the ca-bundle.trust.p11.kit >>>> >>>> As I have p11-kit installed on my Arch Linux system I then ran the comma= nd >>>> >>>> p11-kit extract --format=3Dopenssl-bundle --filter=3Dcertificates --over= write --comment ~/openssl/ca-bundle.trust.crt >>>> >>>> and I successfully created the ca-bundle.trust.crt which is needed by th= e IPFire ca-certificates lfs file. >>>> >>>> I will now try and create a new build script that will do all the new ty= pe stuff and also install the p11-kit library files and see how things go. >>>> >>>> :crossed_fingers: >>>> >>>> Adolf >>>> >>>>> Regards, >>>>> Adolf. >>>>> >>>>>> -Michael >>>>>> >>>>>>> Regards, >>>>>>> Adolf. >>>>>>>> -Michael >>>>>>>>> On 8 Aug 2021, at 14:47, Adolf Belka wro= te: >>>>>>>>> >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> I had another go at the ca-certificates problem, the last barrier t= o getting rid of python2. >>>>>>>>> >>>>>>>>> I found certdata2pem.py files from fedora and 2 from suse. I create= d build subdirectories for each version so I could just test running the buil= d.sh file with each version of certdata2pem.py, including the IPFire current = version after running through the 2to3 convertor. >>>>>>>>> >>>>>>>>> fedora >>>>>>>>> >>>>>>>>> The fedora certdata2pem.py file runs successfully with python3 but = has sed commands built into it which fail to find certain files. The sed comm= ands are not in the IPFire version. >>>>>>>>> >>>>>>>>> The error message is >>>>>>>>> >>>>>>>>> =C2=A0=C2=A0 -> written as 'Certum_Trusted_Root_CA:2.16.30.191.89.8= 0.184.201.128.55.76.6.247.235.85.79.181.237.tmp-p11-kit', trust =3D ['CKA_TRU= ST_SERVER_AUTH', 'CKA_TRUST_EMAIL_PROTECTION'], openssl-trust =3D ['serverAut= h', 'emailProtection'], distrust =3D [], openssl-distrust =3D [] >>>>>>>>> sed: can't read certs/*.crt: No such file or directory >>>>>>>>> >>>>>>>>> >>>>>>>>> suse >>>>>>>>> >>>>>>>>> The first suse version runs successfully with python3 but also has = the sed commands in it with the same error message. >>>>>>>>> >>>>>>>>> The second suse version runs successfully with python3, does not ha= ve the sed commands and completes the build.sh script with no errors. However= this certdata2pem.py file has a section that is in the IPFire version comple= tely missing. >>>>>>>>> >>>>>>>>> >>>>>>>>> IPfire version after running through the 2to3 convertor >>>>>>>>> >>>>>>>>> The following error message occurs >>>>>>>>> >>>>>>>>> producing trust for "GlobalSign Root CA"2.11.4.0.0.0.0.1.21.75.90.1= 95.148 >>>>>>>>> Traceback (most recent call last): >>>>>>>>> =C2=A0=C2=A0=C2=A0 File "/mnt/File_Server/Computers/Linux/ipfire/sa= ndbox/patch in progress/python/ca-certificates/orig-2to3-build/certs/../certd= ata2pem.py", line 224, in >>>>>>>>> f.write("\n".join(textwrap.wrap(base64.b64encode(obj['CKA_VALUE']),= 64))) >>>>>>>>> =C2=A0=C2=A0=C2=A0 File "/usr/lib/python3.9/base64.py", line 58, in= b64encode >>>>>>>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 encoded =3D binascii.b2a_base64(s, n= ewline=3DFalse) >>>>>>>>> TypeError: a bytes-like object is required, not 'str' >>>>>>>>> >>>>>>>>> The section that is failing is the section that is missing in the 2= nd suse version. There is an identical fwrite line at line 206 but that does = not seem to flag up the same TypeError message. >>>>>>>>> >>>>>>>>> >>>>>>>>> As the certdata2pem.py files from the other distributions vary sign= ificantly in content, with some having nearly double the number of lines of c= ode, I think the best alternative is to fix the IPFire version so we stay con= sistent but I am unable to figure out how to fix the python code that is caus= ing the "=C2=A0 TypeError: a bytes-like object is required, not 'str' " error= message and need someone's help with that. >>>>>>>>> >>>>>>>>> Let me know if there is any other information that I need to provid= e. >>>>>>>>> >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Adolf. >>>>>>>>> >>>>>>>>> >>>>>>>>> On 07/08/2021 15:54, Adolf Belka wrote: >>>>>>>>>> Hi All, >>>>>>>>>> >>>>>>>>>> On 04/08/2021 16:45, Michael Tremer wrote: >>>>>>>>>>> Hello, >>>>>>>>>>> >>>>>>>>>>>> On 4 Aug 2021, at 13:40, Adolf Belka = wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi All, >>>>>>>>>>>> >>>>>>>>>>>> I have resolved the frr program build. The version currently in = IPFire (6.0) only works with python2. Python3 support came in with version 7.= 4. I have now built frr with version 8.0 including libyang as a new dependenc= y but only for the build, so nothing installed into IPFire itself, and that h= as successfully built without python2 being present. >>>>>>>>>>> >>>>>>>>>>> Great. This could also resolve Matthias=E2=80=99 problem with bui= lding frr. >>>>>>>>>>> >>>>>>>>>>>> Will now go back and have another go with spice-protocol. >>>>>>>>>>> >>>>>>>>>>> Maybe it has a =E2=80=94-disable-python switch? >>>>>>>>>> I just removed the line in the spice-protocol lfs that ran automak= e/py-compile on the python modules from spice. >>>>>>>>>> Spice and spice-protocol are present for qemu and with the py-comp= ile line removed all three successfully built without python2 being present. = I have submitted a patch for this combined with updating spice and spice-prot= ocol, both from 2017. >>>>>>>>>> >>>>>>>>>> This now only leaves the ca-certificates script that needs to be u= pdated to work with python3. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Adolf. >>>>>>>>>>> >>>>>>>>>>> -Michael >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> >>>>>>>>>>>> Adolf. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 03/08/2021 23:38, Adolf Belka wrote: >>>>>>>>>>>>> Hi Michael & all, >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 03/08/2021 17:11, Michael Tremer wrote: >>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thank you for looking into this. >>>>>>>>>>>>>> >>>>>>>>>>>>>> This is a third-party script that came from either Mozilla or = RedHat. Maybe they have ported it. If not, it should not be rocket science to= do it ourselves. If we do it, we should of course upstream it. >>>>>>>>>>>>> I found an updated script from fedora and gave that a try. This= time the script went all the way through but then the build.sh script failed= at the point where it should find all the .crt files in the certs directory = and it came back and said there weren't any. >>>>>>>>>>>>>> >>>>>>>>>>>>>> However, can you comment out this package and continue the bui= ld? This should be required until you reach the cdrom stage. >>>>>>>>>>>>> I then commented ca-certificates out in make.sh and ran the bui= ld. >>>>>>>>>>>>> This time it stopped at spice-protocol which is an addon and us= es the py-compile script that is in automake to compile some python modules. >>>>>>>>>>>>> py-compile is python2 based and the build stopped because it co= uld not find python >>>>>>>>>>>>> >>>>>>>>>>>>> There is a py_compile.py script that is python3 based but when = I ran that in place of the py-compile script I got a Permission denied error = when it tried to carry out the compile. >>>>>>>>>>>>> >>>>>>>>>>>>> I then commented out spice-protocol and ran the build. >>>>>>>>>>>>> >>>>>>>>>>>>> It then failed on frr which did look for python3-config but the= n failed due to not finding python-config or pkg-config python >>>>>>>>>>>>> It looks like I should be able to tell it to use python3 in the= ./configure >>>>>>>>>>>>> >>>>>>>>>>>>> I commented out frr and nothing else failed before cdrom was re= ached. >>>>>>>>>>>>> >>>>>>>>>>>>> So the packages that need to be made to work with python3 are >>>>>>>>>>>>> ca-certificates >>>>>>>>>>>>> spice-protocol >>>>>>>>>>>>> frr >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I also converted client175 with 2to3 converter and built it and= installed the .ipfire package into a vm and successfully got the WUI page fo= r Media Player IPFire to render. What I haven't tested yet is if the audio wo= rks. I will need to get audio set up in my vm to try that. >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Adolf. >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> If this is the only thing that flags up, we should port the sc= ript. If we find another, stronger reason to keep Python 2 around, we do not = need to bother and can keep the script this way. >>>>>>>>>>>>>> >>>>>>>>>>>>>> -Michael >>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 3 Aug 2021, at 13:31, Adolf Belka wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> So with crda and the remaining python2 modules removed the qu= estion was if removing python2 from the build ran without any problem or if s= omething was flagged up. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ca-certificates was flagged up. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> There is a python2 script, certdata2pem.py, which fails if py= thon2 is not present. Running that script with python3 flags up some invalid = syntax, unsurprisingly. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I found some patches in Debian from 2015 for certdata2pem.py = to provide python3 compatibility. Unfortunately looking at the patch approx h= alf could not be applied because the lines don't exist in the IPFire version = of certdata2pem.py (sections to do with blacklisted certs) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I then ran the 2to3 converter on certdata2pem.py and tried th= at in the build but it came up with the following error. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> TypeError: a bytes-like object is required, not 'str' >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I don't know how to further move forward with this as I am to= tally unfamiliar with the python language. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Adolf. >>> --===============0184025171560781449==--