From mboxrd@z Thu Jan 1 00:00:00 1970 From: Adolf Belka To: development@lists.ipfire.org Subject: Re: First results from running build without python2 Date: Thu, 12 Aug 2021 21:30:33 +0200 Message-ID: <7b0802c3-3d17-6133-8326-1082460fc9ae@ipfire.org> In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============5696223325404466218==" List-Id: --===============5696223325404466218== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi Michael, On 12/08/2021 21:23, Adolf Belka wrote: > Hi Michael, > > On 12/08/2021 15:38, Michael Tremer wrote: >> Hello, >> >> Yes, this is the way to go :) >> >> Reach out here if you need any help. > I am trying to build p11-kit and it has flagged up libtasn1. It is an optio= nal dependency and I can disable it but I would like to check if that is the = correct thing to do or not. > > libtasn1 is said to be required to allow the trust policy module to be buil= t as well as other code that interacts with certificates. > Is this something that would be needed for IPFire or can I disable it in th= e p11-kit build. > Ignore this question. I just did a quick test of disabling libtasn1 and p11-k= it built ok but then my ca-certificates build failed due to the lack of the t= rust tool, so that makes it clear that it is required. I will add it in to th= e build. Slowly getting there. Regards, Adolf. > Regards, > Adolf. >> >> -Michael >> >>> On 12 Aug 2021, at 13:10, Adolf Belka wrote: >>> >>> Hi Michael, >>> >>> On 12/08/2021 13:36, Adolf Belka wrote: >>>> Hi Michael, >>>> >>>> On 12/08/2021 11:17, Michael Tremer wrote: >>>>> Hello, >>>>> >>>>>> On 11 Aug 2021, at 15:03, Adolf Belka wrote: >>>>>> >>>>>> Hi Michael, >>>>>> >>>>>> On 11/08/2021 12:43, Michael Tremer wrote: >>>>>>> Hello, >>>>>>> Is this the one with the broken sed command? >>>>>>> https://src.fedoraproject.org/rpms/ca-certificates/blob/rawhide/f/cer= tdata2pem.py >>>>>> Yes, this is that one. Confirmed with a diff. >>>>>>> This should run if you execute it in the right directory: >>>>>>> pushd %{name}/certs >>>>>>> =C2=A0=C2=A0 pwd >>>>>>> =C2=A0=C2=A0 cp certdata.txt . >>>>>>> =C2=A0=C2=A0 python3 certdata2pem.py >>>>>>> popd >>>>>> I have just learnt about the pushd and popd commands. by doing a quick= search. Never heard of them before. >>>>> >>>>> It is just a version of =E2=80=9Ccd=E2=80=9D that remembers where it ha= s been. >>>>> >>>>> So if you call =E2=80=9Cpushd some-directory=E2=80=9D, then =E2=80=9Cpo= pd=E2=80=9D will bring you back to where you have been before. >>>>> >>>>> =E2=80=9Ccd -=E2=80=9C does the same as popd now. >>>>> >>>>>>> The fedora version no longer has the build.sh script. >>>>>> That was the bit I didn't realise. >>>>> >>>>> No problem. >>>> >>>> Not as simple as I hoped. >>>> >>>> The new certdata2pem.py script no longer creates .crt files but .tmp-p11= -kit files but that is as far as that script goes. >>>> >>>> There is an update-ca-trust file in fedora which splits the various cert= s to their respective locations, with the openssl ones ending up in ca-bundle= .trust.crt but this uses p11-kit. I suspect that it uses ca-bundle.trust.p11-= kit generated later on in the %build section of the ca-certificate.spec file = from the tmp-p11-kit files generated by the certdata2pem.py script. So it loo= ks like p11-kit needs to be installed to make this work. >>>> >>>> Alternatively I have found the following in LFS. >>>> https://www.linuxfromscratch.org/blfs/view/svn/postlfs/make-ca.html >>>> which seems to also create the bundled cert file but also requiring p11-= kit but then talking about different certdata.txt versions that have various = tunings. The one from LFS comes from the mozilla release branch but is modifi= ed to something called the Mercurial revision, or the different ones shipped = by RedHat or OpenSUSE which use the version that comes with NSS. >>>> >>>> All in all I am not sure which approach to use or how to actually build = it in IPFire. >>>> >>>> Definitely help required to know the correct way to go further with this. >>>> >>> Having said I needed help, I thought I would have another go and copied t= he lines from the %build section of the fedora ca-certificate.spec that creat= ed the ca-bundle.trust.p11.kit >>> >>> As I have p11-kit installed on my Arch Linux system I then ran the command >>> >>> p11-kit extract --format=3Dopenssl-bundle --filter=3Dcertificates --overw= rite --comment ~/openssl/ca-bundle.trust.crt >>> >>> and I successfully created the ca-bundle.trust.crt which is needed by the= IPFire ca-certificates lfs file. >>> >>> I will now try and create a new build script that will do all the new typ= e stuff and also install the p11-kit library files and see how things go. >>> >>> :crossed_fingers: >>> >>> Adolf >>> >>>> Regards, >>>> Adolf. >>>> >>>>> -Michael >>>>> >>>>>> Regards, >>>>>> Adolf. >>>>>>> -Michael >>>>>>>> On 8 Aug 2021, at 14:47, Adolf Belka wrot= e: >>>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> I had another go at the ca-certificates problem, the last barrier to= getting rid of python2. >>>>>>>> >>>>>>>> I found certdata2pem.py files from fedora and 2 from suse. I created= build subdirectories for each version so I could just test running the build= .sh file with each version of certdata2pem.py, including the IPFire current v= ersion after running through the 2to3 convertor. >>>>>>>> >>>>>>>> fedora >>>>>>>> >>>>>>>> The fedora certdata2pem.py file runs successfully with python3 but h= as sed commands built into it which fail to find certain files. The sed comma= nds are not in the IPFire version. >>>>>>>> >>>>>>>> The error message is >>>>>>>> >>>>>>>> =C2=A0=C2=A0 -> written as 'Certum_Trusted_Root_CA:2.16.30.191.89.80= .184.201.128.55.76.6.247.235.85.79.181.237.tmp-p11-kit', trust =3D ['CKA_TRUS= T_SERVER_AUTH', 'CKA_TRUST_EMAIL_PROTECTION'], openssl-trust =3D ['serverAuth= ', 'emailProtection'], distrust =3D [], openssl-distrust =3D [] >>>>>>>> sed: can't read certs/*.crt: No such file or directory >>>>>>>> >>>>>>>> >>>>>>>> suse >>>>>>>> >>>>>>>> The first suse version runs successfully with python3 but also has t= he sed commands in it with the same error message. >>>>>>>> >>>>>>>> The second suse version runs successfully with python3, does not hav= e the sed commands and completes the build.sh script with no errors. However = this certdata2pem.py file has a section that is in the IPFire version complet= ely missing. >>>>>>>> >>>>>>>> >>>>>>>> IPfire version after running through the 2to3 convertor >>>>>>>> >>>>>>>> The following error message occurs >>>>>>>> >>>>>>>> producing trust for "GlobalSign Root CA"2.11.4.0.0.0.0.1.21.75.90.19= 5.148 >>>>>>>> Traceback (most recent call last): >>>>>>>> =C2=A0=C2=A0=C2=A0 File "/mnt/File_Server/Computers/Linux/ipfire/san= dbox/patch in progress/python/ca-certificates/orig-2to3-build/certs/../certda= ta2pem.py", line 224, in >>>>>>>> f.write("\n".join(textwrap.wrap(base64.b64encode(obj['CKA_VALUE']), = 64))) >>>>>>>> =C2=A0=C2=A0=C2=A0 File "/usr/lib/python3.9/base64.py", line 58, in = b64encode >>>>>>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 encoded =3D binascii.b2a_base64(s, ne= wline=3DFalse) >>>>>>>> TypeError: a bytes-like object is required, not 'str' >>>>>>>> >>>>>>>> The section that is failing is the section that is missing in the 2n= d suse version. There is an identical fwrite line at line 206 but that does n= ot seem to flag up the same TypeError message. >>>>>>>> >>>>>>>> >>>>>>>> As the certdata2pem.py files from the other distributions vary signi= ficantly in content, with some having nearly double the number of lines of co= de, I think the best alternative is to fix the IPFire version so we stay cons= istent but I am unable to figure out how to fix the python code that is causi= ng the "=C2=A0 TypeError: a bytes-like object is required, not 'str' " error = message and need someone's help with that. >>>>>>>> >>>>>>>> Let me know if there is any other information that I need to provide. >>>>>>>> >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Adolf. >>>>>>>> >>>>>>>> >>>>>>>> On 07/08/2021 15:54, Adolf Belka wrote: >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> On 04/08/2021 16:45, Michael Tremer wrote: >>>>>>>>>> Hello, >>>>>>>>>> >>>>>>>>>>> On 4 Aug 2021, at 13:40, Adolf Belka w= rote: >>>>>>>>>>> >>>>>>>>>>> Hi All, >>>>>>>>>>> >>>>>>>>>>> I have resolved the frr program build. The version currently in I= PFire (6.0) only works with python2. Python3 support came in with version 7.4= . I have now built frr with version 8.0 including libyang as a new dependency= but only for the build, so nothing installed into IPFire itself, and that ha= s successfully built without python2 being present. >>>>>>>>>> >>>>>>>>>> Great. This could also resolve Matthias=E2=80=99 problem with buil= ding frr. >>>>>>>>>> >>>>>>>>>>> Will now go back and have another go with spice-protocol. >>>>>>>>>> >>>>>>>>>> Maybe it has a =E2=80=94-disable-python switch? >>>>>>>>> I just removed the line in the spice-protocol lfs that ran automake= /py-compile on the python modules from spice. >>>>>>>>> Spice and spice-protocol are present for qemu and with the py-compi= le line removed all three successfully built without python2 being present. I= have submitted a patch for this combined with updating spice and spice-proto= col, both from 2017. >>>>>>>>> >>>>>>>>> This now only leaves the ca-certificates script that needs to be up= dated to work with python3. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Adolf. >>>>>>>>>> >>>>>>>>>> -Michael >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> >>>>>>>>>>> Adolf. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 03/08/2021 23:38, Adolf Belka wrote: >>>>>>>>>>>> Hi Michael & all, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 03/08/2021 17:11, Michael Tremer wrote: >>>>>>>>>>>>> Hello, >>>>>>>>>>>>> >>>>>>>>>>>>> Thank you for looking into this. >>>>>>>>>>>>> >>>>>>>>>>>>> This is a third-party script that came from either Mozilla or R= edHat. Maybe they have ported it. If not, it should not be rocket science to = do it ourselves. If we do it, we should of course upstream it. >>>>>>>>>>>> I found an updated script from fedora and gave that a try. This = time the script went all the way through but then the build.sh script failed = at the point where it should find all the .crt files in the certs directory a= nd it came back and said there weren't any. >>>>>>>>>>>>> >>>>>>>>>>>>> However, can you comment out this package and continue the buil= d? This should be required until you reach the cdrom stage. >>>>>>>>>>>> I then commented ca-certificates out in make.sh and ran the buil= d. >>>>>>>>>>>> This time it stopped at spice-protocol which is an addon and use= s the py-compile script that is in automake to compile some python modules. >>>>>>>>>>>> py-compile is python2 based and the build stopped because it cou= ld not find python >>>>>>>>>>>> >>>>>>>>>>>> There is a py_compile.py script that is python3 based but when I= ran that in place of the py-compile script I got a Permission denied error w= hen it tried to carry out the compile. >>>>>>>>>>>> >>>>>>>>>>>> I then commented out spice-protocol and ran the build. >>>>>>>>>>>> >>>>>>>>>>>> It then failed on frr which did look for python3-config but then= failed due to not finding python-config or pkg-config python >>>>>>>>>>>> It looks like I should be able to tell it to use python3 in the = ./configure >>>>>>>>>>>> >>>>>>>>>>>> I commented out frr and nothing else failed before cdrom was rea= ched. >>>>>>>>>>>> >>>>>>>>>>>> So the packages that need to be made to work with python3 are >>>>>>>>>>>> ca-certificates >>>>>>>>>>>> spice-protocol >>>>>>>>>>>> frr >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I also converted client175 with 2to3 converter and built it and = installed the .ipfire package into a vm and successfully got the WUI page for= Media Player IPFire to render. What I haven't tested yet is if the audio wor= ks. I will need to get audio set up in my vm to try that. >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Adolf. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> If this is the only thing that flags up, we should port the scr= ipt. If we find another, stronger reason to keep Python 2 around, we do not n= eed to bother and can keep the script this way. >>>>>>>>>>>>> >>>>>>>>>>>>> -Michael >>>>>>>>>>>>> >>>>>>>>>>>>>> On 3 Aug 2021, at 13:31, Adolf Belka wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>> >>>>>>>>>>>>>> So with crda and the remaining python2 modules removed the que= stion was if removing python2 from the build ran without any problem or if so= mething was flagged up. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> ca-certificates was flagged up. >>>>>>>>>>>>>> >>>>>>>>>>>>>> There is a python2 script, certdata2pem.py, which fails if pyt= hon2 is not present. Running that script with python3 flags up some invalid s= yntax, unsurprisingly. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I found some patches in Debian from 2015 for certdata2pem.py t= o provide python3 compatibility. Unfortunately looking at the patch approx ha= lf could not be applied because the lines don't exist in the IPFire version o= f certdata2pem.py (sections to do with blacklisted certs) >>>>>>>>>>>>>> >>>>>>>>>>>>>> I then ran the 2to3 converter on certdata2pem.py and tried tha= t in the build but it came up with the following error. >>>>>>>>>>>>>> >>>>>>>>>>>>>> TypeError: a bytes-like object is required, not 'str' >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I don't know how to further move forward with this as I am tot= ally unfamiliar with the python language. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Adolf. >> --===============5696223325404466218==--