From mboxrd@z Thu Jan 1 00:00:00 1970 From: Adolf Belka To: development@lists.ipfire.org Subject: Re: First results from running build without python2 Date: Thu, 12 Aug 2021 21:23:38 +0200 Message-ID: In-Reply-To: <55E86324-9EB7-4C3E-B29E-BA16B03FB1E9@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1950210533852696297==" List-Id: --===============1950210533852696297== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi Michael, On 12/08/2021 15:38, Michael Tremer wrote: > Hello, >=20 > Yes, this is the way to go :) >=20 > Reach out here if you need any help. I am trying to build p11-kit and it has flagged up libtasn1. It is an optiona= l dependency and I can disable it but I would like to check if that is the co= rrect thing to do or not. libtasn1 is said to be required to allow the trust policy module to be built = as well as other code that interacts with certificates. Is this something that would be needed for IPFire or can I disable it in the = p11-kit build. Regards, Adolf. >=20 > -Michael >=20 >> On 12 Aug 2021, at 13:10, Adolf Belka wrote: >> >> Hi Michael, >> >> On 12/08/2021 13:36, Adolf Belka wrote: >>> Hi Michael, >>> >>> On 12/08/2021 11:17, Michael Tremer wrote: >>>> Hello, >>>> >>>>> On 11 Aug 2021, at 15:03, Adolf Belka wrote: >>>>> >>>>> Hi Michael, >>>>> >>>>> On 11/08/2021 12:43, Michael Tremer wrote: >>>>>> Hello, >>>>>> Is this the one with the broken sed command? >>>>>> https://src.fedoraproject.org/rpms/ca-certificates/blob/rawhide/f/cert= data2pem.py >>>>> Yes, this is that one. Confirmed with a diff. >>>>>> This should run if you execute it in the right directory: >>>>>> pushd %{name}/certs >>>>>> pwd >>>>>> cp certdata.txt . >>>>>> python3 certdata2pem.py >>>>>> popd >>>>> I have just learnt about the pushd and popd commands. by doing a quick = search. Never heard of them before. >>>> >>>> It is just a version of =E2=80=9Ccd=E2=80=9D that remembers where it has= been. >>>> >>>> So if you call =E2=80=9Cpushd some-directory=E2=80=9D, then =E2=80=9Cpop= d=E2=80=9D will bring you back to where you have been before. >>>> >>>> =E2=80=9Ccd -=E2=80=9C does the same as popd now. >>>> >>>>>> The fedora version no longer has the build.sh script. >>>>> That was the bit I didn't realise. >>>> >>>> No problem. >>> >>> Not as simple as I hoped. >>> >>> The new certdata2pem.py script no longer creates .crt files but .tmp-p11-= kit files but that is as far as that script goes. >>> >>> There is an update-ca-trust file in fedora which splits the various certs= to their respective locations, with the openssl ones ending up in ca-bundle.= trust.crt but this uses p11-kit. I suspect that it uses ca-bundle.trust.p11-k= it generated later on in the %build section of the ca-certificate.spec file f= rom the tmp-p11-kit files generated by the certdata2pem.py script. So it look= s like p11-kit needs to be installed to make this work. >>> >>> Alternatively I have found the following in LFS. >>> https://www.linuxfromscratch.org/blfs/view/svn/postlfs/make-ca.html >>> which seems to also create the bundled cert file but also requiring p11-k= it but then talking about different certdata.txt versions that have various t= unings. The one from LFS comes from the mozilla release branch but is modifie= d to something called the Mercurial revision, or the different ones shipped b= y RedHat or OpenSUSE which use the version that comes with NSS. >>> >>> All in all I am not sure which approach to use or how to actually build i= t in IPFire. >>> >>> Definitely help required to know the correct way to go further with this. >>> >> Having said I needed help, I thought I would have another go and copied th= e lines from the %build section of the fedora ca-certificate.spec that create= d the ca-bundle.trust.p11.kit >> >> As I have p11-kit installed on my Arch Linux system I then ran the command >> >> p11-kit extract --format=3Dopenssl-bundle --filter=3Dcertificates --overwr= ite --comment ~/openssl/ca-bundle.trust.crt >> >> and I successfully created the ca-bundle.trust.crt which is needed by the = IPFire ca-certificates lfs file. >> >> I will now try and create a new build script that will do all the new type= stuff and also install the p11-kit library files and see how things go. >> >> :crossed_fingers: >> >> Adolf >> >>> Regards, >>> Adolf. >>> >>>> -Michael >>>> >>>>> Regards, >>>>> Adolf. >>>>>> -Michael >>>>>>> On 8 Aug 2021, at 14:47, Adolf Belka wrote: >>>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> I had another go at the ca-certificates problem, the last barrier to = getting rid of python2. >>>>>>> >>>>>>> I found certdata2pem.py files from fedora and 2 from suse. I created = build subdirectories for each version so I could just test running the build.= sh file with each version of certdata2pem.py, including the IPFire current ve= rsion after running through the 2to3 convertor. >>>>>>> >>>>>>> fedora >>>>>>> >>>>>>> The fedora certdata2pem.py file runs successfully with python3 but ha= s sed commands built into it which fail to find certain files. The sed comman= ds are not in the IPFire version. >>>>>>> >>>>>>> The error message is >>>>>>> >>>>>>> -> written as 'Certum_Trusted_Root_CA:2.16.30.191.89.80.184.201.12= 8.55.76.6.247.235.85.79.181.237.tmp-p11-kit', trust =3D ['CKA_TRUST_SERVER_AU= TH', 'CKA_TRUST_EMAIL_PROTECTION'], openssl-trust =3D ['serverAuth', 'emailPr= otection'], distrust =3D [], openssl-distrust =3D [] >>>>>>> sed: can't read certs/*.crt: No such file or directory >>>>>>> >>>>>>> >>>>>>> suse >>>>>>> >>>>>>> The first suse version runs successfully with python3 but also has th= e sed commands in it with the same error message. >>>>>>> >>>>>>> The second suse version runs successfully with python3, does not have= the sed commands and completes the build.sh script with no errors. However t= his certdata2pem.py file has a section that is in the IPFire version complete= ly missing. >>>>>>> >>>>>>> >>>>>>> IPfire version after running through the 2to3 convertor >>>>>>> >>>>>>> The following error message occurs >>>>>>> >>>>>>> producing trust for "GlobalSign Root CA"2.11.4.0.0.0.0.1.21.75.90.195= .148 >>>>>>> Traceback (most recent call last): >>>>>>> File "/mnt/File_Server/Computers/Linux/ipfire/sandbox/patch in pr= ogress/python/ca-certificates/orig-2to3-build/certs/../certdata2pem.py", line= 224, in >>>>>>> f.write("\n".join(textwrap.wrap(base64.b64encode(obj['CKA_VALUE']), 6= 4))) >>>>>>> File "/usr/lib/python3.9/base64.py", line 58, in b64encode >>>>>>> encoded =3D binascii.b2a_base64(s, newline=3DFalse) >>>>>>> TypeError: a bytes-like object is required, not 'str' >>>>>>> >>>>>>> The section that is failing is the section that is missing in the 2nd= suse version. There is an identical fwrite line at line 206 but that does no= t seem to flag up the same TypeError message. >>>>>>> >>>>>>> >>>>>>> As the certdata2pem.py files from the other distributions vary signif= icantly in content, with some having nearly double the number of lines of cod= e, I think the best alternative is to fix the IPFire version so we stay consi= stent but I am unable to figure out how to fix the python code that is causin= g the " TypeError: a bytes-like object is required, not 'str' " error messag= e and need someone's help with that. >>>>>>> >>>>>>> Let me know if there is any other information that I need to provide. >>>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Adolf. >>>>>>> >>>>>>> >>>>>>> On 07/08/2021 15:54, Adolf Belka wrote: >>>>>>>> Hi All, >>>>>>>> >>>>>>>> On 04/08/2021 16:45, Michael Tremer wrote: >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>>> On 4 Aug 2021, at 13:40, Adolf Belka wr= ote: >>>>>>>>>> >>>>>>>>>> Hi All, >>>>>>>>>> >>>>>>>>>> I have resolved the frr program build. The version currently in IP= Fire (6.0) only works with python2. Python3 support came in with version 7.4.= I have now built frr with version 8.0 including libyang as a new dependency = but only for the build, so nothing installed into IPFire itself, and that has= successfully built without python2 being present. >>>>>>>>> >>>>>>>>> Great. This could also resolve Matthias=E2=80=99 problem with build= ing frr. >>>>>>>>> >>>>>>>>>> Will now go back and have another go with spice-protocol. >>>>>>>>> >>>>>>>>> Maybe it has a =E2=80=94-disable-python switch? >>>>>>>> I just removed the line in the spice-protocol lfs that ran automake/= py-compile on the python modules from spice. >>>>>>>> Spice and spice-protocol are present for qemu and with the py-compil= e line removed all three successfully built without python2 being present. I = have submitted a patch for this combined with updating spice and spice-protoc= ol, both from 2017. >>>>>>>> >>>>>>>> This now only leaves the ca-certificates script that needs to be upd= ated to work with python3. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Adolf. >>>>>>>>> >>>>>>>>> -Michael >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> >>>>>>>>>> Adolf. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 03/08/2021 23:38, Adolf Belka wrote: >>>>>>>>>>> Hi Michael & all, >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 03/08/2021 17:11, Michael Tremer wrote: >>>>>>>>>>>> Hello, >>>>>>>>>>>> >>>>>>>>>>>> Thank you for looking into this. >>>>>>>>>>>> >>>>>>>>>>>> This is a third-party script that came from either Mozilla or Re= dHat. Maybe they have ported it. If not, it should not be rocket science to d= o it ourselves. If we do it, we should of course upstream it. >>>>>>>>>>> I found an updated script from fedora and gave that a try. This t= ime the script went all the way through but then the build.sh script failed a= t the point where it should find all the .crt files in the certs directory an= d it came back and said there weren't any. >>>>>>>>>>>> >>>>>>>>>>>> However, can you comment out this package and continue the build= ? This should be required until you reach the cdrom stage. >>>>>>>>>>> I then commented ca-certificates out in make.sh and ran the build. >>>>>>>>>>> This time it stopped at spice-protocol which is an addon and uses= the py-compile script that is in automake to compile some python modules. >>>>>>>>>>> py-compile is python2 based and the build stopped because it coul= d not find python >>>>>>>>>>> >>>>>>>>>>> There is a py_compile.py script that is python3 based but when I = ran that in place of the py-compile script I got a Permission denied error wh= en it tried to carry out the compile. >>>>>>>>>>> >>>>>>>>>>> I then commented out spice-protocol and ran the build. >>>>>>>>>>> >>>>>>>>>>> It then failed on frr which did look for python3-config but then = failed due to not finding python-config or pkg-config python >>>>>>>>>>> It looks like I should be able to tell it to use python3 in the .= /configure >>>>>>>>>>> >>>>>>>>>>> I commented out frr and nothing else failed before cdrom was reac= hed. >>>>>>>>>>> >>>>>>>>>>> So the packages that need to be made to work with python3 are >>>>>>>>>>> ca-certificates >>>>>>>>>>> spice-protocol >>>>>>>>>>> frr >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I also converted client175 with 2to3 converter and built it and i= nstalled the .ipfire package into a vm and successfully got the WUI page for = Media Player IPFire to render. What I haven't tested yet is if the audio work= s. I will need to get audio set up in my vm to try that. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Adolf. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> If this is the only thing that flags up, we should port the scri= pt. If we find another, stronger reason to keep Python 2 around, we do not ne= ed to bother and can keep the script this way. >>>>>>>>>>>> >>>>>>>>>>>> -Michael >>>>>>>>>>>> >>>>>>>>>>>>> On 3 Aug 2021, at 13:31, Adolf Belka = wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi All, >>>>>>>>>>>>> >>>>>>>>>>>>> So with crda and the remaining python2 modules removed the ques= tion was if removing python2 from the build ran without any problem or if som= ething was flagged up. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> ca-certificates was flagged up. >>>>>>>>>>>>> >>>>>>>>>>>>> There is a python2 script, certdata2pem.py, which fails if pyth= on2 is not present. Running that script with python3 flags up some invalid sy= ntax, unsurprisingly. >>>>>>>>>>>>> >>>>>>>>>>>>> I found some patches in Debian from 2015 for certdata2pem.py to= provide python3 compatibility. Unfortunately looking at the patch approx hal= f could not be applied because the lines don't exist in the IPFire version of= certdata2pem.py (sections to do with blacklisted certs) >>>>>>>>>>>>> >>>>>>>>>>>>> I then ran the 2to3 converter on certdata2pem.py and tried that= in the build but it came up with the following error. >>>>>>>>>>>>> >>>>>>>>>>>>> TypeError: a bytes-like object is required, not 'str' >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I don't know how to further move forward with this as I am tota= lly unfamiliar with the python language. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> >>>>>>>>>>>>> Adolf. >=20 --===============1950210533852696297==--