From mboxrd@z Thu Jan 1 00:00:00 1970 From: Adolf Belka To: development@lists.ipfire.org Subject: Re: First results from running build without python2 Date: Thu, 12 Aug 2021 13:36:17 +0200 Message-ID: <05382a86-67ee-5a10-d35f-eacea5454791@ipfire.org> In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============6191796481272810666==" List-Id: --===============6191796481272810666== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi Michael, On 12/08/2021 11:17, Michael Tremer wrote: > Hello, >=20 >> On 11 Aug 2021, at 15:03, Adolf Belka wrote: >> >> Hi Michael, >> >> On 11/08/2021 12:43, Michael Tremer wrote: >>> Hello, >>> Is this the one with the broken sed command? >>> https://src.fedoraproject.org/rpms/ca-certificates/blob/rawhide/f/cert= data2pem.py >> Yes, this is that one. Confirmed with a diff. >>> This should run if you execute it in the right directory: >>> pushd %{name}/certs >>> pwd >>> cp certdata.txt . >>> python3 certdata2pem.py >>> popd >> I have just learnt about the pushd and popd commands. by doing a quick sea= rch. Never heard of them before. >=20 > It is just a version of =E2=80=9Ccd=E2=80=9D that remembers where it has be= en. >=20 > So if you call =E2=80=9Cpushd some-directory=E2=80=9D, then =E2=80=9Cpopd= =E2=80=9D will bring you back to where you have been before. >=20 > =E2=80=9Ccd -=E2=80=9C does the same as popd now. >=20 >>> The fedora version no longer has the build.sh script. >> That was the bit I didn't realise. >=20 > No problem. Not as simple as I hoped. The new certdata2pem.py script no longer creates .crt files but .tmp-p11-kit = files but that is as far as that script goes. There is an update-ca-trust file in fedora which splits the various certs to = their respective locations, with the openssl ones ending up in ca-bundle.trus= t.crt but this uses p11-kit. I suspect that it uses ca-bundle.trust.p11-kit g= enerated later on in the %build section of the ca-certificate.spec file from = the tmp-p11-kit files generated by the certdata2pem.py script. So it looks li= ke p11-kit needs to be installed to make this work. Alternatively I have found the following in LFS. https://www.linuxfromscratch.org/blfs/view/svn/postlfs/make-ca.html which seems to also create the bundled cert file but also requiring p11-kit b= ut then talking about different certdata.txt versions that have various tunin= gs. The one from LFS comes from the mozilla release branch but is modified to= something called the Mercurial revision, or the different ones shipped by Re= dHat or OpenSUSE which use the version that comes with NSS. All in all I am not sure which approach to use or how to actually build it in= IPFire. Definitely help required to know the correct way to go further with this. Regards, Adolf. =20 > -Michael >=20 >> Regards, >> Adolf. >>> -Michael >>>> On 8 Aug 2021, at 14:47, Adolf Belka wrote: >>>> >>>> Hi All, >>>> >>>> I had another go at the ca-certificates problem, the last barrier to get= ting rid of python2. >>>> >>>> I found certdata2pem.py files from fedora and 2 from suse. I created bui= ld subdirectories for each version so I could just test running the build.sh = file with each version of certdata2pem.py, including the IPFire current versi= on after running through the 2to3 convertor. >>>> >>>> fedora >>>> >>>> The fedora certdata2pem.py file runs successfully with python3 but has s= ed commands built into it which fail to find certain files. The sed commands = are not in the IPFire version. >>>> >>>> The error message is >>>> >>>> -> written as 'Certum_Trusted_Root_CA:2.16.30.191.89.80.184.201.128.55= .76.6.247.235.85.79.181.237.tmp-p11-kit', trust =3D ['CKA_TRUST_SERVER_AUTH',= 'CKA_TRUST_EMAIL_PROTECTION'], openssl-trust =3D ['serverAuth', 'emailProtec= tion'], distrust =3D [], openssl-distrust =3D [] >>>> sed: can't read certs/*.crt: No such file or directory >>>> >>>> >>>> suse >>>> >>>> The first suse version runs successfully with python3 but also has the s= ed commands in it with the same error message. >>>> >>>> The second suse version runs successfully with python3, does not have th= e sed commands and completes the build.sh script with no errors. However this= certdata2pem.py file has a section that is in the IPFire version completely = missing. >>>> >>>> >>>> IPfire version after running through the 2to3 convertor >>>> >>>> The following error message occurs >>>> >>>> producing trust for "GlobalSign Root CA"2.11.4.0.0.0.0.1.21.75.90.195.148 >>>> Traceback (most recent call last): >>>> File "/mnt/File_Server/Computers/Linux/ipfire/sandbox/patch in progre= ss/python/ca-certificates/orig-2to3-build/certs/../certdata2pem.py", line 224= , in >>>> f.write("\n".join(textwrap.wrap(base64.b64encode(obj['CKA_VALUE']), 64))) >>>> File "/usr/lib/python3.9/base64.py", line 58, in b64encode >>>> encoded =3D binascii.b2a_base64(s, newline=3DFalse) >>>> TypeError: a bytes-like object is required, not 'str' >>>> >>>> The section that is failing is the section that is missing in the 2nd su= se version. There is an identical fwrite line at line 206 but that does not s= eem to flag up the same TypeError message. >>>> >>>> >>>> As the certdata2pem.py files from the other distributions vary significa= ntly in content, with some having nearly double the number of lines of code, = I think the best alternative is to fix the IPFire version so we stay consiste= nt but I am unable to figure out how to fix the python code that is causing t= he " TypeError: a bytes-like object is required, not 'str' " error message a= nd need someone's help with that. >>>> >>>> Let me know if there is any other information that I need to provide. >>>> >>>> >>>> Regards, >>>> >>>> Adolf. >>>> >>>> >>>> On 07/08/2021 15:54, Adolf Belka wrote: >>>>> Hi All, >>>>> >>>>> On 04/08/2021 16:45, Michael Tremer wrote: >>>>>> Hello, >>>>>> >>>>>>> On 4 Aug 2021, at 13:40, Adolf Belka wrote: >>>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> I have resolved the frr program build. The version currently in IPFir= e (6.0) only works with python2. Python3 support came in with version 7.4. I = have now built frr with version 8.0 including libyang as a new dependency but= only for the build, so nothing installed into IPFire itself, and that has su= ccessfully built without python2 being present. >>>>>> >>>>>> Great. This could also resolve Matthias=E2=80=99 problem with building= frr. >>>>>> >>>>>>> Will now go back and have another go with spice-protocol. >>>>>> >>>>>> Maybe it has a =E2=80=94-disable-python switch? >>>>> I just removed the line in the spice-protocol lfs that ran automake/py-= compile on the python modules from spice. >>>>> Spice and spice-protocol are present for qemu and with the py-compile l= ine removed all three successfully built without python2 being present. I hav= e submitted a patch for this combined with updating spice and spice-protocol,= both from 2017. >>>>> >>>>> This now only leaves the ca-certificates script that needs to be update= d to work with python3. >>>>> >>>>> Regards, >>>>> Adolf. >>>>>> >>>>>> -Michael >>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Adolf. >>>>>>> >>>>>>> >>>>>>> On 03/08/2021 23:38, Adolf Belka wrote: >>>>>>>> Hi Michael & all, >>>>>>>> >>>>>>>> >>>>>>>> On 03/08/2021 17:11, Michael Tremer wrote: >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> Thank you for looking into this. >>>>>>>>> >>>>>>>>> This is a third-party script that came from either Mozilla or RedHa= t. Maybe they have ported it. If not, it should not be rocket science to do i= t ourselves. If we do it, we should of course upstream it. >>>>>>>> I found an updated script from fedora and gave that a try. This time= the script went all the way through but then the build.sh script failed at t= he point where it should find all the .crt files in the certs directory and i= t came back and said there weren't any. >>>>>>>>> >>>>>>>>> However, can you comment out this package and continue the build? T= his should be required until you reach the cdrom stage. >>>>>>>> I then commented ca-certificates out in make.sh and ran the build. >>>>>>>> This time it stopped at spice-protocol which is an addon and uses th= e py-compile script that is in automake to compile some python modules. >>>>>>>> py-compile is python2 based and the build stopped because it could n= ot find python >>>>>>>> >>>>>>>> There is a py_compile.py script that is python3 based but when I ran= that in place of the py-compile script I got a Permission denied error when = it tried to carry out the compile. >>>>>>>> >>>>>>>> I then commented out spice-protocol and ran the build. >>>>>>>> >>>>>>>> It then failed on frr which did look for python3-config but then fai= led due to not finding python-config or pkg-config python >>>>>>>> It looks like I should be able to tell it to use python3 in the ./co= nfigure >>>>>>>> >>>>>>>> I commented out frr and nothing else failed before cdrom was reached. >>>>>>>> >>>>>>>> So the packages that need to be made to work with python3 are >>>>>>>> ca-certificates >>>>>>>> spice-protocol >>>>>>>> frr >>>>>>>> >>>>>>>> >>>>>>>> I also converted client175 with 2to3 converter and built it and inst= alled the .ipfire package into a vm and successfully got the WUI page for Med= ia Player IPFire to render. What I haven't tested yet is if the audio works. = I will need to get audio set up in my vm to try that. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Adolf. >>>>>>>> >>>>>>>>> >>>>>>>>> If this is the only thing that flags up, we should port the script.= If we find another, stronger reason to keep Python 2 around, we do not need = to bother and can keep the script this way. >>>>>>>>> >>>>>>>>> -Michael >>>>>>>>> >>>>>>>>>> On 3 Aug 2021, at 13:31, Adolf Belka wr= ote: >>>>>>>>>> >>>>>>>>>> Hi All, >>>>>>>>>> >>>>>>>>>> So with crda and the remaining python2 modules removed the questio= n was if removing python2 from the build ran without any problem or if someth= ing was flagged up. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ca-certificates was flagged up. >>>>>>>>>> >>>>>>>>>> There is a python2 script, certdata2pem.py, which fails if python2= is not present. Running that script with python3 flags up some invalid synta= x, unsurprisingly. >>>>>>>>>> >>>>>>>>>> I found some patches in Debian from 2015 for certdata2pem.py to pr= ovide python3 compatibility. Unfortunately looking at the patch approx half c= ould not be applied because the lines don't exist in the IPFire version of ce= rtdata2pem.py (sections to do with blacklisted certs) >>>>>>>>>> >>>>>>>>>> I then ran the 2to3 converter on certdata2pem.py and tried that in= the build but it came up with the following error. >>>>>>>>>> >>>>>>>>>> TypeError: a bytes-like object is required, not 'str' >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I don't know how to further move forward with this as I am totally= unfamiliar with the python language. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> >>>>>>>>>> Adolf. >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>> >=20 --===============6191796481272810666==--