From mboxrd@z Thu Jan 1 00:00:00 1970 From: Adolf Belka To: development@lists.ipfire.org Subject: Re: First results from running build without python2 Date: Thu, 12 Aug 2021 14:10:33 +0200 Message-ID: In-Reply-To: <05382a86-67ee-5a10-d35f-eacea5454791@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============7548015341104574212==" List-Id: --===============7548015341104574212== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi Michael, On 12/08/2021 13:36, Adolf Belka wrote: > Hi Michael, > > On 12/08/2021 11:17, Michael Tremer wrote: >> Hello, >> >>> On 11 Aug 2021, at 15:03, Adolf Belka wrote: >>> >>> Hi Michael, >>> >>> On 11/08/2021 12:43, Michael Tremer wrote: >>>> Hello, >>>> Is this the one with the broken sed command? >>>> https://src.fedoraproject.org/rpms/ca-certificates/blob/rawhide/f/certda= ta2pem.py >>> Yes, this is that one. Confirmed with a diff. >>>> This should run if you execute it in the right directory: >>>> pushd %{name}/certs >>>> =C2=A0 pwd >>>> =C2=A0 cp certdata.txt . >>>> =C2=A0 python3 certdata2pem.py >>>> popd >>> I have just learnt about the pushd and popd commands. by doing a quick se= arch. Never heard of them before. >> >> It is just a version of =E2=80=9Ccd=E2=80=9D that remembers where it has b= een. >> >> So if you call =E2=80=9Cpushd some-directory=E2=80=9D, then =E2=80=9Cpopd= =E2=80=9D will bring you back to where you have been before. >> >> =E2=80=9Ccd -=E2=80=9C does the same as popd now. >> >>>> The fedora version no longer has the build.sh script. >>> That was the bit I didn't realise. >> >> No problem. > > Not as simple as I hoped. > > The new certdata2pem.py script no longer creates .crt files but .tmp-p11-ki= t files but that is as far as that script goes. > > There is an update-ca-trust file in fedora which splits the various certs t= o their respective locations, with the openssl ones ending up in ca-bundle.tr= ust.crt but this uses p11-kit. I suspect that it uses ca-bundle.trust.p11-kit= generated later on in the %build section of the ca-certificate.spec file fro= m the tmp-p11-kit files generated by the certdata2pem.py script. So it looks = like p11-kit needs to be installed to make this work. > > Alternatively I have found the following in LFS. > https://www.linuxfromscratch.org/blfs/view/svn/postlfs/make-ca.html > which seems to also create the bundled cert file but also requiring p11-kit= but then talking about different certdata.txt versions that have various tun= ings. The one from LFS comes from the mozilla release branch but is modified = to something called the Mercurial revision, or the different ones shipped by = RedHat or OpenSUSE which use the version that comes with NSS. > > All in all I am not sure which approach to use or how to actually build it = in IPFire. > > Definitely help required to know the correct way to go further with this. > Having said I needed help, I thought I would have another go and copied the l= ines from the %build section of the fedora ca-certificate.spec that created t= he ca-bundle.trust.p11.kit As I have p11-kit installed on my Arch Linux system I then ran the command p11-kit extract --format=3Dopenssl-bundle --filter=3Dcertificates --overwrite= --comment ~/openssl/ca-bundle.trust.crt and I successfully created the ca-bundle.trust.crt which is needed by the IPF= ire ca-certificates lfs file. I will now try and create a new build script that will do all the new type st= uff and also install the p11-kit library files and see how things go. :crossed_fingers: Adolf > Regards, > Adolf. > >> -Michael >> >>> Regards, >>> Adolf. >>>> -Michael >>>>> On 8 Aug 2021, at 14:47, Adolf Belka wrote: >>>>> >>>>> Hi All, >>>>> >>>>> I had another go at the ca-certificates problem, the last barrier to ge= tting rid of python2. >>>>> >>>>> I found certdata2pem.py files from fedora and 2 from suse. I created bu= ild subdirectories for each version so I could just test running the build.sh= file with each version of certdata2pem.py, including the IPFire current vers= ion after running through the 2to3 convertor. >>>>> >>>>> fedora >>>>> >>>>> The fedora certdata2pem.py file runs successfully with python3 but has = sed commands built into it which fail to find certain files. The sed commands= are not in the IPFire version. >>>>> >>>>> The error message is >>>>> >>>>> =C2=A0 -> written as 'Certum_Trusted_Root_CA:2.16.30.191.89.80.184.201.= 128.55.76.6.247.235.85.79.181.237.tmp-p11-kit', trust =3D ['CKA_TRUST_SERVER_= AUTH', 'CKA_TRUST_EMAIL_PROTECTION'], openssl-trust =3D ['serverAuth', 'email= Protection'], distrust =3D [], openssl-distrust =3D [] >>>>> sed: can't read certs/*.crt: No such file or directory >>>>> >>>>> >>>>> suse >>>>> >>>>> The first suse version runs successfully with python3 but also has the = sed commands in it with the same error message. >>>>> >>>>> The second suse version runs successfully with python3, does not have t= he sed commands and completes the build.sh script with no errors. However thi= s certdata2pem.py file has a section that is in the IPFire version completely= missing. >>>>> >>>>> >>>>> IPfire version after running through the 2to3 convertor >>>>> >>>>> The following error message occurs >>>>> >>>>> producing trust for "GlobalSign Root CA"2.11.4.0.0.0.0.1.21.75.90.195.1= 48 >>>>> Traceback (most recent call last): >>>>> =C2=A0=C2=A0 File "/mnt/File_Server/Computers/Linux/ipfire/sandbox/patc= h in progress/python/ca-certificates/orig-2to3-build/certs/../certdata2pem.py= ", line 224, in >>>>> f.write("\n".join(textwrap.wrap(base64.b64encode(obj['CKA_VALUE']), 64)= )) >>>>> =C2=A0=C2=A0 File "/usr/lib/python3.9/base64.py", line 58, in b64encode >>>>> =C2=A0=C2=A0=C2=A0=C2=A0 encoded =3D binascii.b2a_base64(s, newline=3DF= alse) >>>>> TypeError: a bytes-like object is required, not 'str' >>>>> >>>>> The section that is failing is the section that is missing in the 2nd s= use version. There is an identical fwrite line at line 206 but that does not = seem to flag up the same TypeError message. >>>>> >>>>> >>>>> As the certdata2pem.py files from the other distributions vary signific= antly in content, with some having nearly double the number of lines of code,= I think the best alternative is to fix the IPFire version so we stay consist= ent but I am unable to figure out how to fix the python code that is causing = the "=C2=A0 TypeError: a bytes-like object is required, not 'str' " error mes= sage and need someone's help with that. >>>>> >>>>> Let me know if there is any other information that I need to provide. >>>>> >>>>> >>>>> Regards, >>>>> >>>>> Adolf. >>>>> >>>>> >>>>> On 07/08/2021 15:54, Adolf Belka wrote: >>>>>> Hi All, >>>>>> >>>>>> On 04/08/2021 16:45, Michael Tremer wrote: >>>>>>> Hello, >>>>>>> >>>>>>>> On 4 Aug 2021, at 13:40, Adolf Belka wrot= e: >>>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> I have resolved the frr program build. The version currently in IPFi= re (6.0) only works with python2. Python3 support came in with version 7.4. I= have now built frr with version 8.0 including libyang as a new dependency bu= t only for the build, so nothing installed into IPFire itself, and that has s= uccessfully built without python2 being present. >>>>>>> >>>>>>> Great. This could also resolve Matthias=E2=80=99 problem with buildin= g frr. >>>>>>> >>>>>>>> Will now go back and have another go with spice-protocol. >>>>>>> >>>>>>> Maybe it has a =E2=80=94-disable-python switch? >>>>>> I just removed the line in the spice-protocol lfs that ran automake/py= -compile on the python modules from spice. >>>>>> Spice and spice-protocol are present for qemu and with the py-compile = line removed all three successfully built without python2 being present. I ha= ve submitted a patch for this combined with updating spice and spice-protocol= , both from 2017. >>>>>> >>>>>> This now only leaves the ca-certificates script that needs to be updat= ed to work with python3. >>>>>> >>>>>> Regards, >>>>>> Adolf. >>>>>>> >>>>>>> -Michael >>>>>>> >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Adolf. >>>>>>>> >>>>>>>> >>>>>>>> On 03/08/2021 23:38, Adolf Belka wrote: >>>>>>>>> Hi Michael & all, >>>>>>>>> >>>>>>>>> >>>>>>>>> On 03/08/2021 17:11, Michael Tremer wrote: >>>>>>>>>> Hello, >>>>>>>>>> >>>>>>>>>> Thank you for looking into this. >>>>>>>>>> >>>>>>>>>> This is a third-party script that came from either Mozilla or RedH= at. Maybe they have ported it. If not, it should not be rocket science to do = it ourselves. If we do it, we should of course upstream it. >>>>>>>>> I found an updated script from fedora and gave that a try. This tim= e the script went all the way through but then the build.sh script failed at = the point where it should find all the .crt files in the certs directory and = it came back and said there weren't any. >>>>>>>>>> >>>>>>>>>> However, can you comment out this package and continue the build? = This should be required until you reach the cdrom stage. >>>>>>>>> I then commented ca-certificates out in make.sh and ran the build. >>>>>>>>> This time it stopped at spice-protocol which is an addon and uses t= he py-compile script that is in automake to compile some python modules. >>>>>>>>> py-compile is python2 based and the build stopped because it could = not find python >>>>>>>>> >>>>>>>>> There is a py_compile.py script that is python3 based but when I ra= n that in place of the py-compile script I got a Permission denied error when= it tried to carry out the compile. >>>>>>>>> >>>>>>>>> I then commented out spice-protocol and ran the build. >>>>>>>>> >>>>>>>>> It then failed on frr which did look for python3-config but then fa= iled due to not finding python-config or pkg-config python >>>>>>>>> It looks like I should be able to tell it to use python3 in the ./c= onfigure >>>>>>>>> >>>>>>>>> I commented out frr and nothing else failed before cdrom was reache= d. >>>>>>>>> >>>>>>>>> So the packages that need to be made to work with python3 are >>>>>>>>> ca-certificates >>>>>>>>> spice-protocol >>>>>>>>> frr >>>>>>>>> >>>>>>>>> >>>>>>>>> I also converted client175 with 2to3 converter and built it and ins= talled the .ipfire package into a vm and successfully got the WUI page for Me= dia Player IPFire to render. What I haven't tested yet is if the audio works.= I will need to get audio set up in my vm to try that. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Adolf. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> If this is the only thing that flags up, we should port the script= . If we find another, stronger reason to keep Python 2 around, we do not need= to bother and can keep the script this way. >>>>>>>>>> >>>>>>>>>> -Michael >>>>>>>>>> >>>>>>>>>>> On 3 Aug 2021, at 13:31, Adolf Belka w= rote: >>>>>>>>>>> >>>>>>>>>>> Hi All, >>>>>>>>>>> >>>>>>>>>>> So with crda and the remaining python2 modules removed the questi= on was if removing python2 from the build ran without any problem or if somet= hing was flagged up. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ca-certificates was flagged up. >>>>>>>>>>> >>>>>>>>>>> There is a python2 script, certdata2pem.py, which fails if python= 2 is not present. Running that script with python3 flags up some invalid synt= ax, unsurprisingly. >>>>>>>>>>> >>>>>>>>>>> I found some patches in Debian from 2015 for certdata2pem.py to p= rovide python3 compatibility. Unfortunately looking at the patch approx half = could not be applied because the lines don't exist in the IPFire version of c= ertdata2pem.py (sections to do with blacklisted certs) >>>>>>>>>>> >>>>>>>>>>> I then ran the 2to3 converter on certdata2pem.py and tried that i= n the build but it came up with the following error. >>>>>>>>>>> >>>>>>>>>>> TypeError: a bytes-like object is required, not 'str' >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I don't know how to further move forward with this as I am totall= y unfamiliar with the python language. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> >>>>>>>>>>> Adolf. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>> >> --===============7548015341104574212==--