From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: First results from running build without python2 Date: Thu, 12 Aug 2021 10:17:37 +0100 Message-ID: In-Reply-To: <12415ddf-2601-7d1e-fd05-8b0388177df2@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============6653822116687522206==" List-Id: --===============6653822116687522206== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello, > On 11 Aug 2021, at 15:03, Adolf Belka wrote: >=20 > Hi Michael, >=20 > On 11/08/2021 12:43, Michael Tremer wrote: >> Hello, >> Is this the one with the broken sed command? >> https://src.fedoraproject.org/rpms/ca-certificates/blob/rawhide/f/certda= ta2pem.py > Yes, this is that one. Confirmed with a diff. >> This should run if you execute it in the right directory: >> pushd %{name}/certs >> pwd >> cp certdata.txt . >> python3 certdata2pem.py >> popd > I have just learnt about the pushd and popd commands. by doing a quick sear= ch. Never heard of them before. It is just a version of =E2=80=9Ccd=E2=80=9D that remembers where it has been. So if you call =E2=80=9Cpushd some-directory=E2=80=9D, then =E2=80=9Cpopd=E2= =80=9D will bring you back to where you have been before. =E2=80=9Ccd -=E2=80=9C does the same as popd now. >> The fedora version no longer has the build.sh script. > That was the bit I didn't realise. No problem. -Michael > Regards, > Adolf. >> -Michael >>> On 8 Aug 2021, at 14:47, Adolf Belka wrote: >>>=20 >>> Hi All, >>>=20 >>> I had another go at the ca-certificates problem, the last barrier to gett= ing rid of python2. >>>=20 >>> I found certdata2pem.py files from fedora and 2 from suse. I created buil= d subdirectories for each version so I could just test running the build.sh f= ile with each version of certdata2pem.py, including the IPFire current versio= n after running through the 2to3 convertor. >>>=20 >>> fedora >>>=20 >>> The fedora certdata2pem.py file runs successfully with python3 but has se= d commands built into it which fail to find certain files. The sed commands a= re not in the IPFire version. >>>=20 >>> The error message is >>>=20 >>> -> written as 'Certum_Trusted_Root_CA:2.16.30.191.89.80.184.201.128.55.7= 6.6.247.235.85.79.181.237.tmp-p11-kit', trust =3D ['CKA_TRUST_SERVER_AUTH', '= CKA_TRUST_EMAIL_PROTECTION'], openssl-trust =3D ['serverAuth', 'emailProtecti= on'], distrust =3D [], openssl-distrust =3D [] >>> sed: can't read certs/*.crt: No such file or directory >>>=20 >>>=20 >>> suse >>>=20 >>> The first suse version runs successfully with python3 but also has the se= d commands in it with the same error message. >>>=20 >>> The second suse version runs successfully with python3, does not have the= sed commands and completes the build.sh script with no errors. However this = certdata2pem.py file has a section that is in the IPFire version completely m= issing. >>>=20 >>>=20 >>> IPfire version after running through the 2to3 convertor >>>=20 >>> The following error message occurs >>>=20 >>> producing trust for "GlobalSign Root CA"2.11.4.0.0.0.0.1.21.75.90.195.148 >>> Traceback (most recent call last): >>> File "/mnt/File_Server/Computers/Linux/ipfire/sandbox/patch in progress= /python/ca-certificates/orig-2to3-build/certs/../certdata2pem.py", line 224, = in >>> f.write("\n".join(textwrap.wrap(base64.b64encode(obj['CKA_VALUE']), 64))) >>> File "/usr/lib/python3.9/base64.py", line 58, in b64encode >>> encoded =3D binascii.b2a_base64(s, newline=3DFalse) >>> TypeError: a bytes-like object is required, not 'str' >>>=20 >>> The section that is failing is the section that is missing in the 2nd sus= e version. There is an identical fwrite line at line 206 but that does not se= em to flag up the same TypeError message. >>>=20 >>>=20 >>> As the certdata2pem.py files from the other distributions vary significan= tly in content, with some having nearly double the number of lines of code, I= think the best alternative is to fix the IPFire version so we stay consisten= t but I am unable to figure out how to fix the python code that is causing th= e " TypeError: a bytes-like object is required, not 'str' " error message an= d need someone's help with that. >>>=20 >>> Let me know if there is any other information that I need to provide. >>>=20 >>>=20 >>> Regards, >>>=20 >>> Adolf. >>>=20 >>>=20 >>> On 07/08/2021 15:54, Adolf Belka wrote: >>>> Hi All, >>>>=20 >>>> On 04/08/2021 16:45, Michael Tremer wrote: >>>>> Hello, >>>>>=20 >>>>>> On 4 Aug 2021, at 13:40, Adolf Belka wrote: >>>>>>=20 >>>>>> Hi All, >>>>>>=20 >>>>>> I have resolved the frr program build. The version currently in IPFire= (6.0) only works with python2. Python3 support came in with version 7.4. I h= ave now built frr with version 8.0 including libyang as a new dependency but = only for the build, so nothing installed into IPFire itself, and that has suc= cessfully built without python2 being present. >>>>>=20 >>>>> Great. This could also resolve Matthias=E2=80=99 problem with building = frr. >>>>>=20 >>>>>> Will now go back and have another go with spice-protocol. >>>>>=20 >>>>> Maybe it has a =E2=80=94-disable-python switch? >>>> I just removed the line in the spice-protocol lfs that ran automake/py-c= ompile on the python modules from spice. >>>> Spice and spice-protocol are present for qemu and with the py-compile li= ne removed all three successfully built without python2 being present. I have= submitted a patch for this combined with updating spice and spice-protocol, = both from 2017. >>>>=20 >>>> This now only leaves the ca-certificates script that needs to be updated= to work with python3. >>>>=20 >>>> Regards, >>>> Adolf. >>>>>=20 >>>>> -Michael >>>>>=20 >>>>>>=20 >>>>>> Regards, >>>>>>=20 >>>>>> Adolf. >>>>>>=20 >>>>>>=20 >>>>>> On 03/08/2021 23:38, Adolf Belka wrote: >>>>>>> Hi Michael & all, >>>>>>>=20 >>>>>>>=20 >>>>>>> On 03/08/2021 17:11, Michael Tremer wrote: >>>>>>>> Hello, >>>>>>>>=20 >>>>>>>> Thank you for looking into this. >>>>>>>>=20 >>>>>>>> This is a third-party script that came from either Mozilla or RedHat= . Maybe they have ported it. If not, it should not be rocket science to do it= ourselves. If we do it, we should of course upstream it. >>>>>>> I found an updated script from fedora and gave that a try. This time = the script went all the way through but then the build.sh script failed at th= e point where it should find all the .crt files in the certs directory and it= came back and said there weren't any. >>>>>>>>=20 >>>>>>>> However, can you comment out this package and continue the build? Th= is should be required until you reach the cdrom stage. >>>>>>> I then commented ca-certificates out in make.sh and ran the build. >>>>>>> This time it stopped at spice-protocol which is an addon and uses the= py-compile script that is in automake to compile some python modules. >>>>>>> py-compile is python2 based and the build stopped because it could no= t find python >>>>>>>=20 >>>>>>> There is a py_compile.py script that is python3 based but when I ran = that in place of the py-compile script I got a Permission denied error when i= t tried to carry out the compile. >>>>>>>=20 >>>>>>> I then commented out spice-protocol and ran the build. >>>>>>>=20 >>>>>>> It then failed on frr which did look for python3-config but then fail= ed due to not finding python-config or pkg-config python >>>>>>> It looks like I should be able to tell it to use python3 in the ./con= figure >>>>>>>=20 >>>>>>> I commented out frr and nothing else failed before cdrom was reached. >>>>>>>=20 >>>>>>> So the packages that need to be made to work with python3 are >>>>>>> ca-certificates >>>>>>> spice-protocol >>>>>>> frr >>>>>>>=20 >>>>>>>=20 >>>>>>> I also converted client175 with 2to3 converter and built it and insta= lled the .ipfire package into a vm and successfully got the WUI page for Medi= a Player IPFire to render. What I haven't tested yet is if the audio works. I= will need to get audio set up in my vm to try that. >>>>>>>=20 >>>>>>> Regards, >>>>>>> Adolf. >>>>>>>=20 >>>>>>>>=20 >>>>>>>> If this is the only thing that flags up, we should port the script. = If we find another, stronger reason to keep Python 2 around, we do not need t= o bother and can keep the script this way. >>>>>>>>=20 >>>>>>>> -Michael >>>>>>>>=20 >>>>>>>>> On 3 Aug 2021, at 13:31, Adolf Belka wro= te: >>>>>>>>>=20 >>>>>>>>> Hi All, >>>>>>>>>=20 >>>>>>>>> So with crda and the remaining python2 modules removed the question= was if removing python2 from the build ran without any problem or if somethi= ng was flagged up. >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>> ca-certificates was flagged up. >>>>>>>>>=20 >>>>>>>>> There is a python2 script, certdata2pem.py, which fails if python2 = is not present. Running that script with python3 flags up some invalid syntax= , unsurprisingly. >>>>>>>>>=20 >>>>>>>>> I found some patches in Debian from 2015 for certdata2pem.py to pro= vide python3 compatibility. Unfortunately looking at the patch approx half co= uld not be applied because the lines don't exist in the IPFire version of cer= tdata2pem.py (sections to do with blacklisted certs) >>>>>>>>>=20 >>>>>>>>> I then ran the 2to3 converter on certdata2pem.py and tried that in = the build but it came up with the following error. >>>>>>>>>=20 >>>>>>>>> TypeError: a bytes-like object is required, not 'str' >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>> I don't know how to further move forward with this as I am totally = unfamiliar with the python language. >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>> Regards, >>>>>>>>>=20 >>>>>>>>> Adolf. >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>=20 >>>>>=20 --===============6653822116687522206==--