From mboxrd@z Thu Jan 1 00:00:00 1970 From: Adolf Belka To: development@lists.ipfire.org Subject: Re: First results from running build without python2 Date: Wed, 11 Aug 2021 16:03:58 +0200 Message-ID: <12415ddf-2601-7d1e-fd05-8b0388177df2@ipfire.org> In-Reply-To: <15B4DB6B-5FD9-4256-9026-A59E936CDF8B@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============3592120030102160260==" List-Id: --===============3592120030102160260== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi Michael, On 11/08/2021 12:43, Michael Tremer wrote: > Hello, >=20 > Is this the one with the broken sed command? >=20 > https://src.fedoraproject.org/rpms/ca-certificates/blob/rawhide/f/certda= ta2pem.py Yes, this is that one. Confirmed with a diff. >=20 > This should run if you execute it in the right directory: >=20 > pushd %{name}/certs > pwd > cp certdata.txt . > python3 certdata2pem.py > popd I have just learnt about the pushd and popd commands. by doing a quick search= . Never heard of them before. >=20 > The fedora version no longer has the build.sh script. That was the bit I didn't realise. Regards, Adolf. >=20 > -Michael >=20 >> On 8 Aug 2021, at 14:47, Adolf Belka wrote: >> >> Hi All, >> >> I had another go at the ca-certificates problem, the last barrier to getti= ng rid of python2. >> >> I found certdata2pem.py files from fedora and 2 from suse. I created build= subdirectories for each version so I could just test running the build.sh fi= le with each version of certdata2pem.py, including the IPFire current version= after running through the 2to3 convertor. >> >> fedora >> >> The fedora certdata2pem.py file runs successfully with python3 but has sed= commands built into it which fail to find certain files. The sed commands ar= e not in the IPFire version. >> >> The error message is >> >> -> written as 'Certum_Trusted_Root_CA:2.16.30.191.89.80.184.201.128.55.7= 6.6.247.235.85.79.181.237.tmp-p11-kit', trust =3D ['CKA_TRUST_SERVER_AUTH', '= CKA_TRUST_EMAIL_PROTECTION'], openssl-trust =3D ['serverAuth', 'emailProtecti= on'], distrust =3D [], openssl-distrust =3D [] >> sed: can't read certs/*.crt: No such file or directory >> >> >> suse >> >> The first suse version runs successfully with python3 but also has the sed= commands in it with the same error message. >> >> The second suse version runs successfully with python3, does not have the = sed commands and completes the build.sh script with no errors. However this c= ertdata2pem.py file has a section that is in the IPFire version completely mi= ssing. >> >> >> IPfire version after running through the 2to3 convertor >> >> The following error message occurs >> >> producing trust for "GlobalSign Root CA"2.11.4.0.0.0.0.1.21.75.90.195.148 >> Traceback (most recent call last): >> File "/mnt/File_Server/Computers/Linux/ipfire/sandbox/patch in progress= /python/ca-certificates/orig-2to3-build/certs/../certdata2pem.py", line 224, = in >> f.write("\n".join(textwrap.wrap(base64.b64encode(obj['CKA_VALUE']), 64))) >> File "/usr/lib/python3.9/base64.py", line 58, in b64encode >> encoded =3D binascii.b2a_base64(s, newline=3DFalse) >> TypeError: a bytes-like object is required, not 'str' >> >> The section that is failing is the section that is missing in the 2nd suse= version. There is an identical fwrite line at line 206 but that does not see= m to flag up the same TypeError message. >> >> >> As the certdata2pem.py files from the other distributions vary significant= ly in content, with some having nearly double the number of lines of code, I = think the best alternative is to fix the IPFire version so we stay consistent= but I am unable to figure out how to fix the python code that is causing the= " TypeError: a bytes-like object is required, not 'str' " error message and= need someone's help with that. >> >> Let me know if there is any other information that I need to provide. >> >> >> Regards, >> >> Adolf. >> >> >> On 07/08/2021 15:54, Adolf Belka wrote: >>> Hi All, >>> >>> On 04/08/2021 16:45, Michael Tremer wrote: >>>> Hello, >>>> >>>>> On 4 Aug 2021, at 13:40, Adolf Belka wrote: >>>>> >>>>> Hi All, >>>>> >>>>> I have resolved the frr program build. The version currently in IPFire = (6.0) only works with python2. Python3 support came in with version 7.4. I ha= ve now built frr with version 8.0 including libyang as a new dependency but o= nly for the build, so nothing installed into IPFire itself, and that has succ= essfully built without python2 being present. >>>> >>>> Great. This could also resolve Matthias=E2=80=99 problem with building f= rr. >>>> >>>>> Will now go back and have another go with spice-protocol. >>>> >>>> Maybe it has a =E2=80=94-disable-python switch? >>> I just removed the line in the spice-protocol lfs that ran automake/py-co= mpile on the python modules from spice. >>> Spice and spice-protocol are present for qemu and with the py-compile lin= e removed all three successfully built without python2 being present. I have = submitted a patch for this combined with updating spice and spice-protocol, b= oth from 2017. >>> >>> This now only leaves the ca-certificates script that needs to be updated = to work with python3. >>> >>> Regards, >>> Adolf. >>>> >>>> -Michael >>>> >>>>> >>>>> Regards, >>>>> >>>>> Adolf. >>>>> >>>>> >>>>> On 03/08/2021 23:38, Adolf Belka wrote: >>>>>> Hi Michael & all, >>>>>> >>>>>> >>>>>> On 03/08/2021 17:11, Michael Tremer wrote: >>>>>>> Hello, >>>>>>> >>>>>>> Thank you for looking into this. >>>>>>> >>>>>>> This is a third-party script that came from either Mozilla or RedHat.= Maybe they have ported it. If not, it should not be rocket science to do it = ourselves. If we do it, we should of course upstream it. >>>>>> I found an updated script from fedora and gave that a try. This time t= he script went all the way through but then the build.sh script failed at the= point where it should find all the .crt files in the certs directory and it = came back and said there weren't any. >>>>>>> >>>>>>> However, can you comment out this package and continue the build? Thi= s should be required until you reach the cdrom stage. >>>>>> I then commented ca-certificates out in make.sh and ran the build. >>>>>> This time it stopped at spice-protocol which is an addon and uses the = py-compile script that is in automake to compile some python modules. >>>>>> py-compile is python2 based and the build stopped because it could not= find python >>>>>> >>>>>> There is a py_compile.py script that is python3 based but when I ran t= hat in place of the py-compile script I got a Permission denied error when it= tried to carry out the compile. >>>>>> >>>>>> I then commented out spice-protocol and ran the build. >>>>>> >>>>>> It then failed on frr which did look for python3-config but then faile= d due to not finding python-config or pkg-config python >>>>>> It looks like I should be able to tell it to use python3 in the ./conf= igure >>>>>> >>>>>> I commented out frr and nothing else failed before cdrom was reached. >>>>>> >>>>>> So the packages that need to be made to work with python3 are >>>>>> ca-certificates >>>>>> spice-protocol >>>>>> frr >>>>>> >>>>>> >>>>>> I also converted client175 with 2to3 converter and built it and instal= led the .ipfire package into a vm and successfully got the WUI page for Media= Player IPFire to render. What I haven't tested yet is if the audio works. I = will need to get audio set up in my vm to try that. >>>>>> >>>>>> Regards, >>>>>> Adolf. >>>>>> >>>>>>> >>>>>>> If this is the only thing that flags up, we should port the script. I= f we find another, stronger reason to keep Python 2 around, we do not need to= bother and can keep the script this way. >>>>>>> >>>>>>> -Michael >>>>>>> >>>>>>>> On 3 Aug 2021, at 13:31, Adolf Belka wrot= e: >>>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> So with crda and the remaining python2 modules removed the question = was if removing python2 from the build ran without any problem or if somethin= g was flagged up. >>>>>>>> >>>>>>>> >>>>>>>> ca-certificates was flagged up. >>>>>>>> >>>>>>>> There is a python2 script, certdata2pem.py, which fails if python2 i= s not present. Running that script with python3 flags up some invalid syntax,= unsurprisingly. >>>>>>>> >>>>>>>> I found some patches in Debian from 2015 for certdata2pem.py to prov= ide python3 compatibility. Unfortunately looking at the patch approx half cou= ld not be applied because the lines don't exist in the IPFire version of cert= data2pem.py (sections to do with blacklisted certs) >>>>>>>> >>>>>>>> I then ran the 2to3 converter on certdata2pem.py and tried that in t= he build but it came up with the following error. >>>>>>>> >>>>>>>> TypeError: a bytes-like object is required, not 'str' >>>>>>>> >>>>>>>> >>>>>>>> I don't know how to further move forward with this as I am totally u= nfamiliar with the python language. >>>>>>>> >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Adolf. >>>>>>>> >>>>>>>> >>>>>>> >>>> >=20 --===============3592120030102160260==--