From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: First results from running build without python2 Date: Thu, 12 Aug 2021 14:38:11 +0100 Message-ID: <55E86324-9EB7-4C3E-B29E-BA16B03FB1E9@ipfire.org> In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2630635274995600129==" List-Id: --===============2630635274995600129== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello, Yes, this is the way to go :) Reach out here if you need any help. -Michael > On 12 Aug 2021, at 13:10, Adolf Belka wrote: >=20 > Hi Michael, >=20 > On 12/08/2021 13:36, Adolf Belka wrote: >> Hi Michael, >>=20 >> On 12/08/2021 11:17, Michael Tremer wrote: >>> Hello, >>>=20 >>>> On 11 Aug 2021, at 15:03, Adolf Belka wrote: >>>>=20 >>>> Hi Michael, >>>>=20 >>>> On 11/08/2021 12:43, Michael Tremer wrote: >>>>> Hello, >>>>> Is this the one with the broken sed command? >>>>> https://src.fedoraproject.org/rpms/ca-certificates/blob/rawhide/f/certd= ata2pem.py >>>> Yes, this is that one. Confirmed with a diff. >>>>> This should run if you execute it in the right directory: >>>>> pushd %{name}/certs >>>>> pwd >>>>> cp certdata.txt . >>>>> python3 certdata2pem.py >>>>> popd >>>> I have just learnt about the pushd and popd commands. by doing a quick s= earch. Never heard of them before. >>>=20 >>> It is just a version of =E2=80=9Ccd=E2=80=9D that remembers where it has = been. >>>=20 >>> So if you call =E2=80=9Cpushd some-directory=E2=80=9D, then =E2=80=9Cpopd= =E2=80=9D will bring you back to where you have been before. >>>=20 >>> =E2=80=9Ccd -=E2=80=9C does the same as popd now. >>>=20 >>>>> The fedora version no longer has the build.sh script. >>>> That was the bit I didn't realise. >>>=20 >>> No problem. >>=20 >> Not as simple as I hoped. >>=20 >> The new certdata2pem.py script no longer creates .crt files but .tmp-p11-k= it files but that is as far as that script goes. >>=20 >> There is an update-ca-trust file in fedora which splits the various certs = to their respective locations, with the openssl ones ending up in ca-bundle.t= rust.crt but this uses p11-kit. I suspect that it uses ca-bundle.trust.p11-ki= t generated later on in the %build section of the ca-certificate.spec file fr= om the tmp-p11-kit files generated by the certdata2pem.py script. So it looks= like p11-kit needs to be installed to make this work. >>=20 >> Alternatively I have found the following in LFS. >> https://www.linuxfromscratch.org/blfs/view/svn/postlfs/make-ca.html >> which seems to also create the bundled cert file but also requiring p11-ki= t but then talking about different certdata.txt versions that have various tu= nings. The one from LFS comes from the mozilla release branch but is modified= to something called the Mercurial revision, or the different ones shipped by= RedHat or OpenSUSE which use the version that comes with NSS. >>=20 >> All in all I am not sure which approach to use or how to actually build it= in IPFire. >>=20 >> Definitely help required to know the correct way to go further with this. >>=20 > Having said I needed help, I thought I would have another go and copied the= lines from the %build section of the fedora ca-certificate.spec that created= the ca-bundle.trust.p11.kit >=20 > As I have p11-kit installed on my Arch Linux system I then ran the command >=20 > p11-kit extract --format=3Dopenssl-bundle --filter=3Dcertificates --overwri= te --comment ~/openssl/ca-bundle.trust.crt >=20 > and I successfully created the ca-bundle.trust.crt which is needed by the I= PFire ca-certificates lfs file. >=20 > I will now try and create a new build script that will do all the new type = stuff and also install the p11-kit library files and see how things go. >=20 > :crossed_fingers: >=20 > Adolf >=20 >> Regards, >> Adolf. >>=20 >>> -Michael >>>=20 >>>> Regards, >>>> Adolf. >>>>> -Michael >>>>>> On 8 Aug 2021, at 14:47, Adolf Belka wrote: >>>>>>=20 >>>>>> Hi All, >>>>>>=20 >>>>>> I had another go at the ca-certificates problem, the last barrier to g= etting rid of python2. >>>>>>=20 >>>>>> I found certdata2pem.py files from fedora and 2 from suse. I created b= uild subdirectories for each version so I could just test running the build.s= h file with each version of certdata2pem.py, including the IPFire current ver= sion after running through the 2to3 convertor. >>>>>>=20 >>>>>> fedora >>>>>>=20 >>>>>> The fedora certdata2pem.py file runs successfully with python3 but has= sed commands built into it which fail to find certain files. The sed command= s are not in the IPFire version. >>>>>>=20 >>>>>> The error message is >>>>>>=20 >>>>>> -> written as 'Certum_Trusted_Root_CA:2.16.30.191.89.80.184.201.128.= 55.76.6.247.235.85.79.181.237.tmp-p11-kit', trust =3D ['CKA_TRUST_SERVER_AUTH= ', 'CKA_TRUST_EMAIL_PROTECTION'], openssl-trust =3D ['serverAuth', 'emailProt= ection'], distrust =3D [], openssl-distrust =3D [] >>>>>> sed: can't read certs/*.crt: No such file or directory >>>>>>=20 >>>>>>=20 >>>>>> suse >>>>>>=20 >>>>>> The first suse version runs successfully with python3 but also has the= sed commands in it with the same error message. >>>>>>=20 >>>>>> The second suse version runs successfully with python3, does not have = the sed commands and completes the build.sh script with no errors. However th= is certdata2pem.py file has a section that is in the IPFire version completel= y missing. >>>>>>=20 >>>>>>=20 >>>>>> IPfire version after running through the 2to3 convertor >>>>>>=20 >>>>>> The following error message occurs >>>>>>=20 >>>>>> producing trust for "GlobalSign Root CA"2.11.4.0.0.0.0.1.21.75.90.195.= 148 >>>>>> Traceback (most recent call last): >>>>>> File "/mnt/File_Server/Computers/Linux/ipfire/sandbox/patch in prog= ress/python/ca-certificates/orig-2to3-build/certs/../certdata2pem.py", line 2= 24, in >>>>>> f.write("\n".join(textwrap.wrap(base64.b64encode(obj['CKA_VALUE']), 64= ))) >>>>>> File "/usr/lib/python3.9/base64.py", line 58, in b64encode >>>>>> encoded =3D binascii.b2a_base64(s, newline=3DFalse) >>>>>> TypeError: a bytes-like object is required, not 'str' >>>>>>=20 >>>>>> The section that is failing is the section that is missing in the 2nd = suse version. There is an identical fwrite line at line 206 but that does not= seem to flag up the same TypeError message. >>>>>>=20 >>>>>>=20 >>>>>> As the certdata2pem.py files from the other distributions vary signifi= cantly in content, with some having nearly double the number of lines of code= , I think the best alternative is to fix the IPFire version so we stay consis= tent but I am unable to figure out how to fix the python code that is causing= the " TypeError: a bytes-like object is required, not 'str' " error message= and need someone's help with that. >>>>>>=20 >>>>>> Let me know if there is any other information that I need to provide. >>>>>>=20 >>>>>>=20 >>>>>> Regards, >>>>>>=20 >>>>>> Adolf. >>>>>>=20 >>>>>>=20 >>>>>> On 07/08/2021 15:54, Adolf Belka wrote: >>>>>>> Hi All, >>>>>>>=20 >>>>>>> On 04/08/2021 16:45, Michael Tremer wrote: >>>>>>>> Hello, >>>>>>>>=20 >>>>>>>>> On 4 Aug 2021, at 13:40, Adolf Belka wro= te: >>>>>>>>>=20 >>>>>>>>> Hi All, >>>>>>>>>=20 >>>>>>>>> I have resolved the frr program build. The version currently in IPF= ire (6.0) only works with python2. Python3 support came in with version 7.4. = I have now built frr with version 8.0 including libyang as a new dependency b= ut only for the build, so nothing installed into IPFire itself, and that has = successfully built without python2 being present. >>>>>>>>=20 >>>>>>>> Great. This could also resolve Matthias=E2=80=99 problem with buildi= ng frr. >>>>>>>>=20 >>>>>>>>> Will now go back and have another go with spice-protocol. >>>>>>>>=20 >>>>>>>> Maybe it has a =E2=80=94-disable-python switch? >>>>>>> I just removed the line in the spice-protocol lfs that ran automake/p= y-compile on the python modules from spice. >>>>>>> Spice and spice-protocol are present for qemu and with the py-compile= line removed all three successfully built without python2 being present. I h= ave submitted a patch for this combined with updating spice and spice-protoco= l, both from 2017. >>>>>>>=20 >>>>>>> This now only leaves the ca-certificates script that needs to be upda= ted to work with python3. >>>>>>>=20 >>>>>>> Regards, >>>>>>> Adolf. >>>>>>>>=20 >>>>>>>> -Michael >>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>> Regards, >>>>>>>>>=20 >>>>>>>>> Adolf. >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>> On 03/08/2021 23:38, Adolf Belka wrote: >>>>>>>>>> Hi Michael & all, >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>> On 03/08/2021 17:11, Michael Tremer wrote: >>>>>>>>>>> Hello, >>>>>>>>>>>=20 >>>>>>>>>>> Thank you for looking into this. >>>>>>>>>>>=20 >>>>>>>>>>> This is a third-party script that came from either Mozilla or Red= Hat. Maybe they have ported it. If not, it should not be rocket science to do= it ourselves. If we do it, we should of course upstream it. >>>>>>>>>> I found an updated script from fedora and gave that a try. This ti= me the script went all the way through but then the build.sh script failed at= the point where it should find all the .crt files in the certs directory and= it came back and said there weren't any. >>>>>>>>>>>=20 >>>>>>>>>>> However, can you comment out this package and continue the build?= This should be required until you reach the cdrom stage. >>>>>>>>>> I then commented ca-certificates out in make.sh and ran the build. >>>>>>>>>> This time it stopped at spice-protocol which is an addon and uses = the py-compile script that is in automake to compile some python modules. >>>>>>>>>> py-compile is python2 based and the build stopped because it could= not find python >>>>>>>>>>=20 >>>>>>>>>> There is a py_compile.py script that is python3 based but when I r= an that in place of the py-compile script I got a Permission denied error whe= n it tried to carry out the compile. >>>>>>>>>>=20 >>>>>>>>>> I then commented out spice-protocol and ran the build. >>>>>>>>>>=20 >>>>>>>>>> It then failed on frr which did look for python3-config but then f= ailed due to not finding python-config or pkg-config python >>>>>>>>>> It looks like I should be able to tell it to use python3 in the ./= configure >>>>>>>>>>=20 >>>>>>>>>> I commented out frr and nothing else failed before cdrom was reach= ed. >>>>>>>>>>=20 >>>>>>>>>> So the packages that need to be made to work with python3 are >>>>>>>>>> ca-certificates >>>>>>>>>> spice-protocol >>>>>>>>>> frr >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>> I also converted client175 with 2to3 converter and built it and in= stalled the .ipfire package into a vm and successfully got the WUI page for M= edia Player IPFire to render. What I haven't tested yet is if the audio works= . I will need to get audio set up in my vm to try that. >>>>>>>>>>=20 >>>>>>>>>> Regards, >>>>>>>>>> Adolf. >>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>> If this is the only thing that flags up, we should port the scrip= t. If we find another, stronger reason to keep Python 2 around, we do not nee= d to bother and can keep the script this way. >>>>>>>>>>>=20 >>>>>>>>>>> -Michael >>>>>>>>>>>=20 >>>>>>>>>>>> On 3 Aug 2021, at 13:31, Adolf Belka = wrote: >>>>>>>>>>>>=20 >>>>>>>>>>>> Hi All, >>>>>>>>>>>>=20 >>>>>>>>>>>> So with crda and the remaining python2 modules removed the quest= ion was if removing python2 from the build ran without any problem or if some= thing was flagged up. >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>> ca-certificates was flagged up. >>>>>>>>>>>>=20 >>>>>>>>>>>> There is a python2 script, certdata2pem.py, which fails if pytho= n2 is not present. Running that script with python3 flags up some invalid syn= tax, unsurprisingly. >>>>>>>>>>>>=20 >>>>>>>>>>>> I found some patches in Debian from 2015 for certdata2pem.py to = provide python3 compatibility. Unfortunately looking at the patch approx half= could not be applied because the lines don't exist in the IPFire version of = certdata2pem.py (sections to do with blacklisted certs) >>>>>>>>>>>>=20 >>>>>>>>>>>> I then ran the 2to3 converter on certdata2pem.py and tried that = in the build but it came up with the following error. >>>>>>>>>>>>=20 >>>>>>>>>>>> TypeError: a bytes-like object is required, not 'str' >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>> I don't know how to further move forward with this as I am total= ly unfamiliar with the python language. >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>> Regards, >>>>>>>>>>>>=20 >>>>>>>>>>>> Adolf. --===============2630635274995600129==--