From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: First results from running build without python2 Date: Thu, 19 Aug 2021 14:54:46 +0100 Message-ID: <1E622E0B-2AED-4AAA-A62E-BC2C8C668A08@ipfire.org> In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1974403223206706441==" List-Id: --===============1974403223206706441== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello Adolf, This sounds very good. > On 18 Aug 2021, at 22:08, Adolf Belka wrote: >=20 > Hi All, >=20 > I have successfully built ca-certificates with the new p11-kit approach. Th= e only difference is that the certificate data such as Version, Serial Number= , Signature Algorithm etc is no longer included in the created bundle files. = There is only the actual certificate itself. I do not exactly understand what you mean. Can you post a file system listing= or something that would help me? > This is also the case for the bundles that Arch Linux has created on my sys= tems, that also uses the p11-kit approach. >=20 > I did a diff on the current ca-bundle.crt/ca-bundle.trust.crt files and the= ones created by the new approach as used by Fedora and all the same certific= ates are present, it is only the certificate data that is missed. >=20 > There is a --comment option in the trust extract command that looks like it= should add the comments that are in the .p11-kit file into the extracted bun= dle files but nothing changes if it is used or not. I will ask into the p11-k= it git repository whether this should work or not. >=20 >=20 > I did a build just with the new ca-certificate files into Core Update 160 a= nd then installed the created iso into a vm on my testbed system. >=20 > Everything worked that I tested out but I am not 100% sure what needs to be= exercised to test the ca-bundle.crt and ca-bundle.trust.crt files. If someon= e can give me some guidance on this then I can do that final test and submit = a patch for this change. If you can download something with wget using HTTPS and the certificate can b= e validated, things should be working fine. > I also then did a test build with all python2 programs removed, including p= ython2 itself and with the python 3 fixes for spice, frr and ca-certificates = installed. The build for every program in the ipfire build section completed = successfully but then I had a stop in the cdrom stage for collectd. It has a = python2 plugin module defined which was of course missing. The module could b= e disabled in the rootfile or in the build config if the python plugin is not= required for IPFire - I could not find any python modules defined. You can just pass =E2=80=9C=E2=80=94-without-python=E2=80=9D or =E2=80=9C=E2= =80=94-disable-python=E2=80=9D and then the module won=E2=80=99t be built any= more. > I have been able to successfully define the path to the python interpreter = in the ./configure statement to point to the python3 program and collectd suc= cessfully built with this so if the python plugin is required then a python3 = version can be created. I have not yet tested this in my vm but will do so. We do not use this module, so I suppose it isn=E2=80=99t worth building it in= the first place. > I also found that freeradius has a python2 module defined. This can be disa= bled. Please do that. If someone needs the Python 3 module, we can still build it. > There is a python3 module available since 2.0.20. To use it with 3.0.21 (cu= rrent version) you have to have --with-experimental-modules defined. I am loo= king at latest version 2.0.23 to see if that also needs the experimental modu= les enabled or if it has become part of the normal modules now and will test = the build to see if it works successfully or not. Thank you for all the research that you invested into this. Looks like we are= not far away to drop Python 2. -Michael >=20 > Regards, >=20 > Adolf. >=20 > On 12/08/2021 21:30, Adolf Belka wrote: >> Hi Michael, >>=20 >> On 12/08/2021 21:23, Adolf Belka wrote: >>> Hi Michael, >>>=20 >>> On 12/08/2021 15:38, Michael Tremer wrote: >>>> Hello, >>>>=20 >>>> Yes, this is the way to go :) >>>>=20 >>>> Reach out here if you need any help. >>> I am trying to build p11-kit and it has flagged up libtasn1. It is an opt= ional dependency and I can disable it but I would like to check if that is th= e correct thing to do or not. >>>=20 >>> libtasn1 is said to be required to allow the trust policy module to be bu= ilt as well as other code that interacts with certificates. >>> Is this something that would be needed for IPFire or can I disable it in = the p11-kit build. >>>=20 >> Ignore this question. I just did a quick test of disabling libtasn1 and p1= 1-kit built ok but then my ca-certificates build failed due to the lack of th= e trust tool, so that makes it clear that it is required. I will add it in to= the build. >>=20 >> Slowly getting there. >>=20 >> Regards, >>=20 >> Adolf. >>=20 >>> Regards, >>> Adolf. >>>>=20 >>>> -Michael >>>>=20 >>>>> On 12 Aug 2021, at 13:10, Adolf Belka wrote: >>>>>=20 >>>>> Hi Michael, >>>>>=20 >>>>> On 12/08/2021 13:36, Adolf Belka wrote: >>>>>> Hi Michael, >>>>>>=20 >>>>>> On 12/08/2021 11:17, Michael Tremer wrote: >>>>>>> Hello, >>>>>>>=20 >>>>>>>> On 11 Aug 2021, at 15:03, Adolf Belka wro= te: >>>>>>>>=20 >>>>>>>> Hi Michael, >>>>>>>>=20 >>>>>>>> On 11/08/2021 12:43, Michael Tremer wrote: >>>>>>>>> Hello, >>>>>>>>> Is this the one with the broken sed command? >>>>>>>>> https://src.fedoraproject.org/rpms/ca-certificates/blob/rawhide/f/c= ertdata2pem.py >>>>>>>> Yes, this is that one. Confirmed with a diff. >>>>>>>>> This should run if you execute it in the right directory: >>>>>>>>> pushd %{name}/certs >>>>>>>>> pwd >>>>>>>>> cp certdata.txt . >>>>>>>>> python3 certdata2pem.py >>>>>>>>> popd >>>>>>>> I have just learnt about the pushd and popd commands. by doing a qui= ck search. Never heard of them before. >>>>>>>=20 >>>>>>> It is just a version of =E2=80=9Ccd=E2=80=9D that remembers where it = has been. >>>>>>>=20 >>>>>>> So if you call =E2=80=9Cpushd some-directory=E2=80=9D, then =E2=80=9C= popd=E2=80=9D will bring you back to where you have been before. >>>>>>>=20 >>>>>>> =E2=80=9Ccd -=E2=80=9C does the same as popd now. >>>>>>>=20 >>>>>>>>> The fedora version no longer has the build.sh script. >>>>>>>> That was the bit I didn't realise. >>>>>>>=20 >>>>>>> No problem. >>>>>>=20 >>>>>> Not as simple as I hoped. >>>>>>=20 >>>>>> The new certdata2pem.py script no longer creates .crt files but .tmp-p= 11-kit files but that is as far as that script goes. >>>>>>=20 >>>>>> There is an update-ca-trust file in fedora which splits the various ce= rts to their respective locations, with the openssl ones ending up in ca-bund= le.trust.crt but this uses p11-kit. I suspect that it uses ca-bundle.trust.p1= 1-kit generated later on in the %build section of the ca-certificate.spec fil= e from the tmp-p11-kit files generated by the certdata2pem.py script. So it l= ooks like p11-kit needs to be installed to make this work. >>>>>>=20 >>>>>> Alternatively I have found the following in LFS. >>>>>> https://www.linuxfromscratch.org/blfs/view/svn/postlfs/make-ca.html >>>>>> which seems to also create the bundled cert file but also requiring p1= 1-kit but then talking about different certdata.txt versions that have variou= s tunings. The one from LFS comes from the mozilla release branch but is modi= fied to something called the Mercurial revision, or the different ones shippe= d by RedHat or OpenSUSE which use the version that comes with NSS. >>>>>>=20 >>>>>> All in all I am not sure which approach to use or how to actually buil= d it in IPFire. >>>>>>=20 >>>>>> Definitely help required to know the correct way to go further with th= is. >>>>>>=20 >>>>> Having said I needed help, I thought I would have another go and copied= the lines from the %build section of the fedora ca-certificate.spec that cre= ated the ca-bundle.trust.p11.kit >>>>>=20 >>>>> As I have p11-kit installed on my Arch Linux system I then ran the comm= and >>>>>=20 >>>>> p11-kit extract --format=3Dopenssl-bundle --filter=3Dcertificates --ove= rwrite --comment ~/openssl/ca-bundle.trust.crt >>>>>=20 >>>>> and I successfully created the ca-bundle.trust.crt which is needed by t= he IPFire ca-certificates lfs file. >>>>>=20 >>>>> I will now try and create a new build script that will do all the new t= ype stuff and also install the p11-kit library files and see how things go. >>>>>=20 >>>>> :crossed_fingers: >>>>>=20 >>>>> Adolf >>>>>=20 >>>>>> Regards, >>>>>> Adolf. >>>>>>=20 >>>>>>> -Michael >>>>>>>=20 >>>>>>>> Regards, >>>>>>>> Adolf. >>>>>>>>> -Michael >>>>>>>>>> On 8 Aug 2021, at 14:47, Adolf Belka wr= ote: >>>>>>>>>>=20 >>>>>>>>>> Hi All, >>>>>>>>>>=20 >>>>>>>>>> I had another go at the ca-certificates problem, the last barrier = to getting rid of python2. >>>>>>>>>>=20 >>>>>>>>>> I found certdata2pem.py files from fedora and 2 from suse. I creat= ed build subdirectories for each version so I could just test running the bui= ld.sh file with each version of certdata2pem.py, including the IPFire current= version after running through the 2to3 convertor. >>>>>>>>>>=20 >>>>>>>>>> fedora >>>>>>>>>>=20 >>>>>>>>>> The fedora certdata2pem.py file runs successfully with python3 but= has sed commands built into it which fail to find certain files. The sed com= mands are not in the IPFire version. >>>>>>>>>>=20 >>>>>>>>>> The error message is >>>>>>>>>>=20 >>>>>>>>>> -> written as 'Certum_Trusted_Root_CA:2.16.30.191.89.80.184.201= .128.55.76.6.247.235.85.79.181.237.tmp-p11-kit', trust =3D ['CKA_TRUST_SERVER= _AUTH', 'CKA_TRUST_EMAIL_PROTECTION'], openssl-trust =3D ['serverAuth', 'emai= lProtection'], distrust =3D [], openssl-distrust =3D [] >>>>>>>>>> sed: can't read certs/*.crt: No such file or directory >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>> suse >>>>>>>>>>=20 >>>>>>>>>> The first suse version runs successfully with python3 but also has= the sed commands in it with the same error message. >>>>>>>>>>=20 >>>>>>>>>> The second suse version runs successfully with python3, does not h= ave the sed commands and completes the build.sh script with no errors. Howeve= r this certdata2pem.py file has a section that is in the IPFire version compl= etely missing. >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>> IPfire version after running through the 2to3 convertor >>>>>>>>>>=20 >>>>>>>>>> The following error message occurs >>>>>>>>>>=20 >>>>>>>>>> producing trust for "GlobalSign Root CA"2.11.4.0.0.0.0.1.21.75.90.= 195.148 >>>>>>>>>> Traceback (most recent call last): >>>>>>>>>> File "/mnt/File_Server/Computers/Linux/ipfire/sandbox/patch in= progress/python/ca-certificates/orig-2to3-build/certs/../certdata2pem.py", l= ine 224, in >>>>>>>>>> f.write("\n".join(textwrap.wrap(base64.b64encode(obj['CKA_VALUE'])= , 64))) >>>>>>>>>> File "/usr/lib/python3.9/base64.py", line 58, in b64encode >>>>>>>>>> encoded =3D binascii.b2a_base64(s, newline=3DFalse) >>>>>>>>>> TypeError: a bytes-like object is required, not 'str' >>>>>>>>>>=20 >>>>>>>>>> The section that is failing is the section that is missing in the = 2nd suse version. There is an identical fwrite line at line 206 but that does= not seem to flag up the same TypeError message. >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>> As the certdata2pem.py files from the other distributions vary sig= nificantly in content, with some having nearly double the number of lines of = code, I think the best alternative is to fix the IPFire version so we stay co= nsistent but I am unable to figure out how to fix the python code that is cau= sing the " TypeError: a bytes-like object is required, not 'str' " error mes= sage and need someone's help with that. >>>>>>>>>>=20 >>>>>>>>>> Let me know if there is any other information that I need to provi= de. >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>> Regards, >>>>>>>>>>=20 >>>>>>>>>> Adolf. >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>> On 07/08/2021 15:54, Adolf Belka wrote: >>>>>>>>>>> Hi All, >>>>>>>>>>>=20 >>>>>>>>>>> On 04/08/2021 16:45, Michael Tremer wrote: >>>>>>>>>>>> Hello, >>>>>>>>>>>>=20 >>>>>>>>>>>>> On 4 Aug 2021, at 13:40, Adolf Belka = wrote: >>>>>>>>>>>>>=20 >>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>=20 >>>>>>>>>>>>> I have resolved the frr program build. The version currently in= IPFire (6.0) only works with python2. Python3 support came in with version 7= .4. I have now built frr with version 8.0 including libyang as a new dependen= cy but only for the build, so nothing installed into IPFire itself, and that = has successfully built without python2 being present. >>>>>>>>>>>>=20 >>>>>>>>>>>> Great. This could also resolve Matthias=E2=80=99 problem with bu= ilding frr. >>>>>>>>>>>>=20 >>>>>>>>>>>>> Will now go back and have another go with spice-protocol. >>>>>>>>>>>>=20 >>>>>>>>>>>> Maybe it has a =E2=80=94-disable-python switch? >>>>>>>>>>> I just removed the line in the spice-protocol lfs that ran automa= ke/py-compile on the python modules from spice. >>>>>>>>>>> Spice and spice-protocol are present for qemu and with the py-com= pile line removed all three successfully built without python2 being present.= I have submitted a patch for this combined with updating spice and spice-pro= tocol, both from 2017. >>>>>>>>>>>=20 >>>>>>>>>>> This now only leaves the ca-certificates script that needs to be = updated to work with python3. >>>>>>>>>>>=20 >>>>>>>>>>> Regards, >>>>>>>>>>> Adolf. >>>>>>>>>>>>=20 >>>>>>>>>>>> -Michael >>>>>>>>>>>>=20 >>>>>>>>>>>>>=20 >>>>>>>>>>>>> Regards, >>>>>>>>>>>>>=20 >>>>>>>>>>>>> Adolf. >>>>>>>>>>>>>=20 >>>>>>>>>>>>>=20 >>>>>>>>>>>>> On 03/08/2021 23:38, Adolf Belka wrote: >>>>>>>>>>>>>> Hi Michael & all, >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> On 03/08/2021 17:11, Michael Tremer wrote: >>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> Thank you for looking into this. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> This is a third-party script that came from either Mozilla or= RedHat. Maybe they have ported it. If not, it should not be rocket science t= o do it ourselves. If we do it, we should of course upstream it. >>>>>>>>>>>>>> I found an updated script from fedora and gave that a try. Thi= s time the script went all the way through but then the build.sh script faile= d at the point where it should find all the .crt files in the certs directory= and it came back and said there weren't any. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> However, can you comment out this package and continue the bu= ild? This should be required until you reach the cdrom stage. >>>>>>>>>>>>>> I then commented ca-certificates out in make.sh and ran the bu= ild. >>>>>>>>>>>>>> This time it stopped at spice-protocol which is an addon and u= ses the py-compile script that is in automake to compile some python modules. >>>>>>>>>>>>>> py-compile is python2 based and the build stopped because it c= ould not find python >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> There is a py_compile.py script that is python3 based but when= I ran that in place of the py-compile script I got a Permission denied error= when it tried to carry out the compile. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> I then commented out spice-protocol and ran the build. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> It then failed on frr which did look for python3-config but th= en failed due to not finding python-config or pkg-config python >>>>>>>>>>>>>> It looks like I should be able to tell it to use python3 in th= e ./configure >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> I commented out frr and nothing else failed before cdrom was r= eached. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> So the packages that need to be made to work with python3 are >>>>>>>>>>>>>> ca-certificates >>>>>>>>>>>>>> spice-protocol >>>>>>>>>>>>>> frr >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> I also converted client175 with 2to3 converter and built it an= d installed the .ipfire package into a vm and successfully got the WUI page f= or Media Player IPFire to render. What I haven't tested yet is if the audio w= orks. I will need to get audio set up in my vm to try that. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> Adolf. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> If this is the only thing that flags up, we should port the s= cript. If we find another, stronger reason to keep Python 2 around, we do not= need to bother and can keep the script this way. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> -Michael >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> On 3 Aug 2021, at 13:31, Adolf Belka wrote: >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> So with crda and the remaining python2 modules removed the q= uestion was if removing python2 from the build ran without any problem or if = something was flagged up. >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> ca-certificates was flagged up. >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> There is a python2 script, certdata2pem.py, which fails if p= ython2 is not present. Running that script with python3 flags up some invalid= syntax, unsurprisingly. >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> I found some patches in Debian from 2015 for certdata2pem.py= to provide python3 compatibility. Unfortunately looking at the patch approx = half could not be applied because the lines don't exist in the IPFire version= of certdata2pem.py (sections to do with blacklisted certs) >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> I then ran the 2to3 converter on certdata2pem.py and tried t= hat in the build but it came up with the following error. >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> TypeError: a bytes-like object is required, not 'str' >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> I don't know how to further move forward with this as I am t= otally unfamiliar with the python language. >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> Adolf. >>>>=20 --===============1974403223206706441==--