Hello Adolf,
This sounds very good.
On 18 Aug 2021, at 22:08, Adolf Belka adolf.belka@ipfire.org wrote:
Hi All,
I have successfully built ca-certificates with the new p11-kit approach. The only difference is that the certificate data such as Version, Serial Number, Signature Algorithm etc is no longer included in the created bundle files. There is only the actual certificate itself.
I do not exactly understand what you mean. Can you post a file system listing or something that would help me?
This is also the case for the bundles that Arch Linux has created on my systems, that also uses the p11-kit approach.
I did a diff on the current ca-bundle.crt/ca-bundle.trust.crt files and the ones created by the new approach as used by Fedora and all the same certificates are present, it is only the certificate data that is missed.
There is a --comment option in the trust extract command that looks like it should add the comments that are in the .p11-kit file into the extracted bundle files but nothing changes if it is used or not. I will ask into the p11-kit git repository whether this should work or not.
I did a build just with the new ca-certificate files into Core Update 160 and then installed the created iso into a vm on my testbed system.
Everything worked that I tested out but I am not 100% sure what needs to be exercised to test the ca-bundle.crt and ca-bundle.trust.crt files. If someone can give me some guidance on this then I can do that final test and submit a patch for this change.
If you can download something with wget using HTTPS and the certificate can be validated, things should be working fine.
I also then did a test build with all python2 programs removed, including python2 itself and with the python 3 fixes for spice, frr and ca-certificates installed. The build for every program in the ipfire build section completed successfully but then I had a stop in the cdrom stage for collectd. It has a python2 plugin module defined which was of course missing. The module could be disabled in the rootfile or in the build config if the python plugin is not required for IPFire - I could not find any python modules defined.
You can just pass “—-without-python” or “—-disable-python” and then the module won’t be built any more.
I have been able to successfully define the path to the python interpreter in the ./configure statement to point to the python3 program and collectd successfully built with this so if the python plugin is required then a python3 version can be created. I have not yet tested this in my vm but will do so.
We do not use this module, so I suppose it isn’t worth building it in the first place.
I also found that freeradius has a python2 module defined. This can be disabled.
Please do that. If someone needs the Python 3 module, we can still build it.
There is a python3 module available since 2.0.20. To use it with 3.0.21 (current version) you have to have --with-experimental-modules defined. I am looking at latest version 2.0.23 to see if that also needs the experimental modules enabled or if it has become part of the normal modules now and will test the build to see if it works successfully or not.
Thank you for all the research that you invested into this. Looks like we are not far away to drop Python 2.
-Michael
Regards,
Adolf.
On 12/08/2021 21:30, Adolf Belka wrote:
Hi Michael,
On 12/08/2021 21:23, Adolf Belka wrote:
Hi Michael,
On 12/08/2021 15:38, Michael Tremer wrote:
Hello,
Yes, this is the way to go :)
Reach out here if you need any help.
I am trying to build p11-kit and it has flagged up libtasn1. It is an optional dependency and I can disable it but I would like to check if that is the correct thing to do or not.
libtasn1 is said to be required to allow the trust policy module to be built as well as other code that interacts with certificates. Is this something that would be needed for IPFire or can I disable it in the p11-kit build.
Ignore this question. I just did a quick test of disabling libtasn1 and p11-kit built ok but then my ca-certificates build failed due to the lack of the trust tool, so that makes it clear that it is required. I will add it in to the build.
Slowly getting there.
Regards,
Adolf.
Regards, Adolf.
-Michael
On 12 Aug 2021, at 13:10, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 12/08/2021 13:36, Adolf Belka wrote:
Hi Michael,
On 12/08/2021 11:17, Michael Tremer wrote: > Hello, > >> On 11 Aug 2021, at 15:03, Adolf Belka adolf.belka@ipfire.org wrote: >> >> Hi Michael, >> >> On 11/08/2021 12:43, Michael Tremer wrote: >>> Hello, >>> Is this the one with the broken sed command? >>> https://src.fedoraproject.org/rpms/ca-certificates/blob/rawhide/f/certdata2p... >> Yes, this is that one. Confirmed with a diff. >>> This should run if you execute it in the right directory: >>> pushd %{name}/certs >>> pwd >>> cp certdata.txt . >>> python3 certdata2pem.py >>> popd >> I have just learnt about the pushd and popd commands. by doing a quick search. Never heard of them before. > > It is just a version of “cd” that remembers where it has been. > > So if you call “pushd some-directory”, then “popd” will bring you back to where you have been before. > > “cd -“ does the same as popd now. > >>> The fedora version no longer has the build.sh script. >> That was the bit I didn't realise. > > No problem.
Not as simple as I hoped.
The new certdata2pem.py script no longer creates .crt files but .tmp-p11-kit files but that is as far as that script goes.
There is an update-ca-trust file in fedora which splits the various certs to their respective locations, with the openssl ones ending up in ca-bundle.trust.crt but this uses p11-kit. I suspect that it uses ca-bundle.trust.p11-kit generated later on in the %build section of the ca-certificate.spec file from the tmp-p11-kit files generated by the certdata2pem.py script. So it looks like p11-kit needs to be installed to make this work.
Alternatively I have found the following in LFS. https://www.linuxfromscratch.org/blfs/view/svn/postlfs/make-ca.html which seems to also create the bundled cert file but also requiring p11-kit but then talking about different certdata.txt versions that have various tunings. The one from LFS comes from the mozilla release branch but is modified to something called the Mercurial revision, or the different ones shipped by RedHat or OpenSUSE which use the version that comes with NSS.
All in all I am not sure which approach to use or how to actually build it in IPFire.
Definitely help required to know the correct way to go further with this.
Having said I needed help, I thought I would have another go and copied the lines from the %build section of the fedora ca-certificate.spec that created the ca-bundle.trust.p11.kit
As I have p11-kit installed on my Arch Linux system I then ran the command
p11-kit extract --format=openssl-bundle --filter=certificates --overwrite --comment ~/openssl/ca-bundle.trust.crt
and I successfully created the ca-bundle.trust.crt which is needed by the IPFire ca-certificates lfs file.
I will now try and create a new build script that will do all the new type stuff and also install the p11-kit library files and see how things go.
:crossed_fingers:
Adolf
Regards, Adolf.
> -Michael > >> Regards, >> Adolf. >>> -Michael >>>> On 8 Aug 2021, at 14:47, Adolf Belka adolf.belka@ipfire.org wrote: >>>> >>>> Hi All, >>>> >>>> I had another go at the ca-certificates problem, the last barrier to getting rid of python2. >>>> >>>> I found certdata2pem.py files from fedora and 2 from suse. I created build subdirectories for each version so I could just test running the build.sh file with each version of certdata2pem.py, including the IPFire current version after running through the 2to3 convertor. >>>> >>>> fedora >>>> >>>> The fedora certdata2pem.py file runs successfully with python3 but has sed commands built into it which fail to find certain files. The sed commands are not in the IPFire version. >>>> >>>> The error message is >>>> >>>> -> written as 'Certum_Trusted_Root_CA:2.16.30.191.89.80.184.201.128.55.76.6.247.235.85.79.181.237.tmp-p11-kit', trust = ['CKA_TRUST_SERVER_AUTH', 'CKA_TRUST_EMAIL_PROTECTION'], openssl-trust = ['serverAuth', 'emailProtection'], distrust = [], openssl-distrust = [] >>>> sed: can't read certs/*.crt: No such file or directory >>>> >>>> >>>> suse >>>> >>>> The first suse version runs successfully with python3 but also has the sed commands in it with the same error message. >>>> >>>> The second suse version runs successfully with python3, does not have the sed commands and completes the build.sh script with no errors. However this certdata2pem.py file has a section that is in the IPFire version completely missing. >>>> >>>> >>>> IPfire version after running through the 2to3 convertor >>>> >>>> The following error message occurs >>>> >>>> producing trust for "GlobalSign Root CA"2.11.4.0.0.0.0.1.21.75.90.195.148 >>>> Traceback (most recent call last): >>>> File "/mnt/File_Server/Computers/Linux/ipfire/sandbox/patch in progress/python/ca-certificates/orig-2to3-build/certs/../certdata2pem.py", line 224, in <module> >>>> f.write("\n".join(textwrap.wrap(base64.b64encode(obj['CKA_VALUE']), 64))) >>>> File "/usr/lib/python3.9/base64.py", line 58, in b64encode >>>> encoded = binascii.b2a_base64(s, newline=False) >>>> TypeError: a bytes-like object is required, not 'str' >>>> >>>> The section that is failing is the section that is missing in the 2nd suse version. There is an identical fwrite line at line 206 but that does not seem to flag up the same TypeError message. >>>> >>>> >>>> As the certdata2pem.py files from the other distributions vary significantly in content, with some having nearly double the number of lines of code, I think the best alternative is to fix the IPFire version so we stay consistent but I am unable to figure out how to fix the python code that is causing the " TypeError: a bytes-like object is required, not 'str' " error message and need someone's help with that. >>>> >>>> Let me know if there is any other information that I need to provide. >>>> >>>> >>>> Regards, >>>> >>>> Adolf. >>>> >>>> >>>> On 07/08/2021 15:54, Adolf Belka wrote: >>>>> Hi All, >>>>> >>>>> On 04/08/2021 16:45, Michael Tremer wrote: >>>>>> Hello, >>>>>> >>>>>>> On 4 Aug 2021, at 13:40, Adolf Belka adolf.belka@ipfire.org wrote: >>>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> I have resolved the frr program build. The version currently in IPFire (6.0) only works with python2. Python3 support came in with version 7.4. I have now built frr with version 8.0 including libyang as a new dependency but only for the build, so nothing installed into IPFire itself, and that has successfully built without python2 being present. >>>>>> >>>>>> Great. This could also resolve Matthias’ problem with building frr. >>>>>> >>>>>>> Will now go back and have another go with spice-protocol. >>>>>> >>>>>> Maybe it has a —-disable-python switch? >>>>> I just removed the line in the spice-protocol lfs that ran automake/py-compile on the python modules from spice. >>>>> Spice and spice-protocol are present for qemu and with the py-compile line removed all three successfully built without python2 being present. I have submitted a patch for this combined with updating spice and spice-protocol, both from 2017. >>>>> >>>>> This now only leaves the ca-certificates script that needs to be updated to work with python3. >>>>> >>>>> Regards, >>>>> Adolf. >>>>>> >>>>>> -Michael >>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Adolf. >>>>>>> >>>>>>> >>>>>>> On 03/08/2021 23:38, Adolf Belka wrote: >>>>>>>> Hi Michael & all, >>>>>>>> >>>>>>>> >>>>>>>> On 03/08/2021 17:11, Michael Tremer wrote: >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> Thank you for looking into this. >>>>>>>>> >>>>>>>>> This is a third-party script that came from either Mozilla or RedHat. Maybe they have ported it. If not, it should not be rocket science to do it ourselves. If we do it, we should of course upstream it. >>>>>>>> I found an updated script from fedora and gave that a try. This time the script went all the way through but then the build.sh script failed at the point where it should find all the .crt files in the certs directory and it came back and said there weren't any. >>>>>>>>> >>>>>>>>> However, can you comment out this package and continue the build? This should be required until you reach the cdrom stage. >>>>>>>> I then commented ca-certificates out in make.sh and ran the build. >>>>>>>> This time it stopped at spice-protocol which is an addon and uses the py-compile script that is in automake to compile some python modules. >>>>>>>> py-compile is python2 based and the build stopped because it could not find python >>>>>>>> >>>>>>>> There is a py_compile.py script that is python3 based but when I ran that in place of the py-compile script I got a Permission denied error when it tried to carry out the compile. >>>>>>>> >>>>>>>> I then commented out spice-protocol and ran the build. >>>>>>>> >>>>>>>> It then failed on frr which did look for python3-config but then failed due to not finding python-config or pkg-config python >>>>>>>> It looks like I should be able to tell it to use python3 in the ./configure >>>>>>>> >>>>>>>> I commented out frr and nothing else failed before cdrom was reached. >>>>>>>> >>>>>>>> So the packages that need to be made to work with python3 are >>>>>>>> ca-certificates >>>>>>>> spice-protocol >>>>>>>> frr >>>>>>>> >>>>>>>> >>>>>>>> I also converted client175 with 2to3 converter and built it and installed the .ipfire package into a vm and successfully got the WUI page for Media Player IPFire to render. What I haven't tested yet is if the audio works. I will need to get audio set up in my vm to try that. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Adolf. >>>>>>>> >>>>>>>>> >>>>>>>>> If this is the only thing that flags up, we should port the script. If we find another, stronger reason to keep Python 2 around, we do not need to bother and can keep the script this way. >>>>>>>>> >>>>>>>>> -Michael >>>>>>>>> >>>>>>>>>> On 3 Aug 2021, at 13:31, Adolf Belka adolf.belka@ipfire.org wrote: >>>>>>>>>> >>>>>>>>>> Hi All, >>>>>>>>>> >>>>>>>>>> So with crda and the remaining python2 modules removed the question was if removing python2 from the build ran without any problem or if something was flagged up. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ca-certificates was flagged up. >>>>>>>>>> >>>>>>>>>> There is a python2 script, certdata2pem.py, which fails if python2 is not present. Running that script with python3 flags up some invalid syntax, unsurprisingly. >>>>>>>>>> >>>>>>>>>> I found some patches in Debian from 2015 for certdata2pem.py to provide python3 compatibility. Unfortunately looking at the patch approx half could not be applied because the lines don't exist in the IPFire version of certdata2pem.py (sections to do with blacklisted certs) >>>>>>>>>> >>>>>>>>>> I then ran the 2to3 converter on certdata2pem.py and tried that in the build but it came up with the following error. >>>>>>>>>> >>>>>>>>>> TypeError: a bytes-like object is required, not 'str' >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I don't know how to further move forward with this as I am totally unfamiliar with the python language. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> >>>>>>>>>> Adolf.