From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: Question about updating python to 3.10 Date: Thu, 13 Jan 2022 09:32:23 +0000 Message-ID: In-Reply-To: <792fc6da-6060-bc0b-c995-e90227839046@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============5143905423667011213==" List-Id: --===============5143905423667011213== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello, > On 12 Jan 2022, at 21:08, Adolf Belka wrote: >=20 > Hi all, >=20 > On 12/01/2022 19:10, Michael Tremer wrote: >> Hey, >>> On 11 Jan 2022, at 14:32, Adolf Belka wrote: >>>=20 >>> Hi Michael and all, >>>=20 >>> python3-six has to stay as python3-dateutil explicitly specifies it in it= s dependencies. >>> So that solves that question. >> Aww :( >>> I have, with quite a bit of work, eventually been able to get all existin= g python modules updated that had updates and built successfully with python3= .10 >> Yay! >>> However later in the IPFire build libvirt failed with a problem parsing s= ome documentation files. >>> Stashing my python3.10 changes and re-running the build had libvirt build= ing without problems. So one of the python changes has caused it. libvirt use= s python3-docutils and that was updated but reverting that back to original o= n its own did not cause libvirt to build successfully. >>>=20 >>> I looked in the libvirt .configure for an option to not build the docs bu= t it is not mentioned under ./configure --help. >>>=20 >>> I tried --without-docs and --disable-docs but both were not recognised in= the build. >>>=20 >>> The current libvirt is from mid 2020 (6.5.0) so it is likely that it is n= ot compatible in some way with the updated python/modules. >>>=20 >>> I tried updating libvirt to the current 7.10.0 >>> libvirt has replaced autotools with meson. I found meson options availabl= e for all existing ./configure options except three. There is nothing in meso= n options for:- >>> --with-virtualport >>> --with-macvtap >>> --without-dbus >> Yay, another project that is using meson :( >> We no longer need macvtap; we have dubs, so that should not hurt; and I ha= ve no idea what virtualport is. >>> I tried building it with all the other options and it successfully built = with the updated python3.10 and modules. All other packages after libvirt bui= lt successfully without any problems. >>>=20 >>> Normally I would then use the updated libvirt, as that is needed to work = with the updated python packages, but I am not sure if those missing options = are now built into libvirt or are no longer available and what impact that mi= ght have for the use of libvirt on IPFire. >> I don=E2=80=99t think this would hurt us much. I have CCed Jonatan who mai= ntains this, so maybe he knows more about this. >>> There is nothing in the libvirt changelog about virtualport, macvtap or d= bus. >>>=20 >>> Searching on the internet, I was also not able to find anything related t= o libvirt and these options. >>>=20 >>> Any suggestions/proposals on how to move forward on this? Should I just b= uild libvirt without those three options or ...? >> Yes, and I think this could be submitted independently of the Python branc= h because it is an add-on. We should be able to ship this at any time and mig= ht want to delay Python until we have enough space in a core update. > Okay, then I will stash my python3.10 changes and submit the libvirt update= on its own. Then Jonaton can test it out. I don't use libvirt or qemu so I c= an't test it easily. >=20 > Then when you think the time is right, let me know and I will submit the py= thon3.10 series of patches. You can submit this at any time. I was just thinking about the conversation b= etween Peter and Arne during the last call regarding an updated kernel and up= dated firmware. Depending on the size of Python, all of this might make an up= date a little bit too large - however, this should not be a patchset that we = cannot merge within two merge windows. >> Thank you for looking into this. Sounds like a not so nice riddle to solve= :) > There were certainly some challenges, like updating python3-daemon which th= en wanted to have twine as a fixed dependency which had nine dependencies som= e of which had more. I had got to 19 additional python packages and still goi= ng and then I said enough was enough and looked at twine. It is required for = uploading python packages to PyPI which IPFire doesn't need. So I tried patch= ing the twine requirement out and that worked and then I found others that ha= d done the same because they had no response to the request to not have twine= as a mandatory build requirement. Python seems to become the hell that Perl used to be. In order to add one per= l package, you had 20 dependencies. And those dependencies had their own depe= ndencies too=E2=80=A6 It would be great to disable or patch out whatever is p= ossible so that we do not create more work for ourselves in the future and ha= ve to update packages that are barely used - we already have enough of these = :) > I found it all challenging but rewarding when I made progress and eventuall= y succeeded in building everything without errors. I am so happy that you are putting all this time and effort into this because= it certainly requires patience - a trade I don=E2=80=99t have. And Python is being used in all sorts of places in the distribution, so it re= ally isn=E2=80=99t anything we should not be up to date on. -Michael >=20 > Regards, > Adolf. >> -Michael >>>=20 >>>=20 >>> Regards, >>>=20 >>> Adolf. >>>=20 >>> On 07/01/2022 17:49, Michael Tremer wrote: >>>> Hello, >>>>> On 4 Jan 2022, at 12:49, Adolf Belka wrote: >>>>>=20 >>>>> Hi Michael, >>>>>=20 >>>>> Following the suggested Arch Linux approach I have got python3-setuptoo= ls-scm building successfully now. Yaaah. >>>>> I had to install python3-toml, followed by python3-pyproject2setuppy fo= llowed by python3-tomli. >>>>>=20 >>>>> However then python3-six which installs after python3-setuptools-scm fa= ils due to missing the python package python3-packaging. >>>>>=20 >>>>> I can install that package, and will do so as I continue testing out th= e python 3.10 build but I have the following question about python3-six. >>>>>=20 >>>>> Is six still needed. It is designed to ensure that programs can work wi= th either python2 or python3 but IPFire only has python3 now. >>>>> I can find python3-six referenced only by aws-cli. In the aws-cli build= python3-six is not listed at all in the build requirements. I tried to searc= h for aws-cli and python3-six being needed in run mode but I could not find a= nything one way or the other. >>>> If aws-cli no longer requires python3-six we can drop it. Last time we t= ouched it I had to bring the package back because aws-cli was still depending= on it. >>>>> If python3-six is really needed for running aws-cli then I can install = it and the new dependency but if it is not needed we could remove two package= s which would partially offset the three new dependencies required to be able= to build the latest python3-setuptools-scm. >>>> Yes, that would be great :) >>>> -Michael >>>>>=20 >>>>> Regards, >>>>> Adolf >>>>>=20 >>>>> On 03/01/2022 12:32, Michael Tremer wrote: >>>>>> Hello, >>>>>>> On 3 Jan 2022, at 11:31, Adolf Belka wrote: >>>>>>>=20 >>>>>>> Hi Michael, >>>>>>>=20 >>>>>>> On 03/01/2022 11:52, Michael Tremer wrote: >>>>>>>> Hello, >>>>>>>> I am not sure if you already looked at what Arch is doing, but as fa= r as I understand it works like this: >>>>>>> No I didn't look at Arch, which is surprising as I have in the past w= ith other packages but I didn't. So thanks very much for the pointer. >>>>>>>> They build Python 3 without pip and setuptools (https://github.com/a= rchlinux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/trun= k/PKGBUILD#L59). >>>>>>>> At this stage we might need to build setuptools separately. >>>>>>>> They use a tool called pyproject2setuppy (great name!) to build toml= i (https://github.com/archlinux/svntogit-packages/blob/packages/python-tomli/= trunk/PKGBUILD). >>>>>>>> Setuptools-scm can then be built with tomli: https://github.com/arch= linux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD >>>>>>>> Would this work or did I overlook anything? >>>>>>> I will have a look at the links and give it a go. >>>>>>>=20 >>>>>>> Thanks very much, I feel re-motivated now. >>>>>> Very good! I got a little bit lost in your last email and I think I co= uld tell. But following a solution that somebody else has come up with can be= a good thing to follow and of course we would customise it to our needs. >>>>>> Best, >>>>>> -Michael >>>>>>>=20 >>>>>>> Regards, >>>>>>> Adolf. >>>>>>>> -Michael >>>>>>>>> On 2 Jan 2022, at 13:44, Adolf Belka wro= te: >>>>>>>>>=20 >>>>>>>>> Hi Michael and All, >>>>>>>>>=20 >>>>>>>>> On 26/12/2021 21:55, Michael Tremer wrote: >>>>>>>>>> Merry Christmas, >>>>>>>>>>> On 26 Dec 2021, at 15:38, Adolf Belka = wrote: >>>>>>>>>>>=20 >>>>>>>>>>> Hi Michael, >>>>>>>>>>>=20 >>>>>>>>>>> On 17/12/2021 12:41, Michael Tremer wrote: >>>>>>>>>>>> Hello Adolf, >>>>>>>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka wrote: >>>>>>>>>>>>>=20 >>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>=20 >>>>>>>>>>>>>=20 >>>>>>>>>>>>> I am working on updating python from 3.8 to 3.10. Python itself= is okay after some other python related programs were also updated due to th= eir older versions not working with 3.10 >>>>>>>>>>>> Thank you for working on this. >>>>>>>>>>>>> I am working through all the programs that reference python-3.8= in their rootfiles, or wherever, to change them to 3.10 >>>>>>>>>>>>>=20 >>>>>>>>>>>>> My question comes with regard to the changes to the rootfiles t= o ensure they will be included properly in an update. >>>>>>>>>>>> We will have to ship everything that puts anything into /usr/lib= */python3.*/site-packages again with the release of Python 3.10. >>>>>>>>>>>> This isn=E2=80=99t fun, but not as much of a pain in the rear as= shipping a new Perl release. >>>>>>>>>>>>> For the addons then when I change the rootfile I am also increm= enting the PAK_VER number in the lfs files and that ensures that they will be= properly updated. >>>>>>>>>>>> Yes. That is required for Pakfire to find a new version. >>>>>>>>>>>>> I don't know what to do for the core programs where I have upda= ted the rootfile. The lfs files for these programs don't have a PAK_VER numbe= r to increment and the actual package version number is not being changed so = I don't know what I need to change in the lfs to ensure that the rootfile cha= nges will be properly included in the update? >>>>>>>>>>>> I would say committing everything into a =E2=80=9Cpython-3.10=E2= =80=9D branch and submitting it all as a patchset is fine. All packages that = have been touched in that branch will have to be shipped again which should b= e easy to identify if it all comes in one large branch. >>>>>>>>>>> Okay, that is clear then.Will follow that approach. >>>>>>>>>>>>> I would appreciate any help on what I need to do for these prog= rams that have had changes only in their rootfiles. >>>>>>>>>>>> Absolutely. Did you find anything that didn=E2=80=99t build yet? >>>>>>>>>>>=20 >>>>>>>>>>> I have not had any problem with python3.10 itself but I found som= e of the older modules would not work with 3.10 so I decided to do a combined= update to 3.10 plus update modules to latest versions. >>>>>>>>>>>=20 >>>>>>>>>>> I found that after python3-setuptools-scm had run that the next m= odule python3-six failed to run due to missing the tomli library. I commented= out six and reran the build and then python3-dateutil had the same problem = and then python3-jmespath >>>>>>>>>> I completely lost track of what the status of setuptools/distutils= and so on is. This has changed multiple times and become independent/part of= the distribution again and so on. >>>>>>>>>>> After some investigation I have found that since version 6.0.0 of= python3-setuptools-scm, tomli has been a required library when it is being u= sed by other modules. So running version 6.0.0 of setuptools-scm worked with = python3.10 and subsequent modules built successfully as tomli not needed. >>>>>>>>>>>=20 >>>>>>>>>>> So the obvious fix is to install tomli but that is when I ran int= o a problem. tomli has no setup.py or setup.cfg files in its tarball. The onl= y way to build it is using pip. >>>>>>>>>> Err. Great. >>>>>>>>>>> Apparently, after some searching, setup.py has been deprecated as= the recommended install method and if setup.py doesn't work the usual fix su= ggestion is to run pip install. >>>>>>>>>> Yes. But apparently the new system is deprecated, too. I really di= slike the state of packaging systems that come with any language these days. >>>>>>>>>>> Having searched on pip for a while, it looks like builds using it= can still be done from tarballs locally without needing to access the intern= et to download the modules. >>>>>>>>>> That is good! >>>>>>>>>>> As pip is installed as part of python3 i tried using that. >>>>>>>>>>>=20 >>>>>>>>>>> I used "python3 -m pip install --root=3D/ ." in place of >>>>>>>>>>> "python3 setup.py build >>>>>>>>>>> python3 setup.py install --root=3D/" >>>>>>>>>>> for the tomli source. >>>>>>>>>>>=20 >>>>>>>>>>> This failed because tomli requires flit_core to be available. >>>>>>>>>>>=20 >>>>>>>>>>> flit_core also has no setup.py and so I tried to build it with pi= p install. This failed with the error message that tomli was required as a de= pendency. So I added --no-deps to the flit_core pip install line and this tim= e flit_core successfully built but the tomli build still came back with not f= inding flit_core. >>>>>>>>>>>=20 >>>>>>>>>>> I have tried a variety of changes to the pip install line but non= e of them has been successful and it has become clear that I am out of my dep= th here and need guidance rather than just trying anything I find from a goog= le search. I am not even really clear if pip install is the correct approach,= except if not then I have no idea how to progress. >>>>>>>>>>>=20 >>>>>>>>>>> Help urgently needed on how I should be approaching this. >>>>>>>>>> Circular dependencies can happen. This won=E2=80=99t be the first = time and it always is an utter nightmare with our buildsystem. >>>>>>>>>> This page states that pip comes by default with all binary package= s: >>>>>>>>>> https://docs.python.org/3/installing/index.html >>>>>>>>>> Is there a chance that it is part of the distribution and can be e= nabled with a configure switch? >>>>>>>>> I wasn't able to find any way to get it enabled that way. >>>>>>>>>=20 >>>>>>>>> I did find that pip install will not install if the package is alre= ady present. You have to add --upgrade to make it update any package already = present. So I tried that. Didn't work. >>>>>>>>>=20 >>>>>>>>> I found a section about needing to use bootstrapping if you want to= install tomli. Followed the information for creating a wheel file for flit_c= ore and then extracting that file into the site-packages directory. Tried tha= t. Didn't work. >>>>>>>>>=20 >>>>>>>>> Rather than doing a complete build every time I changed to trying t= o build flit_core followed by tomli in a ./make.sh shell environment. >>>>>>>>>=20 >>>>>>>>> Doing that I found that I had a pip install command that was creati= ng the flit_core directory in site-packages and also including the .dist-info= directory. The info I found said that the .dist-info directory had to be the= re for pip to be able to find the flit_core package. >>>>>>>>>=20 >>>>>>>>> However the pip install of tomli still ends up not finding any flit= _core package. In the bootstrapping information it mentioned about changing t= he PYTHONPATH environment variable for find the flit_core package so I tried = that in the shell but it also made no difference. >>>>>>>>>=20 >>>>>>>>> I then tried running python3 -c 'import flit_core' in the tomli bui= ld shell before doing the tomli pip install. The import command completed wit= h no error but tomli still could not find any flit_core package. >>>>>>>>>=20 >>>>>>>>> End result of all my work is that flit_core has been built and inst= alled into python3.10/site-packages but nothing seems to want to make the tom= li build find it. >>>>>>>>>=20 >>>>>>>>> Does anyone have any ideas on what command(s) I need to use in the = tomli pip install to have the installed flit_core package visible. >>>>>>>>>=20 >>>>>>>>> Regards and a Happy and Healthy New Year to all, >>>>>>>>>=20 >>>>>>>>> Adolf. >>>>>>>>>=20 >>>>>>>>>> -Michael >>>>>>>>>>>=20 >>>>>>>>>>> Regards, >>>>>>>>>>> Adolf. >>>>>>>>>>>=20 >>>>>>>>>>> PS: I hope everyone had an enjoyable Christmas. >>>>>>>>>> I hope you did, too! >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>>> -Michael >>>>>>>>>>>>>=20 >>>>>>>>>>>>> Regards, >>>>>>>>>>>>>=20 >>>>>>>>>>>>> Adolf. >>>>>>>>>>>>>=20 >>>>>>>>>>>>> --=20 >>>>>>>>>>>>> Sent from my laptop >>>>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>> --=20 >>>>>>>>>>> Sent from my laptop >>>>>>>>>=20 >>>>>>>>> --=20 >>>>>>>>> Sent from my laptop >>>>>=20 >>>>> --=20 >>>>> Sent from my laptop >>>=20 >>> --=20 >>> Sent from my laptop >=20 > --=20 > Sent from my laptop --===============5143905423667011213==--