From mboxrd@z Thu Jan 1 00:00:00 1970 From: Adolf Belka To: development@lists.ipfire.org Subject: Re: Question about updating python to 3.10 Date: Wed, 12 Jan 2022 22:08:57 +0100 Message-ID: <792fc6da-6060-bc0b-c995-e90227839046@ipfire.org> In-Reply-To: <3AA65054-9B42-4834-8D28-9F89B37BA3BF@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============8942690035941495861==" List-Id: --===============8942690035941495861== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi all, On 12/01/2022 19:10, Michael Tremer wrote: > Hey, >=20 >> On 11 Jan 2022, at 14:32, Adolf Belka wrote: >> >> Hi Michael and all, >> >> python3-six has to stay as python3-dateutil explicitly specifies it in its= dependencies. >> So that solves that question. >=20 > Aww :( >=20 >> I have, with quite a bit of work, eventually been able to get all existing= python modules updated that had updates and built successfully with python3.= 10 >=20 > Yay! >=20 >> However later in the IPFire build libvirt failed with a problem parsing so= me documentation files. >=20 >> Stashing my python3.10 changes and re-running the build had libvirt buildi= ng without problems. So one of the python changes has caused it. libvirt uses= python3-docutils and that was updated but reverting that back to original on= its own did not cause libvirt to build successfully. >> >> I looked in the libvirt .configure for an option to not build the docs but= it is not mentioned under ./configure --help. >> >> I tried --without-docs and --disable-docs but both were not recognised in = the build. >> >> The current libvirt is from mid 2020 (6.5.0) so it is likely that it is no= t compatible in some way with the updated python/modules. >> >> I tried updating libvirt to the current 7.10.0 >> libvirt has replaced autotools with meson. I found meson options available= for all existing ./configure options except three. There is nothing in meson= options for:- >> --with-virtualport >> --with-macvtap >> --without-dbus >=20 > Yay, another project that is using meson :( >=20 > We no longer need macvtap; we have dubs, so that should not hurt; and I hav= e no idea what virtualport is. >=20 >> I tried building it with all the other options and it successfully built w= ith the updated python3.10 and modules. All other packages after libvirt buil= t successfully without any problems. >> >> Normally I would then use the updated libvirt, as that is needed to work w= ith the updated python packages, but I am not sure if those missing options a= re now built into libvirt or are no longer available and what impact that mig= ht have for the use of libvirt on IPFire. >=20 > I don=E2=80=99t think this would hurt us much. I have CCed Jonatan who main= tains this, so maybe he knows more about this. >=20 >> There is nothing in the libvirt changelog about virtualport, macvtap or db= us. >> >> Searching on the internet, I was also not able to find anything related to= libvirt and these options. >> >> Any suggestions/proposals on how to move forward on this? Should I just bu= ild libvirt without those three options or ...? >=20 > Yes, and I think this could be submitted independently of the Python branch= because it is an add-on. We should be able to ship this at any time and migh= t want to delay Python until we have enough space in a core update. Okay, then I will stash my python3.10 changes and submit the libvirt=20 update on its own. Then Jonaton can test it out. I don't use libvirt or=20 qemu so I can't test it easily. Then when you think the time is right, let me know and I will submit the=20 python3.10 series of patches. >=20 > Thank you for looking into this. Sounds like a not so nice riddle to solve = :) There were certainly some challenges, like updating python3-daemon which=20 then wanted to have twine as a fixed dependency which had nine=20 dependencies some of which had more. I had got to 19 additional python=20 packages and still going and then I said enough was enough and looked at=20 twine. It is required for uploading python packages to PyPI which IPFire=20 doesn't need. So I tried patching the twine requirement out and that=20 worked and then I found others that had done the same because they had=20 no response to the request to not have twine as a mandatory build=20 requirement. I found it all challenging but rewarding when I made progress and=20 eventually succeeded in building everything without errors. Regards, Adolf. >=20 > -Michael >=20 >> >> >> Regards, >> >> Adolf. >> >> On 07/01/2022 17:49, Michael Tremer wrote: >>> Hello, >>>> On 4 Jan 2022, at 12:49, Adolf Belka wrote: >>>> >>>> Hi Michael, >>>> >>>> Following the suggested Arch Linux approach I have got python3-setuptool= s-scm building successfully now. Yaaah. >>>> I had to install python3-toml, followed by python3-pyproject2setuppy fol= lowed by python3-tomli. >>>> >>>> However then python3-six which installs after python3-setuptools-scm fai= ls due to missing the python package python3-packaging. >>>> >>>> I can install that package, and will do so as I continue testing out the= python 3.10 build but I have the following question about python3-six. >>>> >>>> Is six still needed. It is designed to ensure that programs can work wit= h either python2 or python3 but IPFire only has python3 now. >>>> I can find python3-six referenced only by aws-cli. In the aws-cli build = python3-six is not listed at all in the build requirements. I tried to search= for aws-cli and python3-six being needed in run mode but I could not find an= ything one way or the other. >>> If aws-cli no longer requires python3-six we can drop it. Last time we to= uched it I had to bring the package back because aws-cli was still depending = on it. >>>> If python3-six is really needed for running aws-cli then I can install i= t and the new dependency but if it is not needed we could remove two packages= which would partially offset the three new dependencies required to be able = to build the latest python3-setuptools-scm. >>> Yes, that would be great :) >>> -Michael >>>> >>>> Regards, >>>> Adolf >>>> >>>> On 03/01/2022 12:32, Michael Tremer wrote: >>>>> Hello, >>>>>> On 3 Jan 2022, at 11:31, Adolf Belka wrote: >>>>>> >>>>>> Hi Michael, >>>>>> >>>>>> On 03/01/2022 11:52, Michael Tremer wrote: >>>>>>> Hello, >>>>>>> I am not sure if you already looked at what Arch is doing, but as far= as I understand it works like this: >>>>>> No I didn't look at Arch, which is surprising as I have in the past wi= th other packages but I didn't. So thanks very much for the pointer. >>>>>>> They build Python 3 without pip and setuptools (https://github.com/ar= chlinux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/trunk= /PKGBUILD#L59). >>>>>>> At this stage we might need to build setuptools separately. >>>>>>> They use a tool called pyproject2setuppy (great name!) to build tomli= (https://github.com/archlinux/svntogit-packages/blob/packages/python-tomli/t= runk/PKGBUILD). >>>>>>> Setuptools-scm can then be built with tomli: https://github.com/archl= inux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD >>>>>>> Would this work or did I overlook anything? >>>>>> I will have a look at the links and give it a go. >>>>>> >>>>>> Thanks very much, I feel re-motivated now. >>>>> Very good! I got a little bit lost in your last email and I think I cou= ld tell. But following a solution that somebody else has come up with can be = a good thing to follow and of course we would customise it to our needs. >>>>> Best, >>>>> -Michael >>>>>> >>>>>> Regards, >>>>>> Adolf. >>>>>>> -Michael >>>>>>>> On 2 Jan 2022, at 13:44, Adolf Belka wrot= e: >>>>>>>> >>>>>>>> Hi Michael and All, >>>>>>>> >>>>>>>> On 26/12/2021 21:55, Michael Tremer wrote: >>>>>>>>> Merry Christmas, >>>>>>>>>> On 26 Dec 2021, at 15:38, Adolf Belka w= rote: >>>>>>>>>> >>>>>>>>>> Hi Michael, >>>>>>>>>> >>>>>>>>>> On 17/12/2021 12:41, Michael Tremer wrote: >>>>>>>>>>> Hello Adolf, >>>>>>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka = wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi All, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I am working on updating python from 3.8 to 3.10. Python itself = is okay after some other python related programs were also updated due to the= ir older versions not working with 3.10 >>>>>>>>>>> Thank you for working on this. >>>>>>>>>>>> I am working through all the programs that reference python-3.8 = in their rootfiles, or wherever, to change them to 3.10 >>>>>>>>>>>> >>>>>>>>>>>> My question comes with regard to the changes to the rootfiles to= ensure they will be included properly in an update. >>>>>>>>>>> We will have to ship everything that puts anything into /usr/lib*= /python3.*/site-packages again with the release of Python 3.10. >>>>>>>>>>> This isn=E2=80=99t fun, but not as much of a pain in the rear as = shipping a new Perl release. >>>>>>>>>>>> For the addons then when I change the rootfile I am also increme= nting the PAK_VER number in the lfs files and that ensures that they will be = properly updated. >>>>>>>>>>> Yes. That is required for Pakfire to find a new version. >>>>>>>>>>>> I don't know what to do for the core programs where I have updat= ed the rootfile. The lfs files for these programs don't have a PAK_VER number= to increment and the actual package version number is not being changed so I= don't know what I need to change in the lfs to ensure that the rootfile chan= ges will be properly included in the update? >>>>>>>>>>> I would say committing everything into a =E2=80=9Cpython-3.10=E2= =80=9D branch and submitting it all as a patchset is fine. All packages that = have been touched in that branch will have to be shipped again which should b= e easy to identify if it all comes in one large branch. >>>>>>>>>> Okay, that is clear then.Will follow that approach. >>>>>>>>>>>> I would appreciate any help on what I need to do for these progr= ams that have had changes only in their rootfiles. >>>>>>>>>>> Absolutely. Did you find anything that didn=E2=80=99t build yet? >>>>>>>>>> >>>>>>>>>> I have not had any problem with python3.10 itself but I found some= of the older modules would not work with 3.10 so I decided to do a combined = update to 3.10 plus update modules to latest versions. >>>>>>>>>> >>>>>>>>>> I found that after python3-setuptools-scm had run that the next mo= dule python3-six failed to run due to missing the tomli library. I commented = out six and reran the build and then python3-dateutil had the same problem a= nd then python3-jmespath >>>>>>>>> I completely lost track of what the status of setuptools/distutils = and so on is. This has changed multiple times and become independent/part of = the distribution again and so on. >>>>>>>>>> After some investigation I have found that since version 6.0.0 of = python3-setuptools-scm, tomli has been a required library when it is being us= ed by other modules. So running version 6.0.0 of setuptools-scm worked with p= ython3.10 and subsequent modules built successfully as tomli not needed. >>>>>>>>>> >>>>>>>>>> So the obvious fix is to install tomli but that is when I ran into= a problem. tomli has no setup.py or setup.cfg files in its tarball. The only= way to build it is using pip. >>>>>>>>> Err. Great. >>>>>>>>>> Apparently, after some searching, setup.py has been deprecated as = the recommended install method and if setup.py doesn't work the usual fix sug= gestion is to run pip install. >>>>>>>>> Yes. But apparently the new system is deprecated, too. I really dis= like the state of packaging systems that come with any language these days. >>>>>>>>>> Having searched on pip for a while, it looks like builds using it = can still be done from tarballs locally without needing to access the interne= t to download the modules. >>>>>>>>> That is good! >>>>>>>>>> As pip is installed as part of python3 i tried using that. >>>>>>>>>> >>>>>>>>>> I used "python3 -m pip install --root=3D/ ." in place of >>>>>>>>>> "python3 setup.py build >>>>>>>>>> python3 setup.py install --root=3D/" >>>>>>>>>> for the tomli source. >>>>>>>>>> >>>>>>>>>> This failed because tomli requires flit_core to be available. >>>>>>>>>> >>>>>>>>>> flit_core also has no setup.py and so I tried to build it with pip= install. This failed with the error message that tomli was required as a dep= endency. So I added --no-deps to the flit_core pip install line and this time= flit_core successfully built but the tomli build still came back with not fi= nding flit_core. >>>>>>>>>> >>>>>>>>>> I have tried a variety of changes to the pip install line but none= of them has been successful and it has become clear that I am out of my dept= h here and need guidance rather than just trying anything I find from a googl= e search. I am not even really clear if pip install is the correct approach, = except if not then I have no idea how to progress. >>>>>>>>>> >>>>>>>>>> Help urgently needed on how I should be approaching this. >>>>>>>>> Circular dependencies can happen. This won=E2=80=99t be the first t= ime and it always is an utter nightmare with our buildsystem. >>>>>>>>> This page states that pip comes by default with all binary packages: >>>>>>>>> https://docs.python.org/3/installing/index.html >>>>>>>>> Is there a chance that it is part of the distribution and can be en= abled with a configure switch? >>>>>>>> I wasn't able to find any way to get it enabled that way. >>>>>>>> >>>>>>>> I did find that pip install will not install if the package is alrea= dy present. You have to add --upgrade to make it update any package already p= resent. So I tried that. Didn't work. >>>>>>>> >>>>>>>> I found a section about needing to use bootstrapping if you want to = install tomli. Followed the information for creating a wheel file for flit_co= re and then extracting that file into the site-packages directory. Tried that= . Didn't work. >>>>>>>> >>>>>>>> Rather than doing a complete build every time I changed to trying to= build flit_core followed by tomli in a ./make.sh shell environment. >>>>>>>> >>>>>>>> Doing that I found that I had a pip install command that was creatin= g the flit_core directory in site-packages and also including the .dist-info = directory. The info I found said that the .dist-info directory had to be ther= e for pip to be able to find the flit_core package. >>>>>>>> >>>>>>>> However the pip install of tomli still ends up not finding any flit_= core package. In the bootstrapping information it mentioned about changing th= e PYTHONPATH environment variable for find the flit_core package so I tried t= hat in the shell but it also made no difference. >>>>>>>> >>>>>>>> I then tried running python3 -c 'import flit_core' in the tomli buil= d shell before doing the tomli pip install. The import command completed with= no error but tomli still could not find any flit_core package. >>>>>>>> >>>>>>>> End result of all my work is that flit_core has been built and insta= lled into python3.10/site-packages but nothing seems to want to make the toml= i build find it. >>>>>>>> >>>>>>>> Does anyone have any ideas on what command(s) I need to use in the t= omli pip install to have the installed flit_core package visible. >>>>>>>> >>>>>>>> Regards and a Happy and Healthy New Year to all, >>>>>>>> >>>>>>>> Adolf. >>>>>>>> >>>>>>>>> -Michael >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Adolf. >>>>>>>>>> >>>>>>>>>> PS: I hope everyone had an enjoyable Christmas. >>>>>>>>> I hope you did, too! >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> -Michael >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> >>>>>>>>>>>> Adolf. >>>>>>>>>>>> >>>>>>>>>>>> --=20 >>>>>>>>>>>> Sent from my laptop >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> --=20 >>>>>>>>>> Sent from my laptop >>>>>>>> >>>>>>>> --=20 >>>>>>>> Sent from my laptop >>>> >>>> --=20 >>>> Sent from my laptop >> >> --=20 >> Sent from my laptop >=20 --=20 Sent from my laptop --===============8942690035941495861==--