From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: Question about updating python to 3.10 Date: Wed, 12 Jan 2022 18:10:19 +0000 Message-ID: <3AA65054-9B42-4834-8D28-9F89B37BA3BF@ipfire.org> In-Reply-To: <317de396-dbbf-c79d-ebb8-50f43ae60198@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============3989322371539827023==" List-Id: --===============3989322371539827023== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hey, > On 11 Jan 2022, at 14:32, Adolf Belka wrote: >=20 > Hi Michael and all, >=20 > python3-six has to stay as python3-dateutil explicitly specifies it in its = dependencies. > So that solves that question. Aww :( > I have, with quite a bit of work, eventually been able to get all existing = python modules updated that had updates and built successfully with python3.10 Yay! > However later in the IPFire build libvirt failed with a problem parsing som= e documentation files. > Stashing my python3.10 changes and re-running the build had libvirt buildin= g without problems. So one of the python changes has caused it. libvirt uses = python3-docutils and that was updated but reverting that back to original on = its own did not cause libvirt to build successfully. >=20 > I looked in the libvirt .configure for an option to not build the docs but = it is not mentioned under ./configure --help. >=20 > I tried --without-docs and --disable-docs but both were not recognised in t= he build. >=20 > The current libvirt is from mid 2020 (6.5.0) so it is likely that it is not= compatible in some way with the updated python/modules. >=20 > I tried updating libvirt to the current 7.10.0 > libvirt has replaced autotools with meson. I found meson options available = for all existing ./configure options except three. There is nothing in meson = options for:- > --with-virtualport > --with-macvtap > --without-dbus Yay, another project that is using meson :( We no longer need macvtap; we have dubs, so that should not hurt; and I have = no idea what virtualport is. > I tried building it with all the other options and it successfully built wi= th the updated python3.10 and modules. All other packages after libvirt built= successfully without any problems. >=20 > Normally I would then use the updated libvirt, as that is needed to work wi= th the updated python packages, but I am not sure if those missing options ar= e now built into libvirt or are no longer available and what impact that migh= t have for the use of libvirt on IPFire. I don=E2=80=99t think this would hurt us much. I have CCed Jonatan who mainta= ins this, so maybe he knows more about this. > There is nothing in the libvirt changelog about virtualport, macvtap or dbu= s. >=20 > Searching on the internet, I was also not able to find anything related to = libvirt and these options. >=20 > Any suggestions/proposals on how to move forward on this? Should I just bui= ld libvirt without those three options or ...? Yes, and I think this could be submitted independently of the Python branch b= ecause it is an add-on. We should be able to ship this at any time and might = want to delay Python until we have enough space in a core update. Thank you for looking into this. Sounds like a not so nice riddle to solve :) -Michael >=20 >=20 > Regards, >=20 > Adolf. >=20 > On 07/01/2022 17:49, Michael Tremer wrote: >> Hello, >>> On 4 Jan 2022, at 12:49, Adolf Belka wrote: >>>=20 >>> Hi Michael, >>>=20 >>> Following the suggested Arch Linux approach I have got python3-setuptools= -scm building successfully now. Yaaah. >>> I had to install python3-toml, followed by python3-pyproject2setuppy foll= owed by python3-tomli. >>>=20 >>> However then python3-six which installs after python3-setuptools-scm fail= s due to missing the python package python3-packaging. >>>=20 >>> I can install that package, and will do so as I continue testing out the = python 3.10 build but I have the following question about python3-six. >>>=20 >>> Is six still needed. It is designed to ensure that programs can work with= either python2 or python3 but IPFire only has python3 now. >>> I can find python3-six referenced only by aws-cli. In the aws-cli build p= ython3-six is not listed at all in the build requirements. I tried to search = for aws-cli and python3-six being needed in run mode but I could not find any= thing one way or the other. >> If aws-cli no longer requires python3-six we can drop it. Last time we tou= ched it I had to bring the package back because aws-cli was still depending o= n it. >>> If python3-six is really needed for running aws-cli then I can install it= and the new dependency but if it is not needed we could remove two packages = which would partially offset the three new dependencies required to be able t= o build the latest python3-setuptools-scm. >> Yes, that would be great :) >> -Michael >>>=20 >>> Regards, >>> Adolf >>>=20 >>> On 03/01/2022 12:32, Michael Tremer wrote: >>>> Hello, >>>>> On 3 Jan 2022, at 11:31, Adolf Belka wrote: >>>>>=20 >>>>> Hi Michael, >>>>>=20 >>>>> On 03/01/2022 11:52, Michael Tremer wrote: >>>>>> Hello, >>>>>> I am not sure if you already looked at what Arch is doing, but as far = as I understand it works like this: >>>>> No I didn't look at Arch, which is surprising as I have in the past wit= h other packages but I didn't. So thanks very much for the pointer. >>>>>> They build Python 3 without pip and setuptools (https://github.com/arc= hlinux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/trunk/= PKGBUILD#L59). >>>>>> At this stage we might need to build setuptools separately. >>>>>> They use a tool called pyproject2setuppy (great name!) to build tomli = (https://github.com/archlinux/svntogit-packages/blob/packages/python-tomli/tr= unk/PKGBUILD). >>>>>> Setuptools-scm can then be built with tomli: https://github.com/archli= nux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD >>>>>> Would this work or did I overlook anything? >>>>> I will have a look at the links and give it a go. >>>>>=20 >>>>> Thanks very much, I feel re-motivated now. >>>> Very good! I got a little bit lost in your last email and I think I coul= d tell. But following a solution that somebody else has come up with can be a= good thing to follow and of course we would customise it to our needs. >>>> Best, >>>> -Michael >>>>>=20 >>>>> Regards, >>>>> Adolf. >>>>>> -Michael >>>>>>> On 2 Jan 2022, at 13:44, Adolf Belka wrote: >>>>>>>=20 >>>>>>> Hi Michael and All, >>>>>>>=20 >>>>>>> On 26/12/2021 21:55, Michael Tremer wrote: >>>>>>>> Merry Christmas, >>>>>>>>> On 26 Dec 2021, at 15:38, Adolf Belka wr= ote: >>>>>>>>>=20 >>>>>>>>> Hi Michael, >>>>>>>>>=20 >>>>>>>>> On 17/12/2021 12:41, Michael Tremer wrote: >>>>>>>>>> Hello Adolf, >>>>>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka = wrote: >>>>>>>>>>>=20 >>>>>>>>>>> Hi All, >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>> I am working on updating python from 3.8 to 3.10. Python itself i= s okay after some other python related programs were also updated due to thei= r older versions not working with 3.10 >>>>>>>>>> Thank you for working on this. >>>>>>>>>>> I am working through all the programs that reference python-3.8 i= n their rootfiles, or wherever, to change them to 3.10 >>>>>>>>>>>=20 >>>>>>>>>>> My question comes with regard to the changes to the rootfiles to = ensure they will be included properly in an update. >>>>>>>>>> We will have to ship everything that puts anything into /usr/lib*/= python3.*/site-packages again with the release of Python 3.10. >>>>>>>>>> This isn=E2=80=99t fun, but not as much of a pain in the rear as s= hipping a new Perl release. >>>>>>>>>>> For the addons then when I change the rootfile I am also incremen= ting the PAK_VER number in the lfs files and that ensures that they will be p= roperly updated. >>>>>>>>>> Yes. That is required for Pakfire to find a new version. >>>>>>>>>>> I don't know what to do for the core programs where I have update= d the rootfile. The lfs files for these programs don't have a PAK_VER number = to increment and the actual package version number is not being changed so I = don't know what I need to change in the lfs to ensure that the rootfile chang= es will be properly included in the update? >>>>>>>>>> I would say committing everything into a =E2=80=9Cpython-3.10=E2= =80=9D branch and submitting it all as a patchset is fine. All packages that = have been touched in that branch will have to be shipped again which should b= e easy to identify if it all comes in one large branch. >>>>>>>>> Okay, that is clear then.Will follow that approach. >>>>>>>>>>> I would appreciate any help on what I need to do for these progra= ms that have had changes only in their rootfiles. >>>>>>>>>> Absolutely. Did you find anything that didn=E2=80=99t build yet? >>>>>>>>>=20 >>>>>>>>> I have not had any problem with python3.10 itself but I found some = of the older modules would not work with 3.10 so I decided to do a combined u= pdate to 3.10 plus update modules to latest versions. >>>>>>>>>=20 >>>>>>>>> I found that after python3-setuptools-scm had run that the next mod= ule python3-six failed to run due to missing the tomli library. I commented o= ut six and reran the build and then python3-dateutil had the same problem an= d then python3-jmespath >>>>>>>> I completely lost track of what the status of setuptools/distutils a= nd so on is. This has changed multiple times and become independent/part of t= he distribution again and so on. >>>>>>>>> After some investigation I have found that since version 6.0.0 of p= ython3-setuptools-scm, tomli has been a required library when it is being use= d by other modules. So running version 6.0.0 of setuptools-scm worked with py= thon3.10 and subsequent modules built successfully as tomli not needed. >>>>>>>>>=20 >>>>>>>>> So the obvious fix is to install tomli but that is when I ran into = a problem. tomli has no setup.py or setup.cfg files in its tarball. The only = way to build it is using pip. >>>>>>>> Err. Great. >>>>>>>>> Apparently, after some searching, setup.py has been deprecated as t= he recommended install method and if setup.py doesn't work the usual fix sugg= estion is to run pip install. >>>>>>>> Yes. But apparently the new system is deprecated, too. I really disl= ike the state of packaging systems that come with any language these days. >>>>>>>>> Having searched on pip for a while, it looks like builds using it c= an still be done from tarballs locally without needing to access the internet= to download the modules. >>>>>>>> That is good! >>>>>>>>> As pip is installed as part of python3 i tried using that. >>>>>>>>>=20 >>>>>>>>> I used "python3 -m pip install --root=3D/ ." in place of >>>>>>>>> "python3 setup.py build >>>>>>>>> python3 setup.py install --root=3D/" >>>>>>>>> for the tomli source. >>>>>>>>>=20 >>>>>>>>> This failed because tomli requires flit_core to be available. >>>>>>>>>=20 >>>>>>>>> flit_core also has no setup.py and so I tried to build it with pip = install. This failed with the error message that tomli was required as a depe= ndency. So I added --no-deps to the flit_core pip install line and this time = flit_core successfully built but the tomli build still came back with not fin= ding flit_core. >>>>>>>>>=20 >>>>>>>>> I have tried a variety of changes to the pip install line but none = of them has been successful and it has become clear that I am out of my depth= here and need guidance rather than just trying anything I find from a google= search. I am not even really clear if pip install is the correct approach, e= xcept if not then I have no idea how to progress. >>>>>>>>>=20 >>>>>>>>> Help urgently needed on how I should be approaching this. >>>>>>>> Circular dependencies can happen. This won=E2=80=99t be the first ti= me and it always is an utter nightmare with our buildsystem. >>>>>>>> This page states that pip comes by default with all binary packages: >>>>>>>> https://docs.python.org/3/installing/index.html >>>>>>>> Is there a chance that it is part of the distribution and can be ena= bled with a configure switch? >>>>>>> I wasn't able to find any way to get it enabled that way. >>>>>>>=20 >>>>>>> I did find that pip install will not install if the package is alread= y present. You have to add --upgrade to make it update any package already pr= esent. So I tried that. Didn't work. >>>>>>>=20 >>>>>>> I found a section about needing to use bootstrapping if you want to i= nstall tomli. Followed the information for creating a wheel file for flit_cor= e and then extracting that file into the site-packages directory. Tried that.= Didn't work. >>>>>>>=20 >>>>>>> Rather than doing a complete build every time I changed to trying to = build flit_core followed by tomli in a ./make.sh shell environment. >>>>>>>=20 >>>>>>> Doing that I found that I had a pip install command that was creating= the flit_core directory in site-packages and also including the .dist-info d= irectory. The info I found said that the .dist-info directory had to be there= for pip to be able to find the flit_core package. >>>>>>>=20 >>>>>>> However the pip install of tomli still ends up not finding any flit_c= ore package. In the bootstrapping information it mentioned about changing the= PYTHONPATH environment variable for find the flit_core package so I tried th= at in the shell but it also made no difference. >>>>>>>=20 >>>>>>> I then tried running python3 -c 'import flit_core' in the tomli build= shell before doing the tomli pip install. The import command completed with = no error but tomli still could not find any flit_core package. >>>>>>>=20 >>>>>>> End result of all my work is that flit_core has been built and instal= led into python3.10/site-packages but nothing seems to want to make the tomli= build find it. >>>>>>>=20 >>>>>>> Does anyone have any ideas on what command(s) I need to use in the to= mli pip install to have the installed flit_core package visible. >>>>>>>=20 >>>>>>> Regards and a Happy and Healthy New Year to all, >>>>>>>=20 >>>>>>> Adolf. >>>>>>>=20 >>>>>>>> -Michael >>>>>>>>>=20 >>>>>>>>> Regards, >>>>>>>>> Adolf. >>>>>>>>>=20 >>>>>>>>> PS: I hope everyone had an enjoyable Christmas. >>>>>>>> I hope you did, too! >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>>> -Michael >>>>>>>>>>>=20 >>>>>>>>>>> Regards, >>>>>>>>>>>=20 >>>>>>>>>>> Adolf. >>>>>>>>>>>=20 >>>>>>>>>>> --=20 >>>>>>>>>>> Sent from my laptop >>>>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>> --=20 >>>>>>>>> Sent from my laptop >>>>>>>=20 >>>>>>> --=20 >>>>>>> Sent from my laptop >>>=20 >>> --=20 >>> Sent from my laptop >=20 > --=20 > Sent from my laptop --===============3989322371539827023==--