From mboxrd@z Thu Jan 1 00:00:00 1970 From: Adolf Belka To: development@lists.ipfire.org Subject: Re: Question about updating python to 3.10 Date: Tue, 11 Jan 2022 15:32:55 +0100 Message-ID: <317de396-dbbf-c79d-ebb8-50f43ae60198@ipfire.org> In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============8677258388178268613==" List-Id: --===============8677258388178268613== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi Michael and all, python3-six has to stay as python3-dateutil explicitly specifies it in=20 its dependencies. So that solves that question. I have, with quite a bit of work, eventually been able to get all=20 existing python modules updated that had updates and built successfully=20 with python3.10 However later in the IPFire build libvirt failed with a problem parsing=20 some documentation files. Stashing my python3.10 changes and re-running the build had libvirt=20 building without problems. So one of the python changes has caused it.=20 libvirt uses python3-docutils and that was updated but reverting that=20 back to original on its own did not cause libvirt to build successfully. I looked in the libvirt .configure for an option to not build the docs=20 but it is not mentioned under ./configure --help. I tried --without-docs and --disable-docs but both were not recognised=20 in the build. The current libvirt is from mid 2020 (6.5.0) so it is likely that it is=20 not compatible in some way with the updated python/modules. I tried updating libvirt to the current 7.10.0 libvirt has replaced autotools with meson. I found meson options=20 available for all existing ./configure options except three. There is=20 nothing in meson options for:- --with-virtualport --with-macvtap --without-dbus I tried building it with all the other options and it successfully built=20 with the updated python3.10 and modules. All other packages after=20 libvirt built successfully without any problems. Normally I would then use the updated libvirt, as that is needed to work=20 with the updated python packages, but I am not sure if those missing=20 options are now built into libvirt or are no longer available and what=20 impact that might have for the use of libvirt on IPFire. There is nothing in the libvirt changelog about virtualport, macvtap or=20 dbus. Searching on the internet, I was also not able to find anything related=20 to libvirt and these options. Any suggestions/proposals on how to move forward on this? Should I just=20 build libvirt without those three options or ...? Regards, Adolf. On 07/01/2022 17:49, Michael Tremer wrote: > Hello, >=20 >> On 4 Jan 2022, at 12:49, Adolf Belka wrote: >> >> Hi Michael, >> >> Following the suggested Arch Linux approach I have got python3-setuptools-= scm building successfully now. Yaaah. >> I had to install python3-toml, followed by python3-pyproject2setuppy follo= wed by python3-tomli. >> >> However then python3-six which installs after python3-setuptools-scm fails= due to missing the python package python3-packaging. >> >> I can install that package, and will do so as I continue testing out the p= ython 3.10 build but I have the following question about python3-six. >> >> Is six still needed. It is designed to ensure that programs can work with = either python2 or python3 but IPFire only has python3 now. >> I can find python3-six referenced only by aws-cli. In the aws-cli build py= thon3-six is not listed at all in the build requirements. I tried to search f= or aws-cli and python3-six being needed in run mode but I could not find anyt= hing one way or the other. >=20 > If aws-cli no longer requires python3-six we can drop it. Last time we touc= hed it I had to bring the package back because aws-cli was still depending on= it. >=20 >> If python3-six is really needed for running aws-cli then I can install it = and the new dependency but if it is not needed we could remove two packages w= hich would partially offset the three new dependencies required to be able to= build the latest python3-setuptools-scm. >=20 > Yes, that would be great :) >=20 > -Michael >=20 >> >> Regards, >> Adolf >> >> On 03/01/2022 12:32, Michael Tremer wrote: >>> Hello, >>>> On 3 Jan 2022, at 11:31, Adolf Belka wrote: >>>> >>>> Hi Michael, >>>> >>>> On 03/01/2022 11:52, Michael Tremer wrote: >>>>> Hello, >>>>> I am not sure if you already looked at what Arch is doing, but as far a= s I understand it works like this: >>>> No I didn't look at Arch, which is surprising as I have in the past with= other packages but I didn't. So thanks very much for the pointer. >>>>> They build Python 3 without pip and setuptools (https://github.com/arch= linux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/trunk/P= KGBUILD#L59). >>>>> At this stage we might need to build setuptools separately. >>>>> They use a tool called pyproject2setuppy (great name!) to build tomli (= https://github.com/archlinux/svntogit-packages/blob/packages/python-tomli/tru= nk/PKGBUILD). >>>>> Setuptools-scm can then be built with tomli: https://github.com/archlin= ux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD >>>>> Would this work or did I overlook anything? >>>> I will have a look at the links and give it a go. >>>> >>>> Thanks very much, I feel re-motivated now. >>> Very good! I got a little bit lost in your last email and I think I could= tell. But following a solution that somebody else has come up with can be a = good thing to follow and of course we would customise it to our needs. >>> Best, >>> -Michael >>>> >>>> Regards, >>>> Adolf. >>>>> -Michael >>>>>> On 2 Jan 2022, at 13:44, Adolf Belka wrote: >>>>>> >>>>>> Hi Michael and All, >>>>>> >>>>>> On 26/12/2021 21:55, Michael Tremer wrote: >>>>>>> Merry Christmas, >>>>>>>> On 26 Dec 2021, at 15:38, Adolf Belka wro= te: >>>>>>>> >>>>>>>> Hi Michael, >>>>>>>> >>>>>>>> On 17/12/2021 12:41, Michael Tremer wrote: >>>>>>>>> Hello Adolf, >>>>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka w= rote: >>>>>>>>>> >>>>>>>>>> Hi All, >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I am working on updating python from 3.8 to 3.10. Python itself is= okay after some other python related programs were also updated due to their= older versions not working with 3.10 >>>>>>>>> Thank you for working on this. >>>>>>>>>> I am working through all the programs that reference python-3.8 in= their rootfiles, or wherever, to change them to 3.10 >>>>>>>>>> >>>>>>>>>> My question comes with regard to the changes to the rootfiles to e= nsure they will be included properly in an update. >>>>>>>>> We will have to ship everything that puts anything into /usr/lib*/p= ython3.*/site-packages again with the release of Python 3.10. >>>>>>>>> This isn=E2=80=99t fun, but not as much of a pain in the rear as sh= ipping a new Perl release. >>>>>>>>>> For the addons then when I change the rootfile I am also increment= ing the PAK_VER number in the lfs files and that ensures that they will be pr= operly updated. >>>>>>>>> Yes. That is required for Pakfire to find a new version. >>>>>>>>>> I don't know what to do for the core programs where I have updated= the rootfile. The lfs files for these programs don't have a PAK_VER number t= o increment and the actual package version number is not being changed so I d= on't know what I need to change in the lfs to ensure that the rootfile change= s will be properly included in the update? >>>>>>>>> I would say committing everything into a =E2=80=9Cpython-3.10=E2=80= =9D branch and submitting it all as a patchset is fine. All packages that hav= e been touched in that branch will have to be shipped again which should be e= asy to identify if it all comes in one large branch. >>>>>>>> Okay, that is clear then.Will follow that approach. >>>>>>>>>> I would appreciate any help on what I need to do for these program= s that have had changes only in their rootfiles. >>>>>>>>> Absolutely. Did you find anything that didn=E2=80=99t build yet? >>>>>>>> >>>>>>>> I have not had any problem with python3.10 itself but I found some o= f the older modules would not work with 3.10 so I decided to do a combined up= date to 3.10 plus update modules to latest versions. >>>>>>>> >>>>>>>> I found that after python3-setuptools-scm had run that the next modu= le python3-six failed to run due to missing the tomli library. I commented ou= t six and reran the build and then python3-dateutil had the same problem and= then python3-jmespath >>>>>>> I completely lost track of what the status of setuptools/distutils an= d so on is. This has changed multiple times and become independent/part of th= e distribution again and so on. >>>>>>>> After some investigation I have found that since version 6.0.0 of py= thon3-setuptools-scm, tomli has been a required library when it is being used= by other modules. So running version 6.0.0 of setuptools-scm worked with pyt= hon3.10 and subsequent modules built successfully as tomli not needed. >>>>>>>> >>>>>>>> So the obvious fix is to install tomli but that is when I ran into a= problem. tomli has no setup.py or setup.cfg files in its tarball. The only w= ay to build it is using pip. >>>>>>> Err. Great. >>>>>>>> Apparently, after some searching, setup.py has been deprecated as th= e recommended install method and if setup.py doesn't work the usual fix sugge= stion is to run pip install. >>>>>>> Yes. But apparently the new system is deprecated, too. I really disli= ke the state of packaging systems that come with any language these days. >>>>>>>> Having searched on pip for a while, it looks like builds using it ca= n still be done from tarballs locally without needing to access the internet = to download the modules. >>>>>>> That is good! >>>>>>>> As pip is installed as part of python3 i tried using that. >>>>>>>> >>>>>>>> I used "python3 -m pip install --root=3D/ ." in place of >>>>>>>> "python3 setup.py build >>>>>>>> python3 setup.py install --root=3D/" >>>>>>>> for the tomli source. >>>>>>>> >>>>>>>> This failed because tomli requires flit_core to be available. >>>>>>>> >>>>>>>> flit_core also has no setup.py and so I tried to build it with pip i= nstall. This failed with the error message that tomli was required as a depen= dency. So I added --no-deps to the flit_core pip install line and this time f= lit_core successfully built but the tomli build still came back with not find= ing flit_core. >>>>>>>> >>>>>>>> I have tried a variety of changes to the pip install line but none o= f them has been successful and it has become clear that I am out of my depth = here and need guidance rather than just trying anything I find from a google = search. I am not even really clear if pip install is the correct approach, ex= cept if not then I have no idea how to progress. >>>>>>>> >>>>>>>> Help urgently needed on how I should be approaching this. >>>>>>> Circular dependencies can happen. This won=E2=80=99t be the first tim= e and it always is an utter nightmare with our buildsystem. >>>>>>> This page states that pip comes by default with all binary packages: >>>>>>> https://docs.python.org/3/installing/index.html >>>>>>> Is there a chance that it is part of the distribution and can be enab= led with a configure switch? >>>>>> I wasn't able to find any way to get it enabled that way. >>>>>> >>>>>> I did find that pip install will not install if the package is already= present. You have to add --upgrade to make it update any package already pre= sent. So I tried that. Didn't work. >>>>>> >>>>>> I found a section about needing to use bootstrapping if you want to in= stall tomli. Followed the information for creating a wheel file for flit_core= and then extracting that file into the site-packages directory. Tried that. = Didn't work. >>>>>> >>>>>> Rather than doing a complete build every time I changed to trying to b= uild flit_core followed by tomli in a ./make.sh shell environment. >>>>>> >>>>>> Doing that I found that I had a pip install command that was creating = the flit_core directory in site-packages and also including the .dist-info di= rectory. The info I found said that the .dist-info directory had to be there = for pip to be able to find the flit_core package. >>>>>> >>>>>> However the pip install of tomli still ends up not finding any flit_co= re package. In the bootstrapping information it mentioned about changing the = PYTHONPATH environment variable for find the flit_core package so I tried tha= t in the shell but it also made no difference. >>>>>> >>>>>> I then tried running python3 -c 'import flit_core' in the tomli build = shell before doing the tomli pip install. The import command completed with n= o error but tomli still could not find any flit_core package. >>>>>> >>>>>> End result of all my work is that flit_core has been built and install= ed into python3.10/site-packages but nothing seems to want to make the tomli = build find it. >>>>>> >>>>>> Does anyone have any ideas on what command(s) I need to use in the tom= li pip install to have the installed flit_core package visible. >>>>>> >>>>>> Regards and a Happy and Healthy New Year to all, >>>>>> >>>>>> Adolf. >>>>>> >>>>>>> -Michael >>>>>>>> >>>>>>>> Regards, >>>>>>>> Adolf. >>>>>>>> >>>>>>>> PS: I hope everyone had an enjoyable Christmas. >>>>>>> I hope you did, too! >>>>>>>> >>>>>>>> >>>>>>>>> -Michael >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> >>>>>>>>>> Adolf. >>>>>>>>>> >>>>>>>>>> --=20 >>>>>>>>>> Sent from my laptop >>>>>>>>>> >>>>>>>> >>>>>>>> --=20 >>>>>>>> Sent from my laptop >>>>>> >>>>>> --=20 >>>>>> Sent from my laptop >> >> --=20 >> Sent from my laptop >=20 --=20 Sent from my laptop --===============8677258388178268613==--