From mboxrd@z Thu Jan 1 00:00:00 1970 From: Adolf Belka To: development@lists.ipfire.org Subject: Re: Question about updating python to 3.10 Date: Tue, 04 Jan 2022 13:49:47 +0100 Message-ID: <3167221d-db4c-c38f-5f48-b156f86df2d5@ipfire.org> In-Reply-To: <64901BAC-B399-47BC-BFFB-F1376A3C9E71@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============8753552615342540206==" List-Id: --===============8753552615342540206== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi Michael, Following the suggested Arch Linux approach I have got=20 python3-setuptools-scm building successfully now. Yaaah. I had to install python3-toml, followed by python3-pyproject2setuppy=20 followed by python3-tomli. However then python3-six which installs after python3-setuptools-scm=20 fails due to missing the python package python3-packaging. I can install that package, and will do so as I continue testing out the=20 python 3.10 build but I have the following question about python3-six. Is six still needed. It is designed to ensure that programs can work=20 with either python2 or python3 but IPFire only has python3 now. I can find python3-six referenced only by aws-cli. In the aws-cli build=20 python3-six is not listed at all in the build requirements. I tried to=20 search for aws-cli and python3-six being needed in run mode but I could=20 not find anything one way or the other. If python3-six is really needed for running aws-cli then I can install=20 it and the new dependency but if it is not needed we could remove two=20 packages which would partially offset the three new dependencies=20 required to be able to build the latest python3-setuptools-scm. Regards, Adolf On 03/01/2022 12:32, Michael Tremer wrote: > Hello, >=20 >> On 3 Jan 2022, at 11:31, Adolf Belka wrote: >> >> Hi Michael, >> >> On 03/01/2022 11:52, Michael Tremer wrote: >>> Hello, >>> I am not sure if you already looked at what Arch is doing, but as far as = I understand it works like this: >> No I didn't look at Arch, which is surprising as I have in the past with o= ther packages but I didn't. So thanks very much for the pointer. >>> They build Python 3 without pip and setuptools (https://github.com/archli= nux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/trunk/PKG= BUILD#L59). >>> At this stage we might need to build setuptools separately. >>> They use a tool called pyproject2setuppy (great name!) to build tomli (ht= tps://github.com/archlinux/svntogit-packages/blob/packages/python-tomli/trunk= /PKGBUILD). >>> Setuptools-scm can then be built with tomli: https://github.com/archlinux= /svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD >>> Would this work or did I overlook anything? >> I will have a look at the links and give it a go. >> >> Thanks very much, I feel re-motivated now. >=20 > Very good! I got a little bit lost in your last email and I think I could t= ell. But following a solution that somebody else has come up with can be a go= od thing to follow and of course we would customise it to our needs. >=20 > Best, > -Michael >=20 >> >> Regards, >> Adolf. >>> -Michael >>>> On 2 Jan 2022, at 13:44, Adolf Belka wrote: >>>> >>>> Hi Michael and All, >>>> >>>> On 26/12/2021 21:55, Michael Tremer wrote: >>>>> Merry Christmas, >>>>>> On 26 Dec 2021, at 15:38, Adolf Belka wrote: >>>>>> >>>>>> Hi Michael, >>>>>> >>>>>> On 17/12/2021 12:41, Michael Tremer wrote: >>>>>>> Hello Adolf, >>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka wro= te: >>>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> >>>>>>>> I am working on updating python from 3.8 to 3.10. Python itself is o= kay after some other python related programs were also updated due to their o= lder versions not working with 3.10 >>>>>>> Thank you for working on this. >>>>>>>> I am working through all the programs that reference python-3.8 in t= heir rootfiles, or wherever, to change them to 3.10 >>>>>>>> >>>>>>>> My question comes with regard to the changes to the rootfiles to ens= ure they will be included properly in an update. >>>>>>> We will have to ship everything that puts anything into /usr/lib*/pyt= hon3.*/site-packages again with the release of Python 3.10. >>>>>>> This isn=E2=80=99t fun, but not as much of a pain in the rear as ship= ping a new Perl release. >>>>>>>> For the addons then when I change the rootfile I am also incrementin= g the PAK_VER number in the lfs files and that ensures that they will be prop= erly updated. >>>>>>> Yes. That is required for Pakfire to find a new version. >>>>>>>> I don't know what to do for the core programs where I have updated t= he rootfile. The lfs files for these programs don't have a PAK_VER number to = increment and the actual package version number is not being changed so I don= 't know what I need to change in the lfs to ensure that the rootfile changes = will be properly included in the update? >>>>>>> I would say committing everything into a =E2=80=9Cpython-3.10=E2=80= =9D branch and submitting it all as a patchset is fine. All packages that hav= e been touched in that branch will have to be shipped again which should be e= asy to identify if it all comes in one large branch. >>>>>> Okay, that is clear then.Will follow that approach. >>>>>>>> I would appreciate any help on what I need to do for these programs = that have had changes only in their rootfiles. >>>>>>> Absolutely. Did you find anything that didn=E2=80=99t build yet? >>>>>> >>>>>> I have not had any problem with python3.10 itself but I found some of = the older modules would not work with 3.10 so I decided to do a combined upda= te to 3.10 plus update modules to latest versions. >>>>>> >>>>>> I found that after python3-setuptools-scm had run that the next module= python3-six failed to run due to missing the tomli library. I commented out = six and reran the build and then python3-dateutil had the same problem and t= hen python3-jmespath >>>>> I completely lost track of what the status of setuptools/distutils and = so on is. This has changed multiple times and become independent/part of the = distribution again and so on. >>>>>> After some investigation I have found that since version 6.0.0 of pyth= on3-setuptools-scm, tomli has been a required library when it is being used b= y other modules. So running version 6.0.0 of setuptools-scm worked with pytho= n3.10 and subsequent modules built successfully as tomli not needed. >>>>>> >>>>>> So the obvious fix is to install tomli but that is when I ran into a p= roblem. tomli has no setup.py or setup.cfg files in its tarball. The only way= to build it is using pip. >>>>> Err. Great. >>>>>> Apparently, after some searching, setup.py has been deprecated as the = recommended install method and if setup.py doesn't work the usual fix suggest= ion is to run pip install. >>>>> Yes. But apparently the new system is deprecated, too. I really dislike= the state of packaging systems that come with any language these days. >>>>>> Having searched on pip for a while, it looks like builds using it can = still be done from tarballs locally without needing to access the internet to= download the modules. >>>>> That is good! >>>>>> As pip is installed as part of python3 i tried using that. >>>>>> >>>>>> I used "python3 -m pip install --root=3D/ ." in place of >>>>>> "python3 setup.py build >>>>>> python3 setup.py install --root=3D/" >>>>>> for the tomli source. >>>>>> >>>>>> This failed because tomli requires flit_core to be available. >>>>>> >>>>>> flit_core also has no setup.py and so I tried to build it with pip ins= tall. This failed with the error message that tomli was required as a depende= ncy. So I added --no-deps to the flit_core pip install line and this time fli= t_core successfully built but the tomli build still came back with not findin= g flit_core. >>>>>> >>>>>> I have tried a variety of changes to the pip install line but none of = them has been successful and it has become clear that I am out of my depth he= re and need guidance rather than just trying anything I find from a google se= arch. I am not even really clear if pip install is the correct approach, exce= pt if not then I have no idea how to progress. >>>>>> >>>>>> Help urgently needed on how I should be approaching this. >>>>> Circular dependencies can happen. This won=E2=80=99t be the first time = and it always is an utter nightmare with our buildsystem. >>>>> This page states that pip comes by default with all binary packages: >>>>> https://docs.python.org/3/installing/index.html >>>>> Is there a chance that it is part of the distribution and can be enable= d with a configure switch? >>>> I wasn't able to find any way to get it enabled that way. >>>> >>>> I did find that pip install will not install if the package is already p= resent. You have to add --upgrade to make it update any package already prese= nt. So I tried that. Didn't work. >>>> >>>> I found a section about needing to use bootstrapping if you want to inst= all tomli. Followed the information for creating a wheel file for flit_core a= nd then extracting that file into the site-packages directory. Tried that. Di= dn't work. >>>> >>>> Rather than doing a complete build every time I changed to trying to bui= ld flit_core followed by tomli in a ./make.sh shell environment. >>>> >>>> Doing that I found that I had a pip install command that was creating th= e flit_core directory in site-packages and also including the .dist-info dire= ctory. The info I found said that the .dist-info directory had to be there fo= r pip to be able to find the flit_core package. >>>> >>>> However the pip install of tomli still ends up not finding any flit_core= package. In the bootstrapping information it mentioned about changing the PY= THONPATH environment variable for find the flit_core package so I tried that = in the shell but it also made no difference. >>>> >>>> I then tried running python3 -c 'import flit_core' in the tomli build sh= ell before doing the tomli pip install. The import command completed with no = error but tomli still could not find any flit_core package. >>>> >>>> End result of all my work is that flit_core has been built and installed= into python3.10/site-packages but nothing seems to want to make the tomli bu= ild find it. >>>> >>>> Does anyone have any ideas on what command(s) I need to use in the tomli= pip install to have the installed flit_core package visible. >>>> >>>> Regards and a Happy and Healthy New Year to all, >>>> >>>> Adolf. >>>> >>>>> -Michael >>>>>> >>>>>> Regards, >>>>>> Adolf. >>>>>> >>>>>> PS: I hope everyone had an enjoyable Christmas. >>>>> I hope you did, too! >>>>>> >>>>>> >>>>>>> -Michael >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Adolf. >>>>>>>> >>>>>>>> --=20 >>>>>>>> Sent from my laptop >>>>>>>> >>>>>> >>>>>> --=20 >>>>>> Sent from my laptop >>>> >>>> --=20 >>>> Sent from my laptop >=20 --=20 Sent from my laptop --===============8753552615342540206==--