From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: Question about updating python to 3.10 Date: Fri, 07 Jan 2022 16:49:26 +0000 Message-ID: In-Reply-To: <3167221d-db4c-c38f-5f48-b156f86df2d5@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1297701904811938397==" List-Id: --===============1297701904811938397== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello, > On 4 Jan 2022, at 12:49, Adolf Belka wrote: >=20 > Hi Michael, >=20 > Following the suggested Arch Linux approach I have got python3-setuptools-s= cm building successfully now. Yaaah. > I had to install python3-toml, followed by python3-pyproject2setuppy follow= ed by python3-tomli. >=20 > However then python3-six which installs after python3-setuptools-scm fails = due to missing the python package python3-packaging. >=20 > I can install that package, and will do so as I continue testing out the py= thon 3.10 build but I have the following question about python3-six. >=20 > Is six still needed. It is designed to ensure that programs can work with e= ither python2 or python3 but IPFire only has python3 now. > I can find python3-six referenced only by aws-cli. In the aws-cli build pyt= hon3-six is not listed at all in the build requirements. I tried to search fo= r aws-cli and python3-six being needed in run mode but I could not find anyth= ing one way or the other. If aws-cli no longer requires python3-six we can drop it. Last time we touche= d it I had to bring the package back because aws-cli was still depending on i= t. > If python3-six is really needed for running aws-cli then I can install it a= nd the new dependency but if it is not needed we could remove two packages wh= ich would partially offset the three new dependencies required to be able to = build the latest python3-setuptools-scm. Yes, that would be great :) -Michael >=20 > Regards, > Adolf >=20 > On 03/01/2022 12:32, Michael Tremer wrote: >> Hello, >>> On 3 Jan 2022, at 11:31, Adolf Belka wrote: >>>=20 >>> Hi Michael, >>>=20 >>> On 03/01/2022 11:52, Michael Tremer wrote: >>>> Hello, >>>> I am not sure if you already looked at what Arch is doing, but as far as= I understand it works like this: >>> No I didn't look at Arch, which is surprising as I have in the past with = other packages but I didn't. So thanks very much for the pointer. >>>> They build Python 3 without pip and setuptools (https://github.com/archl= inux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/trunk/PK= GBUILD#L59). >>>> At this stage we might need to build setuptools separately. >>>> They use a tool called pyproject2setuppy (great name!) to build tomli (h= ttps://github.com/archlinux/svntogit-packages/blob/packages/python-tomli/trun= k/PKGBUILD). >>>> Setuptools-scm can then be built with tomli: https://github.com/archlinu= x/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD >>>> Would this work or did I overlook anything? >>> I will have a look at the links and give it a go. >>>=20 >>> Thanks very much, I feel re-motivated now. >> Very good! I got a little bit lost in your last email and I think I could = tell. But following a solution that somebody else has come up with can be a g= ood thing to follow and of course we would customise it to our needs. >> Best, >> -Michael >>>=20 >>> Regards, >>> Adolf. >>>> -Michael >>>>> On 2 Jan 2022, at 13:44, Adolf Belka wrote: >>>>>=20 >>>>> Hi Michael and All, >>>>>=20 >>>>> On 26/12/2021 21:55, Michael Tremer wrote: >>>>>> Merry Christmas, >>>>>>> On 26 Dec 2021, at 15:38, Adolf Belka wrot= e: >>>>>>>=20 >>>>>>> Hi Michael, >>>>>>>=20 >>>>>>> On 17/12/2021 12:41, Michael Tremer wrote: >>>>>>>> Hello Adolf, >>>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka wr= ote: >>>>>>>>>=20 >>>>>>>>> Hi All, >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>> I am working on updating python from 3.8 to 3.10. Python itself is = okay after some other python related programs were also updated due to their = older versions not working with 3.10 >>>>>>>> Thank you for working on this. >>>>>>>>> I am working through all the programs that reference python-3.8 in = their rootfiles, or wherever, to change them to 3.10 >>>>>>>>>=20 >>>>>>>>> My question comes with regard to the changes to the rootfiles to en= sure they will be included properly in an update. >>>>>>>> We will have to ship everything that puts anything into /usr/lib*/py= thon3.*/site-packages again with the release of Python 3.10. >>>>>>>> This isn=E2=80=99t fun, but not as much of a pain in the rear as shi= pping a new Perl release. >>>>>>>>> For the addons then when I change the rootfile I am also incrementi= ng the PAK_VER number in the lfs files and that ensures that they will be pro= perly updated. >>>>>>>> Yes. That is required for Pakfire to find a new version. >>>>>>>>> I don't know what to do for the core programs where I have updated = the rootfile. The lfs files for these programs don't have a PAK_VER number to= increment and the actual package version number is not being changed so I do= n't know what I need to change in the lfs to ensure that the rootfile changes= will be properly included in the update? >>>>>>>> I would say committing everything into a =E2=80=9Cpython-3.10=E2=80= =9D branch and submitting it all as a patchset is fine. All packages that hav= e been touched in that branch will have to be shipped again which should be e= asy to identify if it all comes in one large branch. >>>>>>> Okay, that is clear then.Will follow that approach. >>>>>>>>> I would appreciate any help on what I need to do for these programs= that have had changes only in their rootfiles. >>>>>>>> Absolutely. Did you find anything that didn=E2=80=99t build yet? >>>>>>>=20 >>>>>>> I have not had any problem with python3.10 itself but I found some of= the older modules would not work with 3.10 so I decided to do a combined upd= ate to 3.10 plus update modules to latest versions. >>>>>>>=20 >>>>>>> I found that after python3-setuptools-scm had run that the next modul= e python3-six failed to run due to missing the tomli library. I commented out= six and reran the build and then python3-dateutil had the same problem and = then python3-jmespath >>>>>> I completely lost track of what the status of setuptools/distutils and= so on is. This has changed multiple times and become independent/part of the= distribution again and so on. >>>>>>> After some investigation I have found that since version 6.0.0 of pyt= hon3-setuptools-scm, tomli has been a required library when it is being used = by other modules. So running version 6.0.0 of setuptools-scm worked with pyth= on3.10 and subsequent modules built successfully as tomli not needed. >>>>>>>=20 >>>>>>> So the obvious fix is to install tomli but that is when I ran into a = problem. tomli has no setup.py or setup.cfg files in its tarball. The only wa= y to build it is using pip. >>>>>> Err. Great. >>>>>>> Apparently, after some searching, setup.py has been deprecated as the= recommended install method and if setup.py doesn't work the usual fix sugges= tion is to run pip install. >>>>>> Yes. But apparently the new system is deprecated, too. I really dislik= e the state of packaging systems that come with any language these days. >>>>>>> Having searched on pip for a while, it looks like builds using it can= still be done from tarballs locally without needing to access the internet t= o download the modules. >>>>>> That is good! >>>>>>> As pip is installed as part of python3 i tried using that. >>>>>>>=20 >>>>>>> I used "python3 -m pip install --root=3D/ ." in place of >>>>>>> "python3 setup.py build >>>>>>> python3 setup.py install --root=3D/" >>>>>>> for the tomli source. >>>>>>>=20 >>>>>>> This failed because tomli requires flit_core to be available. >>>>>>>=20 >>>>>>> flit_core also has no setup.py and so I tried to build it with pip in= stall. This failed with the error message that tomli was required as a depend= ency. So I added --no-deps to the flit_core pip install line and this time fl= it_core successfully built but the tomli build still came back with not findi= ng flit_core. >>>>>>>=20 >>>>>>> I have tried a variety of changes to the pip install line but none of= them has been successful and it has become clear that I am out of my depth h= ere and need guidance rather than just trying anything I find from a google s= earch. I am not even really clear if pip install is the correct approach, exc= ept if not then I have no idea how to progress. >>>>>>>=20 >>>>>>> Help urgently needed on how I should be approaching this. >>>>>> Circular dependencies can happen. This won=E2=80=99t be the first time= and it always is an utter nightmare with our buildsystem. >>>>>> This page states that pip comes by default with all binary packages: >>>>>> https://docs.python.org/3/installing/index.html >>>>>> Is there a chance that it is part of the distribution and can be enabl= ed with a configure switch? >>>>> I wasn't able to find any way to get it enabled that way. >>>>>=20 >>>>> I did find that pip install will not install if the package is already = present. You have to add --upgrade to make it update any package already pres= ent. So I tried that. Didn't work. >>>>>=20 >>>>> I found a section about needing to use bootstrapping if you want to ins= tall tomli. Followed the information for creating a wheel file for flit_core = and then extracting that file into the site-packages directory. Tried that. D= idn't work. >>>>>=20 >>>>> Rather than doing a complete build every time I changed to trying to bu= ild flit_core followed by tomli in a ./make.sh shell environment. >>>>>=20 >>>>> Doing that I found that I had a pip install command that was creating t= he flit_core directory in site-packages and also including the .dist-info dir= ectory. The info I found said that the .dist-info directory had to be there f= or pip to be able to find the flit_core package. >>>>>=20 >>>>> However the pip install of tomli still ends up not finding any flit_cor= e package. In the bootstrapping information it mentioned about changing the P= YTHONPATH environment variable for find the flit_core package so I tried that= in the shell but it also made no difference. >>>>>=20 >>>>> I then tried running python3 -c 'import flit_core' in the tomli build s= hell before doing the tomli pip install. The import command completed with no= error but tomli still could not find any flit_core package. >>>>>=20 >>>>> End result of all my work is that flit_core has been built and installe= d into python3.10/site-packages but nothing seems to want to make the tomli b= uild find it. >>>>>=20 >>>>> Does anyone have any ideas on what command(s) I need to use in the toml= i pip install to have the installed flit_core package visible. >>>>>=20 >>>>> Regards and a Happy and Healthy New Year to all, >>>>>=20 >>>>> Adolf. >>>>>=20 >>>>>> -Michael >>>>>>>=20 >>>>>>> Regards, >>>>>>> Adolf. >>>>>>>=20 >>>>>>> PS: I hope everyone had an enjoyable Christmas. >>>>>> I hope you did, too! >>>>>>>=20 >>>>>>>=20 >>>>>>>> -Michael >>>>>>>>>=20 >>>>>>>>> Regards, >>>>>>>>>=20 >>>>>>>>> Adolf. >>>>>>>>>=20 >>>>>>>>> --=20 >>>>>>>>> Sent from my laptop >>>>>>>>>=20 >>>>>>>=20 >>>>>>> --=20 >>>>>>> Sent from my laptop >>>>>=20 >>>>> --=20 >>>>> Sent from my laptop >=20 > --=20 > Sent from my laptop --===============1297701904811938397==--