From mboxrd@z Thu Jan 1 00:00:00 1970 From: Adolf Belka To: development@lists.ipfire.org Subject: Re: Question about updating python to 3.10 Date: Mon, 03 Jan 2022 12:31:17 +0100 Message-ID: <0e39b11a-1b16-b8bc-f83e-37d3f196801c@ipfire.org> In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============3100138734697149039==" List-Id: --===============3100138734697149039== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi Michael, On 03/01/2022 11:52, Michael Tremer wrote: > Hello, >=20 > I am not sure if you already looked at what Arch is doing, but as far as I = understand it works like this: No I didn't look at Arch, which is surprising as I have in the past with othe= r packages but I didn't. So thanks very much for the pointer. >=20 > They build Python 3 without pip and setuptools (https://github.com/archlinu= x/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/trunk/PKGBU= ILD#L59). >=20 > At this stage we might need to build setuptools separately. >=20 > They use a tool called pyproject2setuppy (great name!) to build tomli (http= s://github.com/archlinux/svntogit-packages/blob/packages/python-tomli/trunk/P= KGBUILD). >=20 > Setuptools-scm can then be built with tomli: https://github.com/archlinux/s= vntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD >=20 > Would this work or did I overlook anything? I will have a look at the links and give it a go. Thanks very much, I feel re-motivated now. Regards, Adolf. >=20 > -Michael >=20 >> On 2 Jan 2022, at 13:44, Adolf Belka wrote: >> >> Hi Michael and All, >> >> On 26/12/2021 21:55, Michael Tremer wrote: >>> Merry Christmas, >>>> On 26 Dec 2021, at 15:38, Adolf Belka wrote: >>>> >>>> Hi Michael, >>>> >>>> On 17/12/2021 12:41, Michael Tremer wrote: >>>>> Hello Adolf, >>>>>> On 17 Dec 2021, at 11:55, Adolf Belka wrote: >>>>>> >>>>>> Hi All, >>>>>> >>>>>> >>>>>> I am working on updating python from 3.8 to 3.10. Python itself is oka= y after some other python related programs were also updated due to their old= er versions not working with 3.10 >>>>> Thank you for working on this. >>>>>> I am working through all the programs that reference python-3.8 in the= ir rootfiles, or wherever, to change them to 3.10 >>>>>> >>>>>> My question comes with regard to the changes to the rootfiles to ensur= e they will be included properly in an update. >>>>> We will have to ship everything that puts anything into /usr/lib*/pytho= n3.*/site-packages again with the release of Python 3.10. >>>>> This isn=E2=80=99t fun, but not as much of a pain in the rear as shippi= ng a new Perl release. >>>>>> For the addons then when I change the rootfile I am also incrementing = the PAK_VER number in the lfs files and that ensures that they will be proper= ly updated. >>>>> Yes. That is required for Pakfire to find a new version. >>>>>> I don't know what to do for the core programs where I have updated the= rootfile. The lfs files for these programs don't have a PAK_VER number to in= crement and the actual package version number is not being changed so I don't= know what I need to change in the lfs to ensure that the rootfile changes wi= ll be properly included in the update? >>>>> I would say committing everything into a =E2=80=9Cpython-3.10=E2=80=9D = branch and submitting it all as a patchset is fine. All packages that have be= en touched in that branch will have to be shipped again which should be easy = to identify if it all comes in one large branch. >>>> Okay, that is clear then.Will follow that approach. >>>>>> I would appreciate any help on what I need to do for these programs th= at have had changes only in their rootfiles. >>>>> Absolutely. Did you find anything that didn=E2=80=99t build yet? >>>> >>>> I have not had any problem with python3.10 itself but I found some of th= e older modules would not work with 3.10 so I decided to do a combined update= to 3.10 plus update modules to latest versions. >>>> >>>> I found that after python3-setuptools-scm had run that the next module p= ython3-six failed to run due to missing the tomli library. I commented out si= x and reran the build and then python3-dateutil had the same problem and the= n python3-jmespath >>> I completely lost track of what the status of setuptools/distutils and so= on is. This has changed multiple times and become independent/part of the di= stribution again and so on. >>>> After some investigation I have found that since version 6.0.0 of python= 3-setuptools-scm, tomli has been a required library when it is being used by = other modules. So running version 6.0.0 of setuptools-scm worked with python3= .10 and subsequent modules built successfully as tomli not needed. >>>> >>>> So the obvious fix is to install tomli but that is when I ran into a pro= blem. tomli has no setup.py or setup.cfg files in its tarball. The only way t= o build it is using pip. >>> Err. Great. >>>> Apparently, after some searching, setup.py has been deprecated as the re= commended install method and if setup.py doesn't work the usual fix suggestio= n is to run pip install. >>> Yes. But apparently the new system is deprecated, too. I really dislike t= he state of packaging systems that come with any language these days. >>>> Having searched on pip for a while, it looks like builds using it can st= ill be done from tarballs locally without needing to access the internet to d= ownload the modules. >>> That is good! >>>> As pip is installed as part of python3 i tried using that. >>>> >>>> I used "python3 -m pip install --root=3D/ ." in place of >>>> "python3 setup.py build >>>> python3 setup.py install --root=3D/" >>>> for the tomli source. >>>> >>>> This failed because tomli requires flit_core to be available. >>>> >>>> flit_core also has no setup.py and so I tried to build it with pip insta= ll. This failed with the error message that tomli was required as a dependenc= y. So I added --no-deps to the flit_core pip install line and this time flit_= core successfully built but the tomli build still came back with not finding = flit_core. >>>> >>>> I have tried a variety of changes to the pip install line but none of th= em has been successful and it has become clear that I am out of my depth here= and need guidance rather than just trying anything I find from a google sear= ch. I am not even really clear if pip install is the correct approach, except= if not then I have no idea how to progress. >>>> >>>> Help urgently needed on how I should be approaching this. >>> Circular dependencies can happen. This won=E2=80=99t be the first time an= d it always is an utter nightmare with our buildsystem. >>> This page states that pip comes by default with all binary packages: >>> https://docs.python.org/3/installing/index.html >>> Is there a chance that it is part of the distribution and can be enabled = with a configure switch? >> I wasn't able to find any way to get it enabled that way. >> >> I did find that pip install will not install if the package is already pre= sent. You have to add --upgrade to make it update any package already present= . So I tried that. Didn't work. >> >> I found a section about needing to use bootstrapping if you want to instal= l tomli. Followed the information for creating a wheel file for flit_core and= then extracting that file into the site-packages directory. Tried that. Didn= 't work. >> >> Rather than doing a complete build every time I changed to trying to build= flit_core followed by tomli in a ./make.sh shell environment. >> >> Doing that I found that I had a pip install command that was creating the = flit_core directory in site-packages and also including the .dist-info direct= ory. The info I found said that the .dist-info directory had to be there for = pip to be able to find the flit_core package. >> >> However the pip install of tomli still ends up not finding any flit_core p= ackage. In the bootstrapping information it mentioned about changing the PYTH= ONPATH environment variable for find the flit_core package so I tried that in= the shell but it also made no difference. >> >> I then tried running python3 -c 'import flit_core' in the tomli build shel= l before doing the tomli pip install. The import command completed with no er= ror but tomli still could not find any flit_core package. >> >> End result of all my work is that flit_core has been built and installed i= nto python3.10/site-packages but nothing seems to want to make the tomli buil= d find it. >> >> Does anyone have any ideas on what command(s) I need to use in the tomli p= ip install to have the installed flit_core package visible. >> >> Regards and a Happy and Healthy New Year to all, >> >> Adolf. >> >>> -Michael >>>> >>>> Regards, >>>> Adolf. >>>> >>>> PS: I hope everyone had an enjoyable Christmas. >>> I hope you did, too! >>>> >>>> >>>>> -Michael >>>>>> >>>>>> Regards, >>>>>> >>>>>> Adolf. >>>>>> >>>>>> --=20 >>>>>> Sent from my laptop >>>>>> >>>> >>>> --=20 >>>> Sent from my laptop >> >> --=20 >> Sent from my laptop >=20 --===============3100138734697149039==--