From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: Question about updating python to 3.10 Date: Mon, 03 Jan 2022 11:32:40 +0000 Message-ID: <64901BAC-B399-47BC-BFFB-F1376A3C9E71@ipfire.org> In-Reply-To: <0e39b11a-1b16-b8bc-f83e-37d3f196801c@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============6351814508295650439==" List-Id: --===============6351814508295650439== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello, > On 3 Jan 2022, at 11:31, Adolf Belka wrote: >=20 > Hi Michael, >=20 > On 03/01/2022 11:52, Michael Tremer wrote: >> Hello, >> I am not sure if you already looked at what Arch is doing, but as far as I= understand it works like this: > No I didn't look at Arch, which is surprising as I have in the past with ot= her packages but I didn't. So thanks very much for the pointer. >> They build Python 3 without pip and setuptools (https://github.com/archlin= ux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/trunk/PKGB= UILD#L59). >> At this stage we might need to build setuptools separately. >> They use a tool called pyproject2setuppy (great name!) to build tomli (htt= ps://github.com/archlinux/svntogit-packages/blob/packages/python-tomli/trunk/= PKGBUILD). >> Setuptools-scm can then be built with tomli: https://github.com/archlinux/= svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD >> Would this work or did I overlook anything? > I will have a look at the links and give it a go. >=20 > Thanks very much, I feel re-motivated now. Very good! I got a little bit lost in your last email and I think I could tel= l. But following a solution that somebody else has come up with can be a good= thing to follow and of course we would customise it to our needs. Best, -Michael >=20 > Regards, > Adolf. >> -Michael >>> On 2 Jan 2022, at 13:44, Adolf Belka wrote: >>>=20 >>> Hi Michael and All, >>>=20 >>> On 26/12/2021 21:55, Michael Tremer wrote: >>>> Merry Christmas, >>>>> On 26 Dec 2021, at 15:38, Adolf Belka wrote: >>>>>=20 >>>>> Hi Michael, >>>>>=20 >>>>> On 17/12/2021 12:41, Michael Tremer wrote: >>>>>> Hello Adolf, >>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka wrot= e: >>>>>>>=20 >>>>>>> Hi All, >>>>>>>=20 >>>>>>>=20 >>>>>>> I am working on updating python from 3.8 to 3.10. Python itself is ok= ay after some other python related programs were also updated due to their ol= der versions not working with 3.10 >>>>>> Thank you for working on this. >>>>>>> I am working through all the programs that reference python-3.8 in th= eir rootfiles, or wherever, to change them to 3.10 >>>>>>>=20 >>>>>>> My question comes with regard to the changes to the rootfiles to ensu= re they will be included properly in an update. >>>>>> We will have to ship everything that puts anything into /usr/lib*/pyth= on3.*/site-packages again with the release of Python 3.10. >>>>>> This isn=E2=80=99t fun, but not as much of a pain in the rear as shipp= ing a new Perl release. >>>>>>> For the addons then when I change the rootfile I am also incrementing= the PAK_VER number in the lfs files and that ensures that they will be prope= rly updated. >>>>>> Yes. That is required for Pakfire to find a new version. >>>>>>> I don't know what to do for the core programs where I have updated th= e rootfile. The lfs files for these programs don't have a PAK_VER number to i= ncrement and the actual package version number is not being changed so I don'= t know what I need to change in the lfs to ensure that the rootfile changes w= ill be properly included in the update? >>>>>> I would say committing everything into a =E2=80=9Cpython-3.10=E2=80=9D= branch and submitting it all as a patchset is fine. All packages that have b= een touched in that branch will have to be shipped again which should be easy= to identify if it all comes in one large branch. >>>>> Okay, that is clear then.Will follow that approach. >>>>>>> I would appreciate any help on what I need to do for these programs t= hat have had changes only in their rootfiles. >>>>>> Absolutely. Did you find anything that didn=E2=80=99t build yet? >>>>>=20 >>>>> I have not had any problem with python3.10 itself but I found some of t= he older modules would not work with 3.10 so I decided to do a combined updat= e to 3.10 plus update modules to latest versions. >>>>>=20 >>>>> I found that after python3-setuptools-scm had run that the next module = python3-six failed to run due to missing the tomli library. I commented out s= ix and reran the build and then python3-dateutil had the same problem and th= en python3-jmespath >>>> I completely lost track of what the status of setuptools/distutils and s= o on is. This has changed multiple times and become independent/part of the d= istribution again and so on. >>>>> After some investigation I have found that since version 6.0.0 of pytho= n3-setuptools-scm, tomli has been a required library when it is being used by= other modules. So running version 6.0.0 of setuptools-scm worked with python= 3.10 and subsequent modules built successfully as tomli not needed. >>>>>=20 >>>>> So the obvious fix is to install tomli but that is when I ran into a pr= oblem. tomli has no setup.py or setup.cfg files in its tarball. The only way = to build it is using pip. >>>> Err. Great. >>>>> Apparently, after some searching, setup.py has been deprecated as the r= ecommended install method and if setup.py doesn't work the usual fix suggesti= on is to run pip install. >>>> Yes. But apparently the new system is deprecated, too. I really dislike = the state of packaging systems that come with any language these days. >>>>> Having searched on pip for a while, it looks like builds using it can s= till be done from tarballs locally without needing to access the internet to = download the modules. >>>> That is good! >>>>> As pip is installed as part of python3 i tried using that. >>>>>=20 >>>>> I used "python3 -m pip install --root=3D/ ." in place of >>>>> "python3 setup.py build >>>>> python3 setup.py install --root=3D/" >>>>> for the tomli source. >>>>>=20 >>>>> This failed because tomli requires flit_core to be available. >>>>>=20 >>>>> flit_core also has no setup.py and so I tried to build it with pip inst= all. This failed with the error message that tomli was required as a dependen= cy. So I added --no-deps to the flit_core pip install line and this time flit= _core successfully built but the tomli build still came back with not finding= flit_core. >>>>>=20 >>>>> I have tried a variety of changes to the pip install line but none of t= hem has been successful and it has become clear that I am out of my depth her= e and need guidance rather than just trying anything I find from a google sea= rch. I am not even really clear if pip install is the correct approach, excep= t if not then I have no idea how to progress. >>>>>=20 >>>>> Help urgently needed on how I should be approaching this. >>>> Circular dependencies can happen. This won=E2=80=99t be the first time a= nd it always is an utter nightmare with our buildsystem. >>>> This page states that pip comes by default with all binary packages: >>>> https://docs.python.org/3/installing/index.html >>>> Is there a chance that it is part of the distribution and can be enabled= with a configure switch? >>> I wasn't able to find any way to get it enabled that way. >>>=20 >>> I did find that pip install will not install if the package is already pr= esent. You have to add --upgrade to make it update any package already presen= t. So I tried that. Didn't work. >>>=20 >>> I found a section about needing to use bootstrapping if you want to insta= ll tomli. Followed the information for creating a wheel file for flit_core an= d then extracting that file into the site-packages directory. Tried that. Did= n't work. >>>=20 >>> Rather than doing a complete build every time I changed to trying to buil= d flit_core followed by tomli in a ./make.sh shell environment. >>>=20 >>> Doing that I found that I had a pip install command that was creating the= flit_core directory in site-packages and also including the .dist-info direc= tory. The info I found said that the .dist-info directory had to be there for= pip to be able to find the flit_core package. >>>=20 >>> However the pip install of tomli still ends up not finding any flit_core = package. In the bootstrapping information it mentioned about changing the PYT= HONPATH environment variable for find the flit_core package so I tried that i= n the shell but it also made no difference. >>>=20 >>> I then tried running python3 -c 'import flit_core' in the tomli build she= ll before doing the tomli pip install. The import command completed with no e= rror but tomli still could not find any flit_core package. >>>=20 >>> End result of all my work is that flit_core has been built and installed = into python3.10/site-packages but nothing seems to want to make the tomli bui= ld find it. >>>=20 >>> Does anyone have any ideas on what command(s) I need to use in the tomli = pip install to have the installed flit_core package visible. >>>=20 >>> Regards and a Happy and Healthy New Year to all, >>>=20 >>> Adolf. >>>=20 >>>> -Michael >>>>>=20 >>>>> Regards, >>>>> Adolf. >>>>>=20 >>>>> PS: I hope everyone had an enjoyable Christmas. >>>> I hope you did, too! >>>>>=20 >>>>>=20 >>>>>> -Michael >>>>>>>=20 >>>>>>> Regards, >>>>>>>=20 >>>>>>> Adolf. >>>>>>>=20 >>>>>>> --=20 >>>>>>> Sent from my laptop >>>>>>>=20 >>>>>=20 >>>>> --=20 >>>>> Sent from my laptop >>>=20 >>> --=20 >>> Sent from my laptop --===============6351814508295650439==--