From mboxrd@z Thu Jan 1 00:00:00 1970 From: Adolf Belka To: development@lists.ipfire.org Subject: Re: Question about updating python to 3.10 Date: Tue, 18 Jan 2022 22:21:40 +0100 Message-ID: In-Reply-To: <2D0835FA-9CF5-4833-8182-E4D920823A31@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1395231373591235952==" List-Id: --===============1395231373591235952== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi Michael, On 18/01/2022 19:51, Michael Tremer wrote: > Hello, >=20 >> On 18 Jan 2022, at 16:42, Adolf Belka wrote: >> >> Hi Michael, >> >> On 18/01/2022 16:02, Michael Tremer wrote: >>> Hello Adolf, >>> I have just been trying to build a couple of Python packages that I need = and I finally arrived at needing flit-core. >>> I suppose since you did already build this, there is no point in me doing= all this again and also there is no need to build all my packages on Python = 3.8 when we can have a newer release. >> I am afraid that you will need to build flit-core. I was building it when= I was trying to use pip to install tomli but that approached failed to go fu= rther and that was when I used pyproject2setuppy, from your input, to build t= omli and that did not need flit-core. >=20 > Oh, I suppose that didn=E2=80=99t hold for very long because I now have a p= ackage that needs it :( flit-core also has no setup.py but you will then be able to use the pyproject= 2setuppy package from my python3.10 branch to build flit-core. >=20 >> In terms of the python3.10 branch that I have on my git repository at home= , I presume I could push it onto my personal git repository that you set up f= or me - https://git.ipfire.org/?p=3Dpeople/bonnietwin/ipfire-2.x.git;a=3Dsumm= ary >=20 > Yes, it is always a good idea to backup things to the server. >=20 >> The only thing is I am not sure what command I should use to push the bran= ch from my system here to the IPFire one. >=20 > If you have your repository set up with =E2=80=9Cgit remote add =E2=80=A6= =E2=80=9D then you only need to run =E2=80=9Cgit push bonnietwin python3.10= =E2=80=9D. Aaah. I think when I tried to experiment with this before I created a local n= ew clone of bonnietwin which was then separate from my running local reposito= ry that I used for building and I struggled to figure out how to get branches= between them. Your mention of git remote add indicates that I can add my bonnietwin reposit= ory to my running build repository. Thais is what I was missing before. Thanks for the clue, will give it a try. Regards, Adolf. >=20 > -Michael >=20 >> >> I can have a go at trying it and see how it goes. >> >> Regards, >> >> Adolf. >>> Could you push this branch somewhere for me so that I can base my work on= top of it? >>> Best, >>> -Michael >>>> On 13 Jan 2022, at 09:32, Michael Tremer w= rote: >>>> >>>> Hello, >>>> >>>>> On 12 Jan 2022, at 21:08, Adolf Belka wrote: >>>>> >>>>> Hi all, >>>>> >>>>> On 12/01/2022 19:10, Michael Tremer wrote: >>>>>> Hey, >>>>>>> On 11 Jan 2022, at 14:32, Adolf Belka wrot= e: >>>>>>> >>>>>>> Hi Michael and all, >>>>>>> >>>>>>> python3-six has to stay as python3-dateutil explicitly specifies it i= n its dependencies. >>>>>>> So that solves that question. >>>>>> Aww :( >>>>>>> I have, with quite a bit of work, eventually been able to get all exi= sting python modules updated that had updates and built successfully with pyt= hon3.10 >>>>>> Yay! >>>>>>> However later in the IPFire build libvirt failed with a problem parsi= ng some documentation files. >>>>>>> Stashing my python3.10 changes and re-running the build had libvirt b= uilding without problems. So one of the python changes has caused it. libvirt= uses python3-docutils and that was updated but reverting that back to origin= al on its own did not cause libvirt to build successfully. >>>>>>> >>>>>>> I looked in the libvirt .configure for an option to not build the doc= s but it is not mentioned under ./configure --help. >>>>>>> >>>>>>> I tried --without-docs and --disable-docs but both were not recognise= d in the build. >>>>>>> >>>>>>> The current libvirt is from mid 2020 (6.5.0) so it is likely that it = is not compatible in some way with the updated python/modules. >>>>>>> >>>>>>> I tried updating libvirt to the current 7.10.0 >>>>>>> libvirt has replaced autotools with meson. I found meson options avai= lable for all existing ./configure options except three. There is nothing in = meson options for:- >>>>>>> --with-virtualport >>>>>>> --with-macvtap >>>>>>> --without-dbus >>>>>> Yay, another project that is using meson :( >>>>>> We no longer need macvtap; we have dubs, so that should not hurt; and = I have no idea what virtualport is. >>>>>>> I tried building it with all the other options and it successfully bu= ilt with the updated python3.10 and modules. All other packages after libvirt= built successfully without any problems. >>>>>>> >>>>>>> Normally I would then use the updated libvirt, as that is needed to w= ork with the updated python packages, but I am not sure if those missing opti= ons are now built into libvirt or are no longer available and what impact tha= t might have for the use of libvirt on IPFire. >>>>>> I don=E2=80=99t think this would hurt us much. I have CCed Jonatan who= maintains this, so maybe he knows more about this. >>>>>>> There is nothing in the libvirt changelog about virtualport, macvtap = or dbus. >>>>>>> >>>>>>> Searching on the internet, I was also not able to find anything relat= ed to libvirt and these options. >>>>>>> >>>>>>> Any suggestions/proposals on how to move forward on this? Should I ju= st build libvirt without those three options or ...? >>>>>> Yes, and I think this could be submitted independently of the Python b= ranch because it is an add-on. We should be able to ship this at any time and= might want to delay Python until we have enough space in a core update. >>>>> Okay, then I will stash my python3.10 changes and submit the libvirt up= date on its own. Then Jonaton can test it out. I don't use libvirt or qemu so= I can't test it easily. >>>>> >>>>> Then when you think the time is right, let me know and I will submit th= e python3.10 series of patches. >>>> >>>> You can submit this at any time. I was just thinking about the conversat= ion between Peter and Arne during the last call regarding an updated kernel a= nd updated firmware. Depending on the size of Python, all of this might make = an update a little bit too large - however, this should not be a patchset tha= t we cannot merge within two merge windows. >>>> >>>>>> Thank you for looking into this. Sounds like a not so nice riddle to s= olve :) >>>>> There were certainly some challenges, like updating python3-daemon whic= h then wanted to have twine as a fixed dependency which had nine dependencies= some of which had more. I had got to 19 additional python packages and still= going and then I said enough was enough and looked at twine. It is required = for uploading python packages to PyPI which IPFire doesn't need. So I tried p= atching the twine requirement out and that worked and then I found others tha= t had done the same because they had no response to the request to not have t= wine as a mandatory build requirement. >>>> >>>> Python seems to become the hell that Perl used to be. In order to add on= e perl package, you had 20 dependencies. And those dependencies had their own= dependencies too=E2=80=A6 It would be great to disable or patch out whatever= is possible so that we do not create more work for ourselves in the future a= nd have to update packages that are barely used - we already have enough of t= hese :) >>>> >>>>> I found it all challenging but rewarding when I made progress and event= ually succeeded in building everything without errors. >>>> >>>> I am so happy that you are putting all this time and effort into this be= cause it certainly requires patience - a trade I don=E2=80=99t have. >>>> >>>> And Python is being used in all sorts of places in the distribution, so = it really isn=E2=80=99t anything we should not be up to date on. >>>> >>>> -Michael >>>> >>>>> >>>>> Regards, >>>>> Adolf. >>>>>> -Michael >>>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Adolf. >>>>>>> >>>>>>> On 07/01/2022 17:49, Michael Tremer wrote: >>>>>>>> Hello, >>>>>>>>> On 4 Jan 2022, at 12:49, Adolf Belka wro= te: >>>>>>>>> >>>>>>>>> Hi Michael, >>>>>>>>> >>>>>>>>> Following the suggested Arch Linux approach I have got python3-setu= ptools-scm building successfully now. Yaaah. >>>>>>>>> I had to install python3-toml, followed by python3-pyproject2setupp= y followed by python3-tomli. >>>>>>>>> >>>>>>>>> However then python3-six which installs after python3-setuptools-sc= m fails due to missing the python package python3-packaging. >>>>>>>>> >>>>>>>>> I can install that package, and will do so as I continue testing ou= t the python 3.10 build but I have the following question about python3-six. >>>>>>>>> >>>>>>>>> Is six still needed. It is designed to ensure that programs can wor= k with either python2 or python3 but IPFire only has python3 now. >>>>>>>>> I can find python3-six referenced only by aws-cli. In the aws-cli b= uild python3-six is not listed at all in the build requirements. I tried to s= earch for aws-cli and python3-six being needed in run mode but I could not fi= nd anything one way or the other. >>>>>>>> If aws-cli no longer requires python3-six we can drop it. Last time = we touched it I had to bring the package back because aws-cli was still depen= ding on it. >>>>>>>>> If python3-six is really needed for running aws-cli then I can inst= all it and the new dependency but if it is not needed we could remove two pac= kages which would partially offset the three new dependencies required to be = able to build the latest python3-setuptools-scm. >>>>>>>> Yes, that would be great :) >>>>>>>> -Michael >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Adolf >>>>>>>>> >>>>>>>>> On 03/01/2022 12:32, Michael Tremer wrote: >>>>>>>>>> Hello, >>>>>>>>>>> On 3 Jan 2022, at 11:31, Adolf Belka w= rote: >>>>>>>>>>> >>>>>>>>>>> Hi Michael, >>>>>>>>>>> >>>>>>>>>>> On 03/01/2022 11:52, Michael Tremer wrote: >>>>>>>>>>>> Hello, >>>>>>>>>>>> I am not sure if you already looked at what Arch is doing, but a= s far as I understand it works like this: >>>>>>>>>>> No I didn't look at Arch, which is surprising as I have in the pa= st with other packages but I didn't. So thanks very much for the pointer. >>>>>>>>>>>> They build Python 3 without pip and setuptools (https://github.c= om/archlinux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/= trunk/PKGBUILD#L59). >>>>>>>>>>>> At this stage we might need to build setuptools separately. >>>>>>>>>>>> They use a tool called pyproject2setuppy (great name!) to build = tomli (https://github.com/archlinux/svntogit-packages/blob/packages/python-to= mli/trunk/PKGBUILD). >>>>>>>>>>>> Setuptools-scm can then be built with tomli: https://github.com/= archlinux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUI= LD >>>>>>>>>>>> Would this work or did I overlook anything? >>>>>>>>>>> I will have a look at the links and give it a go. >>>>>>>>>>> >>>>>>>>>>> Thanks very much, I feel re-motivated now. >>>>>>>>>> Very good! I got a little bit lost in your last email and I think = I could tell. But following a solution that somebody else has come up with ca= n be a good thing to follow and of course we would customise it to our needs. >>>>>>>>>> Best, >>>>>>>>>> -Michael >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Adolf. >>>>>>>>>>>> -Michael >>>>>>>>>>>>> On 2 Jan 2022, at 13:44, Adolf Belka = wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Michael and All, >>>>>>>>>>>>> >>>>>>>>>>>>> On 26/12/2021 21:55, Michael Tremer wrote: >>>>>>>>>>>>>> Merry Christmas, >>>>>>>>>>>>>>> On 26 Dec 2021, at 15:38, Adolf Belka wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Michael, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 17/12/2021 12:41, Michael Tremer wrote: >>>>>>>>>>>>>>>> Hello Adolf, >>>>>>>>>>>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I am working on updating python from 3.8 to 3.10. Python it= self is okay after some other python related programs were also updated due t= o their older versions not working with 3.10 >>>>>>>>>>>>>>>> Thank you for working on this. >>>>>>>>>>>>>>>>> I am working through all the programs that reference python= -3.8 in their rootfiles, or wherever, to change them to 3.10 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> My question comes with regard to the changes to the rootfil= es to ensure they will be included properly in an update. >>>>>>>>>>>>>>>> We will have to ship everything that puts anything into /usr= /lib*/python3.*/site-packages again with the release of Python 3.10. >>>>>>>>>>>>>>>> This isn=E2=80=99t fun, but not as much of a pain in the rea= r as shipping a new Perl release. >>>>>>>>>>>>>>>>> For the addons then when I change the rootfile I am also in= crementing the PAK_VER number in the lfs files and that ensures that they wil= l be properly updated. >>>>>>>>>>>>>>>> Yes. That is required for Pakfire to find a new version. >>>>>>>>>>>>>>>>> I don't know what to do for the core programs where I have = updated the rootfile. The lfs files for these programs don't have a PAK_VER n= umber to increment and the actual package version number is not being changed= so I don't know what I need to change in the lfs to ensure that the rootfile= changes will be properly included in the update? >>>>>>>>>>>>>>>> I would say committing everything into a =E2=80=9Cpython-3.1= 0=E2=80=9D branch and submitting it all as a patchset is fine. All packages t= hat have been touched in that branch will have to be shipped again which shou= ld be easy to identify if it all comes in one large branch. >>>>>>>>>>>>>>> Okay, that is clear then.Will follow that approach. >>>>>>>>>>>>>>>>> I would appreciate any help on what I need to do for these = programs that have had changes only in their rootfiles. >>>>>>>>>>>>>>>> Absolutely. Did you find anything that didn=E2=80=99t build = yet? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I have not had any problem with python3.10 itself but I found= some of the older modules would not work with 3.10 so I decided to do a comb= ined update to 3.10 plus update modules to latest versions. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I found that after python3-setuptools-scm had run that the ne= xt module python3-six failed to run due to missing the tomli library. I comme= nted out six and reran the build and then python3-dateutil had the same probl= em and then python3-jmespath >>>>>>>>>>>>>> I completely lost track of what the status of setuptools/distu= tils and so on is. This has changed multiple times and become independent/par= t of the distribution again and so on. >>>>>>>>>>>>>>> After some investigation I have found that since version 6.0.= 0 of python3-setuptools-scm, tomli has been a required library when it is bei= ng used by other modules. So running version 6.0.0 of setuptools-scm worked w= ith python3.10 and subsequent modules built successfully as tomli not needed. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> So the obvious fix is to install tomli but that is when I ran= into a problem. tomli has no setup.py or setup.cfg files in its tarball. The= only way to build it is using pip. >>>>>>>>>>>>>> Err. Great. >>>>>>>>>>>>>>> Apparently, after some searching, setup.py has been deprecate= d as the recommended install method and if setup.py doesn't work the usual fi= x suggestion is to run pip install. >>>>>>>>>>>>>> Yes. But apparently the new system is deprecated, too. I reall= y dislike the state of packaging systems that come with any language these da= ys. >>>>>>>>>>>>>>> Having searched on pip for a while, it looks like builds usin= g it can still be done from tarballs locally without needing to access the in= ternet to download the modules. >>>>>>>>>>>>>> That is good! >>>>>>>>>>>>>>> As pip is installed as part of python3 i tried using that. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I used "python3 -m pip install --root=3D/ ." in place of >>>>>>>>>>>>>>> "python3 setup.py build >>>>>>>>>>>>>>> python3 setup.py install --root=3D/" >>>>>>>>>>>>>>> for the tomli source. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This failed because tomli requires flit_core to be available. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> flit_core also has no setup.py and so I tried to build it wit= h pip install. This failed with the error message that tomli was required as = a dependency. So I added --no-deps to the flit_core pip install line and this= time flit_core successfully built but the tomli build still came back with n= ot finding flit_core. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I have tried a variety of changes to the pip install line but= none of them has been successful and it has become clear that I am out of my= depth here and need guidance rather than just trying anything I find from a = google search. I am not even really clear if pip install is the correct appro= ach, except if not then I have no idea how to progress. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Help urgently needed on how I should be approaching this. >>>>>>>>>>>>>> Circular dependencies can happen. This won=E2=80=99t be the fi= rst time and it always is an utter nightmare with our buildsystem. >>>>>>>>>>>>>> This page states that pip comes by default with all binary pac= kages: >>>>>>>>>>>>>> https://docs.python.org/3/installing/index.html >>>>>>>>>>>>>> Is there a chance that it is part of the distribution and can = be enabled with a configure switch? >>>>>>>>>>>>> I wasn't able to find any way to get it enabled that way. >>>>>>>>>>>>> >>>>>>>>>>>>> I did find that pip install will not install if the package is = already present. You have to add --upgrade to make it update any package alre= ady present. So I tried that. Didn't work. >>>>>>>>>>>>> >>>>>>>>>>>>> I found a section about needing to use bootstrapping if you wan= t to install tomli. Followed the information for creating a wheel file for fl= it_core and then extracting that file into the site-packages directory. Tried= that. Didn't work. >>>>>>>>>>>>> >>>>>>>>>>>>> Rather than doing a complete build every time I changed to tryi= ng to build flit_core followed by tomli in a ./make.sh shell environment. >>>>>>>>>>>>> >>>>>>>>>>>>> Doing that I found that I had a pip install command that was cr= eating the flit_core directory in site-packages and also including the .dist-= info directory. The info I found said that the .dist-info directory had to be= there for pip to be able to find the flit_core package. >>>>>>>>>>>>> >>>>>>>>>>>>> However the pip install of tomli still ends up not finding any = flit_core package. In the bootstrapping information it mentioned about changi= ng the PYTHONPATH environment variable for find the flit_core package so I tr= ied that in the shell but it also made no difference. >>>>>>>>>>>>> >>>>>>>>>>>>> I then tried running python3 -c 'import flit_core' in the tomli= build shell before doing the tomli pip install. The import command completed= with no error but tomli still could not find any flit_core package. >>>>>>>>>>>>> >>>>>>>>>>>>> End result of all my work is that flit_core has been built and = installed into python3.10/site-packages but nothing seems to want to make the= tomli build find it. >>>>>>>>>>>>> >>>>>>>>>>>>> Does anyone have any ideas on what command(s) I need to use in = the tomli pip install to have the installed flit_core package visible. >>>>>>>>>>>>> >>>>>>>>>>>>> Regards and a Happy and Healthy New Year to all, >>>>>>>>>>>>> >>>>>>>>>>>>> Adolf. >>>>>>>>>>>>> >>>>>>>>>>>>>> -Michael >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>> Adolf. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> PS: I hope everyone had an enjoyable Christmas. >>>>>>>>>>>>>> I hope you did, too! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -Michael >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Adolf. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> --=20 >>>>>>>>>>>>>>>>> Sent from my laptop >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> --=20 >>>>>>>>>>>>>>> Sent from my laptop >>>>>>>>>>>>> >>>>>>>>>>>>> --=20 >>>>>>>>>>>>> Sent from my laptop >>>>>>>>> >>>>>>>>> --=20 >>>>>>>>> Sent from my laptop >>>>>>> >>>>>>> --=20 >>>>>>> Sent from my laptop >>>>> >>>>> --=20 >>>>> Sent from my laptop >=20 --===============1395231373591235952==--