From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: Question about updating python to 3.10 Date: Tue, 18 Jan 2022 15:02:10 +0000 Message-ID: In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0269055683714361561==" List-Id: --===============0269055683714361561== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello Adolf, I have just been trying to build a couple of Python packages that I need and = I finally arrived at needing flit-core. I suppose since you did already build this, there is no point in me doing all= this again and also there is no need to build all my packages on Python 3.8 = when we can have a newer release. Could you push this branch somewhere for me so that I can base my work on top= of it? Best, -Michael > On 13 Jan 2022, at 09:32, Michael Tremer wrot= e: >=20 > Hello, >=20 >> On 12 Jan 2022, at 21:08, Adolf Belka wrote: >>=20 >> Hi all, >>=20 >> On 12/01/2022 19:10, Michael Tremer wrote: >>> Hey, >>>> On 11 Jan 2022, at 14:32, Adolf Belka wrote: >>>>=20 >>>> Hi Michael and all, >>>>=20 >>>> python3-six has to stay as python3-dateutil explicitly specifies it in i= ts dependencies. >>>> So that solves that question. >>> Aww :( >>>> I have, with quite a bit of work, eventually been able to get all existi= ng python modules updated that had updates and built successfully with python= 3.10 >>> Yay! >>>> However later in the IPFire build libvirt failed with a problem parsing = some documentation files. >>>> Stashing my python3.10 changes and re-running the build had libvirt buil= ding without problems. So one of the python changes has caused it. libvirt us= es python3-docutils and that was updated but reverting that back to original = on its own did not cause libvirt to build successfully. >>>>=20 >>>> I looked in the libvirt .configure for an option to not build the docs b= ut it is not mentioned under ./configure --help. >>>>=20 >>>> I tried --without-docs and --disable-docs but both were not recognised i= n the build. >>>>=20 >>>> The current libvirt is from mid 2020 (6.5.0) so it is likely that it is = not compatible in some way with the updated python/modules. >>>>=20 >>>> I tried updating libvirt to the current 7.10.0 >>>> libvirt has replaced autotools with meson. I found meson options availab= le for all existing ./configure options except three. There is nothing in mes= on options for:- >>>> --with-virtualport >>>> --with-macvtap >>>> --without-dbus >>> Yay, another project that is using meson :( >>> We no longer need macvtap; we have dubs, so that should not hurt; and I h= ave no idea what virtualport is. >>>> I tried building it with all the other options and it successfully built= with the updated python3.10 and modules. All other packages after libvirt bu= ilt successfully without any problems. >>>>=20 >>>> Normally I would then use the updated libvirt, as that is needed to work= with the updated python packages, but I am not sure if those missing options= are now built into libvirt or are no longer available and what impact that m= ight have for the use of libvirt on IPFire. >>> I don=E2=80=99t think this would hurt us much. I have CCed Jonatan who ma= intains this, so maybe he knows more about this. >>>> There is nothing in the libvirt changelog about virtualport, macvtap or = dbus. >>>>=20 >>>> Searching on the internet, I was also not able to find anything related = to libvirt and these options. >>>>=20 >>>> Any suggestions/proposals on how to move forward on this? Should I just = build libvirt without those three options or ...? >>> Yes, and I think this could be submitted independently of the Python bran= ch because it is an add-on. We should be able to ship this at any time and mi= ght want to delay Python until we have enough space in a core update. >> Okay, then I will stash my python3.10 changes and submit the libvirt updat= e on its own. Then Jonaton can test it out. I don't use libvirt or qemu so I = can't test it easily. >>=20 >> Then when you think the time is right, let me know and I will submit the p= ython3.10 series of patches. >=20 > You can submit this at any time. I was just thinking about the conversation= between Peter and Arne during the last call regarding an updated kernel and = updated firmware. Depending on the size of Python, all of this might make an = update a little bit too large - however, this should not be a patchset that w= e cannot merge within two merge windows. >=20 >>> Thank you for looking into this. Sounds like a not so nice riddle to solv= e :) >> There were certainly some challenges, like updating python3-daemon which t= hen wanted to have twine as a fixed dependency which had nine dependencies so= me of which had more. I had got to 19 additional python packages and still go= ing and then I said enough was enough and looked at twine. It is required for= uploading python packages to PyPI which IPFire doesn't need. So I tried patc= hing the twine requirement out and that worked and then I found others that h= ad done the same because they had no response to the request to not have twin= e as a mandatory build requirement. >=20 > Python seems to become the hell that Perl used to be. In order to add one p= erl package, you had 20 dependencies. And those dependencies had their own de= pendencies too=E2=80=A6 It would be great to disable or patch out whatever is= possible so that we do not create more work for ourselves in the future and = have to update packages that are barely used - we already have enough of thes= e :) >=20 >> I found it all challenging but rewarding when I made progress and eventual= ly succeeded in building everything without errors. >=20 > I am so happy that you are putting all this time and effort into this becau= se it certainly requires patience - a trade I don=E2=80=99t have. >=20 > And Python is being used in all sorts of places in the distribution, so it = really isn=E2=80=99t anything we should not be up to date on. >=20 > -Michael >=20 >>=20 >> Regards, >> Adolf. >>> -Michael >>>>=20 >>>>=20 >>>> Regards, >>>>=20 >>>> Adolf. >>>>=20 >>>> On 07/01/2022 17:49, Michael Tremer wrote: >>>>> Hello, >>>>>> On 4 Jan 2022, at 12:49, Adolf Belka wrote: >>>>>>=20 >>>>>> Hi Michael, >>>>>>=20 >>>>>> Following the suggested Arch Linux approach I have got python3-setupto= ols-scm building successfully now. Yaaah. >>>>>> I had to install python3-toml, followed by python3-pyproject2setuppy f= ollowed by python3-tomli. >>>>>>=20 >>>>>> However then python3-six which installs after python3-setuptools-scm f= ails due to missing the python package python3-packaging. >>>>>>=20 >>>>>> I can install that package, and will do so as I continue testing out t= he python 3.10 build but I have the following question about python3-six. >>>>>>=20 >>>>>> Is six still needed. It is designed to ensure that programs can work w= ith either python2 or python3 but IPFire only has python3 now. >>>>>> I can find python3-six referenced only by aws-cli. In the aws-cli buil= d python3-six is not listed at all in the build requirements. I tried to sear= ch for aws-cli and python3-six being needed in run mode but I could not find = anything one way or the other. >>>>> If aws-cli no longer requires python3-six we can drop it. Last time we = touched it I had to bring the package back because aws-cli was still dependin= g on it. >>>>>> If python3-six is really needed for running aws-cli then I can install= it and the new dependency but if it is not needed we could remove two packag= es which would partially offset the three new dependencies required to be abl= e to build the latest python3-setuptools-scm. >>>>> Yes, that would be great :) >>>>> -Michael >>>>>>=20 >>>>>> Regards, >>>>>> Adolf >>>>>>=20 >>>>>> On 03/01/2022 12:32, Michael Tremer wrote: >>>>>>> Hello, >>>>>>>> On 3 Jan 2022, at 11:31, Adolf Belka wrot= e: >>>>>>>>=20 >>>>>>>> Hi Michael, >>>>>>>>=20 >>>>>>>> On 03/01/2022 11:52, Michael Tremer wrote: >>>>>>>>> Hello, >>>>>>>>> I am not sure if you already looked at what Arch is doing, but as f= ar as I understand it works like this: >>>>>>>> No I didn't look at Arch, which is surprising as I have in the past = with other packages but I didn't. So thanks very much for the pointer. >>>>>>>>> They build Python 3 without pip and setuptools (https://github.com/= archlinux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/tru= nk/PKGBUILD#L59). >>>>>>>>> At this stage we might need to build setuptools separately. >>>>>>>>> They use a tool called pyproject2setuppy (great name!) to build tom= li (https://github.com/archlinux/svntogit-packages/blob/packages/python-tomli= /trunk/PKGBUILD). >>>>>>>>> Setuptools-scm can then be built with tomli: https://github.com/arc= hlinux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD >>>>>>>>> Would this work or did I overlook anything? >>>>>>>> I will have a look at the links and give it a go. >>>>>>>>=20 >>>>>>>> Thanks very much, I feel re-motivated now. >>>>>>> Very good! I got a little bit lost in your last email and I think I c= ould tell. But following a solution that somebody else has come up with can b= e a good thing to follow and of course we would customise it to our needs. >>>>>>> Best, >>>>>>> -Michael >>>>>>>>=20 >>>>>>>> Regards, >>>>>>>> Adolf. >>>>>>>>> -Michael >>>>>>>>>> On 2 Jan 2022, at 13:44, Adolf Belka wr= ote: >>>>>>>>>>=20 >>>>>>>>>> Hi Michael and All, >>>>>>>>>>=20 >>>>>>>>>> On 26/12/2021 21:55, Michael Tremer wrote: >>>>>>>>>>> Merry Christmas, >>>>>>>>>>>> On 26 Dec 2021, at 15:38, Adolf Belka = wrote: >>>>>>>>>>>>=20 >>>>>>>>>>>> Hi Michael, >>>>>>>>>>>>=20 >>>>>>>>>>>> On 17/12/2021 12:41, Michael Tremer wrote: >>>>>>>>>>>>> Hello Adolf, >>>>>>>>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka wrote: >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> I am working on updating python from 3.8 to 3.10. Python itsel= f is okay after some other python related programs were also updated due to t= heir older versions not working with 3.10 >>>>>>>>>>>>> Thank you for working on this. >>>>>>>>>>>>>> I am working through all the programs that reference python-3.= 8 in their rootfiles, or wherever, to change them to 3.10 >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> My question comes with regard to the changes to the rootfiles = to ensure they will be included properly in an update. >>>>>>>>>>>>> We will have to ship everything that puts anything into /usr/li= b*/python3.*/site-packages again with the release of Python 3.10. >>>>>>>>>>>>> This isn=E2=80=99t fun, but not as much of a pain in the rear a= s shipping a new Perl release. >>>>>>>>>>>>>> For the addons then when I change the rootfile I am also incre= menting the PAK_VER number in the lfs files and that ensures that they will b= e properly updated. >>>>>>>>>>>>> Yes. That is required for Pakfire to find a new version. >>>>>>>>>>>>>> I don't know what to do for the core programs where I have upd= ated the rootfile. The lfs files for these programs don't have a PAK_VER numb= er to increment and the actual package version number is not being changed so= I don't know what I need to change in the lfs to ensure that the rootfile ch= anges will be properly included in the update? >>>>>>>>>>>>> I would say committing everything into a =E2=80=9Cpython-3.10= =E2=80=9D branch and submitting it all as a patchset is fine. All packages th= at have been touched in that branch will have to be shipped again which shoul= d be easy to identify if it all comes in one large branch. >>>>>>>>>>>> Okay, that is clear then.Will follow that approach. >>>>>>>>>>>>>> I would appreciate any help on what I need to do for these pro= grams that have had changes only in their rootfiles. >>>>>>>>>>>>> Absolutely. Did you find anything that didn=E2=80=99t build yet? >>>>>>>>>>>>=20 >>>>>>>>>>>> I have not had any problem with python3.10 itself but I found so= me of the older modules would not work with 3.10 so I decided to do a combine= d update to 3.10 plus update modules to latest versions. >>>>>>>>>>>>=20 >>>>>>>>>>>> I found that after python3-setuptools-scm had run that the next = module python3-six failed to run due to missing the tomli library. I commente= d out six and reran the build and then python3-dateutil had the same problem = and then python3-jmespath >>>>>>>>>>> I completely lost track of what the status of setuptools/distutil= s and so on is. This has changed multiple times and become independent/part o= f the distribution again and so on. >>>>>>>>>>>> After some investigation I have found that since version 6.0.0 o= f python3-setuptools-scm, tomli has been a required library when it is being = used by other modules. So running version 6.0.0 of setuptools-scm worked with= python3.10 and subsequent modules built successfully as tomli not needed. >>>>>>>>>>>>=20 >>>>>>>>>>>> So the obvious fix is to install tomli but that is when I ran in= to a problem. tomli has no setup.py or setup.cfg files in its tarball. The on= ly way to build it is using pip. >>>>>>>>>>> Err. Great. >>>>>>>>>>>> Apparently, after some searching, setup.py has been deprecated a= s the recommended install method and if setup.py doesn't work the usual fix s= uggestion is to run pip install. >>>>>>>>>>> Yes. But apparently the new system is deprecated, too. I really d= islike the state of packaging systems that come with any language these days. >>>>>>>>>>>> Having searched on pip for a while, it looks like builds using i= t can still be done from tarballs locally without needing to access the inter= net to download the modules. >>>>>>>>>>> That is good! >>>>>>>>>>>> As pip is installed as part of python3 i tried using that. >>>>>>>>>>>>=20 >>>>>>>>>>>> I used "python3 -m pip install --root=3D/ ." in place of >>>>>>>>>>>> "python3 setup.py build >>>>>>>>>>>> python3 setup.py install --root=3D/" >>>>>>>>>>>> for the tomli source. >>>>>>>>>>>>=20 >>>>>>>>>>>> This failed because tomli requires flit_core to be available. >>>>>>>>>>>>=20 >>>>>>>>>>>> flit_core also has no setup.py and so I tried to build it with p= ip install. This failed with the error message that tomli was required as a d= ependency. So I added --no-deps to the flit_core pip install line and this ti= me flit_core successfully built but the tomli build still came back with not = finding flit_core. >>>>>>>>>>>>=20 >>>>>>>>>>>> I have tried a variety of changes to the pip install line but no= ne of them has been successful and it has become clear that I am out of my de= pth here and need guidance rather than just trying anything I find from a goo= gle search. I am not even really clear if pip install is the correct approach= , except if not then I have no idea how to progress. >>>>>>>>>>>>=20 >>>>>>>>>>>> Help urgently needed on how I should be approaching this. >>>>>>>>>>> Circular dependencies can happen. This won=E2=80=99t be the first= time and it always is an utter nightmare with our buildsystem. >>>>>>>>>>> This page states that pip comes by default with all binary packag= es: >>>>>>>>>>> https://docs.python.org/3/installing/index.html >>>>>>>>>>> Is there a chance that it is part of the distribution and can be = enabled with a configure switch? >>>>>>>>>> I wasn't able to find any way to get it enabled that way. >>>>>>>>>>=20 >>>>>>>>>> I did find that pip install will not install if the package is alr= eady present. You have to add --upgrade to make it update any package already= present. So I tried that. Didn't work. >>>>>>>>>>=20 >>>>>>>>>> I found a section about needing to use bootstrapping if you want t= o install tomli. Followed the information for creating a wheel file for flit_= core and then extracting that file into the site-packages directory. Tried th= at. Didn't work. >>>>>>>>>>=20 >>>>>>>>>> Rather than doing a complete build every time I changed to trying = to build flit_core followed by tomli in a ./make.sh shell environment. >>>>>>>>>>=20 >>>>>>>>>> Doing that I found that I had a pip install command that was creat= ing the flit_core directory in site-packages and also including the .dist-inf= o directory. The info I found said that the .dist-info directory had to be th= ere for pip to be able to find the flit_core package. >>>>>>>>>>=20 >>>>>>>>>> However the pip install of tomli still ends up not finding any fli= t_core package. In the bootstrapping information it mentioned about changing = the PYTHONPATH environment variable for find the flit_core package so I tried= that in the shell but it also made no difference. >>>>>>>>>>=20 >>>>>>>>>> I then tried running python3 -c 'import flit_core' in the tomli bu= ild shell before doing the tomli pip install. The import command completed wi= th no error but tomli still could not find any flit_core package. >>>>>>>>>>=20 >>>>>>>>>> End result of all my work is that flit_core has been built and ins= talled into python3.10/site-packages but nothing seems to want to make the to= mli build find it. >>>>>>>>>>=20 >>>>>>>>>> Does anyone have any ideas on what command(s) I need to use in the= tomli pip install to have the installed flit_core package visible. >>>>>>>>>>=20 >>>>>>>>>> Regards and a Happy and Healthy New Year to all, >>>>>>>>>>=20 >>>>>>>>>> Adolf. >>>>>>>>>>=20 >>>>>>>>>>> -Michael >>>>>>>>>>>>=20 >>>>>>>>>>>> Regards, >>>>>>>>>>>> Adolf. >>>>>>>>>>>>=20 >>>>>>>>>>>> PS: I hope everyone had an enjoyable Christmas. >>>>>>>>>>> I hope you did, too! >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>>> -Michael >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> Adolf. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> --=20 >>>>>>>>>>>>>> Sent from my laptop >>>>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>> --=20 >>>>>>>>>>>> Sent from my laptop >>>>>>>>>>=20 >>>>>>>>>> --=20 >>>>>>>>>> Sent from my laptop >>>>>>=20 >>>>>> --=20 >>>>>> Sent from my laptop >>>>=20 >>>> --=20 >>>> Sent from my laptop >>=20 >> --=20 >> Sent from my laptop --===============0269055683714361561==--