From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: Question about updating python to 3.10 Date: Tue, 18 Jan 2022 18:51:22 +0000 Message-ID: <2D0835FA-9CF5-4833-8182-E4D920823A31@ipfire.org> In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2074495880248803419==" List-Id: --===============2074495880248803419== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello, > On 18 Jan 2022, at 16:42, Adolf Belka wrote: >=20 > Hi Michael, >=20 > On 18/01/2022 16:02, Michael Tremer wrote: >> Hello Adolf, >> I have just been trying to build a couple of Python packages that I need a= nd I finally arrived at needing flit-core. >> I suppose since you did already build this, there is no point in me doing = all this again and also there is no need to build all my packages on Python 3= .8 when we can have a newer release. > I am afraid that you will need to build flit-core. I was building it when = I was trying to use pip to install tomli but that approached failed to go fur= ther and that was when I used pyproject2setuppy, from your input, to build to= mli and that did not need flit-core. Oh, I suppose that didn=E2=80=99t hold for very long because I now have a pac= kage that needs it :( > In terms of the python3.10 branch that I have on my git repository at home,= I presume I could push it onto my personal git repository that you set up fo= r me - https://git.ipfire.org/?p=3Dpeople/bonnietwin/ipfire-2.x.git;a=3Dsumma= ry Yes, it is always a good idea to backup things to the server. > The only thing is I am not sure what command I should use to push the branc= h from my system here to the IPFire one. If you have your repository set up with =E2=80=9Cgit remote add =E2=80=A6=E2= =80=9D then you only need to run =E2=80=9Cgit push bonnietwin python3.10=E2= =80=9D. -Michael >=20 > I can have a go at trying it and see how it goes. >=20 > Regards, >=20 > Adolf. >> Could you push this branch somewhere for me so that I can base my work on = top of it? >> Best, >> -Michael >>> On 13 Jan 2022, at 09:32, Michael Tremer wr= ote: >>>=20 >>> Hello, >>>=20 >>>> On 12 Jan 2022, at 21:08, Adolf Belka wrote: >>>>=20 >>>> Hi all, >>>>=20 >>>> On 12/01/2022 19:10, Michael Tremer wrote: >>>>> Hey, >>>>>> On 11 Jan 2022, at 14:32, Adolf Belka wrote: >>>>>>=20 >>>>>> Hi Michael and all, >>>>>>=20 >>>>>> python3-six has to stay as python3-dateutil explicitly specifies it in= its dependencies. >>>>>> So that solves that question. >>>>> Aww :( >>>>>> I have, with quite a bit of work, eventually been able to get all exis= ting python modules updated that had updates and built successfully with pyth= on3.10 >>>>> Yay! >>>>>> However later in the IPFire build libvirt failed with a problem parsin= g some documentation files. >>>>>> Stashing my python3.10 changes and re-running the build had libvirt bu= ilding without problems. So one of the python changes has caused it. libvirt = uses python3-docutils and that was updated but reverting that back to origina= l on its own did not cause libvirt to build successfully. >>>>>>=20 >>>>>> I looked in the libvirt .configure for an option to not build the docs= but it is not mentioned under ./configure --help. >>>>>>=20 >>>>>> I tried --without-docs and --disable-docs but both were not recognised= in the build. >>>>>>=20 >>>>>> The current libvirt is from mid 2020 (6.5.0) so it is likely that it i= s not compatible in some way with the updated python/modules. >>>>>>=20 >>>>>> I tried updating libvirt to the current 7.10.0 >>>>>> libvirt has replaced autotools with meson. I found meson options avail= able for all existing ./configure options except three. There is nothing in m= eson options for:- >>>>>> --with-virtualport >>>>>> --with-macvtap >>>>>> --without-dbus >>>>> Yay, another project that is using meson :( >>>>> We no longer need macvtap; we have dubs, so that should not hurt; and I= have no idea what virtualport is. >>>>>> I tried building it with all the other options and it successfully bui= lt with the updated python3.10 and modules. All other packages after libvirt = built successfully without any problems. >>>>>>=20 >>>>>> Normally I would then use the updated libvirt, as that is needed to wo= rk with the updated python packages, but I am not sure if those missing optio= ns are now built into libvirt or are no longer available and what impact that= might have for the use of libvirt on IPFire. >>>>> I don=E2=80=99t think this would hurt us much. I have CCed Jonatan who = maintains this, so maybe he knows more about this. >>>>>> There is nothing in the libvirt changelog about virtualport, macvtap o= r dbus. >>>>>>=20 >>>>>> Searching on the internet, I was also not able to find anything relate= d to libvirt and these options. >>>>>>=20 >>>>>> Any suggestions/proposals on how to move forward on this? Should I jus= t build libvirt without those three options or ...? >>>>> Yes, and I think this could be submitted independently of the Python br= anch because it is an add-on. We should be able to ship this at any time and = might want to delay Python until we have enough space in a core update. >>>> Okay, then I will stash my python3.10 changes and submit the libvirt upd= ate on its own. Then Jonaton can test it out. I don't use libvirt or qemu so = I can't test it easily. >>>>=20 >>>> Then when you think the time is right, let me know and I will submit the= python3.10 series of patches. >>>=20 >>> You can submit this at any time. I was just thinking about the conversati= on between Peter and Arne during the last call regarding an updated kernel an= d updated firmware. Depending on the size of Python, all of this might make a= n update a little bit too large - however, this should not be a patchset that= we cannot merge within two merge windows. >>>=20 >>>>> Thank you for looking into this. Sounds like a not so nice riddle to so= lve :) >>>> There were certainly some challenges, like updating python3-daemon which= then wanted to have twine as a fixed dependency which had nine dependencies = some of which had more. I had got to 19 additional python packages and still = going and then I said enough was enough and looked at twine. It is required f= or uploading python packages to PyPI which IPFire doesn't need. So I tried pa= tching the twine requirement out and that worked and then I found others that= had done the same because they had no response to the request to not have tw= ine as a mandatory build requirement. >>>=20 >>> Python seems to become the hell that Perl used to be. In order to add one= perl package, you had 20 dependencies. And those dependencies had their own = dependencies too=E2=80=A6 It would be great to disable or patch out whatever = is possible so that we do not create more work for ourselves in the future an= d have to update packages that are barely used - we already have enough of th= ese :) >>>=20 >>>> I found it all challenging but rewarding when I made progress and eventu= ally succeeded in building everything without errors. >>>=20 >>> I am so happy that you are putting all this time and effort into this bec= ause it certainly requires patience - a trade I don=E2=80=99t have. >>>=20 >>> And Python is being used in all sorts of places in the distribution, so i= t really isn=E2=80=99t anything we should not be up to date on. >>>=20 >>> -Michael >>>=20 >>>>=20 >>>> Regards, >>>> Adolf. >>>>> -Michael >>>>>>=20 >>>>>>=20 >>>>>> Regards, >>>>>>=20 >>>>>> Adolf. >>>>>>=20 >>>>>> On 07/01/2022 17:49, Michael Tremer wrote: >>>>>>> Hello, >>>>>>>> On 4 Jan 2022, at 12:49, Adolf Belka wrot= e: >>>>>>>>=20 >>>>>>>> Hi Michael, >>>>>>>>=20 >>>>>>>> Following the suggested Arch Linux approach I have got python3-setup= tools-scm building successfully now. Yaaah. >>>>>>>> I had to install python3-toml, followed by python3-pyproject2setuppy= followed by python3-tomli. >>>>>>>>=20 >>>>>>>> However then python3-six which installs after python3-setuptools-scm= fails due to missing the python package python3-packaging. >>>>>>>>=20 >>>>>>>> I can install that package, and will do so as I continue testing out= the python 3.10 build but I have the following question about python3-six. >>>>>>>>=20 >>>>>>>> Is six still needed. It is designed to ensure that programs can work= with either python2 or python3 but IPFire only has python3 now. >>>>>>>> I can find python3-six referenced only by aws-cli. In the aws-cli bu= ild python3-six is not listed at all in the build requirements. I tried to se= arch for aws-cli and python3-six being needed in run mode but I could not fin= d anything one way or the other. >>>>>>> If aws-cli no longer requires python3-six we can drop it. Last time w= e touched it I had to bring the package back because aws-cli was still depend= ing on it. >>>>>>>> If python3-six is really needed for running aws-cli then I can insta= ll it and the new dependency but if it is not needed we could remove two pack= ages which would partially offset the three new dependencies required to be a= ble to build the latest python3-setuptools-scm. >>>>>>> Yes, that would be great :) >>>>>>> -Michael >>>>>>>>=20 >>>>>>>> Regards, >>>>>>>> Adolf >>>>>>>>=20 >>>>>>>> On 03/01/2022 12:32, Michael Tremer wrote: >>>>>>>>> Hello, >>>>>>>>>> On 3 Jan 2022, at 11:31, Adolf Belka wr= ote: >>>>>>>>>>=20 >>>>>>>>>> Hi Michael, >>>>>>>>>>=20 >>>>>>>>>> On 03/01/2022 11:52, Michael Tremer wrote: >>>>>>>>>>> Hello, >>>>>>>>>>> I am not sure if you already looked at what Arch is doing, but as= far as I understand it works like this: >>>>>>>>>> No I didn't look at Arch, which is surprising as I have in the pas= t with other packages but I didn't. So thanks very much for the pointer. >>>>>>>>>>> They build Python 3 without pip and setuptools (https://github.co= m/archlinux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/t= runk/PKGBUILD#L59). >>>>>>>>>>> At this stage we might need to build setuptools separately. >>>>>>>>>>> They use a tool called pyproject2setuppy (great name!) to build t= omli (https://github.com/archlinux/svntogit-packages/blob/packages/python-tom= li/trunk/PKGBUILD). >>>>>>>>>>> Setuptools-scm can then be built with tomli: https://github.com/a= rchlinux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD >>>>>>>>>>> Would this work or did I overlook anything? >>>>>>>>>> I will have a look at the links and give it a go. >>>>>>>>>>=20 >>>>>>>>>> Thanks very much, I feel re-motivated now. >>>>>>>>> Very good! I got a little bit lost in your last email and I think I= could tell. But following a solution that somebody else has come up with can= be a good thing to follow and of course we would customise it to our needs. >>>>>>>>> Best, >>>>>>>>> -Michael >>>>>>>>>>=20 >>>>>>>>>> Regards, >>>>>>>>>> Adolf. >>>>>>>>>>> -Michael >>>>>>>>>>>> On 2 Jan 2022, at 13:44, Adolf Belka = wrote: >>>>>>>>>>>>=20 >>>>>>>>>>>> Hi Michael and All, >>>>>>>>>>>>=20 >>>>>>>>>>>> On 26/12/2021 21:55, Michael Tremer wrote: >>>>>>>>>>>>> Merry Christmas, >>>>>>>>>>>>>> On 26 Dec 2021, at 15:38, Adolf Belka wrote: >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> Hi Michael, >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> On 17/12/2021 12:41, Michael Tremer wrote: >>>>>>>>>>>>>>> Hello Adolf, >>>>>>>>>>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka wrote: >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> I am working on updating python from 3.8 to 3.10. Python its= elf is okay after some other python related programs were also updated due to= their older versions not working with 3.10 >>>>>>>>>>>>>>> Thank you for working on this. >>>>>>>>>>>>>>>> I am working through all the programs that reference python-= 3.8 in their rootfiles, or wherever, to change them to 3.10 >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> My question comes with regard to the changes to the rootfile= s to ensure they will be included properly in an update. >>>>>>>>>>>>>>> We will have to ship everything that puts anything into /usr/= lib*/python3.*/site-packages again with the release of Python 3.10. >>>>>>>>>>>>>>> This isn=E2=80=99t fun, but not as much of a pain in the rear= as shipping a new Perl release. >>>>>>>>>>>>>>>> For the addons then when I change the rootfile I am also inc= rementing the PAK_VER number in the lfs files and that ensures that they will= be properly updated. >>>>>>>>>>>>>>> Yes. That is required for Pakfire to find a new version. >>>>>>>>>>>>>>>> I don't know what to do for the core programs where I have u= pdated the rootfile. The lfs files for these programs don't have a PAK_VER nu= mber to increment and the actual package version number is not being changed = so I don't know what I need to change in the lfs to ensure that the rootfile = changes will be properly included in the update? >>>>>>>>>>>>>>> I would say committing everything into a =E2=80=9Cpython-3.10= =E2=80=9D branch and submitting it all as a patchset is fine. All packages th= at have been touched in that branch will have to be shipped again which shoul= d be easy to identify if it all comes in one large branch. >>>>>>>>>>>>>> Okay, that is clear then.Will follow that approach. >>>>>>>>>>>>>>>> I would appreciate any help on what I need to do for these p= rograms that have had changes only in their rootfiles. >>>>>>>>>>>>>>> Absolutely. Did you find anything that didn=E2=80=99t build y= et? >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> I have not had any problem with python3.10 itself but I found = some of the older modules would not work with 3.10 so I decided to do a combi= ned update to 3.10 plus update modules to latest versions. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> I found that after python3-setuptools-scm had run that the nex= t module python3-six failed to run due to missing the tomli library. I commen= ted out six and reran the build and then python3-dateutil had the same proble= m and then python3-jmespath >>>>>>>>>>>>> I completely lost track of what the status of setuptools/distut= ils and so on is. This has changed multiple times and become independent/part= of the distribution again and so on. >>>>>>>>>>>>>> After some investigation I have found that since version 6.0.0= of python3-setuptools-scm, tomli has been a required library when it is bein= g used by other modules. So running version 6.0.0 of setuptools-scm worked wi= th python3.10 and subsequent modules built successfully as tomli not needed. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> So the obvious fix is to install tomli but that is when I ran = into a problem. tomli has no setup.py or setup.cfg files in its tarball. The = only way to build it is using pip. >>>>>>>>>>>>> Err. Great. >>>>>>>>>>>>>> Apparently, after some searching, setup.py has been deprecated= as the recommended install method and if setup.py doesn't work the usual fix= suggestion is to run pip install. >>>>>>>>>>>>> Yes. But apparently the new system is deprecated, too. I really= dislike the state of packaging systems that come with any language these day= s. >>>>>>>>>>>>>> Having searched on pip for a while, it looks like builds using= it can still be done from tarballs locally without needing to access the int= ernet to download the modules. >>>>>>>>>>>>> That is good! >>>>>>>>>>>>>> As pip is installed as part of python3 i tried using that. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> I used "python3 -m pip install --root=3D/ ." in place of >>>>>>>>>>>>>> "python3 setup.py build >>>>>>>>>>>>>> python3 setup.py install --root=3D/" >>>>>>>>>>>>>> for the tomli source. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> This failed because tomli requires flit_core to be available. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> flit_core also has no setup.py and so I tried to build it with= pip install. This failed with the error message that tomli was required as a= dependency. So I added --no-deps to the flit_core pip install line and this = time flit_core successfully built but the tomli build still came back with no= t finding flit_core. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> I have tried a variety of changes to the pip install line but = none of them has been successful and it has become clear that I am out of my = depth here and need guidance rather than just trying anything I find from a g= oogle search. I am not even really clear if pip install is the correct approa= ch, except if not then I have no idea how to progress. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> Help urgently needed on how I should be approaching this. >>>>>>>>>>>>> Circular dependencies can happen. This won=E2=80=99t be the fir= st time and it always is an utter nightmare with our buildsystem. >>>>>>>>>>>>> This page states that pip comes by default with all binary pack= ages: >>>>>>>>>>>>> https://docs.python.org/3/installing/index.html >>>>>>>>>>>>> Is there a chance that it is part of the distribution and can b= e enabled with a configure switch? >>>>>>>>>>>> I wasn't able to find any way to get it enabled that way. >>>>>>>>>>>>=20 >>>>>>>>>>>> I did find that pip install will not install if the package is a= lready present. You have to add --upgrade to make it update any package alrea= dy present. So I tried that. Didn't work. >>>>>>>>>>>>=20 >>>>>>>>>>>> I found a section about needing to use bootstrapping if you want= to install tomli. Followed the information for creating a wheel file for fli= t_core and then extracting that file into the site-packages directory. Tried = that. Didn't work. >>>>>>>>>>>>=20 >>>>>>>>>>>> Rather than doing a complete build every time I changed to tryin= g to build flit_core followed by tomli in a ./make.sh shell environment. >>>>>>>>>>>>=20 >>>>>>>>>>>> Doing that I found that I had a pip install command that was cre= ating the flit_core directory in site-packages and also including the .dist-i= nfo directory. The info I found said that the .dist-info directory had to be = there for pip to be able to find the flit_core package. >>>>>>>>>>>>=20 >>>>>>>>>>>> However the pip install of tomli still ends up not finding any f= lit_core package. In the bootstrapping information it mentioned about changin= g the PYTHONPATH environment variable for find the flit_core package so I tri= ed that in the shell but it also made no difference. >>>>>>>>>>>>=20 >>>>>>>>>>>> I then tried running python3 -c 'import flit_core' in the tomli = build shell before doing the tomli pip install. The import command completed = with no error but tomli still could not find any flit_core package. >>>>>>>>>>>>=20 >>>>>>>>>>>> End result of all my work is that flit_core has been built and i= nstalled into python3.10/site-packages but nothing seems to want to make the = tomli build find it. >>>>>>>>>>>>=20 >>>>>>>>>>>> Does anyone have any ideas on what command(s) I need to use in t= he tomli pip install to have the installed flit_core package visible. >>>>>>>>>>>>=20 >>>>>>>>>>>> Regards and a Happy and Healthy New Year to all, >>>>>>>>>>>>=20 >>>>>>>>>>>> Adolf. >>>>>>>>>>>>=20 >>>>>>>>>>>>> -Michael >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> Adolf. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> PS: I hope everyone had an enjoyable Christmas. >>>>>>>>>>>>> I hope you did, too! >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> -Michael >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> Adolf. >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> --=20 >>>>>>>>>>>>>>>> Sent from my laptop >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> --=20 >>>>>>>>>>>>>> Sent from my laptop >>>>>>>>>>>>=20 >>>>>>>>>>>> --=20 >>>>>>>>>>>> Sent from my laptop >>>>>>>>=20 >>>>>>>> --=20 >>>>>>>> Sent from my laptop >>>>>>=20 >>>>>> --=20 >>>>>> Sent from my laptop >>>>=20 >>>> --=20 >>>> Sent from my laptop --===============2074495880248803419==--