From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: Question about updating python to 3.10 Date: Wed, 19 Jan 2022 08:25:08 +0000 Message-ID: <92927B9F-77B3-466B-A484-C9BE1D919E2B@ipfire.org> In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0964729646696780966==" List-Id: --===============0964729646696780966== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello, I am sure this was documented in the wiki somewhere. For some reason I cannot= find it any more. Please let me know if you need any assistance from me. -Michael > On 18 Jan 2022, at 21:21, Adolf Belka wrote: >=20 > Hi Michael, >=20 > On 18/01/2022 19:51, Michael Tremer wrote: >> Hello, >>> On 18 Jan 2022, at 16:42, Adolf Belka wrote: >>>=20 >>> Hi Michael, >>>=20 >>> On 18/01/2022 16:02, Michael Tremer wrote: >>>> Hello Adolf, >>>> I have just been trying to build a couple of Python packages that I need= and I finally arrived at needing flit-core. >>>> I suppose since you did already build this, there is no point in me doin= g all this again and also there is no need to build all my packages on Python= 3.8 when we can have a newer release. >>> I am afraid that you will need to build flit-core. I was building it whe= n I was trying to use pip to install tomli but that approached failed to go f= urther and that was when I used pyproject2setuppy, from your input, to build = tomli and that did not need flit-core. >> Oh, I suppose that didn=E2=80=99t hold for very long because I now have a = package that needs it :( > flit-core also has no setup.py but you will then be able to use the pyproje= ct2setuppy package from my python3.10 branch to build flit-core. >>> In terms of the python3.10 branch that I have on my git repository at hom= e, I presume I could push it onto my personal git repository that you set up = for me - https://git.ipfire.org/?p=3Dpeople/bonnietwin/ipfire-2.x.git;a=3Dsum= mary >> Yes, it is always a good idea to backup things to the server. >>> The only thing is I am not sure what command I should use to push the bra= nch from my system here to the IPFire one. >> If you have your repository set up with =E2=80=9Cgit remote add =E2=80=A6= =E2=80=9D then you only need to run =E2=80=9Cgit push bonnietwin python3.10= =E2=80=9D. > Aaah. I think when I tried to experiment with this before I created a local= new clone of bonnietwin which was then separate from my running local reposi= tory that I used for building and I struggled to figure out how to get branch= es between them. >=20 > Your mention of git remote add indicates that I can add my bonnietwin repos= itory to my running build repository. Thais is what I was missing before. > Thanks for the clue, will give it a try. >=20 > Regards, >=20 > Adolf. >> -Michael >>>=20 >>> I can have a go at trying it and see how it goes. >>>=20 >>> Regards, >>>=20 >>> Adolf. >>>> Could you push this branch somewhere for me so that I can base my work o= n top of it? >>>> Best, >>>> -Michael >>>>> On 13 Jan 2022, at 09:32, Michael Tremer = wrote: >>>>>=20 >>>>> Hello, >>>>>=20 >>>>>> On 12 Jan 2022, at 21:08, Adolf Belka wrote: >>>>>>=20 >>>>>> Hi all, >>>>>>=20 >>>>>> On 12/01/2022 19:10, Michael Tremer wrote: >>>>>>> Hey, >>>>>>>> On 11 Jan 2022, at 14:32, Adolf Belka wro= te: >>>>>>>>=20 >>>>>>>> Hi Michael and all, >>>>>>>>=20 >>>>>>>> python3-six has to stay as python3-dateutil explicitly specifies it = in its dependencies. >>>>>>>> So that solves that question. >>>>>>> Aww :( >>>>>>>> I have, with quite a bit of work, eventually been able to get all ex= isting python modules updated that had updates and built successfully with py= thon3.10 >>>>>>> Yay! >>>>>>>> However later in the IPFire build libvirt failed with a problem pars= ing some documentation files. >>>>>>>> Stashing my python3.10 changes and re-running the build had libvirt = building without problems. So one of the python changes has caused it. libvir= t uses python3-docutils and that was updated but reverting that back to origi= nal on its own did not cause libvirt to build successfully. >>>>>>>>=20 >>>>>>>> I looked in the libvirt .configure for an option to not build the do= cs but it is not mentioned under ./configure --help. >>>>>>>>=20 >>>>>>>> I tried --without-docs and --disable-docs but both were not recognis= ed in the build. >>>>>>>>=20 >>>>>>>> The current libvirt is from mid 2020 (6.5.0) so it is likely that it= is not compatible in some way with the updated python/modules. >>>>>>>>=20 >>>>>>>> I tried updating libvirt to the current 7.10.0 >>>>>>>> libvirt has replaced autotools with meson. I found meson options ava= ilable for all existing ./configure options except three. There is nothing in= meson options for:- >>>>>>>> --with-virtualport >>>>>>>> --with-macvtap >>>>>>>> --without-dbus >>>>>>> Yay, another project that is using meson :( >>>>>>> We no longer need macvtap; we have dubs, so that should not hurt; and= I have no idea what virtualport is. >>>>>>>> I tried building it with all the other options and it successfully b= uilt with the updated python3.10 and modules. All other packages after libvir= t built successfully without any problems. >>>>>>>>=20 >>>>>>>> Normally I would then use the updated libvirt, as that is needed to = work with the updated python packages, but I am not sure if those missing opt= ions are now built into libvirt or are no longer available and what impact th= at might have for the use of libvirt on IPFire. >>>>>>> I don=E2=80=99t think this would hurt us much. I have CCed Jonatan wh= o maintains this, so maybe he knows more about this. >>>>>>>> There is nothing in the libvirt changelog about virtualport, macvtap= or dbus. >>>>>>>>=20 >>>>>>>> Searching on the internet, I was also not able to find anything rela= ted to libvirt and these options. >>>>>>>>=20 >>>>>>>> Any suggestions/proposals on how to move forward on this? Should I j= ust build libvirt without those three options or ...? >>>>>>> Yes, and I think this could be submitted independently of the Python = branch because it is an add-on. We should be able to ship this at any time an= d might want to delay Python until we have enough space in a core update. >>>>>> Okay, then I will stash my python3.10 changes and submit the libvirt u= pdate on its own. Then Jonaton can test it out. I don't use libvirt or qemu s= o I can't test it easily. >>>>>>=20 >>>>>> Then when you think the time is right, let me know and I will submit t= he python3.10 series of patches. >>>>>=20 >>>>> You can submit this at any time. I was just thinking about the conversa= tion between Peter and Arne during the last call regarding an updated kernel = and updated firmware. Depending on the size of Python, all of this might make= an update a little bit too large - however, this should not be a patchset th= at we cannot merge within two merge windows. >>>>>=20 >>>>>>> Thank you for looking into this. Sounds like a not so nice riddle to = solve :) >>>>>> There were certainly some challenges, like updating python3-daemon whi= ch then wanted to have twine as a fixed dependency which had nine dependencie= s some of which had more. I had got to 19 additional python packages and stil= l going and then I said enough was enough and looked at twine. It is required= for uploading python packages to PyPI which IPFire doesn't need. So I tried = patching the twine requirement out and that worked and then I found others th= at had done the same because they had no response to the request to not have = twine as a mandatory build requirement. >>>>>=20 >>>>> Python seems to become the hell that Perl used to be. In order to add o= ne perl package, you had 20 dependencies. And those dependencies had their ow= n dependencies too=E2=80=A6 It would be great to disable or patch out whateve= r is possible so that we do not create more work for ourselves in the future = and have to update packages that are barely used - we already have enough of = these :) >>>>>=20 >>>>>> I found it all challenging but rewarding when I made progress and even= tually succeeded in building everything without errors. >>>>>=20 >>>>> I am so happy that you are putting all this time and effort into this b= ecause it certainly requires patience - a trade I don=E2=80=99t have. >>>>>=20 >>>>> And Python is being used in all sorts of places in the distribution, so= it really isn=E2=80=99t anything we should not be up to date on. >>>>>=20 >>>>> -Michael >>>>>=20 >>>>>>=20 >>>>>> Regards, >>>>>> Adolf. >>>>>>> -Michael >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> Regards, >>>>>>>>=20 >>>>>>>> Adolf. >>>>>>>>=20 >>>>>>>> On 07/01/2022 17:49, Michael Tremer wrote: >>>>>>>>> Hello, >>>>>>>>>> On 4 Jan 2022, at 12:49, Adolf Belka wr= ote: >>>>>>>>>>=20 >>>>>>>>>> Hi Michael, >>>>>>>>>>=20 >>>>>>>>>> Following the suggested Arch Linux approach I have got python3-set= uptools-scm building successfully now. Yaaah. >>>>>>>>>> I had to install python3-toml, followed by python3-pyproject2setup= py followed by python3-tomli. >>>>>>>>>>=20 >>>>>>>>>> However then python3-six which installs after python3-setuptools-s= cm fails due to missing the python package python3-packaging. >>>>>>>>>>=20 >>>>>>>>>> I can install that package, and will do so as I continue testing o= ut the python 3.10 build but I have the following question about python3-six. >>>>>>>>>>=20 >>>>>>>>>> Is six still needed. It is designed to ensure that programs can wo= rk with either python2 or python3 but IPFire only has python3 now. >>>>>>>>>> I can find python3-six referenced only by aws-cli. In the aws-cli = build python3-six is not listed at all in the build requirements. I tried to = search for aws-cli and python3-six being needed in run mode but I could not f= ind anything one way or the other. >>>>>>>>> If aws-cli no longer requires python3-six we can drop it. Last time= we touched it I had to bring the package back because aws-cli was still depe= nding on it. >>>>>>>>>> If python3-six is really needed for running aws-cli then I can ins= tall it and the new dependency but if it is not needed we could remove two pa= ckages which would partially offset the three new dependencies required to be= able to build the latest python3-setuptools-scm. >>>>>>>>> Yes, that would be great :) >>>>>>>>> -Michael >>>>>>>>>>=20 >>>>>>>>>> Regards, >>>>>>>>>> Adolf >>>>>>>>>>=20 >>>>>>>>>> On 03/01/2022 12:32, Michael Tremer wrote: >>>>>>>>>>> Hello, >>>>>>>>>>>> On 3 Jan 2022, at 11:31, Adolf Belka = wrote: >>>>>>>>>>>>=20 >>>>>>>>>>>> Hi Michael, >>>>>>>>>>>>=20 >>>>>>>>>>>> On 03/01/2022 11:52, Michael Tremer wrote: >>>>>>>>>>>>> Hello, >>>>>>>>>>>>> I am not sure if you already looked at what Arch is doing, but = as far as I understand it works like this: >>>>>>>>>>>> No I didn't look at Arch, which is surprising as I have in the p= ast with other packages but I didn't. So thanks very much for the pointer. >>>>>>>>>>>>> They build Python 3 without pip and setuptools (https://github.= com/archlinux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733= /trunk/PKGBUILD#L59). >>>>>>>>>>>>> At this stage we might need to build setuptools separately. >>>>>>>>>>>>> They use a tool called pyproject2setuppy (great name!) to build= tomli (https://github.com/archlinux/svntogit-packages/blob/packages/python-t= omli/trunk/PKGBUILD). >>>>>>>>>>>>> Setuptools-scm can then be built with tomli: https://github.com= /archlinux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBU= ILD >>>>>>>>>>>>> Would this work or did I overlook anything? >>>>>>>>>>>> I will have a look at the links and give it a go. >>>>>>>>>>>>=20 >>>>>>>>>>>> Thanks very much, I feel re-motivated now. >>>>>>>>>>> Very good! I got a little bit lost in your last email and I think= I could tell. But following a solution that somebody else has come up with c= an be a good thing to follow and of course we would customise it to our needs. >>>>>>>>>>> Best, >>>>>>>>>>> -Michael >>>>>>>>>>>>=20 >>>>>>>>>>>> Regards, >>>>>>>>>>>> Adolf. >>>>>>>>>>>>> -Michael >>>>>>>>>>>>>> On 2 Jan 2022, at 13:44, Adolf Belka wrote: >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> Hi Michael and All, >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> On 26/12/2021 21:55, Michael Tremer wrote: >>>>>>>>>>>>>>> Merry Christmas, >>>>>>>>>>>>>>>> On 26 Dec 2021, at 15:38, Adolf Belka wrote: >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> Hi Michael, >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> On 17/12/2021 12:41, Michael Tremer wrote: >>>>>>>>>>>>>>>>> Hello Adolf, >>>>>>>>>>>>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka wrote: >>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>> I am working on updating python from 3.8 to 3.10. Python i= tself is okay after some other python related programs were also updated due = to their older versions not working with 3.10 >>>>>>>>>>>>>>>>> Thank you for working on this. >>>>>>>>>>>>>>>>>> I am working through all the programs that reference pytho= n-3.8 in their rootfiles, or wherever, to change them to 3.10 >>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>> My question comes with regard to the changes to the rootfi= les to ensure they will be included properly in an update. >>>>>>>>>>>>>>>>> We will have to ship everything that puts anything into /us= r/lib*/python3.*/site-packages again with the release of Python 3.10. >>>>>>>>>>>>>>>>> This isn=E2=80=99t fun, but not as much of a pain in the re= ar as shipping a new Perl release. >>>>>>>>>>>>>>>>>> For the addons then when I change the rootfile I am also i= ncrementing the PAK_VER number in the lfs files and that ensures that they wi= ll be properly updated. >>>>>>>>>>>>>>>>> Yes. That is required for Pakfire to find a new version. >>>>>>>>>>>>>>>>>> I don't know what to do for the core programs where I have= updated the rootfile. The lfs files for these programs don't have a PAK_VER = number to increment and the actual package version number is not being change= d so I don't know what I need to change in the lfs to ensure that the rootfil= e changes will be properly included in the update? >>>>>>>>>>>>>>>>> I would say committing everything into a =E2=80=9Cpython-3.= 10=E2=80=9D branch and submitting it all as a patchset is fine. All packages = that have been touched in that branch will have to be shipped again which sho= uld be easy to identify if it all comes in one large branch. >>>>>>>>>>>>>>>> Okay, that is clear then.Will follow that approach. >>>>>>>>>>>>>>>>>> I would appreciate any help on what I need to do for these= programs that have had changes only in their rootfiles. >>>>>>>>>>>>>>>>> Absolutely. Did you find anything that didn=E2=80=99t build= yet? >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> I have not had any problem with python3.10 itself but I foun= d some of the older modules would not work with 3.10 so I decided to do a com= bined update to 3.10 plus update modules to latest versions. >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> I found that after python3-setuptools-scm had run that the n= ext module python3-six failed to run due to missing the tomli library. I comm= ented out six and reran the build and then python3-dateutil had the same prob= lem and then python3-jmespath >>>>>>>>>>>>>>> I completely lost track of what the status of setuptools/dist= utils and so on is. This has changed multiple times and become independent/pa= rt of the distribution again and so on. >>>>>>>>>>>>>>>> After some investigation I have found that since version 6.0= .0 of python3-setuptools-scm, tomli has been a required library when it is be= ing used by other modules. So running version 6.0.0 of setuptools-scm worked = with python3.10 and subsequent modules built successfully as tomli not needed. >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> So the obvious fix is to install tomli but that is when I ra= n into a problem. tomli has no setup.py or setup.cfg files in its tarball. Th= e only way to build it is using pip. >>>>>>>>>>>>>>> Err. Great. >>>>>>>>>>>>>>>> Apparently, after some searching, setup.py has been deprecat= ed as the recommended install method and if setup.py doesn't work the usual f= ix suggestion is to run pip install. >>>>>>>>>>>>>>> Yes. But apparently the new system is deprecated, too. I real= ly dislike the state of packaging systems that come with any language these d= ays. >>>>>>>>>>>>>>>> Having searched on pip for a while, it looks like builds usi= ng it can still be done from tarballs locally without needing to access the i= nternet to download the modules. >>>>>>>>>>>>>>> That is good! >>>>>>>>>>>>>>>> As pip is installed as part of python3 i tried using that. >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> I used "python3 -m pip install --root=3D/ ." in place of >>>>>>>>>>>>>>>> "python3 setup.py build >>>>>>>>>>>>>>>> python3 setup.py install --root=3D/" >>>>>>>>>>>>>>>> for the tomli source. >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> This failed because tomli requires flit_core to be available. >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> flit_core also has no setup.py and so I tried to build it wi= th pip install. This failed with the error message that tomli was required as= a dependency. So I added --no-deps to the flit_core pip install line and thi= s time flit_core successfully built but the tomli build still came back with = not finding flit_core. >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> I have tried a variety of changes to the pip install line bu= t none of them has been successful and it has become clear that I am out of m= y depth here and need guidance rather than just trying anything I find from a= google search. I am not even really clear if pip install is the correct appr= oach, except if not then I have no idea how to progress. >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> Help urgently needed on how I should be approaching this. >>>>>>>>>>>>>>> Circular dependencies can happen. This won=E2=80=99t be the f= irst time and it always is an utter nightmare with our buildsystem. >>>>>>>>>>>>>>> This page states that pip comes by default with all binary pa= ckages: >>>>>>>>>>>>>>> https://docs.python.org/3/installing/index.html >>>>>>>>>>>>>>> Is there a chance that it is part of the distribution and can= be enabled with a configure switch? >>>>>>>>>>>>>> I wasn't able to find any way to get it enabled that way. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> I did find that pip install will not install if the package is= already present. You have to add --upgrade to make it update any package alr= eady present. So I tried that. Didn't work. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> I found a section about needing to use bootstrapping if you wa= nt to install tomli. Followed the information for creating a wheel file for f= lit_core and then extracting that file into the site-packages directory. Trie= d that. Didn't work. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> Rather than doing a complete build every time I changed to try= ing to build flit_core followed by tomli in a ./make.sh shell environment. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> Doing that I found that I had a pip install command that was c= reating the flit_core directory in site-packages and also including the .dist= -info directory. The info I found said that the .dist-info directory had to b= e there for pip to be able to find the flit_core package. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> However the pip install of tomli still ends up not finding any= flit_core package. In the bootstrapping information it mentioned about chang= ing the PYTHONPATH environment variable for find the flit_core package so I t= ried that in the shell but it also made no difference. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> I then tried running python3 -c 'import flit_core' in the toml= i build shell before doing the tomli pip install. The import command complete= d with no error but tomli still could not find any flit_core package. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> End result of all my work is that flit_core has been built and= installed into python3.10/site-packages but nothing seems to want to make th= e tomli build find it. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> Does anyone have any ideas on what command(s) I need to use in= the tomli pip install to have the installed flit_core package visible. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> Regards and a Happy and Healthy New Year to all, >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> Adolf. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> -Michael >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>> Adolf. >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> PS: I hope everyone had an enjoyable Christmas. >>>>>>>>>>>>>>> I hope you did, too! >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>> -Michael >>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>> Adolf. >>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>> --=20 >>>>>>>>>>>>>>>>>> Sent from my laptop >>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> --=20 >>>>>>>>>>>>>>>> Sent from my laptop >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> --=20 >>>>>>>>>>>>>> Sent from my laptop >>>>>>>>>>=20 >>>>>>>>>> --=20 >>>>>>>>>> Sent from my laptop >>>>>>>>=20 >>>>>>>> --=20 >>>>>>>> Sent from my laptop >>>>>>=20 >>>>>> --=20 >>>>>> Sent from my laptop --===============0964729646696780966==--