From mboxrd@z Thu Jan 1 00:00:00 1970 From: Adolf Belka To: development@lists.ipfire.org Subject: Re: Question about updating python to 3.10 Date: Wed, 19 Jan 2022 20:52:04 +0100 Message-ID: In-Reply-To: <9c34ab1c-5ded-72ce-5b49-a40818256541@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0293345752692458793==" List-Id: --===============0293345752692458793== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi Michael, There will be a bit more time before I have the python3.10 branch available. I made a mistake with one of the commits and didn't notice till some while af= ter. So I did a git reset --soft "commit hash" which I have been successful w= ith in the past but this time something went wrong and messed things up. I have had to re delete my local python3.10 branch with the commits done so f= ar and am now doing a full build again to make sure that my restart doesn't h= ave any errors in it. I will be able to restart doing the commits again tomorrow so probably branch= will be available on Friday. Sorry. Regards, Adolf. On 19/01/2022 13:33, Adolf Belka wrote: > Hi Michael, > > On 19/01/2022 09:25, Michael Tremer wrote: >> Hello, >> >> I am sure this was documented in the wiki somewhere. For some reason I can= not find it any more. >> >> Please let me know if you need any assistance from me. > I got it to work thanks very much. There is now a python3.10 branch in my b= onnietwin repository but it only has a few of the changes so far but I wanted= to confirm that it worked. > > I need to finish doing the remaining commits and then push them. Hopefully = I will be able to complete those later today or tomorrow. > > I will email when I have completed it and the full branch of my python3.10 = changes is pushed into my repository. > > Regards, > > Adolf. >> >> -Michael >> >>> On 18 Jan 2022, at 21:21, Adolf Belka wrote: >>> >>> Hi Michael, >>> >>> On 18/01/2022 19:51, Michael Tremer wrote: >>>> Hello, >>>>> On 18 Jan 2022, at 16:42, Adolf Belka wrote: >>>>> >>>>> Hi Michael, >>>>> >>>>> On 18/01/2022 16:02, Michael Tremer wrote: >>>>>> Hello Adolf, >>>>>> I have just been trying to build a couple of Python packages that I ne= ed and I finally arrived at needing flit-core. >>>>>> I suppose since you did already build this, there is no point in me do= ing all this again and also there is no need to build all my packages on Pyth= on 3.8 when we can have a newer release. >>>>> I am afraid that you will need to build flit-core.=C2=A0 I was building= it when I was trying to use pip to install tomli but that approached failed = to go further and that was when I used pyproject2setuppy, from your input, to= build tomli and that did not need flit-core. >>>> Oh, I suppose that didn=E2=80=99t hold for very long because I now have = a package that needs it :( >>> flit-core also has no setup.py but you will then be able to use the pypro= ject2setuppy package from my python3.10 branch to build flit-core. >>>>> In terms of the python3.10 branch that I have on my git repository at h= ome, I presume I could push it onto my personal git repository that you set u= p for me - https://git.ipfire.org/?p=3Dpeople/bonnietwin/ipfire-2.x.git;a=3Ds= ummary >>>> Yes, it is always a good idea to backup things to the server. >>>>> The only thing is I am not sure what command I should use to push the b= ranch from my system here to the IPFire one. >>>> If you have your repository set up with =E2=80=9Cgit remote add =E2=80= =A6=E2=80=9D then you only need to run =E2=80=9Cgit push bonnietwin python3.1= 0=E2=80=9D. >>> Aaah. I think when I tried to experiment with this before I created a loc= al new clone of bonnietwin which was then separate from my running local repo= sitory that I used for building and I struggled to figure out how to get bran= ches between them. >>> >>> Your mention of git remote add indicates that I can add my bonnietwin rep= ository to my running build repository. Thais is what I was missing before. >>> Thanks for the clue, will give it a try. >>> >>> Regards, >>> >>> Adolf. >>>> -Michael >>>>> >>>>> I can have a go at trying it and see how it goes. >>>>> >>>>> Regards, >>>>> >>>>> Adolf. >>>>>> Could you push this branch somewhere for me so that I can base my work= on top of it? >>>>>> Best, >>>>>> -Michael >>>>>>> On 13 Jan 2022, at 09:32, Michael Tremer wrote: >>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>>> On 12 Jan 2022, at 21:08, Adolf Belka wro= te: >>>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> On 12/01/2022 19:10, Michael Tremer wrote: >>>>>>>>> Hey, >>>>>>>>>> On 11 Jan 2022, at 14:32, Adolf Belka w= rote: >>>>>>>>>> >>>>>>>>>> Hi Michael and all, >>>>>>>>>> >>>>>>>>>> python3-six has to stay as python3-dateutil explicitly specifies i= t in its dependencies. >>>>>>>>>> So that solves that question. >>>>>>>>> Aww :( >>>>>>>>>> I have, with quite a bit of work, eventually been able to get all = existing python modules updated that had updates and built successfully with = python3.10 >>>>>>>>> Yay! >>>>>>>>>> However later in the IPFire build libvirt failed with a problem pa= rsing some documentation files. >>>>>>>>>> Stashing my python3.10 changes and re-running the build had libvir= t building without problems. So one of the python changes has caused it. libv= irt uses python3-docutils and that was updated but reverting that back to ori= ginal on its own did not cause libvirt to build successfully. >>>>>>>>>> >>>>>>>>>> I looked in the libvirt .configure for an option to not build the = docs but it is not mentioned under ./configure --help. >>>>>>>>>> >>>>>>>>>> I tried --without-docs and --disable-docs but both were not recogn= ised in the build. >>>>>>>>>> >>>>>>>>>> The current libvirt is from mid 2020 (6.5.0) so it is likely that = it is not compatible in some way with the updated python/modules. >>>>>>>>>> >>>>>>>>>> I tried updating libvirt to the current 7.10.0 >>>>>>>>>> libvirt has replaced autotools with meson. I found meson options a= vailable for all existing ./configure options except three. There is nothing = in meson options for:- >>>>>>>>>> --with-virtualport >>>>>>>>>> --with-macvtap >>>>>>>>>> --without-dbus >>>>>>>>> Yay, another project that is using meson :( >>>>>>>>> We no longer need macvtap; we have dubs, so that should not hurt; a= nd I have no idea what virtualport is. >>>>>>>>>> I tried building it with all the other options and it successfully= built with the updated python3.10 and modules. All other packages after libv= irt built successfully without any problems. >>>>>>>>>> >>>>>>>>>> Normally I would then use the updated libvirt, as that is needed t= o work with the updated python packages, but I am not sure if those missing o= ptions are now built into libvirt or are no longer available and what impact = that might have for the use of libvirt on IPFire. >>>>>>>>> I don=E2=80=99t think this would hurt us much. I have CCed Jonatan = who maintains this, so maybe he knows more about this. >>>>>>>>>> There is nothing in the libvirt changelog about virtualport, macvt= ap or dbus. >>>>>>>>>> >>>>>>>>>> Searching on the internet, I was also not able to find anything re= lated to libvirt and these options. >>>>>>>>>> >>>>>>>>>> Any suggestions/proposals on how to move forward on this? Should I= just build libvirt without those three options or ...? >>>>>>>>> Yes, and I think this could be submitted independently of the Pytho= n branch because it is an add-on. We should be able to ship this at any time = and might want to delay Python until we have enough space in a core update. >>>>>>>> Okay, then I will stash my python3.10 changes and submit the libvirt= update on its own. Then Jonaton can test it out. I don't use libvirt or qemu= so I can't test it easily. >>>>>>>> >>>>>>>> Then when you think the time is right, let me know and I will submit= the python3.10 series of patches. >>>>>>> >>>>>>> You can submit this at any time. I was just thinking about the conver= sation between Peter and Arne during the last call regarding an updated kerne= l and updated firmware. Depending on the size of Python, all of this might ma= ke an update a little bit too large - however, this should not be a patchset = that we cannot merge within two merge windows. >>>>>>> >>>>>>>>> Thank you for looking into this. Sounds like a not so nice riddle t= o solve :) >>>>>>>> There were certainly some challenges, like updating python3-daemon w= hich then wanted to have twine as a fixed dependency which had nine dependenc= ies some of which had more. I had got to 19 additional python packages and st= ill going and then I said enough was enough and looked at twine. It is requir= ed for uploading python packages to PyPI which IPFire doesn't need. So I trie= d patching the twine requirement out and that worked and then I found others = that had done the same because they had no response to the request to not hav= e twine as a mandatory build requirement. >>>>>>> >>>>>>> Python seems to become the hell that Perl used to be. In order to add= one perl package, you had 20 dependencies. And those dependencies had their = own dependencies too=E2=80=A6 It would be great to disable or patch out whate= ver is possible so that we do not create more work for ourselves in the futur= e and have to update packages that are barely used - we already have enough o= f these :) >>>>>>> >>>>>>>> I found it all challenging but rewarding when I made progress and ev= entually succeeded in building everything without errors. >>>>>>> >>>>>>> I am so happy that you are putting all this time and effort into this= because it certainly requires patience - a trade I don=E2=80=99t have. >>>>>>> >>>>>>> And Python is being used in all sorts of places in the distribution, = so it really isn=E2=80=99t anything we should not be up to date on. >>>>>>> >>>>>>> -Michael >>>>>>> >>>>>>>> >>>>>>>> Regards, >>>>>>>> Adolf. >>>>>>>>> -Michael >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> >>>>>>>>>> Adolf. >>>>>>>>>> >>>>>>>>>> On 07/01/2022 17:49, Michael Tremer wrote: >>>>>>>>>>> Hello, >>>>>>>>>>>> On 4 Jan 2022, at 12:49, Adolf Belka = wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi Michael, >>>>>>>>>>>> >>>>>>>>>>>> Following the suggested Arch Linux approach I have got python3-s= etuptools-scm building successfully now. Yaaah. >>>>>>>>>>>> I had to install python3-toml, followed by python3-pyproject2set= uppy followed by python3-tomli. >>>>>>>>>>>> >>>>>>>>>>>> However then python3-six which installs after python3-setuptools= -scm fails due to missing the python package python3-packaging. >>>>>>>>>>>> >>>>>>>>>>>> I can install that package, and will do so as I continue testing= out the python 3.10 build but I have the following question about python3-si= x. >>>>>>>>>>>> >>>>>>>>>>>> Is six still needed. It is designed to ensure that programs can = work with either python2 or python3 but IPFire only has python3 now. >>>>>>>>>>>> I can find python3-six referenced only by aws-cli. In the aws-cl= i build python3-six is not listed at all in the build requirements. I tried t= o search for aws-cli and python3-six being needed in run mode but I could not= find anything one way or the other. >>>>>>>>>>> If aws-cli no longer requires python3-six we can drop it. Last ti= me we touched it I had to bring the package back because aws-cli was still de= pending on it. >>>>>>>>>>>> If python3-six is really needed for running aws-cli then I can i= nstall it and the new dependency but if it is not needed we could remove two = packages which would partially offset the three new dependencies required to = be able to build the latest python3-setuptools-scm. >>>>>>>>>>> Yes, that would be great :) >>>>>>>>>>> -Michael >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Adolf >>>>>>>>>>>> >>>>>>>>>>>> On 03/01/2022 12:32, Michael Tremer wrote: >>>>>>>>>>>>> Hello, >>>>>>>>>>>>>> On 3 Jan 2022, at 11:31, Adolf Belka wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Michael, >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 03/01/2022 11:52, Michael Tremer wrote: >>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>> I am not sure if you already looked at what Arch is doing, bu= t as far as I understand it works like this: >>>>>>>>>>>>>> No I didn't look at Arch, which is surprising as I have in the= past with other packages but I didn't. So thanks very much for the pointer. >>>>>>>>>>>>>>> They build Python 3 without pip and setuptools (https://githu= b.com/archlinux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e7= 33/trunk/PKGBUILD#L59). >>>>>>>>>>>>>>> At this stage we might need to build setuptools separately. >>>>>>>>>>>>>>> They use a tool called pyproject2setuppy (great name!) to bui= ld tomli (https://github.com/archlinux/svntogit-packages/blob/packages/python= -tomli/trunk/PKGBUILD). >>>>>>>>>>>>>>> Setuptools-scm can then be built with tomli: https://github.c= om/archlinux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKG= BUILD >>>>>>>>>>>>>>> Would this work or did I overlook anything? >>>>>>>>>>>>>> I will have a look at the links and give it a go. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks very much, I feel re-motivated now. >>>>>>>>>>>>> Very good! I got a little bit lost in your last email and I thi= nk I could tell. But following a solution that somebody else has come up with= can be a good thing to follow and of course we would customise it to our nee= ds. >>>>>>>>>>>>> Best, >>>>>>>>>>>>> -Michael >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> Adolf. >>>>>>>>>>>>>>> -Michael >>>>>>>>>>>>>>>> On 2 Jan 2022, at 13:44, Adolf Belka wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Michael and All, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 26/12/2021 21:55, Michael Tremer wrote: >>>>>>>>>>>>>>>>> Merry Christmas, >>>>>>>>>>>>>>>>>> On 26 Dec 2021, at 15:38, Adolf Belka wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi Michael, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On 17/12/2021 12:41, Michael Tremer wrote: >>>>>>>>>>>>>>>>>>> Hello Adolf, >>>>>>>>>>>>>>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I am working on updating python from 3.8 to 3.10. Python= itself is okay after some other python related programs were also updated du= e to their older versions not working with 3.10 >>>>>>>>>>>>>>>>>>> Thank you for working on this. >>>>>>>>>>>>>>>>>>>> I am working through all the programs that reference pyt= hon-3.8 in their rootfiles, or wherever, to change them to 3.10 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> My question comes with regard to the changes to the root= files to ensure they will be included properly in an update. >>>>>>>>>>>>>>>>>>> We will have to ship everything that puts anything into /= usr/lib*/python3.*/site-packages again with the release of Python 3.10. >>>>>>>>>>>>>>>>>>> This isn=E2=80=99t fun, but not as much of a pain in the = rear as shipping a new Perl release. >>>>>>>>>>>>>>>>>>>> For the addons then when I change the rootfile I am also= incrementing the PAK_VER number in the lfs files and that ensures that they = will be properly updated. >>>>>>>>>>>>>>>>>>> Yes. That is required for Pakfire to find a new version. >>>>>>>>>>>>>>>>>>>> I don't know what to do for the core programs where I ha= ve updated the rootfile. The lfs files for these programs don't have a PAK_VE= R number to increment and the actual package version number is not being chan= ged so I don't know what I need to change in the lfs to ensure that the rootf= ile changes will be properly included in the update? >>>>>>>>>>>>>>>>>>> I would say committing everything into a =E2=80=9Cpython-= 3.10=E2=80=9D branch and submitting it all as a patchset is fine. All package= s that have been touched in that branch will have to be shipped again which s= hould be easy to identify if it all comes in one large branch. >>>>>>>>>>>>>>>>>> Okay, that is clear then.Will follow that approach. >>>>>>>>>>>>>>>>>>>> I would appreciate any help on what I need to do for the= se programs that have had changes only in their rootfiles. >>>>>>>>>>>>>>>>>>> Absolutely. Did you find anything that didn=E2=80=99t bui= ld yet? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I have not had any problem with python3.10 itself but I fo= und some of the older modules would not work with 3.10 so I decided to do a c= ombined update to 3.10 plus update modules to latest versions. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I found that after python3-setuptools-scm had run that the= next module python3-six failed to run due to missing the tomli library. I co= mmented out six and reran the build and then python3-dateutil had the same pr= oblem=C2=A0 and then python3-jmespath >>>>>>>>>>>>>>>>> I completely lost track of what the status of setuptools/di= stutils and so on is. This has changed multiple times and become independent/= part of the distribution again and so on. >>>>>>>>>>>>>>>>>> After some investigation I have found that since version 6= .0.0 of python3-setuptools-scm, tomli has been a required library when it is = being used by other modules. So running version 6.0.0 of setuptools-scm worke= d with python3.10 and subsequent modules built successfully as tomli not need= ed. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> So the obvious fix is to install tomli but that is when I = ran into a problem. tomli has no setup.py or setup.cfg files in its tarball. = The only way to build it is using pip. >>>>>>>>>>>>>>>>> Err. Great. >>>>>>>>>>>>>>>>>> Apparently, after some searching, setup.py has been deprec= ated as the recommended install method and if setup.py doesn't work the usual= fix suggestion is to run pip install. >>>>>>>>>>>>>>>>> Yes. But apparently the new system is deprecated, too. I re= ally dislike the state of packaging systems that come with any language these= days. >>>>>>>>>>>>>>>>>> Having searched on pip for a while, it looks like builds u= sing it can still be done from tarballs locally without needing to access the= internet to download the modules. >>>>>>>>>>>>>>>>> That is good! >>>>>>>>>>>>>>>>>> As pip is installed as part of python3 i tried using that. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I used "python3 -m pip install --root=3D/ ."=C2=A0 in plac= e of >>>>>>>>>>>>>>>>>> "python3 setup.py build >>>>>>>>>>>>>>>>>> python3 setup.py install --root=3D/" >>>>>>>>>>>>>>>>>> for the tomli source. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> This failed because tomli requires flit_core to be availab= le. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> flit_core also has no setup.py and so I tried to build it = with pip install. This failed with the error message that tomli was required = as a dependency. So I added --no-deps to the flit_core pip install line and t= his time flit_core successfully built but the tomli build still came back wit= h not finding flit_core. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I have tried a variety of changes to the pip install line = but none of them has been successful and it has become clear that I am out of= my depth here and need guidance rather than just trying anything I find from= a google search. I am not even really clear if pip install is the correct ap= proach, except if not then I have no idea how to progress. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Help urgently needed on how I should be approaching this. >>>>>>>>>>>>>>>>> Circular dependencies can happen. This won=E2=80=99t be the= first time and it always is an utter nightmare with our buildsystem. >>>>>>>>>>>>>>>>> This page states that pip comes by default with all binary = packages: >>>>>>>>>>>>>>>>> https://docs.python.org/3/installing/index.html >>>>>>>>>>>>>>>>> Is there a chance that it is part of the distribution and c= an be enabled with a configure switch? >>>>>>>>>>>>>>>> I wasn't able to find any way to get it enabled that way. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I did find that pip install will not install if the package = is already present. You have to add --upgrade to make it update any package a= lready present. So I tried that. Didn't work. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I found a section about needing to use bootstrapping if you = want to install tomli. Followed the information for creating a wheel file for= flit_core and then extracting that file into the site-packages directory. Tr= ied that. Didn't work. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Rather than doing a complete build every time I changed to t= rying to build flit_core followed by tomli in a ./make.sh shell environment. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Doing that I found that I had a pip install command that was= creating the flit_core directory in site-packages and also including the .di= st-info directory. The info I found said that the .dist-info directory had to= be there for pip to be able to find the flit_core package. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> However the pip install of tomli still ends up not finding a= ny flit_core package. In the bootstrapping information it mentioned about cha= nging the PYTHONPATH environment variable for find the flit_core package so I= tried that in the shell but it also made no difference. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I then tried running python3 -c 'import flit_core' in the to= mli build shell before doing the tomli pip install. The import command comple= ted with no error but tomli still could not find any flit_core package. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> End result of all my work is that flit_core has been built a= nd installed into python3.10/site-packages but nothing seems to want to make = the tomli build find it. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Does anyone have any ideas on what command(s) I need to use = in the tomli pip install to have the installed flit_core package visible. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Regards and a Happy and Healthy New Year to all, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Adolf. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -Michael >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>> Adolf. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> PS:=C2=A0 I hope everyone had an enjoyable Christmas. >>>>>>>>>>>>>>>>> I hope you did, too! >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> -Michael >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Adolf. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> --=20 >>>>>>>>>>>>>>>>>>>> Sent from my laptop >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> --=20 >>>>>>>>>>>>>>>>>> Sent from my laptop >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> --=20 >>>>>>>>>>>>>>>> Sent from my laptop >>>>>>>>>>>> >>>>>>>>>>>> --=20 >>>>>>>>>>>> Sent from my laptop >>>>>>>>>> >>>>>>>>>> --=20 >>>>>>>>>> Sent from my laptop >>>>>>>> >>>>>>>> --=20 >>>>>>>> Sent from my laptop >> --===============0293345752692458793==--