From mboxrd@z Thu Jan 1 00:00:00 1970 From: Adolf Belka To: development@lists.ipfire.org Subject: Re: Question about updating python to 3.10 Date: Tue, 18 Jan 2022 17:42:22 +0100 Message-ID: In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============8015118941701990162==" List-Id: --===============8015118941701990162== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi Michael, On 18/01/2022 16:02, Michael Tremer wrote: > Hello Adolf, >=20 > I have just been trying to build a couple of Python packages that I need an= d I finally arrived at needing flit-core. >=20 > I suppose since you did already build this, there is no point in me doing a= ll this again and also there is no need to build all my packages on Python 3.= 8 when we can have a newer release. I am afraid that you will need to build flit-core. I was building it when I = was trying to use pip to install tomli but that approached failed to go furth= er and that was when I used pyproject2setuppy, from your input, to build toml= i and that did not need flit-core. In terms of the python3.10 branch that I have on my git repository at home, I= presume I could push it onto my personal git repository that you set up for = me - https://git.ipfire.org/?p=3Dpeople/bonnietwin/ipfire-2.x.git;a=3Dsummary The only thing is I am not sure what command I should use to push the branch = from my system here to the IPFire one. I can have a go at trying it and see how it goes. Regards, Adolf. >=20 > Could you push this branch somewhere for me so that I can base my work on t= op of it? >=20 > Best, > -Michael >=20 >> On 13 Jan 2022, at 09:32, Michael Tremer wro= te: >> >> Hello, >> >>> On 12 Jan 2022, at 21:08, Adolf Belka wrote: >>> >>> Hi all, >>> >>> On 12/01/2022 19:10, Michael Tremer wrote: >>>> Hey, >>>>> On 11 Jan 2022, at 14:32, Adolf Belka wrote: >>>>> >>>>> Hi Michael and all, >>>>> >>>>> python3-six has to stay as python3-dateutil explicitly specifies it in = its dependencies. >>>>> So that solves that question. >>>> Aww :( >>>>> I have, with quite a bit of work, eventually been able to get all exist= ing python modules updated that had updates and built successfully with pytho= n3.10 >>>> Yay! >>>>> However later in the IPFire build libvirt failed with a problem parsing= some documentation files. >>>>> Stashing my python3.10 changes and re-running the build had libvirt bui= lding without problems. So one of the python changes has caused it. libvirt u= ses python3-docutils and that was updated but reverting that back to original= on its own did not cause libvirt to build successfully. >>>>> >>>>> I looked in the libvirt .configure for an option to not build the docs = but it is not mentioned under ./configure --help. >>>>> >>>>> I tried --without-docs and --disable-docs but both were not recognised = in the build. >>>>> >>>>> The current libvirt is from mid 2020 (6.5.0) so it is likely that it is= not compatible in some way with the updated python/modules. >>>>> >>>>> I tried updating libvirt to the current 7.10.0 >>>>> libvirt has replaced autotools with meson. I found meson options availa= ble for all existing ./configure options except three. There is nothing in me= son options for:- >>>>> --with-virtualport >>>>> --with-macvtap >>>>> --without-dbus >>>> Yay, another project that is using meson :( >>>> We no longer need macvtap; we have dubs, so that should not hurt; and I = have no idea what virtualport is. >>>>> I tried building it with all the other options and it successfully buil= t with the updated python3.10 and modules. All other packages after libvirt b= uilt successfully without any problems. >>>>> >>>>> Normally I would then use the updated libvirt, as that is needed to wor= k with the updated python packages, but I am not sure if those missing option= s are now built into libvirt or are no longer available and what impact that = might have for the use of libvirt on IPFire. >>>> I don=E2=80=99t think this would hurt us much. I have CCed Jonatan who m= aintains this, so maybe he knows more about this. >>>>> There is nothing in the libvirt changelog about virtualport, macvtap or= dbus. >>>>> >>>>> Searching on the internet, I was also not able to find anything related= to libvirt and these options. >>>>> >>>>> Any suggestions/proposals on how to move forward on this? Should I just= build libvirt without those three options or ...? >>>> Yes, and I think this could be submitted independently of the Python bra= nch because it is an add-on. We should be able to ship this at any time and m= ight want to delay Python until we have enough space in a core update. >>> Okay, then I will stash my python3.10 changes and submit the libvirt upda= te on its own. Then Jonaton can test it out. I don't use libvirt or qemu so I= can't test it easily. >>> >>> Then when you think the time is right, let me know and I will submit the = python3.10 series of patches. >> >> You can submit this at any time. I was just thinking about the conversatio= n between Peter and Arne during the last call regarding an updated kernel and= updated firmware. Depending on the size of Python, all of this might make an= update a little bit too large - however, this should not be a patchset that = we cannot merge within two merge windows. >> >>>> Thank you for looking into this. Sounds like a not so nice riddle to sol= ve :) >>> There were certainly some challenges, like updating python3-daemon which = then wanted to have twine as a fixed dependency which had nine dependencies s= ome of which had more. I had got to 19 additional python packages and still g= oing and then I said enough was enough and looked at twine. It is required fo= r uploading python packages to PyPI which IPFire doesn't need. So I tried pat= ching the twine requirement out and that worked and then I found others that = had done the same because they had no response to the request to not have twi= ne as a mandatory build requirement. >> >> Python seems to become the hell that Perl used to be. In order to add one = perl package, you had 20 dependencies. And those dependencies had their own d= ependencies too=E2=80=A6 It would be great to disable or patch out whatever i= s possible so that we do not create more work for ourselves in the future and= have to update packages that are barely used - we already have enough of the= se :) >> >>> I found it all challenging but rewarding when I made progress and eventua= lly succeeded in building everything without errors. >> >> I am so happy that you are putting all this time and effort into this beca= use it certainly requires patience - a trade I don=E2=80=99t have. >> >> And Python is being used in all sorts of places in the distribution, so it= really isn=E2=80=99t anything we should not be up to date on. >> >> -Michael >> >>> >>> Regards, >>> Adolf. >>>> -Michael >>>>> >>>>> >>>>> Regards, >>>>> >>>>> Adolf. >>>>> >>>>> On 07/01/2022 17:49, Michael Tremer wrote: >>>>>> Hello, >>>>>>> On 4 Jan 2022, at 12:49, Adolf Belka wrote: >>>>>>> >>>>>>> Hi Michael, >>>>>>> >>>>>>> Following the suggested Arch Linux approach I have got python3-setupt= ools-scm building successfully now. Yaaah. >>>>>>> I had to install python3-toml, followed by python3-pyproject2setuppy = followed by python3-tomli. >>>>>>> >>>>>>> However then python3-six which installs after python3-setuptools-scm = fails due to missing the python package python3-packaging. >>>>>>> >>>>>>> I can install that package, and will do so as I continue testing out = the python 3.10 build but I have the following question about python3-six. >>>>>>> >>>>>>> Is six still needed. It is designed to ensure that programs can work = with either python2 or python3 but IPFire only has python3 now. >>>>>>> I can find python3-six referenced only by aws-cli. In the aws-cli bui= ld python3-six is not listed at all in the build requirements. I tried to sea= rch for aws-cli and python3-six being needed in run mode but I could not find= anything one way or the other. >>>>>> If aws-cli no longer requires python3-six we can drop it. Last time we= touched it I had to bring the package back because aws-cli was still dependi= ng on it. >>>>>>> If python3-six is really needed for running aws-cli then I can instal= l it and the new dependency but if it is not needed we could remove two packa= ges which would partially offset the three new dependencies required to be ab= le to build the latest python3-setuptools-scm. >>>>>> Yes, that would be great :) >>>>>> -Michael >>>>>>> >>>>>>> Regards, >>>>>>> Adolf >>>>>>> >>>>>>> On 03/01/2022 12:32, Michael Tremer wrote: >>>>>>>> Hello, >>>>>>>>> On 3 Jan 2022, at 11:31, Adolf Belka wro= te: >>>>>>>>> >>>>>>>>> Hi Michael, >>>>>>>>> >>>>>>>>> On 03/01/2022 11:52, Michael Tremer wrote: >>>>>>>>>> Hello, >>>>>>>>>> I am not sure if you already looked at what Arch is doing, but as = far as I understand it works like this: >>>>>>>>> No I didn't look at Arch, which is surprising as I have in the past= with other packages but I didn't. So thanks very much for the pointer. >>>>>>>>>> They build Python 3 without pip and setuptools (https://github.com= /archlinux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/tr= unk/PKGBUILD#L59). >>>>>>>>>> At this stage we might need to build setuptools separately. >>>>>>>>>> They use a tool called pyproject2setuppy (great name!) to build to= mli (https://github.com/archlinux/svntogit-packages/blob/packages/python-toml= i/trunk/PKGBUILD). >>>>>>>>>> Setuptools-scm can then be built with tomli: https://github.com/ar= chlinux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD >>>>>>>>>> Would this work or did I overlook anything? >>>>>>>>> I will have a look at the links and give it a go. >>>>>>>>> >>>>>>>>> Thanks very much, I feel re-motivated now. >>>>>>>> Very good! I got a little bit lost in your last email and I think I = could tell. But following a solution that somebody else has come up with can = be a good thing to follow and of course we would customise it to our needs. >>>>>>>> Best, >>>>>>>> -Michael >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Adolf. >>>>>>>>>> -Michael >>>>>>>>>>> On 2 Jan 2022, at 13:44, Adolf Belka w= rote: >>>>>>>>>>> >>>>>>>>>>> Hi Michael and All, >>>>>>>>>>> >>>>>>>>>>> On 26/12/2021 21:55, Michael Tremer wrote: >>>>>>>>>>>> Merry Christmas, >>>>>>>>>>>>> On 26 Dec 2021, at 15:38, Adolf Belka wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Michael, >>>>>>>>>>>>> >>>>>>>>>>>>> On 17/12/2021 12:41, Michael Tremer wrote: >>>>>>>>>>>>>> Hello Adolf, >>>>>>>>>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I am working on updating python from 3.8 to 3.10. Python itse= lf is okay after some other python related programs were also updated due to = their older versions not working with 3.10 >>>>>>>>>>>>>> Thank you for working on this. >>>>>>>>>>>>>>> I am working through all the programs that reference python-3= .8 in their rootfiles, or wherever, to change them to 3.10 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> My question comes with regard to the changes to the rootfiles= to ensure they will be included properly in an update. >>>>>>>>>>>>>> We will have to ship everything that puts anything into /usr/l= ib*/python3.*/site-packages again with the release of Python 3.10. >>>>>>>>>>>>>> This isn=E2=80=99t fun, but not as much of a pain in the rear = as shipping a new Perl release. >>>>>>>>>>>>>>> For the addons then when I change the rootfile I am also incr= ementing the PAK_VER number in the lfs files and that ensures that they will = be properly updated. >>>>>>>>>>>>>> Yes. That is required for Pakfire to find a new version. >>>>>>>>>>>>>>> I don't know what to do for the core programs where I have up= dated the rootfile. The lfs files for these programs don't have a PAK_VER num= ber to increment and the actual package version number is not being changed s= o I don't know what I need to change in the lfs to ensure that the rootfile c= hanges will be properly included in the update? >>>>>>>>>>>>>> I would say committing everything into a =E2=80=9Cpython-3.10= =E2=80=9D branch and submitting it all as a patchset is fine. All packages th= at have been touched in that branch will have to be shipped again which shoul= d be easy to identify if it all comes in one large branch. >>>>>>>>>>>>> Okay, that is clear then.Will follow that approach. >>>>>>>>>>>>>>> I would appreciate any help on what I need to do for these pr= ograms that have had changes only in their rootfiles. >>>>>>>>>>>>>> Absolutely. Did you find anything that didn=E2=80=99t build ye= t? >>>>>>>>>>>>> >>>>>>>>>>>>> I have not had any problem with python3.10 itself but I found s= ome of the older modules would not work with 3.10 so I decided to do a combin= ed update to 3.10 plus update modules to latest versions. >>>>>>>>>>>>> >>>>>>>>>>>>> I found that after python3-setuptools-scm had run that the next= module python3-six failed to run due to missing the tomli library. I comment= ed out six and reran the build and then python3-dateutil had the same problem= and then python3-jmespath >>>>>>>>>>>> I completely lost track of what the status of setuptools/distuti= ls and so on is. This has changed multiple times and become independent/part = of the distribution again and so on. >>>>>>>>>>>>> After some investigation I have found that since version 6.0.0 = of python3-setuptools-scm, tomli has been a required library when it is being= used by other modules. So running version 6.0.0 of setuptools-scm worked wit= h python3.10 and subsequent modules built successfully as tomli not needed. >>>>>>>>>>>>> >>>>>>>>>>>>> So the obvious fix is to install tomli but that is when I ran i= nto a problem. tomli has no setup.py or setup.cfg files in its tarball. The o= nly way to build it is using pip. >>>>>>>>>>>> Err. Great. >>>>>>>>>>>>> Apparently, after some searching, setup.py has been deprecated = as the recommended install method and if setup.py doesn't work the usual fix = suggestion is to run pip install. >>>>>>>>>>>> Yes. But apparently the new system is deprecated, too. I really = dislike the state of packaging systems that come with any language these days. >>>>>>>>>>>>> Having searched on pip for a while, it looks like builds using = it can still be done from tarballs locally without needing to access the inte= rnet to download the modules. >>>>>>>>>>>> That is good! >>>>>>>>>>>>> As pip is installed as part of python3 i tried using that. >>>>>>>>>>>>> >>>>>>>>>>>>> I used "python3 -m pip install --root=3D/ ." in place of >>>>>>>>>>>>> "python3 setup.py build >>>>>>>>>>>>> python3 setup.py install --root=3D/" >>>>>>>>>>>>> for the tomli source. >>>>>>>>>>>>> >>>>>>>>>>>>> This failed because tomli requires flit_core to be available. >>>>>>>>>>>>> >>>>>>>>>>>>> flit_core also has no setup.py and so I tried to build it with = pip install. This failed with the error message that tomli was required as a = dependency. So I added --no-deps to the flit_core pip install line and this t= ime flit_core successfully built but the tomli build still came back with not= finding flit_core. >>>>>>>>>>>>> >>>>>>>>>>>>> I have tried a variety of changes to the pip install line but n= one of them has been successful and it has become clear that I am out of my d= epth here and need guidance rather than just trying anything I find from a go= ogle search. I am not even really clear if pip install is the correct approac= h, except if not then I have no idea how to progress. >>>>>>>>>>>>> >>>>>>>>>>>>> Help urgently needed on how I should be approaching this. >>>>>>>>>>>> Circular dependencies can happen. This won=E2=80=99t be the firs= t time and it always is an utter nightmare with our buildsystem. >>>>>>>>>>>> This page states that pip comes by default with all binary packa= ges: >>>>>>>>>>>> https://docs.python.org/3/installing/index.html >>>>>>>>>>>> Is there a chance that it is part of the distribution and can be= enabled with a configure switch? >>>>>>>>>>> I wasn't able to find any way to get it enabled that way. >>>>>>>>>>> >>>>>>>>>>> I did find that pip install will not install if the package is al= ready present. You have to add --upgrade to make it update any package alread= y present. So I tried that. Didn't work. >>>>>>>>>>> >>>>>>>>>>> I found a section about needing to use bootstrapping if you want = to install tomli. Followed the information for creating a wheel file for flit= _core and then extracting that file into the site-packages directory. Tried t= hat. Didn't work. >>>>>>>>>>> >>>>>>>>>>> Rather than doing a complete build every time I changed to trying= to build flit_core followed by tomli in a ./make.sh shell environment. >>>>>>>>>>> >>>>>>>>>>> Doing that I found that I had a pip install command that was crea= ting the flit_core directory in site-packages and also including the .dist-in= fo directory. The info I found said that the .dist-info directory had to be t= here for pip to be able to find the flit_core package. >>>>>>>>>>> >>>>>>>>>>> However the pip install of tomli still ends up not finding any fl= it_core package. In the bootstrapping information it mentioned about changing= the PYTHONPATH environment variable for find the flit_core package so I trie= d that in the shell but it also made no difference. >>>>>>>>>>> >>>>>>>>>>> I then tried running python3 -c 'import flit_core' in the tomli b= uild shell before doing the tomli pip install. The import command completed w= ith no error but tomli still could not find any flit_core package. >>>>>>>>>>> >>>>>>>>>>> End result of all my work is that flit_core has been built and in= stalled into python3.10/site-packages but nothing seems to want to make the t= omli build find it. >>>>>>>>>>> >>>>>>>>>>> Does anyone have any ideas on what command(s) I need to use in th= e tomli pip install to have the installed flit_core package visible. >>>>>>>>>>> >>>>>>>>>>> Regards and a Happy and Healthy New Year to all, >>>>>>>>>>> >>>>>>>>>>> Adolf. >>>>>>>>>>> >>>>>>>>>>>> -Michael >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Adolf. >>>>>>>>>>>>> >>>>>>>>>>>>> PS: I hope everyone had an enjoyable Christmas. >>>>>>>>>>>> I hope you did, too! >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> -Michael >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Adolf. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> --=20 >>>>>>>>>>>>>>> Sent from my laptop >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> --=20 >>>>>>>>>>>>> Sent from my laptop >>>>>>>>>>> >>>>>>>>>>> --=20 >>>>>>>>>>> Sent from my laptop >>>>>>> >>>>>>> --=20 >>>>>>> Sent from my laptop >>>>> >>>>> --=20 >>>>> Sent from my laptop >>> >>> --=20 >>> Sent from my laptop >=20 --===============8015118941701990162==--