From mboxrd@z Thu Jan 1 00:00:00 1970 From: Adolf Belka To: development@lists.ipfire.org Subject: Re: Question about updating python to 3.10 Date: Wed, 19 Jan 2022 13:33:24 +0100 Message-ID: <9c34ab1c-5ded-72ce-5b49-a40818256541@ipfire.org> In-Reply-To: <92927B9F-77B3-466B-A484-C9BE1D919E2B@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============3625694098376568189==" List-Id: --===============3625694098376568189== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi Michael, On 19/01/2022 09:25, Michael Tremer wrote: > Hello, >=20 > I am sure this was documented in the wiki somewhere. For some reason I cann= ot find it any more. >=20 > Please let me know if you need any assistance from me. I got it to work thanks very much. There is now a python3.10 branch in my bon= nietwin repository but it only has a few of the changes so far but I wanted t= o confirm that it worked. I need to finish doing the remaining commits and then push them. Hopefully I = will be able to complete those later today or tomorrow. I will email when I have completed it and the full branch of my python3.10 ch= anges is pushed into my repository. Regards, Adolf. >=20 > -Michael >=20 >> On 18 Jan 2022, at 21:21, Adolf Belka wrote: >> >> Hi Michael, >> >> On 18/01/2022 19:51, Michael Tremer wrote: >>> Hello, >>>> On 18 Jan 2022, at 16:42, Adolf Belka wrote: >>>> >>>> Hi Michael, >>>> >>>> On 18/01/2022 16:02, Michael Tremer wrote: >>>>> Hello Adolf, >>>>> I have just been trying to build a couple of Python packages that I nee= d and I finally arrived at needing flit-core. >>>>> I suppose since you did already build this, there is no point in me doi= ng all this again and also there is no need to build all my packages on Pytho= n 3.8 when we can have a newer release. >>>> I am afraid that you will need to build flit-core. I was building it wh= en I was trying to use pip to install tomli but that approached failed to go = further and that was when I used pyproject2setuppy, from your input, to build= tomli and that did not need flit-core. >>> Oh, I suppose that didn=E2=80=99t hold for very long because I now have a= package that needs it :( >> flit-core also has no setup.py but you will then be able to use the pyproj= ect2setuppy package from my python3.10 branch to build flit-core. >>>> In terms of the python3.10 branch that I have on my git repository at ho= me, I presume I could push it onto my personal git repository that you set up= for me - https://git.ipfire.org/?p=3Dpeople/bonnietwin/ipfire-2.x.git;a=3Dsu= mmary >>> Yes, it is always a good idea to backup things to the server. >>>> The only thing is I am not sure what command I should use to push the br= anch from my system here to the IPFire one. >>> If you have your repository set up with =E2=80=9Cgit remote add =E2=80=A6= =E2=80=9D then you only need to run =E2=80=9Cgit push bonnietwin python3.10= =E2=80=9D. >> Aaah. I think when I tried to experiment with this before I created a loca= l new clone of bonnietwin which was then separate from my running local repos= itory that I used for building and I struggled to figure out how to get branc= hes between them. >> >> Your mention of git remote add indicates that I can add my bonnietwin repo= sitory to my running build repository. Thais is what I was missing before. >> Thanks for the clue, will give it a try. >> >> Regards, >> >> Adolf. >>> -Michael >>>> >>>> I can have a go at trying it and see how it goes. >>>> >>>> Regards, >>>> >>>> Adolf. >>>>> Could you push this branch somewhere for me so that I can base my work = on top of it? >>>>> Best, >>>>> -Michael >>>>>> On 13 Jan 2022, at 09:32, Michael Tremer = wrote: >>>>>> >>>>>> Hello, >>>>>> >>>>>>> On 12 Jan 2022, at 21:08, Adolf Belka wrot= e: >>>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> On 12/01/2022 19:10, Michael Tremer wrote: >>>>>>>> Hey, >>>>>>>>> On 11 Jan 2022, at 14:32, Adolf Belka wr= ote: >>>>>>>>> >>>>>>>>> Hi Michael and all, >>>>>>>>> >>>>>>>>> python3-six has to stay as python3-dateutil explicitly specifies it= in its dependencies. >>>>>>>>> So that solves that question. >>>>>>>> Aww :( >>>>>>>>> I have, with quite a bit of work, eventually been able to get all e= xisting python modules updated that had updates and built successfully with p= ython3.10 >>>>>>>> Yay! >>>>>>>>> However later in the IPFire build libvirt failed with a problem par= sing some documentation files. >>>>>>>>> Stashing my python3.10 changes and re-running the build had libvirt= building without problems. So one of the python changes has caused it. libvi= rt uses python3-docutils and that was updated but reverting that back to orig= inal on its own did not cause libvirt to build successfully. >>>>>>>>> >>>>>>>>> I looked in the libvirt .configure for an option to not build the d= ocs but it is not mentioned under ./configure --help. >>>>>>>>> >>>>>>>>> I tried --without-docs and --disable-docs but both were not recogni= sed in the build. >>>>>>>>> >>>>>>>>> The current libvirt is from mid 2020 (6.5.0) so it is likely that i= t is not compatible in some way with the updated python/modules. >>>>>>>>> >>>>>>>>> I tried updating libvirt to the current 7.10.0 >>>>>>>>> libvirt has replaced autotools with meson. I found meson options av= ailable for all existing ./configure options except three. There is nothing i= n meson options for:- >>>>>>>>> --with-virtualport >>>>>>>>> --with-macvtap >>>>>>>>> --without-dbus >>>>>>>> Yay, another project that is using meson :( >>>>>>>> We no longer need macvtap; we have dubs, so that should not hurt; an= d I have no idea what virtualport is. >>>>>>>>> I tried building it with all the other options and it successfully = built with the updated python3.10 and modules. All other packages after libvi= rt built successfully without any problems. >>>>>>>>> >>>>>>>>> Normally I would then use the updated libvirt, as that is needed to= work with the updated python packages, but I am not sure if those missing op= tions are now built into libvirt or are no longer available and what impact t= hat might have for the use of libvirt on IPFire. >>>>>>>> I don=E2=80=99t think this would hurt us much. I have CCed Jonatan w= ho maintains this, so maybe he knows more about this. >>>>>>>>> There is nothing in the libvirt changelog about virtualport, macvta= p or dbus. >>>>>>>>> >>>>>>>>> Searching on the internet, I was also not able to find anything rel= ated to libvirt and these options. >>>>>>>>> >>>>>>>>> Any suggestions/proposals on how to move forward on this? Should I = just build libvirt without those three options or ...? >>>>>>>> Yes, and I think this could be submitted independently of the Python= branch because it is an add-on. We should be able to ship this at any time a= nd might want to delay Python until we have enough space in a core update. >>>>>>> Okay, then I will stash my python3.10 changes and submit the libvirt = update on its own. Then Jonaton can test it out. I don't use libvirt or qemu = so I can't test it easily. >>>>>>> >>>>>>> Then when you think the time is right, let me know and I will submit = the python3.10 series of patches. >>>>>> >>>>>> You can submit this at any time. I was just thinking about the convers= ation between Peter and Arne during the last call regarding an updated kernel= and updated firmware. Depending on the size of Python, all of this might mak= e an update a little bit too large - however, this should not be a patchset t= hat we cannot merge within two merge windows. >>>>>> >>>>>>>> Thank you for looking into this. Sounds like a not so nice riddle to= solve :) >>>>>>> There were certainly some challenges, like updating python3-daemon wh= ich then wanted to have twine as a fixed dependency which had nine dependenci= es some of which had more. I had got to 19 additional python packages and sti= ll going and then I said enough was enough and looked at twine. It is require= d for uploading python packages to PyPI which IPFire doesn't need. So I tried= patching the twine requirement out and that worked and then I found others t= hat had done the same because they had no response to the request to not have= twine as a mandatory build requirement. >>>>>> >>>>>> Python seems to become the hell that Perl used to be. In order to add = one perl package, you had 20 dependencies. And those dependencies had their o= wn dependencies too=E2=80=A6 It would be great to disable or patch out whatev= er is possible so that we do not create more work for ourselves in the future= and have to update packages that are barely used - we already have enough of= these :) >>>>>> >>>>>>> I found it all challenging but rewarding when I made progress and eve= ntually succeeded in building everything without errors. >>>>>> >>>>>> I am so happy that you are putting all this time and effort into this = because it certainly requires patience - a trade I don=E2=80=99t have. >>>>>> >>>>>> And Python is being used in all sorts of places in the distribution, s= o it really isn=E2=80=99t anything we should not be up to date on. >>>>>> >>>>>> -Michael >>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> Adolf. >>>>>>>> -Michael >>>>>>>>> >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Adolf. >>>>>>>>> >>>>>>>>> On 07/01/2022 17:49, Michael Tremer wrote: >>>>>>>>>> Hello, >>>>>>>>>>> On 4 Jan 2022, at 12:49, Adolf Belka w= rote: >>>>>>>>>>> >>>>>>>>>>> Hi Michael, >>>>>>>>>>> >>>>>>>>>>> Following the suggested Arch Linux approach I have got python3-se= tuptools-scm building successfully now. Yaaah. >>>>>>>>>>> I had to install python3-toml, followed by python3-pyproject2setu= ppy followed by python3-tomli. >>>>>>>>>>> >>>>>>>>>>> However then python3-six which installs after python3-setuptools-= scm fails due to missing the python package python3-packaging. >>>>>>>>>>> >>>>>>>>>>> I can install that package, and will do so as I continue testing = out the python 3.10 build but I have the following question about python3-six. >>>>>>>>>>> >>>>>>>>>>> Is six still needed. It is designed to ensure that programs can w= ork with either python2 or python3 but IPFire only has python3 now. >>>>>>>>>>> I can find python3-six referenced only by aws-cli. In the aws-cli= build python3-six is not listed at all in the build requirements. I tried to= search for aws-cli and python3-six being needed in run mode but I could not = find anything one way or the other. >>>>>>>>>> If aws-cli no longer requires python3-six we can drop it. Last tim= e we touched it I had to bring the package back because aws-cli was still dep= ending on it. >>>>>>>>>>> If python3-six is really needed for running aws-cli then I can in= stall it and the new dependency but if it is not needed we could remove two p= ackages which would partially offset the three new dependencies required to b= e able to build the latest python3-setuptools-scm. >>>>>>>>>> Yes, that would be great :) >>>>>>>>>> -Michael >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Adolf >>>>>>>>>>> >>>>>>>>>>> On 03/01/2022 12:32, Michael Tremer wrote: >>>>>>>>>>>> Hello, >>>>>>>>>>>>> On 3 Jan 2022, at 11:31, Adolf Belka = wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Michael, >>>>>>>>>>>>> >>>>>>>>>>>>> On 03/01/2022 11:52, Michael Tremer wrote: >>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>> I am not sure if you already looked at what Arch is doing, but= as far as I understand it works like this: >>>>>>>>>>>>> No I didn't look at Arch, which is surprising as I have in the = past with other packages but I didn't. So thanks very much for the pointer. >>>>>>>>>>>>>> They build Python 3 without pip and setuptools (https://github= .com/archlinux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e73= 3/trunk/PKGBUILD#L59). >>>>>>>>>>>>>> At this stage we might need to build setuptools separately. >>>>>>>>>>>>>> They use a tool called pyproject2setuppy (great name!) to buil= d tomli (https://github.com/archlinux/svntogit-packages/blob/packages/python-= tomli/trunk/PKGBUILD). >>>>>>>>>>>>>> Setuptools-scm can then be built with tomli: https://github.co= m/archlinux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGB= UILD >>>>>>>>>>>>>> Would this work or did I overlook anything? >>>>>>>>>>>>> I will have a look at the links and give it a go. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks very much, I feel re-motivated now. >>>>>>>>>>>> Very good! I got a little bit lost in your last email and I thin= k I could tell. But following a solution that somebody else has come up with = can be a good thing to follow and of course we would customise it to our need= s. >>>>>>>>>>>> Best, >>>>>>>>>>>> -Michael >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Adolf. >>>>>>>>>>>>>> -Michael >>>>>>>>>>>>>>> On 2 Jan 2022, at 13:44, Adolf Belka wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Michael and All, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 26/12/2021 21:55, Michael Tremer wrote: >>>>>>>>>>>>>>>> Merry Christmas, >>>>>>>>>>>>>>>>> On 26 Dec 2021, at 15:38, Adolf Belka wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Michael, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 17/12/2021 12:41, Michael Tremer wrote: >>>>>>>>>>>>>>>>>> Hello Adolf, >>>>>>>>>>>>>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I am working on updating python from 3.8 to 3.10. Python = itself is okay after some other python related programs were also updated due= to their older versions not working with 3.10 >>>>>>>>>>>>>>>>>> Thank you for working on this. >>>>>>>>>>>>>>>>>>> I am working through all the programs that reference pyth= on-3.8 in their rootfiles, or wherever, to change them to 3.10 >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> My question comes with regard to the changes to the rootf= iles to ensure they will be included properly in an update. >>>>>>>>>>>>>>>>>> We will have to ship everything that puts anything into /u= sr/lib*/python3.*/site-packages again with the release of Python 3.10. >>>>>>>>>>>>>>>>>> This isn=E2=80=99t fun, but not as much of a pain in the r= ear as shipping a new Perl release. >>>>>>>>>>>>>>>>>>> For the addons then when I change the rootfile I am also = incrementing the PAK_VER number in the lfs files and that ensures that they w= ill be properly updated. >>>>>>>>>>>>>>>>>> Yes. That is required for Pakfire to find a new version. >>>>>>>>>>>>>>>>>>> I don't know what to do for the core programs where I hav= e updated the rootfile. The lfs files for these programs don't have a PAK_VER= number to increment and the actual package version number is not being chang= ed so I don't know what I need to change in the lfs to ensure that the rootfi= le changes will be properly included in the update? >>>>>>>>>>>>>>>>>> I would say committing everything into a =E2=80=9Cpython-3= .10=E2=80=9D branch and submitting it all as a patchset is fine. All packages= that have been touched in that branch will have to be shipped again which sh= ould be easy to identify if it all comes in one large branch. >>>>>>>>>>>>>>>>> Okay, that is clear then.Will follow that approach. >>>>>>>>>>>>>>>>>>> I would appreciate any help on what I need to do for thes= e programs that have had changes only in their rootfiles. >>>>>>>>>>>>>>>>>> Absolutely. Did you find anything that didn=E2=80=99t buil= d yet? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I have not had any problem with python3.10 itself but I fou= nd some of the older modules would not work with 3.10 so I decided to do a co= mbined update to 3.10 plus update modules to latest versions. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I found that after python3-setuptools-scm had run that the = next module python3-six failed to run due to missing the tomli library. I com= mented out six and reran the build and then python3-dateutil had the same pro= blem and then python3-jmespath >>>>>>>>>>>>>>>> I completely lost track of what the status of setuptools/dis= tutils and so on is. This has changed multiple times and become independent/p= art of the distribution again and so on. >>>>>>>>>>>>>>>>> After some investigation I have found that since version 6.= 0.0 of python3-setuptools-scm, tomli has been a required library when it is b= eing used by other modules. So running version 6.0.0 of setuptools-scm worked= with python3.10 and subsequent modules built successfully as tomli not neede= d. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> So the obvious fix is to install tomli but that is when I r= an into a problem. tomli has no setup.py or setup.cfg files in its tarball. T= he only way to build it is using pip. >>>>>>>>>>>>>>>> Err. Great. >>>>>>>>>>>>>>>>> Apparently, after some searching, setup.py has been depreca= ted as the recommended install method and if setup.py doesn't work the usual = fix suggestion is to run pip install. >>>>>>>>>>>>>>>> Yes. But apparently the new system is deprecated, too. I rea= lly dislike the state of packaging systems that come with any language these = days. >>>>>>>>>>>>>>>>> Having searched on pip for a while, it looks like builds us= ing it can still be done from tarballs locally without needing to access the = internet to download the modules. >>>>>>>>>>>>>>>> That is good! >>>>>>>>>>>>>>>>> As pip is installed as part of python3 i tried using that. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I used "python3 -m pip install --root=3D/ ." in place of >>>>>>>>>>>>>>>>> "python3 setup.py build >>>>>>>>>>>>>>>>> python3 setup.py install --root=3D/" >>>>>>>>>>>>>>>>> for the tomli source. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> This failed because tomli requires flit_core to be availabl= e. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> flit_core also has no setup.py and so I tried to build it w= ith pip install. This failed with the error message that tomli was required a= s a dependency. So I added --no-deps to the flit_core pip install line and th= is time flit_core successfully built but the tomli build still came back with= not finding flit_core. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I have tried a variety of changes to the pip install line b= ut none of them has been successful and it has become clear that I am out of = my depth here and need guidance rather than just trying anything I find from = a google search. I am not even really clear if pip install is the correct app= roach, except if not then I have no idea how to progress. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Help urgently needed on how I should be approaching this. >>>>>>>>>>>>>>>> Circular dependencies can happen. This won=E2=80=99t be the = first time and it always is an utter nightmare with our buildsystem. >>>>>>>>>>>>>>>> This page states that pip comes by default with all binary p= ackages: >>>>>>>>>>>>>>>> https://docs.python.org/3/installing/index.html >>>>>>>>>>>>>>>> Is there a chance that it is part of the distribution and ca= n be enabled with a configure switch? >>>>>>>>>>>>>>> I wasn't able to find any way to get it enabled that way. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I did find that pip install will not install if the package i= s already present. You have to add --upgrade to make it update any package al= ready present. So I tried that. Didn't work. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I found a section about needing to use bootstrapping if you w= ant to install tomli. Followed the information for creating a wheel file for = flit_core and then extracting that file into the site-packages directory. Tri= ed that. Didn't work. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Rather than doing a complete build every time I changed to tr= ying to build flit_core followed by tomli in a ./make.sh shell environment. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Doing that I found that I had a pip install command that was = creating the flit_core directory in site-packages and also including the .dis= t-info directory. The info I found said that the .dist-info directory had to = be there for pip to be able to find the flit_core package. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> However the pip install of tomli still ends up not finding an= y flit_core package. In the bootstrapping information it mentioned about chan= ging the PYTHONPATH environment variable for find the flit_core package so I = tried that in the shell but it also made no difference. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I then tried running python3 -c 'import flit_core' in the tom= li build shell before doing the tomli pip install. The import command complet= ed with no error but tomli still could not find any flit_core package. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> End result of all my work is that flit_core has been built an= d installed into python3.10/site-packages but nothing seems to want to make t= he tomli build find it. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Does anyone have any ideas on what command(s) I need to use i= n the tomli pip install to have the installed flit_core package visible. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards and a Happy and Healthy New Year to all, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Adolf. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -Michael >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>> Adolf. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> PS: I hope everyone had an enjoyable Christmas. >>>>>>>>>>>>>>>> I hope you did, too! >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -Michael >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Adolf. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> --=20 >>>>>>>>>>>>>>>>>>> Sent from my laptop >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> --=20 >>>>>>>>>>>>>>>>> Sent from my laptop >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> --=20 >>>>>>>>>>>>>>> Sent from my laptop >>>>>>>>>>> >>>>>>>>>>> --=20 >>>>>>>>>>> Sent from my laptop >>>>>>>>> >>>>>>>>> --=20 >>>>>>>>> Sent from my laptop >>>>>>> >>>>>>> --=20 >>>>>>> Sent from my laptop >=20 --===============3625694098376568189==--