Hey,
On 11 Jan 2022, at 14:32, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael and all,
python3-six has to stay as python3-dateutil explicitly specifies it in its dependencies. So that solves that question.
Aww :(
I have, with quite a bit of work, eventually been able to get all existing python modules updated that had updates and built successfully with python3.10
Yay!
However later in the IPFire build libvirt failed with a problem parsing some documentation files.
Stashing my python3.10 changes and re-running the build had libvirt building without problems. So one of the python changes has caused it. libvirt uses python3-docutils and that was updated but reverting that back to original on its own did not cause libvirt to build successfully.
I looked in the libvirt .configure for an option to not build the docs but it is not mentioned under ./configure --help.
I tried --without-docs and --disable-docs but both were not recognised in the build.
The current libvirt is from mid 2020 (6.5.0) so it is likely that it is not compatible in some way with the updated python/modules.
I tried updating libvirt to the current 7.10.0 libvirt has replaced autotools with meson. I found meson options available for all existing ./configure options except three. There is nothing in meson options for:- --with-virtualport --with-macvtap --without-dbus
Yay, another project that is using meson :(
We no longer need macvtap; we have dubs, so that should not hurt; and I have no idea what virtualport is.
I tried building it with all the other options and it successfully built with the updated python3.10 and modules. All other packages after libvirt built successfully without any problems.
Normally I would then use the updated libvirt, as that is needed to work with the updated python packages, but I am not sure if those missing options are now built into libvirt or are no longer available and what impact that might have for the use of libvirt on IPFire.
I don’t think this would hurt us much. I have CCed Jonatan who maintains this, so maybe he knows more about this.
There is nothing in the libvirt changelog about virtualport, macvtap or dbus.
Searching on the internet, I was also not able to find anything related to libvirt and these options.
Any suggestions/proposals on how to move forward on this? Should I just build libvirt without those three options or ...?
Yes, and I think this could be submitted independently of the Python branch because it is an add-on. We should be able to ship this at any time and might want to delay Python until we have enough space in a core update.
Thank you for looking into this. Sounds like a not so nice riddle to solve :)
-Michael
Regards,
Adolf.
On 07/01/2022 17:49, Michael Tremer wrote:
Hello,
On 4 Jan 2022, at 12:49, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
Following the suggested Arch Linux approach I have got python3-setuptools-scm building successfully now. Yaaah. I had to install python3-toml, followed by python3-pyproject2setuppy followed by python3-tomli.
However then python3-six which installs after python3-setuptools-scm fails due to missing the python package python3-packaging.
I can install that package, and will do so as I continue testing out the python 3.10 build but I have the following question about python3-six.
Is six still needed. It is designed to ensure that programs can work with either python2 or python3 but IPFire only has python3 now. I can find python3-six referenced only by aws-cli. In the aws-cli build python3-six is not listed at all in the build requirements. I tried to search for aws-cli and python3-six being needed in run mode but I could not find anything one way or the other.
If aws-cli no longer requires python3-six we can drop it. Last time we touched it I had to bring the package back because aws-cli was still depending on it.
If python3-six is really needed for running aws-cli then I can install it and the new dependency but if it is not needed we could remove two packages which would partially offset the three new dependencies required to be able to build the latest python3-setuptools-scm.
Yes, that would be great :) -Michael
Regards, Adolf
On 03/01/2022 12:32, Michael Tremer wrote:
Hello,
On 3 Jan 2022, at 11:31, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 03/01/2022 11:52, Michael Tremer wrote:
Hello, I am not sure if you already looked at what Arch is doing, but as far as I understand it works like this:
No I didn't look at Arch, which is surprising as I have in the past with other packages but I didn't. So thanks very much for the pointer.
They build Python 3 without pip and setuptools (https://github.com/archlinux/svntogit-packages/blob/912617035af94b1466595eed...). At this stage we might need to build setuptools separately. They use a tool called pyproject2setuppy (great name!) to build tomli (https://github.com/archlinux/svntogit-packages/blob/packages/python-tomli/tr...). Setuptools-scm can then be built with tomli: https://github.com/archlinux/svntogit-community/blob/packages/python-setupto... Would this work or did I overlook anything?
I will have a look at the links and give it a go.
Thanks very much, I feel re-motivated now.
Very good! I got a little bit lost in your last email and I think I could tell. But following a solution that somebody else has come up with can be a good thing to follow and of course we would customise it to our needs. Best, -Michael
Regards, Adolf.
-Michael > On 2 Jan 2022, at 13:44, Adolf Belka adolf.belka@ipfire.org wrote: > > Hi Michael and All, > > On 26/12/2021 21:55, Michael Tremer wrote: >> Merry Christmas, >>> On 26 Dec 2021, at 15:38, Adolf Belka adolf.belka@ipfire.org wrote: >>> >>> Hi Michael, >>> >>> On 17/12/2021 12:41, Michael Tremer wrote: >>>> Hello Adolf, >>>>> On 17 Dec 2021, at 11:55, Adolf Belka adolf.belka@ipfire.org wrote: >>>>> >>>>> Hi All, >>>>> >>>>> >>>>> I am working on updating python from 3.8 to 3.10. Python itself is okay after some other python related programs were also updated due to their older versions not working with 3.10 >>>> Thank you for working on this. >>>>> I am working through all the programs that reference python-3.8 in their rootfiles, or wherever, to change them to 3.10 >>>>> >>>>> My question comes with regard to the changes to the rootfiles to ensure they will be included properly in an update. >>>> We will have to ship everything that puts anything into /usr/lib*/python3.*/site-packages again with the release of Python 3.10. >>>> This isn’t fun, but not as much of a pain in the rear as shipping a new Perl release. >>>>> For the addons then when I change the rootfile I am also incrementing the PAK_VER number in the lfs files and that ensures that they will be properly updated. >>>> Yes. That is required for Pakfire to find a new version. >>>>> I don't know what to do for the core programs where I have updated the rootfile. The lfs files for these programs don't have a PAK_VER number to increment and the actual package version number is not being changed so I don't know what I need to change in the lfs to ensure that the rootfile changes will be properly included in the update? >>>> I would say committing everything into a “python-3.10” branch and submitting it all as a patchset is fine. All packages that have been touched in that branch will have to be shipped again which should be easy to identify if it all comes in one large branch. >>> Okay, that is clear then.Will follow that approach. >>>>> I would appreciate any help on what I need to do for these programs that have had changes only in their rootfiles. >>>> Absolutely. Did you find anything that didn’t build yet? >>> >>> I have not had any problem with python3.10 itself but I found some of the older modules would not work with 3.10 so I decided to do a combined update to 3.10 plus update modules to latest versions. >>> >>> I found that after python3-setuptools-scm had run that the next module python3-six failed to run due to missing the tomli library. I commented out six and reran the build and then python3-dateutil had the same problem and then python3-jmespath >> I completely lost track of what the status of setuptools/distutils and so on is. This has changed multiple times and become independent/part of the distribution again and so on. >>> After some investigation I have found that since version 6.0.0 of python3-setuptools-scm, tomli has been a required library when it is being used by other modules. So running version 6.0.0 of setuptools-scm worked with python3.10 and subsequent modules built successfully as tomli not needed. >>> >>> So the obvious fix is to install tomli but that is when I ran into a problem. tomli has no setup.py or setup.cfg files in its tarball. The only way to build it is using pip. >> Err. Great. >>> Apparently, after some searching, setup.py has been deprecated as the recommended install method and if setup.py doesn't work the usual fix suggestion is to run pip install. >> Yes. But apparently the new system is deprecated, too. I really dislike the state of packaging systems that come with any language these days. >>> Having searched on pip for a while, it looks like builds using it can still be done from tarballs locally without needing to access the internet to download the modules. >> That is good! >>> As pip is installed as part of python3 i tried using that. >>> >>> I used "python3 -m pip install --root=/ ." in place of >>> "python3 setup.py build >>> python3 setup.py install --root=/" >>> for the tomli source. >>> >>> This failed because tomli requires flit_core to be available. >>> >>> flit_core also has no setup.py and so I tried to build it with pip install. This failed with the error message that tomli was required as a dependency. So I added --no-deps to the flit_core pip install line and this time flit_core successfully built but the tomli build still came back with not finding flit_core. >>> >>> I have tried a variety of changes to the pip install line but none of them has been successful and it has become clear that I am out of my depth here and need guidance rather than just trying anything I find from a google search. I am not even really clear if pip install is the correct approach, except if not then I have no idea how to progress. >>> >>> Help urgently needed on how I should be approaching this. >> Circular dependencies can happen. This won’t be the first time and it always is an utter nightmare with our buildsystem. >> This page states that pip comes by default with all binary packages: >> https://docs.python.org/3/installing/index.html >> Is there a chance that it is part of the distribution and can be enabled with a configure switch? > I wasn't able to find any way to get it enabled that way. > > I did find that pip install will not install if the package is already present. You have to add --upgrade to make it update any package already present. So I tried that. Didn't work. > > I found a section about needing to use bootstrapping if you want to install tomli. Followed the information for creating a wheel file for flit_core and then extracting that file into the site-packages directory. Tried that. Didn't work. > > Rather than doing a complete build every time I changed to trying to build flit_core followed by tomli in a ./make.sh shell environment. > > Doing that I found that I had a pip install command that was creating the flit_core directory in site-packages and also including the .dist-info directory. The info I found said that the .dist-info directory had to be there for pip to be able to find the flit_core package. > > However the pip install of tomli still ends up not finding any flit_core package. In the bootstrapping information it mentioned about changing the PYTHONPATH environment variable for find the flit_core package so I tried that in the shell but it also made no difference. > > I then tried running python3 -c 'import flit_core' in the tomli build shell before doing the tomli pip install. The import command completed with no error but tomli still could not find any flit_core package. > > End result of all my work is that flit_core has been built and installed into python3.10/site-packages but nothing seems to want to make the tomli build find it. > > Does anyone have any ideas on what command(s) I need to use in the tomli pip install to have the installed flit_core package visible. > > Regards and a Happy and Healthy New Year to all, > > Adolf. > >> -Michael >>> >>> Regards, >>> Adolf. >>> >>> PS: I hope everyone had an enjoyable Christmas. >> I hope you did, too! >>> >>> >>>> -Michael >>>>> >>>>> Regards, >>>>> >>>>> Adolf. >>>>> >>>>> -- >>>>> Sent from my laptop >>>>> >>> >>> -- >>> Sent from my laptop > > -- > Sent from my laptop
-- Sent from my laptop
-- Sent from my laptop