Hello,
I am not sure if you already looked at what Arch is doing, but as far as I understand it works like this:
They build Python 3 without pip and setuptools (https://github.com/archlinux/svntogit-packages/blob/912617035af94b1466595eed...).
At this stage we might need to build setuptools separately.
They use a tool called pyproject2setuppy (great name!) to build tomli (https://github.com/archlinux/svntogit-packages/blob/packages/python-tomli/tr...).
Setuptools-scm can then be built with tomli: https://github.com/archlinux/svntogit-community/blob/packages/python-setupto...
Would this work or did I overlook anything?
-Michael
On 2 Jan 2022, at 13:44, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael and All,
On 26/12/2021 21:55, Michael Tremer wrote:
Merry Christmas,
On 26 Dec 2021, at 15:38, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 17/12/2021 12:41, Michael Tremer wrote:
Hello Adolf,
On 17 Dec 2021, at 11:55, Adolf Belka adolf.belka@ipfire.org wrote:
Hi All,
I am working on updating python from 3.8 to 3.10. Python itself is okay after some other python related programs were also updated due to their older versions not working with 3.10
Thank you for working on this.
I am working through all the programs that reference python-3.8 in their rootfiles, or wherever, to change them to 3.10
My question comes with regard to the changes to the rootfiles to ensure they will be included properly in an update.
We will have to ship everything that puts anything into /usr/lib*/python3.*/site-packages again with the release of Python 3.10. This isn’t fun, but not as much of a pain in the rear as shipping a new Perl release.
For the addons then when I change the rootfile I am also incrementing the PAK_VER number in the lfs files and that ensures that they will be properly updated.
Yes. That is required for Pakfire to find a new version.
I don't know what to do for the core programs where I have updated the rootfile. The lfs files for these programs don't have a PAK_VER number to increment and the actual package version number is not being changed so I don't know what I need to change in the lfs to ensure that the rootfile changes will be properly included in the update?
I would say committing everything into a “python-3.10” branch and submitting it all as a patchset is fine. All packages that have been touched in that branch will have to be shipped again which should be easy to identify if it all comes in one large branch.
Okay, that is clear then.Will follow that approach.
I would appreciate any help on what I need to do for these programs that have had changes only in their rootfiles.
Absolutely. Did you find anything that didn’t build yet?
I have not had any problem with python3.10 itself but I found some of the older modules would not work with 3.10 so I decided to do a combined update to 3.10 plus update modules to latest versions.
I found that after python3-setuptools-scm had run that the next module python3-six failed to run due to missing the tomli library. I commented out six and reran the build and then python3-dateutil had the same problem and then python3-jmespath
I completely lost track of what the status of setuptools/distutils and so on is. This has changed multiple times and become independent/part of the distribution again and so on.
After some investigation I have found that since version 6.0.0 of python3-setuptools-scm, tomli has been a required library when it is being used by other modules. So running version 6.0.0 of setuptools-scm worked with python3.10 and subsequent modules built successfully as tomli not needed.
So the obvious fix is to install tomli but that is when I ran into a problem. tomli has no setup.py or setup.cfg files in its tarball. The only way to build it is using pip.
Err. Great.
Apparently, after some searching, setup.py has been deprecated as the recommended install method and if setup.py doesn't work the usual fix suggestion is to run pip install.
Yes. But apparently the new system is deprecated, too. I really dislike the state of packaging systems that come with any language these days.
Having searched on pip for a while, it looks like builds using it can still be done from tarballs locally without needing to access the internet to download the modules.
That is good!
As pip is installed as part of python3 i tried using that.
I used "python3 -m pip install --root=/ ." in place of "python3 setup.py build python3 setup.py install --root=/" for the tomli source.
This failed because tomli requires flit_core to be available.
flit_core also has no setup.py and so I tried to build it with pip install. This failed with the error message that tomli was required as a dependency. So I added --no-deps to the flit_core pip install line and this time flit_core successfully built but the tomli build still came back with not finding flit_core.
I have tried a variety of changes to the pip install line but none of them has been successful and it has become clear that I am out of my depth here and need guidance rather than just trying anything I find from a google search. I am not even really clear if pip install is the correct approach, except if not then I have no idea how to progress.
Help urgently needed on how I should be approaching this.
Circular dependencies can happen. This won’t be the first time and it always is an utter nightmare with our buildsystem. This page states that pip comes by default with all binary packages: https://docs.python.org/3/installing/index.html Is there a chance that it is part of the distribution and can be enabled with a configure switch?
I wasn't able to find any way to get it enabled that way.
I did find that pip install will not install if the package is already present. You have to add --upgrade to make it update any package already present. So I tried that. Didn't work.
I found a section about needing to use bootstrapping if you want to install tomli. Followed the information for creating a wheel file for flit_core and then extracting that file into the site-packages directory. Tried that. Didn't work.
Rather than doing a complete build every time I changed to trying to build flit_core followed by tomli in a ./make.sh shell environment.
Doing that I found that I had a pip install command that was creating the flit_core directory in site-packages and also including the .dist-info directory. The info I found said that the .dist-info directory had to be there for pip to be able to find the flit_core package.
However the pip install of tomli still ends up not finding any flit_core package. In the bootstrapping information it mentioned about changing the PYTHONPATH environment variable for find the flit_core package so I tried that in the shell but it also made no difference.
I then tried running python3 -c 'import flit_core' in the tomli build shell before doing the tomli pip install. The import command completed with no error but tomli still could not find any flit_core package.
End result of all my work is that flit_core has been built and installed into python3.10/site-packages but nothing seems to want to make the tomli build find it.
Does anyone have any ideas on what command(s) I need to use in the tomli pip install to have the installed flit_core package visible.
Regards and a Happy and Healthy New Year to all,
Adolf.
-Michael
Regards, Adolf.
PS: I hope everyone had an enjoyable Christmas.
I hope you did, too!
-Michael
Regards,
Adolf.
-- Sent from my laptop
-- Sent from my laptop
-- Sent from my laptop