Question about updating python to 3.10

Question about updating python to 3.10

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 1103 bytes --]

Hi All,


I am working on updating python from 3.8 to 3.10. Python itself is okay 
after some other python related programs were also updated due to their 
older versions not working with 3.10

I am working through all the programs that reference python-3.8 in their 
rootfiles, or wherever, to change them to 3.10

My question comes with regard to the changes to the rootfiles to ensure 
they will be included properly in an update.

For the addons then when I change the rootfile I am also incrementing 
the PAK_VER number in the lfs files and that ensures that they will be 
properly updated.

I don't know what to do for the core programs where I have updated the 
rootfile. The lfs files for these programs don't have a PAK_VER number 
to increment and the actual package version number is not being changed 
so I don't know what I need to change in the lfs to ensure that the 
rootfile changes will be properly included in the update?

I would appreciate any help on what I need to do for these programs that 
have had changes only in their rootfiles.


Regards,

Adolf.

-- 
Sent from my laptop

Re: Question about updating python to 3.10

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 1904 bytes --]

Hello Adolf,

> On 17 Dec 2021, at 11:55, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi All,
> 
> 
> I am working on updating python from 3.8 to 3.10. Python itself is okay after some other python related programs were also updated due to their older versions not working with 3.10

Thank you for working on this.

> I am working through all the programs that reference python-3.8 in their rootfiles, or wherever, to change them to 3.10
> 
> My question comes with regard to the changes to the rootfiles to ensure they will be included properly in an update.

We will have to ship everything that puts anything into /usr/lib*/python3.*/site-packages again with the release of Python 3.10.

This isn’t fun, but not as much of a pain in the rear as shipping a new Perl release.

> For the addons then when I change the rootfile I am also incrementing the PAK_VER number in the lfs files and that ensures that they will be properly updated.

Yes. That is required for Pakfire to find a new version.

> I don't know what to do for the core programs where I have updated the rootfile. The lfs files for these programs don't have a PAK_VER number to increment and the actual package version number is not being changed so I don't know what I need to change in the lfs to ensure that the rootfile changes will be properly included in the update?

I would say committing everything into a “python-3.10” branch and submitting it all as a patchset is fine. All packages that have been touched in that branch will have to be shipped again which should be easy to identify if it all comes in one large branch.

> I would appreciate any help on what I need to do for these programs that have had changes only in their rootfiles.

Absolutely. Did you find anything that didn’t build yet?

-Michael

> 
> Regards,
> 
> Adolf.
> 
> -- 
> Sent from my laptop
> 

Re: Question about updating python to 3.10

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 4497 bytes --]

Hi Michael,

On 17/12/2021 12:41, Michael Tremer wrote:
> Hello Adolf,
> 
>> On 17 Dec 2021, at 11:55, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>
>> Hi All,
>>
>>
>> I am working on updating python from 3.8 to 3.10. Python itself is okay after some other python related programs were also updated due to their older versions not working with 3.10
> 
> Thank you for working on this.
> 
>> I am working through all the programs that reference python-3.8 in their rootfiles, or wherever, to change them to 3.10
>>
>> My question comes with regard to the changes to the rootfiles to ensure they will be included properly in an update.
> 
> We will have to ship everything that puts anything into /usr/lib*/python3.*/site-packages again with the release of Python 3.10.
> 
> This isn’t fun, but not as much of a pain in the rear as shipping a new Perl release.
> 
>> For the addons then when I change the rootfile I am also incrementing the PAK_VER number in the lfs files and that ensures that they will be properly updated.
> 
> Yes. That is required for Pakfire to find a new version.
> 
>> I don't know what to do for the core programs where I have updated the rootfile. The lfs files for these programs don't have a PAK_VER number to increment and the actual package version number is not being changed so I don't know what I need to change in the lfs to ensure that the rootfile changes will be properly included in the update?
> 
> I would say committing everything into a “python-3.10” branch and submitting it all as a patchset is fine. All packages that have been touched in that branch will have to be shipped again which should be easy to identify if it all comes in one large branch.
Okay, that is clear then.Will follow that approach.
> 
>> I would appreciate any help on what I need to do for these programs that have had changes only in their rootfiles.
> 
> Absolutely. Did you find anything that didn’t build yet?

I have not had any problem with python3.10 itself but I found some of 
the older modules would not work with 3.10 so I decided to do a combined 
update to 3.10 plus update modules to latest versions.

I found that after python3-setuptools-scm had run that the next module 
python3-six failed to run due to missing the tomli library. I commented 
out six and reran the build and then python3-dateutil had the same 
problem  and then python3-jmespath

After some investigation I have found that since version 6.0.0 of 
python3-setuptools-scm, tomli has been a required library when it is 
being used by other modules. So running version 6.0.0 of setuptools-scm 
worked with python3.10 and subsequent modules built successfully as 
tomli not needed.

So the obvious fix is to install tomli but that is when I ran into a 
problem. tomli has no setup.py or setup.cfg files in its tarball. The 
only way to build it is using pip.

Apparently, after some searching, setup.py has been deprecated as the 
recommended install method and if setup.py doesn't work the usual fix 
suggestion is to run pip install.


Having searched on pip for a while, it looks like builds using it can 
still be done from tarballs locally without needing to access the 
internet to download the modules.

As pip is installed as part of python3 i tried using that.

I used "python3 -m pip install --root=/ ."  in place of
"python3 setup.py build
  python3 setup.py install --root=/"
for the tomli source.

This failed because tomli requires flit_core to be available.

flit_core also has no setup.py and so I tried to build it with pip 
install. This failed with the error message that tomli was required as a 
dependency. So I added --no-deps to the flit_core pip install line and 
this time flit_core successfully built but the tomli build still came 
back with not finding flit_core.

I have tried a variety of changes to the pip install line but none of 
them has been successful and it has become clear that I am out of my 
depth here and need guidance rather than just trying anything I find 
from a google search. I am not even really clear if pip install is the 
correct approach, except if not then I have no idea how to progress.

Help urgently needed on how I should be approaching this.

Regards,
Adolf.

PS:  I hope everyone had an enjoyable Christmas.


> 
> -Michael
> 
>>
>> Regards,
>>
>> Adolf.
>>
>> -- 
>> Sent from my laptop
>>
> 

-- 
Sent from my laptop

Re: Question about updating python to 3.10

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 5355 bytes --]

Merry Christmas,

> On 26 Dec 2021, at 15:38, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi Michael,
> 
> On 17/12/2021 12:41, Michael Tremer wrote:
>> Hello Adolf,
>>> On 17 Dec 2021, at 11:55, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>> 
>>> Hi All,
>>> 
>>> 
>>> I am working on updating python from 3.8 to 3.10. Python itself is okay after some other python related programs were also updated due to their older versions not working with 3.10
>> Thank you for working on this.
>>> I am working through all the programs that reference python-3.8 in their rootfiles, or wherever, to change them to 3.10
>>> 
>>> My question comes with regard to the changes to the rootfiles to ensure they will be included properly in an update.
>> We will have to ship everything that puts anything into /usr/lib*/python3.*/site-packages again with the release of Python 3.10.
>> This isn’t fun, but not as much of a pain in the rear as shipping a new Perl release.
>>> For the addons then when I change the rootfile I am also incrementing the PAK_VER number in the lfs files and that ensures that they will be properly updated.
>> Yes. That is required for Pakfire to find a new version.
>>> I don't know what to do for the core programs where I have updated the rootfile. The lfs files for these programs don't have a PAK_VER number to increment and the actual package version number is not being changed so I don't know what I need to change in the lfs to ensure that the rootfile changes will be properly included in the update?
>> I would say committing everything into a “python-3.10” branch and submitting it all as a patchset is fine. All packages that have been touched in that branch will have to be shipped again which should be easy to identify if it all comes in one large branch.
> Okay, that is clear then.Will follow that approach.
>>> I would appreciate any help on what I need to do for these programs that have had changes only in their rootfiles.
>> Absolutely. Did you find anything that didn’t build yet?
> 
> I have not had any problem with python3.10 itself but I found some of the older modules would not work with 3.10 so I decided to do a combined update to 3.10 plus update modules to latest versions.
> 
> I found that after python3-setuptools-scm had run that the next module python3-six failed to run due to missing the tomli library. I commented out six and reran the build and then python3-dateutil had the same problem  and then python3-jmespath

I completely lost track of what the status of setuptools/distutils and so on is. This has changed multiple times and become independent/part of the distribution again and so on.

> After some investigation I have found that since version 6.0.0 of python3-setuptools-scm, tomli has been a required library when it is being used by other modules. So running version 6.0.0 of setuptools-scm worked with python3.10 and subsequent modules built successfully as tomli not needed.
> 
> So the obvious fix is to install tomli but that is when I ran into a problem. tomli has no setup.py or setup.cfg files in its tarball. The only way to build it is using pip.

Err. Great.

> Apparently, after some searching, setup.py has been deprecated as the recommended install method and if setup.py doesn't work the usual fix suggestion is to run pip install.

Yes. But apparently the new system is deprecated, too. I really dislike the state of packaging systems that come with any language these days.

> Having searched on pip for a while, it looks like builds using it can still be done from tarballs locally without needing to access the internet to download the modules.

That is good!

> As pip is installed as part of python3 i tried using that.
> 
> I used "python3 -m pip install --root=/ ."  in place of
> "python3 setup.py build
> python3 setup.py install --root=/"
> for the tomli source.
> 
> This failed because tomli requires flit_core to be available.
> 
> flit_core also has no setup.py and so I tried to build it with pip install. This failed with the error message that tomli was required as a dependency. So I added --no-deps to the flit_core pip install line and this time flit_core successfully built but the tomli build still came back with not finding flit_core.
> 
> I have tried a variety of changes to the pip install line but none of them has been successful and it has become clear that I am out of my depth here and need guidance rather than just trying anything I find from a google search. I am not even really clear if pip install is the correct approach, except if not then I have no idea how to progress.
> 
> Help urgently needed on how I should be approaching this.

Circular dependencies can happen. This won’t be the first time and it always is an utter nightmare with our buildsystem.

This page states that pip comes by default with all binary packages:

  https://docs.python.org/3/installing/index.html

Is there a chance that it is part of the distribution and can be enabled with a configure switch?

-Michael

> 
> Regards,
> Adolf.
> 
> PS:  I hope everyone had an enjoyable Christmas.

I hope you did, too!


> 
> 
>> -Michael
>>> 
>>> Regards,
>>> 
>>> Adolf.
>>> 
>>> -- 
>>> Sent from my laptop
>>> 
> 
> -- 
> Sent from my laptop

Re: Question about updating python to 3.10

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 7343 bytes --]

Hi Michael and All,

On 26/12/2021 21:55, Michael Tremer wrote:
> Merry Christmas,
> 
>> On 26 Dec 2021, at 15:38, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>
>> Hi Michael,
>>
>> On 17/12/2021 12:41, Michael Tremer wrote:
>>> Hello Adolf,
>>>> On 17 Dec 2021, at 11:55, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>
>>>> Hi All,
>>>>
>>>>
>>>> I am working on updating python from 3.8 to 3.10. Python itself is okay after some other python related programs were also updated due to their older versions not working with 3.10
>>> Thank you for working on this.
>>>> I am working through all the programs that reference python-3.8 in their rootfiles, or wherever, to change them to 3.10
>>>>
>>>> My question comes with regard to the changes to the rootfiles to ensure they will be included properly in an update.
>>> We will have to ship everything that puts anything into /usr/lib*/python3.*/site-packages again with the release of Python 3.10.
>>> This isn’t fun, but not as much of a pain in the rear as shipping a new Perl release.
>>>> For the addons then when I change the rootfile I am also incrementing the PAK_VER number in the lfs files and that ensures that they will be properly updated.
>>> Yes. That is required for Pakfire to find a new version.
>>>> I don't know what to do for the core programs where I have updated the rootfile. The lfs files for these programs don't have a PAK_VER number to increment and the actual package version number is not being changed so I don't know what I need to change in the lfs to ensure that the rootfile changes will be properly included in the update?
>>> I would say committing everything into a “python-3.10” branch and submitting it all as a patchset is fine. All packages that have been touched in that branch will have to be shipped again which should be easy to identify if it all comes in one large branch.
>> Okay, that is clear then.Will follow that approach.
>>>> I would appreciate any help on what I need to do for these programs that have had changes only in their rootfiles.
>>> Absolutely. Did you find anything that didn’t build yet?
>>
>> I have not had any problem with python3.10 itself but I found some of the older modules would not work with 3.10 so I decided to do a combined update to 3.10 plus update modules to latest versions.
>>
>> I found that after python3-setuptools-scm had run that the next module python3-six failed to run due to missing the tomli library. I commented out six and reran the build and then python3-dateutil had the same problem  and then python3-jmespath
> 
> I completely lost track of what the status of setuptools/distutils and so on is. This has changed multiple times and become independent/part of the distribution again and so on.
> 
>> After some investigation I have found that since version 6.0.0 of python3-setuptools-scm, tomli has been a required library when it is being used by other modules. So running version 6.0.0 of setuptools-scm worked with python3.10 and subsequent modules built successfully as tomli not needed.
>>
>> So the obvious fix is to install tomli but that is when I ran into a problem. tomli has no setup.py or setup.cfg files in its tarball. The only way to build it is using pip.
> 
> Err. Great.
> 
>> Apparently, after some searching, setup.py has been deprecated as the recommended install method and if setup.py doesn't work the usual fix suggestion is to run pip install.
> 
> Yes. But apparently the new system is deprecated, too. I really dislike the state of packaging systems that come with any language these days.
> 
>> Having searched on pip for a while, it looks like builds using it can still be done from tarballs locally without needing to access the internet to download the modules.
> 
> That is good!
> 
>> As pip is installed as part of python3 i tried using that.
>>
>> I used "python3 -m pip install --root=/ ."  in place of
>> "python3 setup.py build
>> python3 setup.py install --root=/"
>> for the tomli source.
>>
>> This failed because tomli requires flit_core to be available.
>>
>> flit_core also has no setup.py and so I tried to build it with pip install. This failed with the error message that tomli was required as a dependency. So I added --no-deps to the flit_core pip install line and this time flit_core successfully built but the tomli build still came back with not finding flit_core.
>>
>> I have tried a variety of changes to the pip install line but none of them has been successful and it has become clear that I am out of my depth here and need guidance rather than just trying anything I find from a google search. I am not even really clear if pip install is the correct approach, except if not then I have no idea how to progress.
>>
>> Help urgently needed on how I should be approaching this.
> 
> Circular dependencies can happen. This won’t be the first time and it always is an utter nightmare with our buildsystem.
> 
> This page states that pip comes by default with all binary packages:
> 
>    https://docs.python.org/3/installing/index.html
> 
> Is there a chance that it is part of the distribution and can be enabled with a configure switch?
I wasn't able to find any way to get it enabled that way.

I did find that pip install will not install if the package is already 
present. You have to add --upgrade to make it update any package already 
present. So I tried that. Didn't work.

I found a section about needing to use bootstrapping if you want to 
install tomli. Followed the information for creating a wheel file for 
flit_core and then extracting that file into the site-packages 
directory. Tried that. Didn't work.

Rather than doing a complete build every time I changed to trying to 
build flit_core followed by tomli in a ./make.sh shell environment.

Doing that I found that I had a pip install command that was creating 
the flit_core directory in site-packages and also including the 
.dist-info directory. The info I found said that the .dist-info 
directory had to be there for pip to be able to find the flit_core package.

However the pip install of tomli still ends up not finding any flit_core 
package. In the bootstrapping information it mentioned about changing 
the PYTHONPATH environment variable for find the flit_core package so I 
tried that in the shell but it also made no difference.

I then tried running python3 -c 'import flit_core' in the tomli build 
shell before doing the tomli pip install. The import command completed 
with no error but tomli still could not find any flit_core package.

End result of all my work is that flit_core has been built and installed 
into python3.10/site-packages but nothing seems to want to make the 
tomli build find it.

Does anyone have any ideas on what command(s) I need to use in the tomli 
pip install to have the installed flit_core package visible.

Regards and a Happy and Healthy New Year to all,

Adolf.

> 
> -Michael
> 
>>
>> Regards,
>> Adolf.
>>
>> PS:  I hope everyone had an enjoyable Christmas.
> 
> I hope you did, too!
> 
> 
>>
>>
>>> -Michael
>>>>
>>>> Regards,
>>>>
>>>> Adolf.
>>>>
>>>> -- 
>>>> Sent from my laptop
>>>>
>>
>> -- 
>> Sent from my laptop
> 

-- 
Sent from my laptop

Re: Question about updating python to 3.10

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 8187 bytes --]

Hello,

I am not sure if you already looked at what Arch is doing, but as far as I understand it works like this:

They build Python 3 without pip and setuptools (https://github.com/archlinux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/trunk/PKGBUILD#L59).

At this stage we might need to build setuptools separately.

They use a tool called pyproject2setuppy (great name!) to build tomli (https://github.com/archlinux/svntogit-packages/blob/packages/python-tomli/trunk/PKGBUILD).

Setuptools-scm can then be built with tomli: https://github.com/archlinux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD

Would this work or did I overlook anything?

-Michael

> On 2 Jan 2022, at 13:44, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi Michael and All,
> 
> On 26/12/2021 21:55, Michael Tremer wrote:
>> Merry Christmas,
>>> On 26 Dec 2021, at 15:38, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>> 
>>> Hi Michael,
>>> 
>>> On 17/12/2021 12:41, Michael Tremer wrote:
>>>> Hello Adolf,
>>>>> On 17 Dec 2021, at 11:55, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>> 
>>>>> Hi All,
>>>>> 
>>>>> 
>>>>> I am working on updating python from 3.8 to 3.10. Python itself is okay after some other python related programs were also updated due to their older versions not working with 3.10
>>>> Thank you for working on this.
>>>>> I am working through all the programs that reference python-3.8 in their rootfiles, or wherever, to change them to 3.10
>>>>> 
>>>>> My question comes with regard to the changes to the rootfiles to ensure they will be included properly in an update.
>>>> We will have to ship everything that puts anything into /usr/lib*/python3.*/site-packages again with the release of Python 3.10.
>>>> This isn’t fun, but not as much of a pain in the rear as shipping a new Perl release.
>>>>> For the addons then when I change the rootfile I am also incrementing the PAK_VER number in the lfs files and that ensures that they will be properly updated.
>>>> Yes. That is required for Pakfire to find a new version.
>>>>> I don't know what to do for the core programs where I have updated the rootfile. The lfs files for these programs don't have a PAK_VER number to increment and the actual package version number is not being changed so I don't know what I need to change in the lfs to ensure that the rootfile changes will be properly included in the update?
>>>> I would say committing everything into a “python-3.10” branch and submitting it all as a patchset is fine. All packages that have been touched in that branch will have to be shipped again which should be easy to identify if it all comes in one large branch.
>>> Okay, that is clear then.Will follow that approach.
>>>>> I would appreciate any help on what I need to do for these programs that have had changes only in their rootfiles.
>>>> Absolutely. Did you find anything that didn’t build yet?
>>> 
>>> I have not had any problem with python3.10 itself but I found some of the older modules would not work with 3.10 so I decided to do a combined update to 3.10 plus update modules to latest versions.
>>> 
>>> I found that after python3-setuptools-scm had run that the next module python3-six failed to run due to missing the tomli library. I commented out six and reran the build and then python3-dateutil had the same problem  and then python3-jmespath
>> I completely lost track of what the status of setuptools/distutils and so on is. This has changed multiple times and become independent/part of the distribution again and so on.
>>> After some investigation I have found that since version 6.0.0 of python3-setuptools-scm, tomli has been a required library when it is being used by other modules. So running version 6.0.0 of setuptools-scm worked with python3.10 and subsequent modules built successfully as tomli not needed.
>>> 
>>> So the obvious fix is to install tomli but that is when I ran into a problem. tomli has no setup.py or setup.cfg files in its tarball. The only way to build it is using pip.
>> Err. Great.
>>> Apparently, after some searching, setup.py has been deprecated as the recommended install method and if setup.py doesn't work the usual fix suggestion is to run pip install.
>> Yes. But apparently the new system is deprecated, too. I really dislike the state of packaging systems that come with any language these days.
>>> Having searched on pip for a while, it looks like builds using it can still be done from tarballs locally without needing to access the internet to download the modules.
>> That is good!
>>> As pip is installed as part of python3 i tried using that.
>>> 
>>> I used "python3 -m pip install --root=/ ."  in place of
>>> "python3 setup.py build
>>> python3 setup.py install --root=/"
>>> for the tomli source.
>>> 
>>> This failed because tomli requires flit_core to be available.
>>> 
>>> flit_core also has no setup.py and so I tried to build it with pip install. This failed with the error message that tomli was required as a dependency. So I added --no-deps to the flit_core pip install line and this time flit_core successfully built but the tomli build still came back with not finding flit_core.
>>> 
>>> I have tried a variety of changes to the pip install line but none of them has been successful and it has become clear that I am out of my depth here and need guidance rather than just trying anything I find from a google search. I am not even really clear if pip install is the correct approach, except if not then I have no idea how to progress.
>>> 
>>> Help urgently needed on how I should be approaching this.
>> Circular dependencies can happen. This won’t be the first time and it always is an utter nightmare with our buildsystem.
>> This page states that pip comes by default with all binary packages:
>>   https://docs.python.org/3/installing/index.html
>> Is there a chance that it is part of the distribution and can be enabled with a configure switch?
> I wasn't able to find any way to get it enabled that way.
> 
> I did find that pip install will not install if the package is already present. You have to add --upgrade to make it update any package already present. So I tried that. Didn't work.
> 
> I found a section about needing to use bootstrapping if you want to install tomli. Followed the information for creating a wheel file for flit_core and then extracting that file into the site-packages directory. Tried that. Didn't work.
> 
> Rather than doing a complete build every time I changed to trying to build flit_core followed by tomli in a ./make.sh shell environment.
> 
> Doing that I found that I had a pip install command that was creating the flit_core directory in site-packages and also including the .dist-info directory. The info I found said that the .dist-info directory had to be there for pip to be able to find the flit_core package.
> 
> However the pip install of tomli still ends up not finding any flit_core package. In the bootstrapping information it mentioned about changing the PYTHONPATH environment variable for find the flit_core package so I tried that in the shell but it also made no difference.
> 
> I then tried running python3 -c 'import flit_core' in the tomli build shell before doing the tomli pip install. The import command completed with no error but tomli still could not find any flit_core package.
> 
> End result of all my work is that flit_core has been built and installed into python3.10/site-packages but nothing seems to want to make the tomli build find it.
> 
> Does anyone have any ideas on what command(s) I need to use in the tomli pip install to have the installed flit_core package visible.
> 
> Regards and a Happy and Healthy New Year to all,
> 
> Adolf.
> 
>> -Michael
>>> 
>>> Regards,
>>> Adolf.
>>> 
>>> PS:  I hope everyone had an enjoyable Christmas.
>> I hope you did, too!
>>> 
>>> 
>>>> -Michael
>>>>> 
>>>>> Regards,
>>>>> 
>>>>> Adolf.
>>>>> 
>>>>> -- 
>>>>> Sent from my laptop
>>>>> 
>>> 
>>> -- 
>>> Sent from my laptop
> 
> -- 
> Sent from my laptop

Re: Question about updating python to 3.10

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 8609 bytes --]

Hi Michael,

On 03/01/2022 11:52, Michael Tremer wrote:
> Hello,
> 
> I am not sure if you already looked at what Arch is doing, but as far as I understand it works like this:
No I didn't look at Arch, which is surprising as I have in the past with other packages but I didn't. So thanks very much for the pointer.
> 
> They build Python 3 without pip and setuptools (https://github.com/archlinux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/trunk/PKGBUILD#L59).
> 
> At this stage we might need to build setuptools separately.
> 
> They use a tool called pyproject2setuppy (great name!) to build tomli (https://github.com/archlinux/svntogit-packages/blob/packages/python-tomli/trunk/PKGBUILD).
> 
> Setuptools-scm can then be built with tomli: https://github.com/archlinux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD
> 
> Would this work or did I overlook anything?
I will have a look at the links and give it a go.

Thanks very much, I feel re-motivated now.

Regards,
Adolf.
> 
> -Michael
> 
>> On 2 Jan 2022, at 13:44, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>
>> Hi Michael and All,
>>
>> On 26/12/2021 21:55, Michael Tremer wrote:
>>> Merry Christmas,
>>>> On 26 Dec 2021, at 15:38, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>
>>>> Hi Michael,
>>>>
>>>> On 17/12/2021 12:41, Michael Tremer wrote:
>>>>> Hello Adolf,
>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>>
>>>>>> I am working on updating python from 3.8 to 3.10. Python itself is okay after some other python related programs were also updated due to their older versions not working with 3.10
>>>>> Thank you for working on this.
>>>>>> I am working through all the programs that reference python-3.8 in their rootfiles, or wherever, to change them to 3.10
>>>>>>
>>>>>> My question comes with regard to the changes to the rootfiles to ensure they will be included properly in an update.
>>>>> We will have to ship everything that puts anything into /usr/lib*/python3.*/site-packages again with the release of Python 3.10.
>>>>> This isn’t fun, but not as much of a pain in the rear as shipping a new Perl release.
>>>>>> For the addons then when I change the rootfile I am also incrementing the PAK_VER number in the lfs files and that ensures that they will be properly updated.
>>>>> Yes. That is required for Pakfire to find a new version.
>>>>>> I don't know what to do for the core programs where I have updated the rootfile. The lfs files for these programs don't have a PAK_VER number to increment and the actual package version number is not being changed so I don't know what I need to change in the lfs to ensure that the rootfile changes will be properly included in the update?
>>>>> I would say committing everything into a “python-3.10” branch and submitting it all as a patchset is fine. All packages that have been touched in that branch will have to be shipped again which should be easy to identify if it all comes in one large branch.
>>>> Okay, that is clear then.Will follow that approach.
>>>>>> I would appreciate any help on what I need to do for these programs that have had changes only in their rootfiles.
>>>>> Absolutely. Did you find anything that didn’t build yet?
>>>>
>>>> I have not had any problem with python3.10 itself but I found some of the older modules would not work with 3.10 so I decided to do a combined update to 3.10 plus update modules to latest versions.
>>>>
>>>> I found that after python3-setuptools-scm had run that the next module python3-six failed to run due to missing the tomli library. I commented out six and reran the build and then python3-dateutil had the same problem  and then python3-jmespath
>>> I completely lost track of what the status of setuptools/distutils and so on is. This has changed multiple times and become independent/part of the distribution again and so on.
>>>> After some investigation I have found that since version 6.0.0 of python3-setuptools-scm, tomli has been a required library when it is being used by other modules. So running version 6.0.0 of setuptools-scm worked with python3.10 and subsequent modules built successfully as tomli not needed.
>>>>
>>>> So the obvious fix is to install tomli but that is when I ran into a problem. tomli has no setup.py or setup.cfg files in its tarball. The only way to build it is using pip.
>>> Err. Great.
>>>> Apparently, after some searching, setup.py has been deprecated as the recommended install method and if setup.py doesn't work the usual fix suggestion is to run pip install.
>>> Yes. But apparently the new system is deprecated, too. I really dislike the state of packaging systems that come with any language these days.
>>>> Having searched on pip for a while, it looks like builds using it can still be done from tarballs locally without needing to access the internet to download the modules.
>>> That is good!
>>>> As pip is installed as part of python3 i tried using that.
>>>>
>>>> I used "python3 -m pip install --root=/ ."  in place of
>>>> "python3 setup.py build
>>>> python3 setup.py install --root=/"
>>>> for the tomli source.
>>>>
>>>> This failed because tomli requires flit_core to be available.
>>>>
>>>> flit_core also has no setup.py and so I tried to build it with pip install. This failed with the error message that tomli was required as a dependency. So I added --no-deps to the flit_core pip install line and this time flit_core successfully built but the tomli build still came back with not finding flit_core.
>>>>
>>>> I have tried a variety of changes to the pip install line but none of them has been successful and it has become clear that I am out of my depth here and need guidance rather than just trying anything I find from a google search. I am not even really clear if pip install is the correct approach, except if not then I have no idea how to progress.
>>>>
>>>> Help urgently needed on how I should be approaching this.
>>> Circular dependencies can happen. This won’t be the first time and it always is an utter nightmare with our buildsystem.
>>> This page states that pip comes by default with all binary packages:
>>>    https://docs.python.org/3/installing/index.html
>>> Is there a chance that it is part of the distribution and can be enabled with a configure switch?
>> I wasn't able to find any way to get it enabled that way.
>>
>> I did find that pip install will not install if the package is already present. You have to add --upgrade to make it update any package already present. So I tried that. Didn't work.
>>
>> I found a section about needing to use bootstrapping if you want to install tomli. Followed the information for creating a wheel file for flit_core and then extracting that file into the site-packages directory. Tried that. Didn't work.
>>
>> Rather than doing a complete build every time I changed to trying to build flit_core followed by tomli in a ./make.sh shell environment.
>>
>> Doing that I found that I had a pip install command that was creating the flit_core directory in site-packages and also including the .dist-info directory. The info I found said that the .dist-info directory had to be there for pip to be able to find the flit_core package.
>>
>> However the pip install of tomli still ends up not finding any flit_core package. In the bootstrapping information it mentioned about changing the PYTHONPATH environment variable for find the flit_core package so I tried that in the shell but it also made no difference.
>>
>> I then tried running python3 -c 'import flit_core' in the tomli build shell before doing the tomli pip install. The import command completed with no error but tomli still could not find any flit_core package.
>>
>> End result of all my work is that flit_core has been built and installed into python3.10/site-packages but nothing seems to want to make the tomli build find it.
>>
>> Does anyone have any ideas on what command(s) I need to use in the tomli pip install to have the installed flit_core package visible.
>>
>> Regards and a Happy and Healthy New Year to all,
>>
>> Adolf.
>>
>>> -Michael
>>>>
>>>> Regards,
>>>> Adolf.
>>>>
>>>> PS:  I hope everyone had an enjoyable Christmas.
>>> I hope you did, too!
>>>>
>>>>
>>>>> -Michael
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Adolf.
>>>>>>
>>>>>> -- 
>>>>>> Sent from my laptop
>>>>>>
>>>>
>>>> -- 
>>>> Sent from my laptop
>>
>> -- 
>> Sent from my laptop
> 

Re: Question about updating python to 3.10

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 9080 bytes --]

Hello,

> On 3 Jan 2022, at 11:31, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi Michael,
> 
> On 03/01/2022 11:52, Michael Tremer wrote:
>> Hello,
>> I am not sure if you already looked at what Arch is doing, but as far as I understand it works like this:
> No I didn't look at Arch, which is surprising as I have in the past with other packages but I didn't. So thanks very much for the pointer.
>> They build Python 3 without pip and setuptools (https://github.com/archlinux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/trunk/PKGBUILD#L59).
>> At this stage we might need to build setuptools separately.
>> They use a tool called pyproject2setuppy (great name!) to build tomli (https://github.com/archlinux/svntogit-packages/blob/packages/python-tomli/trunk/PKGBUILD).
>> Setuptools-scm can then be built with tomli: https://github.com/archlinux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD
>> Would this work or did I overlook anything?
> I will have a look at the links and give it a go.
> 
> Thanks very much, I feel re-motivated now.

Very good! I got a little bit lost in your last email and I think I could tell. But following a solution that somebody else has come up with can be a good thing to follow and of course we would customise it to our needs.

Best,
-Michael

> 
> Regards,
> Adolf.
>> -Michael
>>> On 2 Jan 2022, at 13:44, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>> 
>>> Hi Michael and All,
>>> 
>>> On 26/12/2021 21:55, Michael Tremer wrote:
>>>> Merry Christmas,
>>>>> On 26 Dec 2021, at 15:38, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>> 
>>>>> Hi Michael,
>>>>> 
>>>>> On 17/12/2021 12:41, Michael Tremer wrote:
>>>>>> Hello Adolf,
>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>> 
>>>>>>> Hi All,
>>>>>>> 
>>>>>>> 
>>>>>>> I am working on updating python from 3.8 to 3.10. Python itself is okay after some other python related programs were also updated due to their older versions not working with 3.10
>>>>>> Thank you for working on this.
>>>>>>> I am working through all the programs that reference python-3.8 in their rootfiles, or wherever, to change them to 3.10
>>>>>>> 
>>>>>>> My question comes with regard to the changes to the rootfiles to ensure they will be included properly in an update.
>>>>>> We will have to ship everything that puts anything into /usr/lib*/python3.*/site-packages again with the release of Python 3.10.
>>>>>> This isn’t fun, but not as much of a pain in the rear as shipping a new Perl release.
>>>>>>> For the addons then when I change the rootfile I am also incrementing the PAK_VER number in the lfs files and that ensures that they will be properly updated.
>>>>>> Yes. That is required for Pakfire to find a new version.
>>>>>>> I don't know what to do for the core programs where I have updated the rootfile. The lfs files for these programs don't have a PAK_VER number to increment and the actual package version number is not being changed so I don't know what I need to change in the lfs to ensure that the rootfile changes will be properly included in the update?
>>>>>> I would say committing everything into a “python-3.10” branch and submitting it all as a patchset is fine. All packages that have been touched in that branch will have to be shipped again which should be easy to identify if it all comes in one large branch.
>>>>> Okay, that is clear then.Will follow that approach.
>>>>>>> I would appreciate any help on what I need to do for these programs that have had changes only in their rootfiles.
>>>>>> Absolutely. Did you find anything that didn’t build yet?
>>>>> 
>>>>> I have not had any problem with python3.10 itself but I found some of the older modules would not work with 3.10 so I decided to do a combined update to 3.10 plus update modules to latest versions.
>>>>> 
>>>>> I found that after python3-setuptools-scm had run that the next module python3-six failed to run due to missing the tomli library. I commented out six and reran the build and then python3-dateutil had the same problem  and then python3-jmespath
>>>> I completely lost track of what the status of setuptools/distutils and so on is. This has changed multiple times and become independent/part of the distribution again and so on.
>>>>> After some investigation I have found that since version 6.0.0 of python3-setuptools-scm, tomli has been a required library when it is being used by other modules. So running version 6.0.0 of setuptools-scm worked with python3.10 and subsequent modules built successfully as tomli not needed.
>>>>> 
>>>>> So the obvious fix is to install tomli but that is when I ran into a problem. tomli has no setup.py or setup.cfg files in its tarball. The only way to build it is using pip.
>>>> Err. Great.
>>>>> Apparently, after some searching, setup.py has been deprecated as the recommended install method and if setup.py doesn't work the usual fix suggestion is to run pip install.
>>>> Yes. But apparently the new system is deprecated, too. I really dislike the state of packaging systems that come with any language these days.
>>>>> Having searched on pip for a while, it looks like builds using it can still be done from tarballs locally without needing to access the internet to download the modules.
>>>> That is good!
>>>>> As pip is installed as part of python3 i tried using that.
>>>>> 
>>>>> I used "python3 -m pip install --root=/ ."  in place of
>>>>> "python3 setup.py build
>>>>> python3 setup.py install --root=/"
>>>>> for the tomli source.
>>>>> 
>>>>> This failed because tomli requires flit_core to be available.
>>>>> 
>>>>> flit_core also has no setup.py and so I tried to build it with pip install. This failed with the error message that tomli was required as a dependency. So I added --no-deps to the flit_core pip install line and this time flit_core successfully built but the tomli build still came back with not finding flit_core.
>>>>> 
>>>>> I have tried a variety of changes to the pip install line but none of them has been successful and it has become clear that I am out of my depth here and need guidance rather than just trying anything I find from a google search. I am not even really clear if pip install is the correct approach, except if not then I have no idea how to progress.
>>>>> 
>>>>> Help urgently needed on how I should be approaching this.
>>>> Circular dependencies can happen. This won’t be the first time and it always is an utter nightmare with our buildsystem.
>>>> This page states that pip comes by default with all binary packages:
>>>>   https://docs.python.org/3/installing/index.html
>>>> Is there a chance that it is part of the distribution and can be enabled with a configure switch?
>>> I wasn't able to find any way to get it enabled that way.
>>> 
>>> I did find that pip install will not install if the package is already present. You have to add --upgrade to make it update any package already present. So I tried that. Didn't work.
>>> 
>>> I found a section about needing to use bootstrapping if you want to install tomli. Followed the information for creating a wheel file for flit_core and then extracting that file into the site-packages directory. Tried that. Didn't work.
>>> 
>>> Rather than doing a complete build every time I changed to trying to build flit_core followed by tomli in a ./make.sh shell environment.
>>> 
>>> Doing that I found that I had a pip install command that was creating the flit_core directory in site-packages and also including the .dist-info directory. The info I found said that the .dist-info directory had to be there for pip to be able to find the flit_core package.
>>> 
>>> However the pip install of tomli still ends up not finding any flit_core package. In the bootstrapping information it mentioned about changing the PYTHONPATH environment variable for find the flit_core package so I tried that in the shell but it also made no difference.
>>> 
>>> I then tried running python3 -c 'import flit_core' in the tomli build shell before doing the tomli pip install. The import command completed with no error but tomli still could not find any flit_core package.
>>> 
>>> End result of all my work is that flit_core has been built and installed into python3.10/site-packages but nothing seems to want to make the tomli build find it.
>>> 
>>> Does anyone have any ideas on what command(s) I need to use in the tomli pip install to have the installed flit_core package visible.
>>> 
>>> Regards and a Happy and Healthy New Year to all,
>>> 
>>> Adolf.
>>> 
>>>> -Michael
>>>>> 
>>>>> Regards,
>>>>> Adolf.
>>>>> 
>>>>> PS:  I hope everyone had an enjoyable Christmas.
>>>> I hope you did, too!
>>>>> 
>>>>> 
>>>>>> -Michael
>>>>>>> 
>>>>>>> Regards,
>>>>>>> 
>>>>>>> Adolf.
>>>>>>> 
>>>>>>> -- 
>>>>>>> Sent from my laptop
>>>>>>> 
>>>>> 
>>>>> -- 
>>>>> Sent from my laptop
>>> 
>>> -- 
>>> Sent from my laptop

Re: Question about updating python to 3.10

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 10469 bytes --]

Hi Michael,

Following the suggested Arch Linux approach I have got 
python3-setuptools-scm building successfully now. Yaaah.
I had to install python3-toml, followed by python3-pyproject2setuppy 
followed by python3-tomli.

However then python3-six which installs after python3-setuptools-scm 
fails due to missing the python package python3-packaging.

I can install that package, and will do so as I continue testing out the 
python 3.10 build but I have the following question about python3-six.

Is six still needed. It is designed to ensure that programs can work 
with either python2 or python3 but IPFire only has python3 now.
I can find python3-six referenced only by aws-cli. In the aws-cli build 
python3-six is not listed at all in the build requirements. I tried to 
search for aws-cli and python3-six being needed in run mode but I could 
not find anything one way or the other.

If python3-six is really needed for running aws-cli then I can install 
it and the new dependency but if it is not needed we could remove two 
packages which would partially offset the three new dependencies 
required to be able to build the latest python3-setuptools-scm.

Regards,
Adolf

On 03/01/2022 12:32, Michael Tremer wrote:
> Hello,
> 
>> On 3 Jan 2022, at 11:31, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>
>> Hi Michael,
>>
>> On 03/01/2022 11:52, Michael Tremer wrote:
>>> Hello,
>>> I am not sure if you already looked at what Arch is doing, but as far as I understand it works like this:
>> No I didn't look at Arch, which is surprising as I have in the past with other packages but I didn't. So thanks very much for the pointer.
>>> They build Python 3 without pip and setuptools (https://github.com/archlinux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/trunk/PKGBUILD#L59).
>>> At this stage we might need to build setuptools separately.
>>> They use a tool called pyproject2setuppy (great name!) to build tomli (https://github.com/archlinux/svntogit-packages/blob/packages/python-tomli/trunk/PKGBUILD).
>>> Setuptools-scm can then be built with tomli: https://github.com/archlinux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD
>>> Would this work or did I overlook anything?
>> I will have a look at the links and give it a go.
>>
>> Thanks very much, I feel re-motivated now.
> 
> Very good! I got a little bit lost in your last email and I think I could tell. But following a solution that somebody else has come up with can be a good thing to follow and of course we would customise it to our needs.
> 
> Best,
> -Michael
> 
>>
>> Regards,
>> Adolf.
>>> -Michael
>>>> On 2 Jan 2022, at 13:44, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>
>>>> Hi Michael and All,
>>>>
>>>> On 26/12/2021 21:55, Michael Tremer wrote:
>>>>> Merry Christmas,
>>>>>> On 26 Dec 2021, at 15:38, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>
>>>>>> Hi Michael,
>>>>>>
>>>>>> On 17/12/2021 12:41, Michael Tremer wrote:
>>>>>>> Hello Adolf,
>>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>>
>>>>>>>> I am working on updating python from 3.8 to 3.10. Python itself is okay after some other python related programs were also updated due to their older versions not working with 3.10
>>>>>>> Thank you for working on this.
>>>>>>>> I am working through all the programs that reference python-3.8 in their rootfiles, or wherever, to change them to 3.10
>>>>>>>>
>>>>>>>> My question comes with regard to the changes to the rootfiles to ensure they will be included properly in an update.
>>>>>>> We will have to ship everything that puts anything into /usr/lib*/python3.*/site-packages again with the release of Python 3.10.
>>>>>>> This isn’t fun, but not as much of a pain in the rear as shipping a new Perl release.
>>>>>>>> For the addons then when I change the rootfile I am also incrementing the PAK_VER number in the lfs files and that ensures that they will be properly updated.
>>>>>>> Yes. That is required for Pakfire to find a new version.
>>>>>>>> I don't know what to do for the core programs where I have updated the rootfile. The lfs files for these programs don't have a PAK_VER number to increment and the actual package version number is not being changed so I don't know what I need to change in the lfs to ensure that the rootfile changes will be properly included in the update?
>>>>>>> I would say committing everything into a “python-3.10” branch and submitting it all as a patchset is fine. All packages that have been touched in that branch will have to be shipped again which should be easy to identify if it all comes in one large branch.
>>>>>> Okay, that is clear then.Will follow that approach.
>>>>>>>> I would appreciate any help on what I need to do for these programs that have had changes only in their rootfiles.
>>>>>>> Absolutely. Did you find anything that didn’t build yet?
>>>>>>
>>>>>> I have not had any problem with python3.10 itself but I found some of the older modules would not work with 3.10 so I decided to do a combined update to 3.10 plus update modules to latest versions.
>>>>>>
>>>>>> I found that after python3-setuptools-scm had run that the next module python3-six failed to run due to missing the tomli library. I commented out six and reran the build and then python3-dateutil had the same problem  and then python3-jmespath
>>>>> I completely lost track of what the status of setuptools/distutils and so on is. This has changed multiple times and become independent/part of the distribution again and so on.
>>>>>> After some investigation I have found that since version 6.0.0 of python3-setuptools-scm, tomli has been a required library when it is being used by other modules. So running version 6.0.0 of setuptools-scm worked with python3.10 and subsequent modules built successfully as tomli not needed.
>>>>>>
>>>>>> So the obvious fix is to install tomli but that is when I ran into a problem. tomli has no setup.py or setup.cfg files in its tarball. The only way to build it is using pip.
>>>>> Err. Great.
>>>>>> Apparently, after some searching, setup.py has been deprecated as the recommended install method and if setup.py doesn't work the usual fix suggestion is to run pip install.
>>>>> Yes. But apparently the new system is deprecated, too. I really dislike the state of packaging systems that come with any language these days.
>>>>>> Having searched on pip for a while, it looks like builds using it can still be done from tarballs locally without needing to access the internet to download the modules.
>>>>> That is good!
>>>>>> As pip is installed as part of python3 i tried using that.
>>>>>>
>>>>>> I used "python3 -m pip install --root=/ ."  in place of
>>>>>> "python3 setup.py build
>>>>>> python3 setup.py install --root=/"
>>>>>> for the tomli source.
>>>>>>
>>>>>> This failed because tomli requires flit_core to be available.
>>>>>>
>>>>>> flit_core also has no setup.py and so I tried to build it with pip install. This failed with the error message that tomli was required as a dependency. So I added --no-deps to the flit_core pip install line and this time flit_core successfully built but the tomli build still came back with not finding flit_core.
>>>>>>
>>>>>> I have tried a variety of changes to the pip install line but none of them has been successful and it has become clear that I am out of my depth here and need guidance rather than just trying anything I find from a google search. I am not even really clear if pip install is the correct approach, except if not then I have no idea how to progress.
>>>>>>
>>>>>> Help urgently needed on how I should be approaching this.
>>>>> Circular dependencies can happen. This won’t be the first time and it always is an utter nightmare with our buildsystem.
>>>>> This page states that pip comes by default with all binary packages:
>>>>>    https://docs.python.org/3/installing/index.html
>>>>> Is there a chance that it is part of the distribution and can be enabled with a configure switch?
>>>> I wasn't able to find any way to get it enabled that way.
>>>>
>>>> I did find that pip install will not install if the package is already present. You have to add --upgrade to make it update any package already present. So I tried that. Didn't work.
>>>>
>>>> I found a section about needing to use bootstrapping if you want to install tomli. Followed the information for creating a wheel file for flit_core and then extracting that file into the site-packages directory. Tried that. Didn't work.
>>>>
>>>> Rather than doing a complete build every time I changed to trying to build flit_core followed by tomli in a ./make.sh shell environment.
>>>>
>>>> Doing that I found that I had a pip install command that was creating the flit_core directory in site-packages and also including the .dist-info directory. The info I found said that the .dist-info directory had to be there for pip to be able to find the flit_core package.
>>>>
>>>> However the pip install of tomli still ends up not finding any flit_core package. In the bootstrapping information it mentioned about changing the PYTHONPATH environment variable for find the flit_core package so I tried that in the shell but it also made no difference.
>>>>
>>>> I then tried running python3 -c 'import flit_core' in the tomli build shell before doing the tomli pip install. The import command completed with no error but tomli still could not find any flit_core package.
>>>>
>>>> End result of all my work is that flit_core has been built and installed into python3.10/site-packages but nothing seems to want to make the tomli build find it.
>>>>
>>>> Does anyone have any ideas on what command(s) I need to use in the tomli pip install to have the installed flit_core package visible.
>>>>
>>>> Regards and a Happy and Healthy New Year to all,
>>>>
>>>> Adolf.
>>>>
>>>>> -Michael
>>>>>>
>>>>>> Regards,
>>>>>> Adolf.
>>>>>>
>>>>>> PS:  I hope everyone had an enjoyable Christmas.
>>>>> I hope you did, too!
>>>>>>
>>>>>>
>>>>>>> -Michael
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Adolf.
>>>>>>>>
>>>>>>>> -- 
>>>>>>>> Sent from my laptop
>>>>>>>>
>>>>>>
>>>>>> -- 
>>>>>> Sent from my laptop
>>>>
>>>> -- 
>>>> Sent from my laptop
> 

-- 
Sent from my laptop

Re: Question about updating python to 3.10

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 10934 bytes --]

Hello,

> On 4 Jan 2022, at 12:49, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi Michael,
> 
> Following the suggested Arch Linux approach I have got python3-setuptools-scm building successfully now. Yaaah.
> I had to install python3-toml, followed by python3-pyproject2setuppy followed by python3-tomli.
> 
> However then python3-six which installs after python3-setuptools-scm fails due to missing the python package python3-packaging.
> 
> I can install that package, and will do so as I continue testing out the python 3.10 build but I have the following question about python3-six.
> 
> Is six still needed. It is designed to ensure that programs can work with either python2 or python3 but IPFire only has python3 now.
> I can find python3-six referenced only by aws-cli. In the aws-cli build python3-six is not listed at all in the build requirements. I tried to search for aws-cli and python3-six being needed in run mode but I could not find anything one way or the other.

If aws-cli no longer requires python3-six we can drop it. Last time we touched it I had to bring the package back because aws-cli was still depending on it.

> If python3-six is really needed for running aws-cli then I can install it and the new dependency but if it is not needed we could remove two packages which would partially offset the three new dependencies required to be able to build the latest python3-setuptools-scm.

Yes, that would be great :)

-Michael

> 
> Regards,
> Adolf
> 
> On 03/01/2022 12:32, Michael Tremer wrote:
>> Hello,
>>> On 3 Jan 2022, at 11:31, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>> 
>>> Hi Michael,
>>> 
>>> On 03/01/2022 11:52, Michael Tremer wrote:
>>>> Hello,
>>>> I am not sure if you already looked at what Arch is doing, but as far as I understand it works like this:
>>> No I didn't look at Arch, which is surprising as I have in the past with other packages but I didn't. So thanks very much for the pointer.
>>>> They build Python 3 without pip and setuptools (https://github.com/archlinux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/trunk/PKGBUILD#L59).
>>>> At this stage we might need to build setuptools separately.
>>>> They use a tool called pyproject2setuppy (great name!) to build tomli (https://github.com/archlinux/svntogit-packages/blob/packages/python-tomli/trunk/PKGBUILD).
>>>> Setuptools-scm can then be built with tomli: https://github.com/archlinux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD
>>>> Would this work or did I overlook anything?
>>> I will have a look at the links and give it a go.
>>> 
>>> Thanks very much, I feel re-motivated now.
>> Very good! I got a little bit lost in your last email and I think I could tell. But following a solution that somebody else has come up with can be a good thing to follow and of course we would customise it to our needs.
>> Best,
>> -Michael
>>> 
>>> Regards,
>>> Adolf.
>>>> -Michael
>>>>> On 2 Jan 2022, at 13:44, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>> 
>>>>> Hi Michael and All,
>>>>> 
>>>>> On 26/12/2021 21:55, Michael Tremer wrote:
>>>>>> Merry Christmas,
>>>>>>> On 26 Dec 2021, at 15:38, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>> 
>>>>>>> Hi Michael,
>>>>>>> 
>>>>>>> On 17/12/2021 12:41, Michael Tremer wrote:
>>>>>>>> Hello Adolf,
>>>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>> 
>>>>>>>>> Hi All,
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> I am working on updating python from 3.8 to 3.10. Python itself is okay after some other python related programs were also updated due to their older versions not working with 3.10
>>>>>>>> Thank you for working on this.
>>>>>>>>> I am working through all the programs that reference python-3.8 in their rootfiles, or wherever, to change them to 3.10
>>>>>>>>> 
>>>>>>>>> My question comes with regard to the changes to the rootfiles to ensure they will be included properly in an update.
>>>>>>>> We will have to ship everything that puts anything into /usr/lib*/python3.*/site-packages again with the release of Python 3.10.
>>>>>>>> This isn’t fun, but not as much of a pain in the rear as shipping a new Perl release.
>>>>>>>>> For the addons then when I change the rootfile I am also incrementing the PAK_VER number in the lfs files and that ensures that they will be properly updated.
>>>>>>>> Yes. That is required for Pakfire to find a new version.
>>>>>>>>> I don't know what to do for the core programs where I have updated the rootfile. The lfs files for these programs don't have a PAK_VER number to increment and the actual package version number is not being changed so I don't know what I need to change in the lfs to ensure that the rootfile changes will be properly included in the update?
>>>>>>>> I would say committing everything into a “python-3.10” branch and submitting it all as a patchset is fine. All packages that have been touched in that branch will have to be shipped again which should be easy to identify if it all comes in one large branch.
>>>>>>> Okay, that is clear then.Will follow that approach.
>>>>>>>>> I would appreciate any help on what I need to do for these programs that have had changes only in their rootfiles.
>>>>>>>> Absolutely. Did you find anything that didn’t build yet?
>>>>>>> 
>>>>>>> I have not had any problem with python3.10 itself but I found some of the older modules would not work with 3.10 so I decided to do a combined update to 3.10 plus update modules to latest versions.
>>>>>>> 
>>>>>>> I found that after python3-setuptools-scm had run that the next module python3-six failed to run due to missing the tomli library. I commented out six and reran the build and then python3-dateutil had the same problem  and then python3-jmespath
>>>>>> I completely lost track of what the status of setuptools/distutils and so on is. This has changed multiple times and become independent/part of the distribution again and so on.
>>>>>>> After some investigation I have found that since version 6.0.0 of python3-setuptools-scm, tomli has been a required library when it is being used by other modules. So running version 6.0.0 of setuptools-scm worked with python3.10 and subsequent modules built successfully as tomli not needed.
>>>>>>> 
>>>>>>> So the obvious fix is to install tomli but that is when I ran into a problem. tomli has no setup.py or setup.cfg files in its tarball. The only way to build it is using pip.
>>>>>> Err. Great.
>>>>>>> Apparently, after some searching, setup.py has been deprecated as the recommended install method and if setup.py doesn't work the usual fix suggestion is to run pip install.
>>>>>> Yes. But apparently the new system is deprecated, too. I really dislike the state of packaging systems that come with any language these days.
>>>>>>> Having searched on pip for a while, it looks like builds using it can still be done from tarballs locally without needing to access the internet to download the modules.
>>>>>> That is good!
>>>>>>> As pip is installed as part of python3 i tried using that.
>>>>>>> 
>>>>>>> I used "python3 -m pip install --root=/ ."  in place of
>>>>>>> "python3 setup.py build
>>>>>>> python3 setup.py install --root=/"
>>>>>>> for the tomli source.
>>>>>>> 
>>>>>>> This failed because tomli requires flit_core to be available.
>>>>>>> 
>>>>>>> flit_core also has no setup.py and so I tried to build it with pip install. This failed with the error message that tomli was required as a dependency. So I added --no-deps to the flit_core pip install line and this time flit_core successfully built but the tomli build still came back with not finding flit_core.
>>>>>>> 
>>>>>>> I have tried a variety of changes to the pip install line but none of them has been successful and it has become clear that I am out of my depth here and need guidance rather than just trying anything I find from a google search. I am not even really clear if pip install is the correct approach, except if not then I have no idea how to progress.
>>>>>>> 
>>>>>>> Help urgently needed on how I should be approaching this.
>>>>>> Circular dependencies can happen. This won’t be the first time and it always is an utter nightmare with our buildsystem.
>>>>>> This page states that pip comes by default with all binary packages:
>>>>>>   https://docs.python.org/3/installing/index.html
>>>>>> Is there a chance that it is part of the distribution and can be enabled with a configure switch?
>>>>> I wasn't able to find any way to get it enabled that way.
>>>>> 
>>>>> I did find that pip install will not install if the package is already present. You have to add --upgrade to make it update any package already present. So I tried that. Didn't work.
>>>>> 
>>>>> I found a section about needing to use bootstrapping if you want to install tomli. Followed the information for creating a wheel file for flit_core and then extracting that file into the site-packages directory. Tried that. Didn't work.
>>>>> 
>>>>> Rather than doing a complete build every time I changed to trying to build flit_core followed by tomli in a ./make.sh shell environment.
>>>>> 
>>>>> Doing that I found that I had a pip install command that was creating the flit_core directory in site-packages and also including the .dist-info directory. The info I found said that the .dist-info directory had to be there for pip to be able to find the flit_core package.
>>>>> 
>>>>> However the pip install of tomli still ends up not finding any flit_core package. In the bootstrapping information it mentioned about changing the PYTHONPATH environment variable for find the flit_core package so I tried that in the shell but it also made no difference.
>>>>> 
>>>>> I then tried running python3 -c 'import flit_core' in the tomli build shell before doing the tomli pip install. The import command completed with no error but tomli still could not find any flit_core package.
>>>>> 
>>>>> End result of all my work is that flit_core has been built and installed into python3.10/site-packages but nothing seems to want to make the tomli build find it.
>>>>> 
>>>>> Does anyone have any ideas on what command(s) I need to use in the tomli pip install to have the installed flit_core package visible.
>>>>> 
>>>>> Regards and a Happy and Healthy New Year to all,
>>>>> 
>>>>> Adolf.
>>>>> 
>>>>>> -Michael
>>>>>>> 
>>>>>>> Regards,
>>>>>>> Adolf.
>>>>>>> 
>>>>>>> PS:  I hope everyone had an enjoyable Christmas.
>>>>>> I hope you did, too!
>>>>>>> 
>>>>>>> 
>>>>>>>> -Michael
>>>>>>>>> 
>>>>>>>>> Regards,
>>>>>>>>> 
>>>>>>>>> Adolf.
>>>>>>>>> 
>>>>>>>>> -- 
>>>>>>>>> Sent from my laptop
>>>>>>>>> 
>>>>>>> 
>>>>>>> -- 
>>>>>>> Sent from my laptop
>>>>> 
>>>>> -- 
>>>>> Sent from my laptop
> 
> -- 
> Sent from my laptop

Re: Question about updating python to 3.10

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 13276 bytes --]

Hi Michael and all,

python3-six has to stay as python3-dateutil explicitly specifies it in 
its dependencies.
So that solves that question.

I have, with quite a bit of work, eventually been able to get all 
existing python modules updated that had updates and built successfully 
with python3.10
However later in the IPFire build libvirt failed with a problem parsing 
some documentation files.

Stashing my python3.10 changes and re-running the build had libvirt 
building without problems. So one of the python changes has caused it. 
libvirt uses python3-docutils and that was updated but reverting that 
back to original on its own did not cause libvirt to build successfully.

I looked in the libvirt .configure for an option to not build the docs 
but it is not mentioned under ./configure --help.

I tried --without-docs and --disable-docs but both were not recognised 
in the build.

The current libvirt is from mid 2020 (6.5.0) so it is likely that it is 
not compatible in some way with the updated python/modules.

I tried updating libvirt to the current 7.10.0
libvirt has replaced autotools with meson. I found meson options 
available for all existing ./configure options except three. There is 
nothing in meson options for:-
--with-virtualport
--with-macvtap
--without-dbus

I tried building it with all the other options and it successfully built 
with the updated python3.10 and modules. All other packages after 
libvirt built successfully without any problems.

Normally I would then use the updated libvirt, as that is needed to work 
with the updated python packages, but I am not sure if those missing 
options are now built into libvirt or are no longer available and what 
impact that might have for the use of libvirt on IPFire.

There is nothing in the libvirt changelog about virtualport, macvtap or 
dbus.

Searching on the internet, I was also not able to find anything related 
to libvirt and these options.

Any suggestions/proposals on how to move forward on this? Should I just 
build libvirt without those three options or ...?


Regards,

Adolf.

On 07/01/2022 17:49, Michael Tremer wrote:
> Hello,
> 
>> On 4 Jan 2022, at 12:49, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>
>> Hi Michael,
>>
>> Following the suggested Arch Linux approach I have got python3-setuptools-scm building successfully now. Yaaah.
>> I had to install python3-toml, followed by python3-pyproject2setuppy followed by python3-tomli.
>>
>> However then python3-six which installs after python3-setuptools-scm fails due to missing the python package python3-packaging.
>>
>> I can install that package, and will do so as I continue testing out the python 3.10 build but I have the following question about python3-six.
>>
>> Is six still needed. It is designed to ensure that programs can work with either python2 or python3 but IPFire only has python3 now.
>> I can find python3-six referenced only by aws-cli. In the aws-cli build python3-six is not listed at all in the build requirements. I tried to search for aws-cli and python3-six being needed in run mode but I could not find anything one way or the other.
> 
> If aws-cli no longer requires python3-six we can drop it. Last time we touched it I had to bring the package back because aws-cli was still depending on it.
> 
>> If python3-six is really needed for running aws-cli then I can install it and the new dependency but if it is not needed we could remove two packages which would partially offset the three new dependencies required to be able to build the latest python3-setuptools-scm.
> 
> Yes, that would be great :)
> 
> -Michael
> 
>>
>> Regards,
>> Adolf
>>
>> On 03/01/2022 12:32, Michael Tremer wrote:
>>> Hello,
>>>> On 3 Jan 2022, at 11:31, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>
>>>> Hi Michael,
>>>>
>>>> On 03/01/2022 11:52, Michael Tremer wrote:
>>>>> Hello,
>>>>> I am not sure if you already looked at what Arch is doing, but as far as I understand it works like this:
>>>> No I didn't look at Arch, which is surprising as I have in the past with other packages but I didn't. So thanks very much for the pointer.
>>>>> They build Python 3 without pip and setuptools (https://github.com/archlinux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/trunk/PKGBUILD#L59).
>>>>> At this stage we might need to build setuptools separately.
>>>>> They use a tool called pyproject2setuppy (great name!) to build tomli (https://github.com/archlinux/svntogit-packages/blob/packages/python-tomli/trunk/PKGBUILD).
>>>>> Setuptools-scm can then be built with tomli: https://github.com/archlinux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD
>>>>> Would this work or did I overlook anything?
>>>> I will have a look at the links and give it a go.
>>>>
>>>> Thanks very much, I feel re-motivated now.
>>> Very good! I got a little bit lost in your last email and I think I could tell. But following a solution that somebody else has come up with can be a good thing to follow and of course we would customise it to our needs.
>>> Best,
>>> -Michael
>>>>
>>>> Regards,
>>>> Adolf.
>>>>> -Michael
>>>>>> On 2 Jan 2022, at 13:44, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>
>>>>>> Hi Michael and All,
>>>>>>
>>>>>> On 26/12/2021 21:55, Michael Tremer wrote:
>>>>>>> Merry Christmas,
>>>>>>>> On 26 Dec 2021, at 15:38, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>
>>>>>>>> Hi Michael,
>>>>>>>>
>>>>>>>> On 17/12/2021 12:41, Michael Tremer wrote:
>>>>>>>>> Hello Adolf,
>>>>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi All,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I am working on updating python from 3.8 to 3.10. Python itself is okay after some other python related programs were also updated due to their older versions not working with 3.10
>>>>>>>>> Thank you for working on this.
>>>>>>>>>> I am working through all the programs that reference python-3.8 in their rootfiles, or wherever, to change them to 3.10
>>>>>>>>>>
>>>>>>>>>> My question comes with regard to the changes to the rootfiles to ensure they will be included properly in an update.
>>>>>>>>> We will have to ship everything that puts anything into /usr/lib*/python3.*/site-packages again with the release of Python 3.10.
>>>>>>>>> This isn’t fun, but not as much of a pain in the rear as shipping a new Perl release.
>>>>>>>>>> For the addons then when I change the rootfile I am also incrementing the PAK_VER number in the lfs files and that ensures that they will be properly updated.
>>>>>>>>> Yes. That is required for Pakfire to find a new version.
>>>>>>>>>> I don't know what to do for the core programs where I have updated the rootfile. The lfs files for these programs don't have a PAK_VER number to increment and the actual package version number is not being changed so I don't know what I need to change in the lfs to ensure that the rootfile changes will be properly included in the update?
>>>>>>>>> I would say committing everything into a “python-3.10” branch and submitting it all as a patchset is fine. All packages that have been touched in that branch will have to be shipped again which should be easy to identify if it all comes in one large branch.
>>>>>>>> Okay, that is clear then.Will follow that approach.
>>>>>>>>>> I would appreciate any help on what I need to do for these programs that have had changes only in their rootfiles.
>>>>>>>>> Absolutely. Did you find anything that didn’t build yet?
>>>>>>>>
>>>>>>>> I have not had any problem with python3.10 itself but I found some of the older modules would not work with 3.10 so I decided to do a combined update to 3.10 plus update modules to latest versions.
>>>>>>>>
>>>>>>>> I found that after python3-setuptools-scm had run that the next module python3-six failed to run due to missing the tomli library. I commented out six and reran the build and then python3-dateutil had the same problem  and then python3-jmespath
>>>>>>> I completely lost track of what the status of setuptools/distutils and so on is. This has changed multiple times and become independent/part of the distribution again and so on.
>>>>>>>> After some investigation I have found that since version 6.0.0 of python3-setuptools-scm, tomli has been a required library when it is being used by other modules. So running version 6.0.0 of setuptools-scm worked with python3.10 and subsequent modules built successfully as tomli not needed.
>>>>>>>>
>>>>>>>> So the obvious fix is to install tomli but that is when I ran into a problem. tomli has no setup.py or setup.cfg files in its tarball. The only way to build it is using pip.
>>>>>>> Err. Great.
>>>>>>>> Apparently, after some searching, setup.py has been deprecated as the recommended install method and if setup.py doesn't work the usual fix suggestion is to run pip install.
>>>>>>> Yes. But apparently the new system is deprecated, too. I really dislike the state of packaging systems that come with any language these days.
>>>>>>>> Having searched on pip for a while, it looks like builds using it can still be done from tarballs locally without needing to access the internet to download the modules.
>>>>>>> That is good!
>>>>>>>> As pip is installed as part of python3 i tried using that.
>>>>>>>>
>>>>>>>> I used "python3 -m pip install --root=/ ."  in place of
>>>>>>>> "python3 setup.py build
>>>>>>>> python3 setup.py install --root=/"
>>>>>>>> for the tomli source.
>>>>>>>>
>>>>>>>> This failed because tomli requires flit_core to be available.
>>>>>>>>
>>>>>>>> flit_core also has no setup.py and so I tried to build it with pip install. This failed with the error message that tomli was required as a dependency. So I added --no-deps to the flit_core pip install line and this time flit_core successfully built but the tomli build still came back with not finding flit_core.
>>>>>>>>
>>>>>>>> I have tried a variety of changes to the pip install line but none of them has been successful and it has become clear that I am out of my depth here and need guidance rather than just trying anything I find from a google search. I am not even really clear if pip install is the correct approach, except if not then I have no idea how to progress.
>>>>>>>>
>>>>>>>> Help urgently needed on how I should be approaching this.
>>>>>>> Circular dependencies can happen. This won’t be the first time and it always is an utter nightmare with our buildsystem.
>>>>>>> This page states that pip comes by default with all binary packages:
>>>>>>>    https://docs.python.org/3/installing/index.html
>>>>>>> Is there a chance that it is part of the distribution and can be enabled with a configure switch?
>>>>>> I wasn't able to find any way to get it enabled that way.
>>>>>>
>>>>>> I did find that pip install will not install if the package is already present. You have to add --upgrade to make it update any package already present. So I tried that. Didn't work.
>>>>>>
>>>>>> I found a section about needing to use bootstrapping if you want to install tomli. Followed the information for creating a wheel file for flit_core and then extracting that file into the site-packages directory. Tried that. Didn't work.
>>>>>>
>>>>>> Rather than doing a complete build every time I changed to trying to build flit_core followed by tomli in a ./make.sh shell environment.
>>>>>>
>>>>>> Doing that I found that I had a pip install command that was creating the flit_core directory in site-packages and also including the .dist-info directory. The info I found said that the .dist-info directory had to be there for pip to be able to find the flit_core package.
>>>>>>
>>>>>> However the pip install of tomli still ends up not finding any flit_core package. In the bootstrapping information it mentioned about changing the PYTHONPATH environment variable for find the flit_core package so I tried that in the shell but it also made no difference.
>>>>>>
>>>>>> I then tried running python3 -c 'import flit_core' in the tomli build shell before doing the tomli pip install. The import command completed with no error but tomli still could not find any flit_core package.
>>>>>>
>>>>>> End result of all my work is that flit_core has been built and installed into python3.10/site-packages but nothing seems to want to make the tomli build find it.
>>>>>>
>>>>>> Does anyone have any ideas on what command(s) I need to use in the tomli pip install to have the installed flit_core package visible.
>>>>>>
>>>>>> Regards and a Happy and Healthy New Year to all,
>>>>>>
>>>>>> Adolf.
>>>>>>
>>>>>>> -Michael
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Adolf.
>>>>>>>>
>>>>>>>> PS:  I hope everyone had an enjoyable Christmas.
>>>>>>> I hope you did, too!
>>>>>>>>
>>>>>>>>
>>>>>>>>> -Michael
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>> Adolf.
>>>>>>>>>>
>>>>>>>>>> -- 
>>>>>>>>>> Sent from my laptop
>>>>>>>>>>
>>>>>>>>
>>>>>>>> -- 
>>>>>>>> Sent from my laptop
>>>>>>
>>>>>> -- 
>>>>>> Sent from my laptop
>>
>> -- 
>> Sent from my laptop
> 

-- 
Sent from my laptop

Re: Question about updating python to 3.10

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 14190 bytes --]

Hey,

> On 11 Jan 2022, at 14:32, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi Michael and all,
> 
> python3-six has to stay as python3-dateutil explicitly specifies it in its dependencies.
> So that solves that question.

Aww :(

> I have, with quite a bit of work, eventually been able to get all existing python modules updated that had updates and built successfully with python3.10

Yay!

> However later in the IPFire build libvirt failed with a problem parsing some documentation files.

> Stashing my python3.10 changes and re-running the build had libvirt building without problems. So one of the python changes has caused it. libvirt uses python3-docutils and that was updated but reverting that back to original on its own did not cause libvirt to build successfully.
> 
> I looked in the libvirt .configure for an option to not build the docs but it is not mentioned under ./configure --help.
> 
> I tried --without-docs and --disable-docs but both were not recognised in the build.
> 
> The current libvirt is from mid 2020 (6.5.0) so it is likely that it is not compatible in some way with the updated python/modules.
> 
> I tried updating libvirt to the current 7.10.0
> libvirt has replaced autotools with meson. I found meson options available for all existing ./configure options except three. There is nothing in meson options for:-
> --with-virtualport
> --with-macvtap
> --without-dbus

Yay, another project that is using meson :(

We no longer need macvtap; we have dubs, so that should not hurt; and I have no idea what virtualport is.

> I tried building it with all the other options and it successfully built with the updated python3.10 and modules. All other packages after libvirt built successfully without any problems.
> 
> Normally I would then use the updated libvirt, as that is needed to work with the updated python packages, but I am not sure if those missing options are now built into libvirt or are no longer available and what impact that might have for the use of libvirt on IPFire.

I don’t think this would hurt us much. I have CCed Jonatan who maintains this, so maybe he knows more about this.

> There is nothing in the libvirt changelog about virtualport, macvtap or dbus.
> 
> Searching on the internet, I was also not able to find anything related to libvirt and these options.
> 
> Any suggestions/proposals on how to move forward on this? Should I just build libvirt without those three options or ...?

Yes, and I think this could be submitted independently of the Python branch because it is an add-on. We should be able to ship this at any time and might want to delay Python until we have enough space in a core update.

Thank you for looking into this. Sounds like a not so nice riddle to solve :)

-Michael

> 
> 
> Regards,
> 
> Adolf.
> 
> On 07/01/2022 17:49, Michael Tremer wrote:
>> Hello,
>>> On 4 Jan 2022, at 12:49, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>> 
>>> Hi Michael,
>>> 
>>> Following the suggested Arch Linux approach I have got python3-setuptools-scm building successfully now. Yaaah.
>>> I had to install python3-toml, followed by python3-pyproject2setuppy followed by python3-tomli.
>>> 
>>> However then python3-six which installs after python3-setuptools-scm fails due to missing the python package python3-packaging.
>>> 
>>> I can install that package, and will do so as I continue testing out the python 3.10 build but I have the following question about python3-six.
>>> 
>>> Is six still needed. It is designed to ensure that programs can work with either python2 or python3 but IPFire only has python3 now.
>>> I can find python3-six referenced only by aws-cli. In the aws-cli build python3-six is not listed at all in the build requirements. I tried to search for aws-cli and python3-six being needed in run mode but I could not find anything one way or the other.
>> If aws-cli no longer requires python3-six we can drop it. Last time we touched it I had to bring the package back because aws-cli was still depending on it.
>>> If python3-six is really needed for running aws-cli then I can install it and the new dependency but if it is not needed we could remove two packages which would partially offset the three new dependencies required to be able to build the latest python3-setuptools-scm.
>> Yes, that would be great :)
>> -Michael
>>> 
>>> Regards,
>>> Adolf
>>> 
>>> On 03/01/2022 12:32, Michael Tremer wrote:
>>>> Hello,
>>>>> On 3 Jan 2022, at 11:31, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>> 
>>>>> Hi Michael,
>>>>> 
>>>>> On 03/01/2022 11:52, Michael Tremer wrote:
>>>>>> Hello,
>>>>>> I am not sure if you already looked at what Arch is doing, but as far as I understand it works like this:
>>>>> No I didn't look at Arch, which is surprising as I have in the past with other packages but I didn't. So thanks very much for the pointer.
>>>>>> They build Python 3 without pip and setuptools (https://github.com/archlinux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/trunk/PKGBUILD#L59).
>>>>>> At this stage we might need to build setuptools separately.
>>>>>> They use a tool called pyproject2setuppy (great name!) to build tomli (https://github.com/archlinux/svntogit-packages/blob/packages/python-tomli/trunk/PKGBUILD).
>>>>>> Setuptools-scm can then be built with tomli: https://github.com/archlinux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD
>>>>>> Would this work or did I overlook anything?
>>>>> I will have a look at the links and give it a go.
>>>>> 
>>>>> Thanks very much, I feel re-motivated now.
>>>> Very good! I got a little bit lost in your last email and I think I could tell. But following a solution that somebody else has come up with can be a good thing to follow and of course we would customise it to our needs.
>>>> Best,
>>>> -Michael
>>>>> 
>>>>> Regards,
>>>>> Adolf.
>>>>>> -Michael
>>>>>>> On 2 Jan 2022, at 13:44, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>> 
>>>>>>> Hi Michael and All,
>>>>>>> 
>>>>>>> On 26/12/2021 21:55, Michael Tremer wrote:
>>>>>>>> Merry Christmas,
>>>>>>>>> On 26 Dec 2021, at 15:38, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>> 
>>>>>>>>> Hi Michael,
>>>>>>>>> 
>>>>>>>>> On 17/12/2021 12:41, Michael Tremer wrote:
>>>>>>>>>> Hello Adolf,
>>>>>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Hi All,
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> I am working on updating python from 3.8 to 3.10. Python itself is okay after some other python related programs were also updated due to their older versions not working with 3.10
>>>>>>>>>> Thank you for working on this.
>>>>>>>>>>> I am working through all the programs that reference python-3.8 in their rootfiles, or wherever, to change them to 3.10
>>>>>>>>>>> 
>>>>>>>>>>> My question comes with regard to the changes to the rootfiles to ensure they will be included properly in an update.
>>>>>>>>>> We will have to ship everything that puts anything into /usr/lib*/python3.*/site-packages again with the release of Python 3.10.
>>>>>>>>>> This isn’t fun, but not as much of a pain in the rear as shipping a new Perl release.
>>>>>>>>>>> For the addons then when I change the rootfile I am also incrementing the PAK_VER number in the lfs files and that ensures that they will be properly updated.
>>>>>>>>>> Yes. That is required for Pakfire to find a new version.
>>>>>>>>>>> I don't know what to do for the core programs where I have updated the rootfile. The lfs files for these programs don't have a PAK_VER number to increment and the actual package version number is not being changed so I don't know what I need to change in the lfs to ensure that the rootfile changes will be properly included in the update?
>>>>>>>>>> I would say committing everything into a “python-3.10” branch and submitting it all as a patchset is fine. All packages that have been touched in that branch will have to be shipped again which should be easy to identify if it all comes in one large branch.
>>>>>>>>> Okay, that is clear then.Will follow that approach.
>>>>>>>>>>> I would appreciate any help on what I need to do for these programs that have had changes only in their rootfiles.
>>>>>>>>>> Absolutely. Did you find anything that didn’t build yet?
>>>>>>>>> 
>>>>>>>>> I have not had any problem with python3.10 itself but I found some of the older modules would not work with 3.10 so I decided to do a combined update to 3.10 plus update modules to latest versions.
>>>>>>>>> 
>>>>>>>>> I found that after python3-setuptools-scm had run that the next module python3-six failed to run due to missing the tomli library. I commented out six and reran the build and then python3-dateutil had the same problem  and then python3-jmespath
>>>>>>>> I completely lost track of what the status of setuptools/distutils and so on is. This has changed multiple times and become independent/part of the distribution again and so on.
>>>>>>>>> After some investigation I have found that since version 6.0.0 of python3-setuptools-scm, tomli has been a required library when it is being used by other modules. So running version 6.0.0 of setuptools-scm worked with python3.10 and subsequent modules built successfully as tomli not needed.
>>>>>>>>> 
>>>>>>>>> So the obvious fix is to install tomli but that is when I ran into a problem. tomli has no setup.py or setup.cfg files in its tarball. The only way to build it is using pip.
>>>>>>>> Err. Great.
>>>>>>>>> Apparently, after some searching, setup.py has been deprecated as the recommended install method and if setup.py doesn't work the usual fix suggestion is to run pip install.
>>>>>>>> Yes. But apparently the new system is deprecated, too. I really dislike the state of packaging systems that come with any language these days.
>>>>>>>>> Having searched on pip for a while, it looks like builds using it can still be done from tarballs locally without needing to access the internet to download the modules.
>>>>>>>> That is good!
>>>>>>>>> As pip is installed as part of python3 i tried using that.
>>>>>>>>> 
>>>>>>>>> I used "python3 -m pip install --root=/ ."  in place of
>>>>>>>>> "python3 setup.py build
>>>>>>>>> python3 setup.py install --root=/"
>>>>>>>>> for the tomli source.
>>>>>>>>> 
>>>>>>>>> This failed because tomli requires flit_core to be available.
>>>>>>>>> 
>>>>>>>>> flit_core also has no setup.py and so I tried to build it with pip install. This failed with the error message that tomli was required as a dependency. So I added --no-deps to the flit_core pip install line and this time flit_core successfully built but the tomli build still came back with not finding flit_core.
>>>>>>>>> 
>>>>>>>>> I have tried a variety of changes to the pip install line but none of them has been successful and it has become clear that I am out of my depth here and need guidance rather than just trying anything I find from a google search. I am not even really clear if pip install is the correct approach, except if not then I have no idea how to progress.
>>>>>>>>> 
>>>>>>>>> Help urgently needed on how I should be approaching this.
>>>>>>>> Circular dependencies can happen. This won’t be the first time and it always is an utter nightmare with our buildsystem.
>>>>>>>> This page states that pip comes by default with all binary packages:
>>>>>>>>   https://docs.python.org/3/installing/index.html
>>>>>>>> Is there a chance that it is part of the distribution and can be enabled with a configure switch?
>>>>>>> I wasn't able to find any way to get it enabled that way.
>>>>>>> 
>>>>>>> I did find that pip install will not install if the package is already present. You have to add --upgrade to make it update any package already present. So I tried that. Didn't work.
>>>>>>> 
>>>>>>> I found a section about needing to use bootstrapping if you want to install tomli. Followed the information for creating a wheel file for flit_core and then extracting that file into the site-packages directory. Tried that. Didn't work.
>>>>>>> 
>>>>>>> Rather than doing a complete build every time I changed to trying to build flit_core followed by tomli in a ./make.sh shell environment.
>>>>>>> 
>>>>>>> Doing that I found that I had a pip install command that was creating the flit_core directory in site-packages and also including the .dist-info directory. The info I found said that the .dist-info directory had to be there for pip to be able to find the flit_core package.
>>>>>>> 
>>>>>>> However the pip install of tomli still ends up not finding any flit_core package. In the bootstrapping information it mentioned about changing the PYTHONPATH environment variable for find the flit_core package so I tried that in the shell but it also made no difference.
>>>>>>> 
>>>>>>> I then tried running python3 -c 'import flit_core' in the tomli build shell before doing the tomli pip install. The import command completed with no error but tomli still could not find any flit_core package.
>>>>>>> 
>>>>>>> End result of all my work is that flit_core has been built and installed into python3.10/site-packages but nothing seems to want to make the tomli build find it.
>>>>>>> 
>>>>>>> Does anyone have any ideas on what command(s) I need to use in the tomli pip install to have the installed flit_core package visible.
>>>>>>> 
>>>>>>> Regards and a Happy and Healthy New Year to all,
>>>>>>> 
>>>>>>> Adolf.
>>>>>>> 
>>>>>>>> -Michael
>>>>>>>>> 
>>>>>>>>> Regards,
>>>>>>>>> Adolf.
>>>>>>>>> 
>>>>>>>>> PS:  I hope everyone had an enjoyable Christmas.
>>>>>>>> I hope you did, too!
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> -Michael
>>>>>>>>>>> 
>>>>>>>>>>> Regards,
>>>>>>>>>>> 
>>>>>>>>>>> Adolf.
>>>>>>>>>>> 
>>>>>>>>>>> -- 
>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> -- 
>>>>>>>>> Sent from my laptop
>>>>>>> 
>>>>>>> -- 
>>>>>>> Sent from my laptop
>>> 
>>> -- 
>>> Sent from my laptop
> 
> -- 
> Sent from my laptop

Re: Question about updating python to 3.10

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 15484 bytes --]

Hi all,

On 12/01/2022 19:10, Michael Tremer wrote:
> Hey,
> 
>> On 11 Jan 2022, at 14:32, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>
>> Hi Michael and all,
>>
>> python3-six has to stay as python3-dateutil explicitly specifies it in its dependencies.
>> So that solves that question.
> 
> Aww :(
> 
>> I have, with quite a bit of work, eventually been able to get all existing python modules updated that had updates and built successfully with python3.10
> 
> Yay!
> 
>> However later in the IPFire build libvirt failed with a problem parsing some documentation files.
> 
>> Stashing my python3.10 changes and re-running the build had libvirt building without problems. So one of the python changes has caused it. libvirt uses python3-docutils and that was updated but reverting that back to original on its own did not cause libvirt to build successfully.
>>
>> I looked in the libvirt .configure for an option to not build the docs but it is not mentioned under ./configure --help.
>>
>> I tried --without-docs and --disable-docs but both were not recognised in the build.
>>
>> The current libvirt is from mid 2020 (6.5.0) so it is likely that it is not compatible in some way with the updated python/modules.
>>
>> I tried updating libvirt to the current 7.10.0
>> libvirt has replaced autotools with meson. I found meson options available for all existing ./configure options except three. There is nothing in meson options for:-
>> --with-virtualport
>> --with-macvtap
>> --without-dbus
> 
> Yay, another project that is using meson :(
> 
> We no longer need macvtap; we have dubs, so that should not hurt; and I have no idea what virtualport is.
> 
>> I tried building it with all the other options and it successfully built with the updated python3.10 and modules. All other packages after libvirt built successfully without any problems.
>>
>> Normally I would then use the updated libvirt, as that is needed to work with the updated python packages, but I am not sure if those missing options are now built into libvirt or are no longer available and what impact that might have for the use of libvirt on IPFire.
> 
> I don’t think this would hurt us much. I have CCed Jonatan who maintains this, so maybe he knows more about this.
> 
>> There is nothing in the libvirt changelog about virtualport, macvtap or dbus.
>>
>> Searching on the internet, I was also not able to find anything related to libvirt and these options.
>>
>> Any suggestions/proposals on how to move forward on this? Should I just build libvirt without those three options or ...?
> 
> Yes, and I think this could be submitted independently of the Python branch because it is an add-on. We should be able to ship this at any time and might want to delay Python until we have enough space in a core update.
Okay, then I will stash my python3.10 changes and submit the libvirt 
update on its own. Then Jonaton can test it out. I don't use libvirt or 
qemu so I can't test it easily.

Then when you think the time is right, let me know and I will submit the 
python3.10 series of patches.
> 
> Thank you for looking into this. Sounds like a not so nice riddle to solve :)
There were certainly some challenges, like updating python3-daemon which 
then wanted to have twine as a fixed dependency which had nine 
dependencies some of which had more. I had got to 19 additional python 
packages and still going and then I said enough was enough and looked at 
twine. It is required for uploading python packages to PyPI which IPFire 
doesn't need. So I tried patching the twine requirement out and that 
worked and then I found others that had done the same because they had 
no response to the request to not have twine as a mandatory build 
requirement.

I found it all challenging but rewarding when I made progress and 
eventually succeeded in building everything without errors.

Regards,
Adolf.
> 
> -Michael
> 
>>
>>
>> Regards,
>>
>> Adolf.
>>
>> On 07/01/2022 17:49, Michael Tremer wrote:
>>> Hello,
>>>> On 4 Jan 2022, at 12:49, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>
>>>> Hi Michael,
>>>>
>>>> Following the suggested Arch Linux approach I have got python3-setuptools-scm building successfully now. Yaaah.
>>>> I had to install python3-toml, followed by python3-pyproject2setuppy followed by python3-tomli.
>>>>
>>>> However then python3-six which installs after python3-setuptools-scm fails due to missing the python package python3-packaging.
>>>>
>>>> I can install that package, and will do so as I continue testing out the python 3.10 build but I have the following question about python3-six.
>>>>
>>>> Is six still needed. It is designed to ensure that programs can work with either python2 or python3 but IPFire only has python3 now.
>>>> I can find python3-six referenced only by aws-cli. In the aws-cli build python3-six is not listed at all in the build requirements. I tried to search for aws-cli and python3-six being needed in run mode but I could not find anything one way or the other.
>>> If aws-cli no longer requires python3-six we can drop it. Last time we touched it I had to bring the package back because aws-cli was still depending on it.
>>>> If python3-six is really needed for running aws-cli then I can install it and the new dependency but if it is not needed we could remove two packages which would partially offset the three new dependencies required to be able to build the latest python3-setuptools-scm.
>>> Yes, that would be great :)
>>> -Michael
>>>>
>>>> Regards,
>>>> Adolf
>>>>
>>>> On 03/01/2022 12:32, Michael Tremer wrote:
>>>>> Hello,
>>>>>> On 3 Jan 2022, at 11:31, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>
>>>>>> Hi Michael,
>>>>>>
>>>>>> On 03/01/2022 11:52, Michael Tremer wrote:
>>>>>>> Hello,
>>>>>>> I am not sure if you already looked at what Arch is doing, but as far as I understand it works like this:
>>>>>> No I didn't look at Arch, which is surprising as I have in the past with other packages but I didn't. So thanks very much for the pointer.
>>>>>>> They build Python 3 without pip and setuptools (https://github.com/archlinux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/trunk/PKGBUILD#L59).
>>>>>>> At this stage we might need to build setuptools separately.
>>>>>>> They use a tool called pyproject2setuppy (great name!) to build tomli (https://github.com/archlinux/svntogit-packages/blob/packages/python-tomli/trunk/PKGBUILD).
>>>>>>> Setuptools-scm can then be built with tomli: https://github.com/archlinux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD
>>>>>>> Would this work or did I overlook anything?
>>>>>> I will have a look at the links and give it a go.
>>>>>>
>>>>>> Thanks very much, I feel re-motivated now.
>>>>> Very good! I got a little bit lost in your last email and I think I could tell. But following a solution that somebody else has come up with can be a good thing to follow and of course we would customise it to our needs.
>>>>> Best,
>>>>> -Michael
>>>>>>
>>>>>> Regards,
>>>>>> Adolf.
>>>>>>> -Michael
>>>>>>>> On 2 Jan 2022, at 13:44, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>
>>>>>>>> Hi Michael and All,
>>>>>>>>
>>>>>>>> On 26/12/2021 21:55, Michael Tremer wrote:
>>>>>>>>> Merry Christmas,
>>>>>>>>>> On 26 Dec 2021, at 15:38, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Michael,
>>>>>>>>>>
>>>>>>>>>> On 17/12/2021 12:41, Michael Tremer wrote:
>>>>>>>>>>> Hello Adolf,
>>>>>>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I am working on updating python from 3.8 to 3.10. Python itself is okay after some other python related programs were also updated due to their older versions not working with 3.10
>>>>>>>>>>> Thank you for working on this.
>>>>>>>>>>>> I am working through all the programs that reference python-3.8 in their rootfiles, or wherever, to change them to 3.10
>>>>>>>>>>>>
>>>>>>>>>>>> My question comes with regard to the changes to the rootfiles to ensure they will be included properly in an update.
>>>>>>>>>>> We will have to ship everything that puts anything into /usr/lib*/python3.*/site-packages again with the release of Python 3.10.
>>>>>>>>>>> This isn’t fun, but not as much of a pain in the rear as shipping a new Perl release.
>>>>>>>>>>>> For the addons then when I change the rootfile I am also incrementing the PAK_VER number in the lfs files and that ensures that they will be properly updated.
>>>>>>>>>>> Yes. That is required for Pakfire to find a new version.
>>>>>>>>>>>> I don't know what to do for the core programs where I have updated the rootfile. The lfs files for these programs don't have a PAK_VER number to increment and the actual package version number is not being changed so I don't know what I need to change in the lfs to ensure that the rootfile changes will be properly included in the update?
>>>>>>>>>>> I would say committing everything into a “python-3.10” branch and submitting it all as a patchset is fine. All packages that have been touched in that branch will have to be shipped again which should be easy to identify if it all comes in one large branch.
>>>>>>>>>> Okay, that is clear then.Will follow that approach.
>>>>>>>>>>>> I would appreciate any help on what I need to do for these programs that have had changes only in their rootfiles.
>>>>>>>>>>> Absolutely. Did you find anything that didn’t build yet?
>>>>>>>>>>
>>>>>>>>>> I have not had any problem with python3.10 itself but I found some of the older modules would not work with 3.10 so I decided to do a combined update to 3.10 plus update modules to latest versions.
>>>>>>>>>>
>>>>>>>>>> I found that after python3-setuptools-scm had run that the next module python3-six failed to run due to missing the tomli library. I commented out six and reran the build and then python3-dateutil had the same problem  and then python3-jmespath
>>>>>>>>> I completely lost track of what the status of setuptools/distutils and so on is. This has changed multiple times and become independent/part of the distribution again and so on.
>>>>>>>>>> After some investigation I have found that since version 6.0.0 of python3-setuptools-scm, tomli has been a required library when it is being used by other modules. So running version 6.0.0 of setuptools-scm worked with python3.10 and subsequent modules built successfully as tomli not needed.
>>>>>>>>>>
>>>>>>>>>> So the obvious fix is to install tomli but that is when I ran into a problem. tomli has no setup.py or setup.cfg files in its tarball. The only way to build it is using pip.
>>>>>>>>> Err. Great.
>>>>>>>>>> Apparently, after some searching, setup.py has been deprecated as the recommended install method and if setup.py doesn't work the usual fix suggestion is to run pip install.
>>>>>>>>> Yes. But apparently the new system is deprecated, too. I really dislike the state of packaging systems that come with any language these days.
>>>>>>>>>> Having searched on pip for a while, it looks like builds using it can still be done from tarballs locally without needing to access the internet to download the modules.
>>>>>>>>> That is good!
>>>>>>>>>> As pip is installed as part of python3 i tried using that.
>>>>>>>>>>
>>>>>>>>>> I used "python3 -m pip install --root=/ ."  in place of
>>>>>>>>>> "python3 setup.py build
>>>>>>>>>> python3 setup.py install --root=/"
>>>>>>>>>> for the tomli source.
>>>>>>>>>>
>>>>>>>>>> This failed because tomli requires flit_core to be available.
>>>>>>>>>>
>>>>>>>>>> flit_core also has no setup.py and so I tried to build it with pip install. This failed with the error message that tomli was required as a dependency. So I added --no-deps to the flit_core pip install line and this time flit_core successfully built but the tomli build still came back with not finding flit_core.
>>>>>>>>>>
>>>>>>>>>> I have tried a variety of changes to the pip install line but none of them has been successful and it has become clear that I am out of my depth here and need guidance rather than just trying anything I find from a google search. I am not even really clear if pip install is the correct approach, except if not then I have no idea how to progress.
>>>>>>>>>>
>>>>>>>>>> Help urgently needed on how I should be approaching this.
>>>>>>>>> Circular dependencies can happen. This won’t be the first time and it always is an utter nightmare with our buildsystem.
>>>>>>>>> This page states that pip comes by default with all binary packages:
>>>>>>>>>    https://docs.python.org/3/installing/index.html
>>>>>>>>> Is there a chance that it is part of the distribution and can be enabled with a configure switch?
>>>>>>>> I wasn't able to find any way to get it enabled that way.
>>>>>>>>
>>>>>>>> I did find that pip install will not install if the package is already present. You have to add --upgrade to make it update any package already present. So I tried that. Didn't work.
>>>>>>>>
>>>>>>>> I found a section about needing to use bootstrapping if you want to install tomli. Followed the information for creating a wheel file for flit_core and then extracting that file into the site-packages directory. Tried that. Didn't work.
>>>>>>>>
>>>>>>>> Rather than doing a complete build every time I changed to trying to build flit_core followed by tomli in a ./make.sh shell environment.
>>>>>>>>
>>>>>>>> Doing that I found that I had a pip install command that was creating the flit_core directory in site-packages and also including the .dist-info directory. The info I found said that the .dist-info directory had to be there for pip to be able to find the flit_core package.
>>>>>>>>
>>>>>>>> However the pip install of tomli still ends up not finding any flit_core package. In the bootstrapping information it mentioned about changing the PYTHONPATH environment variable for find the flit_core package so I tried that in the shell but it also made no difference.
>>>>>>>>
>>>>>>>> I then tried running python3 -c 'import flit_core' in the tomli build shell before doing the tomli pip install. The import command completed with no error but tomli still could not find any flit_core package.
>>>>>>>>
>>>>>>>> End result of all my work is that flit_core has been built and installed into python3.10/site-packages but nothing seems to want to make the tomli build find it.
>>>>>>>>
>>>>>>>> Does anyone have any ideas on what command(s) I need to use in the tomli pip install to have the installed flit_core package visible.
>>>>>>>>
>>>>>>>> Regards and a Happy and Healthy New Year to all,
>>>>>>>>
>>>>>>>> Adolf.
>>>>>>>>
>>>>>>>>> -Michael
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Adolf.
>>>>>>>>>>
>>>>>>>>>> PS:  I hope everyone had an enjoyable Christmas.
>>>>>>>>> I hope you did, too!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> -Michael
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>
>>>>>>>>>>>> -- 
>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -- 
>>>>>>>>>> Sent from my laptop
>>>>>>>>
>>>>>>>> -- 
>>>>>>>> Sent from my laptop
>>>>
>>>> -- 
>>>> Sent from my laptop
>>
>> -- 
>> Sent from my laptop
> 

-- 
Sent from my laptop

Re: Question about updating python to 3.10

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 16806 bytes --]

Hello,

> On 12 Jan 2022, at 21:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi all,
> 
> On 12/01/2022 19:10, Michael Tremer wrote:
>> Hey,
>>> On 11 Jan 2022, at 14:32, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>> 
>>> Hi Michael and all,
>>> 
>>> python3-six has to stay as python3-dateutil explicitly specifies it in its dependencies.
>>> So that solves that question.
>> Aww :(
>>> I have, with quite a bit of work, eventually been able to get all existing python modules updated that had updates and built successfully with python3.10
>> Yay!
>>> However later in the IPFire build libvirt failed with a problem parsing some documentation files.
>>> Stashing my python3.10 changes and re-running the build had libvirt building without problems. So one of the python changes has caused it. libvirt uses python3-docutils and that was updated but reverting that back to original on its own did not cause libvirt to build successfully.
>>> 
>>> I looked in the libvirt .configure for an option to not build the docs but it is not mentioned under ./configure --help.
>>> 
>>> I tried --without-docs and --disable-docs but both were not recognised in the build.
>>> 
>>> The current libvirt is from mid 2020 (6.5.0) so it is likely that it is not compatible in some way with the updated python/modules.
>>> 
>>> I tried updating libvirt to the current 7.10.0
>>> libvirt has replaced autotools with meson. I found meson options available for all existing ./configure options except three. There is nothing in meson options for:-
>>> --with-virtualport
>>> --with-macvtap
>>> --without-dbus
>> Yay, another project that is using meson :(
>> We no longer need macvtap; we have dubs, so that should not hurt; and I have no idea what virtualport is.
>>> I tried building it with all the other options and it successfully built with the updated python3.10 and modules. All other packages after libvirt built successfully without any problems.
>>> 
>>> Normally I would then use the updated libvirt, as that is needed to work with the updated python packages, but I am not sure if those missing options are now built into libvirt or are no longer available and what impact that might have for the use of libvirt on IPFire.
>> I don’t think this would hurt us much. I have CCed Jonatan who maintains this, so maybe he knows more about this.
>>> There is nothing in the libvirt changelog about virtualport, macvtap or dbus.
>>> 
>>> Searching on the internet, I was also not able to find anything related to libvirt and these options.
>>> 
>>> Any suggestions/proposals on how to move forward on this? Should I just build libvirt without those three options or ...?
>> Yes, and I think this could be submitted independently of the Python branch because it is an add-on. We should be able to ship this at any time and might want to delay Python until we have enough space in a core update.
> Okay, then I will stash my python3.10 changes and submit the libvirt update on its own. Then Jonaton can test it out. I don't use libvirt or qemu so I can't test it easily.
> 
> Then when you think the time is right, let me know and I will submit the python3.10 series of patches.

You can submit this at any time. I was just thinking about the conversation between Peter and Arne during the last call regarding an updated kernel and updated firmware. Depending on the size of Python, all of this might make an update a little bit too large - however, this should not be a patchset that we cannot merge within two merge windows.

>> Thank you for looking into this. Sounds like a not so nice riddle to solve :)
> There were certainly some challenges, like updating python3-daemon which then wanted to have twine as a fixed dependency which had nine dependencies some of which had more. I had got to 19 additional python packages and still going and then I said enough was enough and looked at twine. It is required for uploading python packages to PyPI which IPFire doesn't need. So I tried patching the twine requirement out and that worked and then I found others that had done the same because they had no response to the request to not have twine as a mandatory build requirement.

Python seems to become the hell that Perl used to be. In order to add one perl package, you had 20 dependencies. And those dependencies had their own dependencies too… It would be great to disable or patch out whatever is possible so that we do not create more work for ourselves in the future and have to update packages that are barely used - we already have enough of these :)

> I found it all challenging but rewarding when I made progress and eventually succeeded in building everything without errors.

I am so happy that you are putting all this time and effort into this because it certainly requires patience - a trade I don’t have.

And Python is being used in all sorts of places in the distribution, so it really isn’t anything we should not be up to date on.

-Michael

> 
> Regards,
> Adolf.
>> -Michael
>>> 
>>> 
>>> Regards,
>>> 
>>> Adolf.
>>> 
>>> On 07/01/2022 17:49, Michael Tremer wrote:
>>>> Hello,
>>>>> On 4 Jan 2022, at 12:49, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>> 
>>>>> Hi Michael,
>>>>> 
>>>>> Following the suggested Arch Linux approach I have got python3-setuptools-scm building successfully now. Yaaah.
>>>>> I had to install python3-toml, followed by python3-pyproject2setuppy followed by python3-tomli.
>>>>> 
>>>>> However then python3-six which installs after python3-setuptools-scm fails due to missing the python package python3-packaging.
>>>>> 
>>>>> I can install that package, and will do so as I continue testing out the python 3.10 build but I have the following question about python3-six.
>>>>> 
>>>>> Is six still needed. It is designed to ensure that programs can work with either python2 or python3 but IPFire only has python3 now.
>>>>> I can find python3-six referenced only by aws-cli. In the aws-cli build python3-six is not listed at all in the build requirements. I tried to search for aws-cli and python3-six being needed in run mode but I could not find anything one way or the other.
>>>> If aws-cli no longer requires python3-six we can drop it. Last time we touched it I had to bring the package back because aws-cli was still depending on it.
>>>>> If python3-six is really needed for running aws-cli then I can install it and the new dependency but if it is not needed we could remove two packages which would partially offset the three new dependencies required to be able to build the latest python3-setuptools-scm.
>>>> Yes, that would be great :)
>>>> -Michael
>>>>> 
>>>>> Regards,
>>>>> Adolf
>>>>> 
>>>>> On 03/01/2022 12:32, Michael Tremer wrote:
>>>>>> Hello,
>>>>>>> On 3 Jan 2022, at 11:31, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>> 
>>>>>>> Hi Michael,
>>>>>>> 
>>>>>>> On 03/01/2022 11:52, Michael Tremer wrote:
>>>>>>>> Hello,
>>>>>>>> I am not sure if you already looked at what Arch is doing, but as far as I understand it works like this:
>>>>>>> No I didn't look at Arch, which is surprising as I have in the past with other packages but I didn't. So thanks very much for the pointer.
>>>>>>>> They build Python 3 without pip and setuptools (https://github.com/archlinux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/trunk/PKGBUILD#L59).
>>>>>>>> At this stage we might need to build setuptools separately.
>>>>>>>> They use a tool called pyproject2setuppy (great name!) to build tomli (https://github.com/archlinux/svntogit-packages/blob/packages/python-tomli/trunk/PKGBUILD).
>>>>>>>> Setuptools-scm can then be built with tomli: https://github.com/archlinux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD
>>>>>>>> Would this work or did I overlook anything?
>>>>>>> I will have a look at the links and give it a go.
>>>>>>> 
>>>>>>> Thanks very much, I feel re-motivated now.
>>>>>> Very good! I got a little bit lost in your last email and I think I could tell. But following a solution that somebody else has come up with can be a good thing to follow and of course we would customise it to our needs.
>>>>>> Best,
>>>>>> -Michael
>>>>>>> 
>>>>>>> Regards,
>>>>>>> Adolf.
>>>>>>>> -Michael
>>>>>>>>> On 2 Jan 2022, at 13:44, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>> 
>>>>>>>>> Hi Michael and All,
>>>>>>>>> 
>>>>>>>>> On 26/12/2021 21:55, Michael Tremer wrote:
>>>>>>>>>> Merry Christmas,
>>>>>>>>>>> On 26 Dec 2021, at 15:38, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Hi Michael,
>>>>>>>>>>> 
>>>>>>>>>>> On 17/12/2021 12:41, Michael Tremer wrote:
>>>>>>>>>>>> Hello Adolf,
>>>>>>>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I am working on updating python from 3.8 to 3.10. Python itself is okay after some other python related programs were also updated due to their older versions not working with 3.10
>>>>>>>>>>>> Thank you for working on this.
>>>>>>>>>>>>> I am working through all the programs that reference python-3.8 in their rootfiles, or wherever, to change them to 3.10
>>>>>>>>>>>>> 
>>>>>>>>>>>>> My question comes with regard to the changes to the rootfiles to ensure they will be included properly in an update.
>>>>>>>>>>>> We will have to ship everything that puts anything into /usr/lib*/python3.*/site-packages again with the release of Python 3.10.
>>>>>>>>>>>> This isn’t fun, but not as much of a pain in the rear as shipping a new Perl release.
>>>>>>>>>>>>> For the addons then when I change the rootfile I am also incrementing the PAK_VER number in the lfs files and that ensures that they will be properly updated.
>>>>>>>>>>>> Yes. That is required for Pakfire to find a new version.
>>>>>>>>>>>>> I don't know what to do for the core programs where I have updated the rootfile. The lfs files for these programs don't have a PAK_VER number to increment and the actual package version number is not being changed so I don't know what I need to change in the lfs to ensure that the rootfile changes will be properly included in the update?
>>>>>>>>>>>> I would say committing everything into a “python-3.10” branch and submitting it all as a patchset is fine. All packages that have been touched in that branch will have to be shipped again which should be easy to identify if it all comes in one large branch.
>>>>>>>>>>> Okay, that is clear then.Will follow that approach.
>>>>>>>>>>>>> I would appreciate any help on what I need to do for these programs that have had changes only in their rootfiles.
>>>>>>>>>>>> Absolutely. Did you find anything that didn’t build yet?
>>>>>>>>>>> 
>>>>>>>>>>> I have not had any problem with python3.10 itself but I found some of the older modules would not work with 3.10 so I decided to do a combined update to 3.10 plus update modules to latest versions.
>>>>>>>>>>> 
>>>>>>>>>>> I found that after python3-setuptools-scm had run that the next module python3-six failed to run due to missing the tomli library. I commented out six and reran the build and then python3-dateutil had the same problem  and then python3-jmespath
>>>>>>>>>> I completely lost track of what the status of setuptools/distutils and so on is. This has changed multiple times and become independent/part of the distribution again and so on.
>>>>>>>>>>> After some investigation I have found that since version 6.0.0 of python3-setuptools-scm, tomli has been a required library when it is being used by other modules. So running version 6.0.0 of setuptools-scm worked with python3.10 and subsequent modules built successfully as tomli not needed.
>>>>>>>>>>> 
>>>>>>>>>>> So the obvious fix is to install tomli but that is when I ran into a problem. tomli has no setup.py or setup.cfg files in its tarball. The only way to build it is using pip.
>>>>>>>>>> Err. Great.
>>>>>>>>>>> Apparently, after some searching, setup.py has been deprecated as the recommended install method and if setup.py doesn't work the usual fix suggestion is to run pip install.
>>>>>>>>>> Yes. But apparently the new system is deprecated, too. I really dislike the state of packaging systems that come with any language these days.
>>>>>>>>>>> Having searched on pip for a while, it looks like builds using it can still be done from tarballs locally without needing to access the internet to download the modules.
>>>>>>>>>> That is good!
>>>>>>>>>>> As pip is installed as part of python3 i tried using that.
>>>>>>>>>>> 
>>>>>>>>>>> I used "python3 -m pip install --root=/ ."  in place of
>>>>>>>>>>> "python3 setup.py build
>>>>>>>>>>> python3 setup.py install --root=/"
>>>>>>>>>>> for the tomli source.
>>>>>>>>>>> 
>>>>>>>>>>> This failed because tomli requires flit_core to be available.
>>>>>>>>>>> 
>>>>>>>>>>> flit_core also has no setup.py and so I tried to build it with pip install. This failed with the error message that tomli was required as a dependency. So I added --no-deps to the flit_core pip install line and this time flit_core successfully built but the tomli build still came back with not finding flit_core.
>>>>>>>>>>> 
>>>>>>>>>>> I have tried a variety of changes to the pip install line but none of them has been successful and it has become clear that I am out of my depth here and need guidance rather than just trying anything I find from a google search. I am not even really clear if pip install is the correct approach, except if not then I have no idea how to progress.
>>>>>>>>>>> 
>>>>>>>>>>> Help urgently needed on how I should be approaching this.
>>>>>>>>>> Circular dependencies can happen. This won’t be the first time and it always is an utter nightmare with our buildsystem.
>>>>>>>>>> This page states that pip comes by default with all binary packages:
>>>>>>>>>>   https://docs.python.org/3/installing/index.html
>>>>>>>>>> Is there a chance that it is part of the distribution and can be enabled with a configure switch?
>>>>>>>>> I wasn't able to find any way to get it enabled that way.
>>>>>>>>> 
>>>>>>>>> I did find that pip install will not install if the package is already present. You have to add --upgrade to make it update any package already present. So I tried that. Didn't work.
>>>>>>>>> 
>>>>>>>>> I found a section about needing to use bootstrapping if you want to install tomli. Followed the information for creating a wheel file for flit_core and then extracting that file into the site-packages directory. Tried that. Didn't work.
>>>>>>>>> 
>>>>>>>>> Rather than doing a complete build every time I changed to trying to build flit_core followed by tomli in a ./make.sh shell environment.
>>>>>>>>> 
>>>>>>>>> Doing that I found that I had a pip install command that was creating the flit_core directory in site-packages and also including the .dist-info directory. The info I found said that the .dist-info directory had to be there for pip to be able to find the flit_core package.
>>>>>>>>> 
>>>>>>>>> However the pip install of tomli still ends up not finding any flit_core package. In the bootstrapping information it mentioned about changing the PYTHONPATH environment variable for find the flit_core package so I tried that in the shell but it also made no difference.
>>>>>>>>> 
>>>>>>>>> I then tried running python3 -c 'import flit_core' in the tomli build shell before doing the tomli pip install. The import command completed with no error but tomli still could not find any flit_core package.
>>>>>>>>> 
>>>>>>>>> End result of all my work is that flit_core has been built and installed into python3.10/site-packages but nothing seems to want to make the tomli build find it.
>>>>>>>>> 
>>>>>>>>> Does anyone have any ideas on what command(s) I need to use in the tomli pip install to have the installed flit_core package visible.
>>>>>>>>> 
>>>>>>>>> Regards and a Happy and Healthy New Year to all,
>>>>>>>>> 
>>>>>>>>> Adolf.
>>>>>>>>> 
>>>>>>>>>> -Michael
>>>>>>>>>>> 
>>>>>>>>>>> Regards,
>>>>>>>>>>> Adolf.
>>>>>>>>>>> 
>>>>>>>>>>> PS:  I hope everyone had an enjoyable Christmas.
>>>>>>>>>> I hope you did, too!
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> -Michael
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> -- 
>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> -- 
>>>>>>>>>>> Sent from my laptop
>>>>>>>>> 
>>>>>>>>> -- 
>>>>>>>>> Sent from my laptop
>>>>> 
>>>>> -- 
>>>>> Sent from my laptop
>>> 
>>> -- 
>>> Sent from my laptop
> 
> -- 
> Sent from my laptop

Re: Question about updating python to 3.10

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 17576 bytes --]

Hello Adolf,

I have just been trying to build a couple of Python packages that I need and I finally arrived at needing flit-core.

I suppose since you did already build this, there is no point in me doing all this again and also there is no need to build all my packages on Python 3.8 when we can have a newer release.

Could you push this branch somewhere for me so that I can base my work on top of it?

Best,
-Michael

> On 13 Jan 2022, at 09:32, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
> 
> Hello,
> 
>> On 12 Jan 2022, at 21:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>> 
>> Hi all,
>> 
>> On 12/01/2022 19:10, Michael Tremer wrote:
>>> Hey,
>>>> On 11 Jan 2022, at 14:32, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>> 
>>>> Hi Michael and all,
>>>> 
>>>> python3-six has to stay as python3-dateutil explicitly specifies it in its dependencies.
>>>> So that solves that question.
>>> Aww :(
>>>> I have, with quite a bit of work, eventually been able to get all existing python modules updated that had updates and built successfully with python3.10
>>> Yay!
>>>> However later in the IPFire build libvirt failed with a problem parsing some documentation files.
>>>> Stashing my python3.10 changes and re-running the build had libvirt building without problems. So one of the python changes has caused it. libvirt uses python3-docutils and that was updated but reverting that back to original on its own did not cause libvirt to build successfully.
>>>> 
>>>> I looked in the libvirt .configure for an option to not build the docs but it is not mentioned under ./configure --help.
>>>> 
>>>> I tried --without-docs and --disable-docs but both were not recognised in the build.
>>>> 
>>>> The current libvirt is from mid 2020 (6.5.0) so it is likely that it is not compatible in some way with the updated python/modules.
>>>> 
>>>> I tried updating libvirt to the current 7.10.0
>>>> libvirt has replaced autotools with meson. I found meson options available for all existing ./configure options except three. There is nothing in meson options for:-
>>>> --with-virtualport
>>>> --with-macvtap
>>>> --without-dbus
>>> Yay, another project that is using meson :(
>>> We no longer need macvtap; we have dubs, so that should not hurt; and I have no idea what virtualport is.
>>>> I tried building it with all the other options and it successfully built with the updated python3.10 and modules. All other packages after libvirt built successfully without any problems.
>>>> 
>>>> Normally I would then use the updated libvirt, as that is needed to work with the updated python packages, but I am not sure if those missing options are now built into libvirt or are no longer available and what impact that might have for the use of libvirt on IPFire.
>>> I don’t think this would hurt us much. I have CCed Jonatan who maintains this, so maybe he knows more about this.
>>>> There is nothing in the libvirt changelog about virtualport, macvtap or dbus.
>>>> 
>>>> Searching on the internet, I was also not able to find anything related to libvirt and these options.
>>>> 
>>>> Any suggestions/proposals on how to move forward on this? Should I just build libvirt without those three options or ...?
>>> Yes, and I think this could be submitted independently of the Python branch because it is an add-on. We should be able to ship this at any time and might want to delay Python until we have enough space in a core update.
>> Okay, then I will stash my python3.10 changes and submit the libvirt update on its own. Then Jonaton can test it out. I don't use libvirt or qemu so I can't test it easily.
>> 
>> Then when you think the time is right, let me know and I will submit the python3.10 series of patches.
> 
> You can submit this at any time. I was just thinking about the conversation between Peter and Arne during the last call regarding an updated kernel and updated firmware. Depending on the size of Python, all of this might make an update a little bit too large - however, this should not be a patchset that we cannot merge within two merge windows.
> 
>>> Thank you for looking into this. Sounds like a not so nice riddle to solve :)
>> There were certainly some challenges, like updating python3-daemon which then wanted to have twine as a fixed dependency which had nine dependencies some of which had more. I had got to 19 additional python packages and still going and then I said enough was enough and looked at twine. It is required for uploading python packages to PyPI which IPFire doesn't need. So I tried patching the twine requirement out and that worked and then I found others that had done the same because they had no response to the request to not have twine as a mandatory build requirement.
> 
> Python seems to become the hell that Perl used to be. In order to add one perl package, you had 20 dependencies. And those dependencies had their own dependencies too… It would be great to disable or patch out whatever is possible so that we do not create more work for ourselves in the future and have to update packages that are barely used - we already have enough of these :)
> 
>> I found it all challenging but rewarding when I made progress and eventually succeeded in building everything without errors.
> 
> I am so happy that you are putting all this time and effort into this because it certainly requires patience - a trade I don’t have.
> 
> And Python is being used in all sorts of places in the distribution, so it really isn’t anything we should not be up to date on.
> 
> -Michael
> 
>> 
>> Regards,
>> Adolf.
>>> -Michael
>>>> 
>>>> 
>>>> Regards,
>>>> 
>>>> Adolf.
>>>> 
>>>> On 07/01/2022 17:49, Michael Tremer wrote:
>>>>> Hello,
>>>>>> On 4 Jan 2022, at 12:49, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>> 
>>>>>> Hi Michael,
>>>>>> 
>>>>>> Following the suggested Arch Linux approach I have got python3-setuptools-scm building successfully now. Yaaah.
>>>>>> I had to install python3-toml, followed by python3-pyproject2setuppy followed by python3-tomli.
>>>>>> 
>>>>>> However then python3-six which installs after python3-setuptools-scm fails due to missing the python package python3-packaging.
>>>>>> 
>>>>>> I can install that package, and will do so as I continue testing out the python 3.10 build but I have the following question about python3-six.
>>>>>> 
>>>>>> Is six still needed. It is designed to ensure that programs can work with either python2 or python3 but IPFire only has python3 now.
>>>>>> I can find python3-six referenced only by aws-cli. In the aws-cli build python3-six is not listed at all in the build requirements. I tried to search for aws-cli and python3-six being needed in run mode but I could not find anything one way or the other.
>>>>> If aws-cli no longer requires python3-six we can drop it. Last time we touched it I had to bring the package back because aws-cli was still depending on it.
>>>>>> If python3-six is really needed for running aws-cli then I can install it and the new dependency but if it is not needed we could remove two packages which would partially offset the three new dependencies required to be able to build the latest python3-setuptools-scm.
>>>>> Yes, that would be great :)
>>>>> -Michael
>>>>>> 
>>>>>> Regards,
>>>>>> Adolf
>>>>>> 
>>>>>> On 03/01/2022 12:32, Michael Tremer wrote:
>>>>>>> Hello,
>>>>>>>> On 3 Jan 2022, at 11:31, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>> 
>>>>>>>> Hi Michael,
>>>>>>>> 
>>>>>>>> On 03/01/2022 11:52, Michael Tremer wrote:
>>>>>>>>> Hello,
>>>>>>>>> I am not sure if you already looked at what Arch is doing, but as far as I understand it works like this:
>>>>>>>> No I didn't look at Arch, which is surprising as I have in the past with other packages but I didn't. So thanks very much for the pointer.
>>>>>>>>> They build Python 3 without pip and setuptools (https://github.com/archlinux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/trunk/PKGBUILD#L59).
>>>>>>>>> At this stage we might need to build setuptools separately.
>>>>>>>>> They use a tool called pyproject2setuppy (great name!) to build tomli (https://github.com/archlinux/svntogit-packages/blob/packages/python-tomli/trunk/PKGBUILD).
>>>>>>>>> Setuptools-scm can then be built with tomli: https://github.com/archlinux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD
>>>>>>>>> Would this work or did I overlook anything?
>>>>>>>> I will have a look at the links and give it a go.
>>>>>>>> 
>>>>>>>> Thanks very much, I feel re-motivated now.
>>>>>>> Very good! I got a little bit lost in your last email and I think I could tell. But following a solution that somebody else has come up with can be a good thing to follow and of course we would customise it to our needs.
>>>>>>> Best,
>>>>>>> -Michael
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> Adolf.
>>>>>>>>> -Michael
>>>>>>>>>> On 2 Jan 2022, at 13:44, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hi Michael and All,
>>>>>>>>>> 
>>>>>>>>>> On 26/12/2021 21:55, Michael Tremer wrote:
>>>>>>>>>>> Merry Christmas,
>>>>>>>>>>>> On 26 Dec 2021, at 15:38, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>> 
>>>>>>>>>>>> On 17/12/2021 12:41, Michael Tremer wrote:
>>>>>>>>>>>>> Hello Adolf,
>>>>>>>>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I am working on updating python from 3.8 to 3.10. Python itself is okay after some other python related programs were also updated due to their older versions not working with 3.10
>>>>>>>>>>>>> Thank you for working on this.
>>>>>>>>>>>>>> I am working through all the programs that reference python-3.8 in their rootfiles, or wherever, to change them to 3.10
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> My question comes with regard to the changes to the rootfiles to ensure they will be included properly in an update.
>>>>>>>>>>>>> We will have to ship everything that puts anything into /usr/lib*/python3.*/site-packages again with the release of Python 3.10.
>>>>>>>>>>>>> This isn’t fun, but not as much of a pain in the rear as shipping a new Perl release.
>>>>>>>>>>>>>> For the addons then when I change the rootfile I am also incrementing the PAK_VER number in the lfs files and that ensures that they will be properly updated.
>>>>>>>>>>>>> Yes. That is required for Pakfire to find a new version.
>>>>>>>>>>>>>> I don't know what to do for the core programs where I have updated the rootfile. The lfs files for these programs don't have a PAK_VER number to increment and the actual package version number is not being changed so I don't know what I need to change in the lfs to ensure that the rootfile changes will be properly included in the update?
>>>>>>>>>>>>> I would say committing everything into a “python-3.10” branch and submitting it all as a patchset is fine. All packages that have been touched in that branch will have to be shipped again which should be easy to identify if it all comes in one large branch.
>>>>>>>>>>>> Okay, that is clear then.Will follow that approach.
>>>>>>>>>>>>>> I would appreciate any help on what I need to do for these programs that have had changes only in their rootfiles.
>>>>>>>>>>>>> Absolutely. Did you find anything that didn’t build yet?
>>>>>>>>>>>> 
>>>>>>>>>>>> I have not had any problem with python3.10 itself but I found some of the older modules would not work with 3.10 so I decided to do a combined update to 3.10 plus update modules to latest versions.
>>>>>>>>>>>> 
>>>>>>>>>>>> I found that after python3-setuptools-scm had run that the next module python3-six failed to run due to missing the tomli library. I commented out six and reran the build and then python3-dateutil had the same problem  and then python3-jmespath
>>>>>>>>>>> I completely lost track of what the status of setuptools/distutils and so on is. This has changed multiple times and become independent/part of the distribution again and so on.
>>>>>>>>>>>> After some investigation I have found that since version 6.0.0 of python3-setuptools-scm, tomli has been a required library when it is being used by other modules. So running version 6.0.0 of setuptools-scm worked with python3.10 and subsequent modules built successfully as tomli not needed.
>>>>>>>>>>>> 
>>>>>>>>>>>> So the obvious fix is to install tomli but that is when I ran into a problem. tomli has no setup.py or setup.cfg files in its tarball. The only way to build it is using pip.
>>>>>>>>>>> Err. Great.
>>>>>>>>>>>> Apparently, after some searching, setup.py has been deprecated as the recommended install method and if setup.py doesn't work the usual fix suggestion is to run pip install.
>>>>>>>>>>> Yes. But apparently the new system is deprecated, too. I really dislike the state of packaging systems that come with any language these days.
>>>>>>>>>>>> Having searched on pip for a while, it looks like builds using it can still be done from tarballs locally without needing to access the internet to download the modules.
>>>>>>>>>>> That is good!
>>>>>>>>>>>> As pip is installed as part of python3 i tried using that.
>>>>>>>>>>>> 
>>>>>>>>>>>> I used "python3 -m pip install --root=/ ."  in place of
>>>>>>>>>>>> "python3 setup.py build
>>>>>>>>>>>> python3 setup.py install --root=/"
>>>>>>>>>>>> for the tomli source.
>>>>>>>>>>>> 
>>>>>>>>>>>> This failed because tomli requires flit_core to be available.
>>>>>>>>>>>> 
>>>>>>>>>>>> flit_core also has no setup.py and so I tried to build it with pip install. This failed with the error message that tomli was required as a dependency. So I added --no-deps to the flit_core pip install line and this time flit_core successfully built but the tomli build still came back with not finding flit_core.
>>>>>>>>>>>> 
>>>>>>>>>>>> I have tried a variety of changes to the pip install line but none of them has been successful and it has become clear that I am out of my depth here and need guidance rather than just trying anything I find from a google search. I am not even really clear if pip install is the correct approach, except if not then I have no idea how to progress.
>>>>>>>>>>>> 
>>>>>>>>>>>> Help urgently needed on how I should be approaching this.
>>>>>>>>>>> Circular dependencies can happen. This won’t be the first time and it always is an utter nightmare with our buildsystem.
>>>>>>>>>>> This page states that pip comes by default with all binary packages:
>>>>>>>>>>>  https://docs.python.org/3/installing/index.html
>>>>>>>>>>> Is there a chance that it is part of the distribution and can be enabled with a configure switch?
>>>>>>>>>> I wasn't able to find any way to get it enabled that way.
>>>>>>>>>> 
>>>>>>>>>> I did find that pip install will not install if the package is already present. You have to add --upgrade to make it update any package already present. So I tried that. Didn't work.
>>>>>>>>>> 
>>>>>>>>>> I found a section about needing to use bootstrapping if you want to install tomli. Followed the information for creating a wheel file for flit_core and then extracting that file into the site-packages directory. Tried that. Didn't work.
>>>>>>>>>> 
>>>>>>>>>> Rather than doing a complete build every time I changed to trying to build flit_core followed by tomli in a ./make.sh shell environment.
>>>>>>>>>> 
>>>>>>>>>> Doing that I found that I had a pip install command that was creating the flit_core directory in site-packages and also including the .dist-info directory. The info I found said that the .dist-info directory had to be there for pip to be able to find the flit_core package.
>>>>>>>>>> 
>>>>>>>>>> However the pip install of tomli still ends up not finding any flit_core package. In the bootstrapping information it mentioned about changing the PYTHONPATH environment variable for find the flit_core package so I tried that in the shell but it also made no difference.
>>>>>>>>>> 
>>>>>>>>>> I then tried running python3 -c 'import flit_core' in the tomli build shell before doing the tomli pip install. The import command completed with no error but tomli still could not find any flit_core package.
>>>>>>>>>> 
>>>>>>>>>> End result of all my work is that flit_core has been built and installed into python3.10/site-packages but nothing seems to want to make the tomli build find it.
>>>>>>>>>> 
>>>>>>>>>> Does anyone have any ideas on what command(s) I need to use in the tomli pip install to have the installed flit_core package visible.
>>>>>>>>>> 
>>>>>>>>>> Regards and a Happy and Healthy New Year to all,
>>>>>>>>>> 
>>>>>>>>>> Adolf.
>>>>>>>>>> 
>>>>>>>>>>> -Michael
>>>>>>>>>>>> 
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Adolf.
>>>>>>>>>>>> 
>>>>>>>>>>>> PS:  I hope everyone had an enjoyable Christmas.
>>>>>>>>>>> I hope you did, too!
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> -- 
>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>> 
>>>>>>>>>> -- 
>>>>>>>>>> Sent from my laptop
>>>>>> 
>>>>>> -- 
>>>>>> Sent from my laptop
>>>> 
>>>> -- 
>>>> Sent from my laptop
>> 
>> -- 
>> Sent from my laptop

Re: Question about updating python to 3.10

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 18513 bytes --]

Hi Michael,

On 18/01/2022 16:02, Michael Tremer wrote:
> Hello Adolf,
> 
> I have just been trying to build a couple of Python packages that I need and I finally arrived at needing flit-core.
> 
> I suppose since you did already build this, there is no point in me doing all this again and also there is no need to build all my packages on Python 3.8 when we can have a newer release.
I am afraid that you will need to build flit-core.  I was building it when I was trying to use pip to install tomli but that approached failed to go further and that was when I used pyproject2setuppy, from your input, to build tomli and that did not need flit-core.

In terms of the python3.10 branch that I have on my git repository at home, I presume I could push it onto my personal git repository that you set up for me - https://git.ipfire.org/?p=people/bonnietwin/ipfire-2.x.git;a=summary

The only thing is I am not sure what command I should use to push the branch from my system here to the IPFire one.

I can have a go at trying it and see how it goes.

Regards,

Adolf.
> 
> Could you push this branch somewhere for me so that I can base my work on top of it?
> 
> Best,
> -Michael
> 
>> On 13 Jan 2022, at 09:32, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>
>> Hello,
>>
>>> On 12 Jan 2022, at 21:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>
>>> Hi all,
>>>
>>> On 12/01/2022 19:10, Michael Tremer wrote:
>>>> Hey,
>>>>> On 11 Jan 2022, at 14:32, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>
>>>>> Hi Michael and all,
>>>>>
>>>>> python3-six has to stay as python3-dateutil explicitly specifies it in its dependencies.
>>>>> So that solves that question.
>>>> Aww :(
>>>>> I have, with quite a bit of work, eventually been able to get all existing python modules updated that had updates and built successfully with python3.10
>>>> Yay!
>>>>> However later in the IPFire build libvirt failed with a problem parsing some documentation files.
>>>>> Stashing my python3.10 changes and re-running the build had libvirt building without problems. So one of the python changes has caused it. libvirt uses python3-docutils and that was updated but reverting that back to original on its own did not cause libvirt to build successfully.
>>>>>
>>>>> I looked in the libvirt .configure for an option to not build the docs but it is not mentioned under ./configure --help.
>>>>>
>>>>> I tried --without-docs and --disable-docs but both were not recognised in the build.
>>>>>
>>>>> The current libvirt is from mid 2020 (6.5.0) so it is likely that it is not compatible in some way with the updated python/modules.
>>>>>
>>>>> I tried updating libvirt to the current 7.10.0
>>>>> libvirt has replaced autotools with meson. I found meson options available for all existing ./configure options except three. There is nothing in meson options for:-
>>>>> --with-virtualport
>>>>> --with-macvtap
>>>>> --without-dbus
>>>> Yay, another project that is using meson :(
>>>> We no longer need macvtap; we have dubs, so that should not hurt; and I have no idea what virtualport is.
>>>>> I tried building it with all the other options and it successfully built with the updated python3.10 and modules. All other packages after libvirt built successfully without any problems.
>>>>>
>>>>> Normally I would then use the updated libvirt, as that is needed to work with the updated python packages, but I am not sure if those missing options are now built into libvirt or are no longer available and what impact that might have for the use of libvirt on IPFire.
>>>> I don’t think this would hurt us much. I have CCed Jonatan who maintains this, so maybe he knows more about this.
>>>>> There is nothing in the libvirt changelog about virtualport, macvtap or dbus.
>>>>>
>>>>> Searching on the internet, I was also not able to find anything related to libvirt and these options.
>>>>>
>>>>> Any suggestions/proposals on how to move forward on this? Should I just build libvirt without those three options or ...?
>>>> Yes, and I think this could be submitted independently of the Python branch because it is an add-on. We should be able to ship this at any time and might want to delay Python until we have enough space in a core update.
>>> Okay, then I will stash my python3.10 changes and submit the libvirt update on its own. Then Jonaton can test it out. I don't use libvirt or qemu so I can't test it easily.
>>>
>>> Then when you think the time is right, let me know and I will submit the python3.10 series of patches.
>>
>> You can submit this at any time. I was just thinking about the conversation between Peter and Arne during the last call regarding an updated kernel and updated firmware. Depending on the size of Python, all of this might make an update a little bit too large - however, this should not be a patchset that we cannot merge within two merge windows.
>>
>>>> Thank you for looking into this. Sounds like a not so nice riddle to solve :)
>>> There were certainly some challenges, like updating python3-daemon which then wanted to have twine as a fixed dependency which had nine dependencies some of which had more. I had got to 19 additional python packages and still going and then I said enough was enough and looked at twine. It is required for uploading python packages to PyPI which IPFire doesn't need. So I tried patching the twine requirement out and that worked and then I found others that had done the same because they had no response to the request to not have twine as a mandatory build requirement.
>>
>> Python seems to become the hell that Perl used to be. In order to add one perl package, you had 20 dependencies. And those dependencies had their own dependencies too… It would be great to disable or patch out whatever is possible so that we do not create more work for ourselves in the future and have to update packages that are barely used - we already have enough of these :)
>>
>>> I found it all challenging but rewarding when I made progress and eventually succeeded in building everything without errors.
>>
>> I am so happy that you are putting all this time and effort into this because it certainly requires patience - a trade I don’t have.
>>
>> And Python is being used in all sorts of places in the distribution, so it really isn’t anything we should not be up to date on.
>>
>> -Michael
>>
>>>
>>> Regards,
>>> Adolf.
>>>> -Michael
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> Adolf.
>>>>>
>>>>> On 07/01/2022 17:49, Michael Tremer wrote:
>>>>>> Hello,
>>>>>>> On 4 Jan 2022, at 12:49, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>
>>>>>>> Hi Michael,
>>>>>>>
>>>>>>> Following the suggested Arch Linux approach I have got python3-setuptools-scm building successfully now. Yaaah.
>>>>>>> I had to install python3-toml, followed by python3-pyproject2setuppy followed by python3-tomli.
>>>>>>>
>>>>>>> However then python3-six which installs after python3-setuptools-scm fails due to missing the python package python3-packaging.
>>>>>>>
>>>>>>> I can install that package, and will do so as I continue testing out the python 3.10 build but I have the following question about python3-six.
>>>>>>>
>>>>>>> Is six still needed. It is designed to ensure that programs can work with either python2 or python3 but IPFire only has python3 now.
>>>>>>> I can find python3-six referenced only by aws-cli. In the aws-cli build python3-six is not listed at all in the build requirements. I tried to search for aws-cli and python3-six being needed in run mode but I could not find anything one way or the other.
>>>>>> If aws-cli no longer requires python3-six we can drop it. Last time we touched it I had to bring the package back because aws-cli was still depending on it.
>>>>>>> If python3-six is really needed for running aws-cli then I can install it and the new dependency but if it is not needed we could remove two packages which would partially offset the three new dependencies required to be able to build the latest python3-setuptools-scm.
>>>>>> Yes, that would be great :)
>>>>>> -Michael
>>>>>>>
>>>>>>> Regards,
>>>>>>> Adolf
>>>>>>>
>>>>>>> On 03/01/2022 12:32, Michael Tremer wrote:
>>>>>>>> Hello,
>>>>>>>>> On 3 Jan 2022, at 11:31, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>
>>>>>>>>> Hi Michael,
>>>>>>>>>
>>>>>>>>> On 03/01/2022 11:52, Michael Tremer wrote:
>>>>>>>>>> Hello,
>>>>>>>>>> I am not sure if you already looked at what Arch is doing, but as far as I understand it works like this:
>>>>>>>>> No I didn't look at Arch, which is surprising as I have in the past with other packages but I didn't. So thanks very much for the pointer.
>>>>>>>>>> They build Python 3 without pip and setuptools (https://github.com/archlinux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/trunk/PKGBUILD#L59).
>>>>>>>>>> At this stage we might need to build setuptools separately.
>>>>>>>>>> They use a tool called pyproject2setuppy (great name!) to build tomli (https://github.com/archlinux/svntogit-packages/blob/packages/python-tomli/trunk/PKGBUILD).
>>>>>>>>>> Setuptools-scm can then be built with tomli: https://github.com/archlinux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD
>>>>>>>>>> Would this work or did I overlook anything?
>>>>>>>>> I will have a look at the links and give it a go.
>>>>>>>>>
>>>>>>>>> Thanks very much, I feel re-motivated now.
>>>>>>>> Very good! I got a little bit lost in your last email and I think I could tell. But following a solution that somebody else has come up with can be a good thing to follow and of course we would customise it to our needs.
>>>>>>>> Best,
>>>>>>>> -Michael
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Adolf.
>>>>>>>>>> -Michael
>>>>>>>>>>> On 2 Jan 2022, at 13:44, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Michael and All,
>>>>>>>>>>>
>>>>>>>>>>> On 26/12/2021 21:55, Michael Tremer wrote:
>>>>>>>>>>>> Merry Christmas,
>>>>>>>>>>>>> On 26 Dec 2021, at 15:38, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 17/12/2021 12:41, Michael Tremer wrote:
>>>>>>>>>>>>>> Hello Adolf,
>>>>>>>>>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I am working on updating python from 3.8 to 3.10. Python itself is okay after some other python related programs were also updated due to their older versions not working with 3.10
>>>>>>>>>>>>>> Thank you for working on this.
>>>>>>>>>>>>>>> I am working through all the programs that reference python-3.8 in their rootfiles, or wherever, to change them to 3.10
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> My question comes with regard to the changes to the rootfiles to ensure they will be included properly in an update.
>>>>>>>>>>>>>> We will have to ship everything that puts anything into /usr/lib*/python3.*/site-packages again with the release of Python 3.10.
>>>>>>>>>>>>>> This isn’t fun, but not as much of a pain in the rear as shipping a new Perl release.
>>>>>>>>>>>>>>> For the addons then when I change the rootfile I am also incrementing the PAK_VER number in the lfs files and that ensures that they will be properly updated.
>>>>>>>>>>>>>> Yes. That is required for Pakfire to find a new version.
>>>>>>>>>>>>>>> I don't know what to do for the core programs where I have updated the rootfile. The lfs files for these programs don't have a PAK_VER number to increment and the actual package version number is not being changed so I don't know what I need to change in the lfs to ensure that the rootfile changes will be properly included in the update?
>>>>>>>>>>>>>> I would say committing everything into a “python-3.10” branch and submitting it all as a patchset is fine. All packages that have been touched in that branch will have to be shipped again which should be easy to identify if it all comes in one large branch.
>>>>>>>>>>>>> Okay, that is clear then.Will follow that approach.
>>>>>>>>>>>>>>> I would appreciate any help on what I need to do for these programs that have had changes only in their rootfiles.
>>>>>>>>>>>>>> Absolutely. Did you find anything that didn’t build yet?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have not had any problem with python3.10 itself but I found some of the older modules would not work with 3.10 so I decided to do a combined update to 3.10 plus update modules to latest versions.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I found that after python3-setuptools-scm had run that the next module python3-six failed to run due to missing the tomli library. I commented out six and reran the build and then python3-dateutil had the same problem  and then python3-jmespath
>>>>>>>>>>>> I completely lost track of what the status of setuptools/distutils and so on is. This has changed multiple times and become independent/part of the distribution again and so on.
>>>>>>>>>>>>> After some investigation I have found that since version 6.0.0 of python3-setuptools-scm, tomli has been a required library when it is being used by other modules. So running version 6.0.0 of setuptools-scm worked with python3.10 and subsequent modules built successfully as tomli not needed.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So the obvious fix is to install tomli but that is when I ran into a problem. tomli has no setup.py or setup.cfg files in its tarball. The only way to build it is using pip.
>>>>>>>>>>>> Err. Great.
>>>>>>>>>>>>> Apparently, after some searching, setup.py has been deprecated as the recommended install method and if setup.py doesn't work the usual fix suggestion is to run pip install.
>>>>>>>>>>>> Yes. But apparently the new system is deprecated, too. I really dislike the state of packaging systems that come with any language these days.
>>>>>>>>>>>>> Having searched on pip for a while, it looks like builds using it can still be done from tarballs locally without needing to access the internet to download the modules.
>>>>>>>>>>>> That is good!
>>>>>>>>>>>>> As pip is installed as part of python3 i tried using that.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I used "python3 -m pip install --root=/ ."  in place of
>>>>>>>>>>>>> "python3 setup.py build
>>>>>>>>>>>>> python3 setup.py install --root=/"
>>>>>>>>>>>>> for the tomli source.
>>>>>>>>>>>>>
>>>>>>>>>>>>> This failed because tomli requires flit_core to be available.
>>>>>>>>>>>>>
>>>>>>>>>>>>> flit_core also has no setup.py and so I tried to build it with pip install. This failed with the error message that tomli was required as a dependency. So I added --no-deps to the flit_core pip install line and this time flit_core successfully built but the tomli build still came back with not finding flit_core.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have tried a variety of changes to the pip install line but none of them has been successful and it has become clear that I am out of my depth here and need guidance rather than just trying anything I find from a google search. I am not even really clear if pip install is the correct approach, except if not then I have no idea how to progress.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Help urgently needed on how I should be approaching this.
>>>>>>>>>>>> Circular dependencies can happen. This won’t be the first time and it always is an utter nightmare with our buildsystem.
>>>>>>>>>>>> This page states that pip comes by default with all binary packages:
>>>>>>>>>>>>   https://docs.python.org/3/installing/index.html
>>>>>>>>>>>> Is there a chance that it is part of the distribution and can be enabled with a configure switch?
>>>>>>>>>>> I wasn't able to find any way to get it enabled that way.
>>>>>>>>>>>
>>>>>>>>>>> I did find that pip install will not install if the package is already present. You have to add --upgrade to make it update any package already present. So I tried that. Didn't work.
>>>>>>>>>>>
>>>>>>>>>>> I found a section about needing to use bootstrapping if you want to install tomli. Followed the information for creating a wheel file for flit_core and then extracting that file into the site-packages directory. Tried that. Didn't work.
>>>>>>>>>>>
>>>>>>>>>>> Rather than doing a complete build every time I changed to trying to build flit_core followed by tomli in a ./make.sh shell environment.
>>>>>>>>>>>
>>>>>>>>>>> Doing that I found that I had a pip install command that was creating the flit_core directory in site-packages and also including the .dist-info directory. The info I found said that the .dist-info directory had to be there for pip to be able to find the flit_core package.
>>>>>>>>>>>
>>>>>>>>>>> However the pip install of tomli still ends up not finding any flit_core package. In the bootstrapping information it mentioned about changing the PYTHONPATH environment variable for find the flit_core package so I tried that in the shell but it also made no difference.
>>>>>>>>>>>
>>>>>>>>>>> I then tried running python3 -c 'import flit_core' in the tomli build shell before doing the tomli pip install. The import command completed with no error but tomli still could not find any flit_core package.
>>>>>>>>>>>
>>>>>>>>>>> End result of all my work is that flit_core has been built and installed into python3.10/site-packages but nothing seems to want to make the tomli build find it.
>>>>>>>>>>>
>>>>>>>>>>> Does anyone have any ideas on what command(s) I need to use in the tomli pip install to have the installed flit_core package visible.
>>>>>>>>>>>
>>>>>>>>>>> Regards and a Happy and Healthy New Year to all,
>>>>>>>>>>>
>>>>>>>>>>> Adolf.
>>>>>>>>>>>
>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>
>>>>>>>>>>>>> PS:  I hope everyone had an enjoyable Christmas.
>>>>>>>>>>>> I hope you did, too!
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> -- 
>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>
>>>>>>>>>>> -- 
>>>>>>>>>>> Sent from my laptop
>>>>>>>
>>>>>>> -- 
>>>>>>> Sent from my laptop
>>>>>
>>>>> -- 
>>>>> Sent from my laptop
>>>
>>> -- 
>>> Sent from my laptop
> 

Re: Question about updating python to 3.10

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 19233 bytes --]

Hello,

> On 18 Jan 2022, at 16:42, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi Michael,
> 
> On 18/01/2022 16:02, Michael Tremer wrote:
>> Hello Adolf,
>> I have just been trying to build a couple of Python packages that I need and I finally arrived at needing flit-core.
>> I suppose since you did already build this, there is no point in me doing all this again and also there is no need to build all my packages on Python 3.8 when we can have a newer release.
> I am afraid that you will need to build flit-core.  I was building it when I was trying to use pip to install tomli but that approached failed to go further and that was when I used pyproject2setuppy, from your input, to build tomli and that did not need flit-core.

Oh, I suppose that didn’t hold for very long because I now have a package that needs it :(

> In terms of the python3.10 branch that I have on my git repository at home, I presume I could push it onto my personal git repository that you set up for me - https://git.ipfire.org/?p=people/bonnietwin/ipfire-2.x.git;a=summary

Yes, it is always a good idea to backup things to the server.

> The only thing is I am not sure what command I should use to push the branch from my system here to the IPFire one.

If you have your repository set up with “git remote add …” then you only need to run “git push bonnietwin python3.10”.

-Michael

> 
> I can have a go at trying it and see how it goes.
> 
> Regards,
> 
> Adolf.
>> Could you push this branch somewhere for me so that I can base my work on top of it?
>> Best,
>> -Michael
>>> On 13 Jan 2022, at 09:32, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>> 
>>> Hello,
>>> 
>>>> On 12 Jan 2022, at 21:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>> 
>>>> Hi all,
>>>> 
>>>> On 12/01/2022 19:10, Michael Tremer wrote:
>>>>> Hey,
>>>>>> On 11 Jan 2022, at 14:32, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>> 
>>>>>> Hi Michael and all,
>>>>>> 
>>>>>> python3-six has to stay as python3-dateutil explicitly specifies it in its dependencies.
>>>>>> So that solves that question.
>>>>> Aww :(
>>>>>> I have, with quite a bit of work, eventually been able to get all existing python modules updated that had updates and built successfully with python3.10
>>>>> Yay!
>>>>>> However later in the IPFire build libvirt failed with a problem parsing some documentation files.
>>>>>> Stashing my python3.10 changes and re-running the build had libvirt building without problems. So one of the python changes has caused it. libvirt uses python3-docutils and that was updated but reverting that back to original on its own did not cause libvirt to build successfully.
>>>>>> 
>>>>>> I looked in the libvirt .configure for an option to not build the docs but it is not mentioned under ./configure --help.
>>>>>> 
>>>>>> I tried --without-docs and --disable-docs but both were not recognised in the build.
>>>>>> 
>>>>>> The current libvirt is from mid 2020 (6.5.0) so it is likely that it is not compatible in some way with the updated python/modules.
>>>>>> 
>>>>>> I tried updating libvirt to the current 7.10.0
>>>>>> libvirt has replaced autotools with meson. I found meson options available for all existing ./configure options except three. There is nothing in meson options for:-
>>>>>> --with-virtualport
>>>>>> --with-macvtap
>>>>>> --without-dbus
>>>>> Yay, another project that is using meson :(
>>>>> We no longer need macvtap; we have dubs, so that should not hurt; and I have no idea what virtualport is.
>>>>>> I tried building it with all the other options and it successfully built with the updated python3.10 and modules. All other packages after libvirt built successfully without any problems.
>>>>>> 
>>>>>> Normally I would then use the updated libvirt, as that is needed to work with the updated python packages, but I am not sure if those missing options are now built into libvirt or are no longer available and what impact that might have for the use of libvirt on IPFire.
>>>>> I don’t think this would hurt us much. I have CCed Jonatan who maintains this, so maybe he knows more about this.
>>>>>> There is nothing in the libvirt changelog about virtualport, macvtap or dbus.
>>>>>> 
>>>>>> Searching on the internet, I was also not able to find anything related to libvirt and these options.
>>>>>> 
>>>>>> Any suggestions/proposals on how to move forward on this? Should I just build libvirt without those three options or ...?
>>>>> Yes, and I think this could be submitted independently of the Python branch because it is an add-on. We should be able to ship this at any time and might want to delay Python until we have enough space in a core update.
>>>> Okay, then I will stash my python3.10 changes and submit the libvirt update on its own. Then Jonaton can test it out. I don't use libvirt or qemu so I can't test it easily.
>>>> 
>>>> Then when you think the time is right, let me know and I will submit the python3.10 series of patches.
>>> 
>>> You can submit this at any time. I was just thinking about the conversation between Peter and Arne during the last call regarding an updated kernel and updated firmware. Depending on the size of Python, all of this might make an update a little bit too large - however, this should not be a patchset that we cannot merge within two merge windows.
>>> 
>>>>> Thank you for looking into this. Sounds like a not so nice riddle to solve :)
>>>> There were certainly some challenges, like updating python3-daemon which then wanted to have twine as a fixed dependency which had nine dependencies some of which had more. I had got to 19 additional python packages and still going and then I said enough was enough and looked at twine. It is required for uploading python packages to PyPI which IPFire doesn't need. So I tried patching the twine requirement out and that worked and then I found others that had done the same because they had no response to the request to not have twine as a mandatory build requirement.
>>> 
>>> Python seems to become the hell that Perl used to be. In order to add one perl package, you had 20 dependencies. And those dependencies had their own dependencies too… It would be great to disable or patch out whatever is possible so that we do not create more work for ourselves in the future and have to update packages that are barely used - we already have enough of these :)
>>> 
>>>> I found it all challenging but rewarding when I made progress and eventually succeeded in building everything without errors.
>>> 
>>> I am so happy that you are putting all this time and effort into this because it certainly requires patience - a trade I don’t have.
>>> 
>>> And Python is being used in all sorts of places in the distribution, so it really isn’t anything we should not be up to date on.
>>> 
>>> -Michael
>>> 
>>>> 
>>>> Regards,
>>>> Adolf.
>>>>> -Michael
>>>>>> 
>>>>>> 
>>>>>> Regards,
>>>>>> 
>>>>>> Adolf.
>>>>>> 
>>>>>> On 07/01/2022 17:49, Michael Tremer wrote:
>>>>>>> Hello,
>>>>>>>> On 4 Jan 2022, at 12:49, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>> 
>>>>>>>> Hi Michael,
>>>>>>>> 
>>>>>>>> Following the suggested Arch Linux approach I have got python3-setuptools-scm building successfully now. Yaaah.
>>>>>>>> I had to install python3-toml, followed by python3-pyproject2setuppy followed by python3-tomli.
>>>>>>>> 
>>>>>>>> However then python3-six which installs after python3-setuptools-scm fails due to missing the python package python3-packaging.
>>>>>>>> 
>>>>>>>> I can install that package, and will do so as I continue testing out the python 3.10 build but I have the following question about python3-six.
>>>>>>>> 
>>>>>>>> Is six still needed. It is designed to ensure that programs can work with either python2 or python3 but IPFire only has python3 now.
>>>>>>>> I can find python3-six referenced only by aws-cli. In the aws-cli build python3-six is not listed at all in the build requirements. I tried to search for aws-cli and python3-six being needed in run mode but I could not find anything one way or the other.
>>>>>>> If aws-cli no longer requires python3-six we can drop it. Last time we touched it I had to bring the package back because aws-cli was still depending on it.
>>>>>>>> If python3-six is really needed for running aws-cli then I can install it and the new dependency but if it is not needed we could remove two packages which would partially offset the three new dependencies required to be able to build the latest python3-setuptools-scm.
>>>>>>> Yes, that would be great :)
>>>>>>> -Michael
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> Adolf
>>>>>>>> 
>>>>>>>> On 03/01/2022 12:32, Michael Tremer wrote:
>>>>>>>>> Hello,
>>>>>>>>>> On 3 Jan 2022, at 11:31, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hi Michael,
>>>>>>>>>> 
>>>>>>>>>> On 03/01/2022 11:52, Michael Tremer wrote:
>>>>>>>>>>> Hello,
>>>>>>>>>>> I am not sure if you already looked at what Arch is doing, but as far as I understand it works like this:
>>>>>>>>>> No I didn't look at Arch, which is surprising as I have in the past with other packages but I didn't. So thanks very much for the pointer.
>>>>>>>>>>> They build Python 3 without pip and setuptools (https://github.com/archlinux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/trunk/PKGBUILD#L59).
>>>>>>>>>>> At this stage we might need to build setuptools separately.
>>>>>>>>>>> They use a tool called pyproject2setuppy (great name!) to build tomli (https://github.com/archlinux/svntogit-packages/blob/packages/python-tomli/trunk/PKGBUILD).
>>>>>>>>>>> Setuptools-scm can then be built with tomli: https://github.com/archlinux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD
>>>>>>>>>>> Would this work or did I overlook anything?
>>>>>>>>>> I will have a look at the links and give it a go.
>>>>>>>>>> 
>>>>>>>>>> Thanks very much, I feel re-motivated now.
>>>>>>>>> Very good! I got a little bit lost in your last email and I think I could tell. But following a solution that somebody else has come up with can be a good thing to follow and of course we would customise it to our needs.
>>>>>>>>> Best,
>>>>>>>>> -Michael
>>>>>>>>>> 
>>>>>>>>>> Regards,
>>>>>>>>>> Adolf.
>>>>>>>>>>> -Michael
>>>>>>>>>>>> On 2 Jan 2022, at 13:44, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi Michael and All,
>>>>>>>>>>>> 
>>>>>>>>>>>> On 26/12/2021 21:55, Michael Tremer wrote:
>>>>>>>>>>>>> Merry Christmas,
>>>>>>>>>>>>>> On 26 Dec 2021, at 15:38, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 17/12/2021 12:41, Michael Tremer wrote:
>>>>>>>>>>>>>>> Hello Adolf,
>>>>>>>>>>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I am working on updating python from 3.8 to 3.10. Python itself is okay after some other python related programs were also updated due to their older versions not working with 3.10
>>>>>>>>>>>>>>> Thank you for working on this.
>>>>>>>>>>>>>>>> I am working through all the programs that reference python-3.8 in their rootfiles, or wherever, to change them to 3.10
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> My question comes with regard to the changes to the rootfiles to ensure they will be included properly in an update.
>>>>>>>>>>>>>>> We will have to ship everything that puts anything into /usr/lib*/python3.*/site-packages again with the release of Python 3.10.
>>>>>>>>>>>>>>> This isn’t fun, but not as much of a pain in the rear as shipping a new Perl release.
>>>>>>>>>>>>>>>> For the addons then when I change the rootfile I am also incrementing the PAK_VER number in the lfs files and that ensures that they will be properly updated.
>>>>>>>>>>>>>>> Yes. That is required for Pakfire to find a new version.
>>>>>>>>>>>>>>>> I don't know what to do for the core programs where I have updated the rootfile. The lfs files for these programs don't have a PAK_VER number to increment and the actual package version number is not being changed so I don't know what I need to change in the lfs to ensure that the rootfile changes will be properly included in the update?
>>>>>>>>>>>>>>> I would say committing everything into a “python-3.10” branch and submitting it all as a patchset is fine. All packages that have been touched in that branch will have to be shipped again which should be easy to identify if it all comes in one large branch.
>>>>>>>>>>>>>> Okay, that is clear then.Will follow that approach.
>>>>>>>>>>>>>>>> I would appreciate any help on what I need to do for these programs that have had changes only in their rootfiles.
>>>>>>>>>>>>>>> Absolutely. Did you find anything that didn’t build yet?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I have not had any problem with python3.10 itself but I found some of the older modules would not work with 3.10 so I decided to do a combined update to 3.10 plus update modules to latest versions.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I found that after python3-setuptools-scm had run that the next module python3-six failed to run due to missing the tomli library. I commented out six and reran the build and then python3-dateutil had the same problem  and then python3-jmespath
>>>>>>>>>>>>> I completely lost track of what the status of setuptools/distutils and so on is. This has changed multiple times and become independent/part of the distribution again and so on.
>>>>>>>>>>>>>> After some investigation I have found that since version 6.0.0 of python3-setuptools-scm, tomli has been a required library when it is being used by other modules. So running version 6.0.0 of setuptools-scm worked with python3.10 and subsequent modules built successfully as tomli not needed.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> So the obvious fix is to install tomli but that is when I ran into a problem. tomli has no setup.py or setup.cfg files in its tarball. The only way to build it is using pip.
>>>>>>>>>>>>> Err. Great.
>>>>>>>>>>>>>> Apparently, after some searching, setup.py has been deprecated as the recommended install method and if setup.py doesn't work the usual fix suggestion is to run pip install.
>>>>>>>>>>>>> Yes. But apparently the new system is deprecated, too. I really dislike the state of packaging systems that come with any language these days.
>>>>>>>>>>>>>> Having searched on pip for a while, it looks like builds using it can still be done from tarballs locally without needing to access the internet to download the modules.
>>>>>>>>>>>>> That is good!
>>>>>>>>>>>>>> As pip is installed as part of python3 i tried using that.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I used "python3 -m pip install --root=/ ."  in place of
>>>>>>>>>>>>>> "python3 setup.py build
>>>>>>>>>>>>>> python3 setup.py install --root=/"
>>>>>>>>>>>>>> for the tomli source.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> This failed because tomli requires flit_core to be available.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> flit_core also has no setup.py and so I tried to build it with pip install. This failed with the error message that tomli was required as a dependency. So I added --no-deps to the flit_core pip install line and this time flit_core successfully built but the tomli build still came back with not finding flit_core.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I have tried a variety of changes to the pip install line but none of them has been successful and it has become clear that I am out of my depth here and need guidance rather than just trying anything I find from a google search. I am not even really clear if pip install is the correct approach, except if not then I have no idea how to progress.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Help urgently needed on how I should be approaching this.
>>>>>>>>>>>>> Circular dependencies can happen. This won’t be the first time and it always is an utter nightmare with our buildsystem.
>>>>>>>>>>>>> This page states that pip comes by default with all binary packages:
>>>>>>>>>>>>>  https://docs.python.org/3/installing/index.html
>>>>>>>>>>>>> Is there a chance that it is part of the distribution and can be enabled with a configure switch?
>>>>>>>>>>>> I wasn't able to find any way to get it enabled that way.
>>>>>>>>>>>> 
>>>>>>>>>>>> I did find that pip install will not install if the package is already present. You have to add --upgrade to make it update any package already present. So I tried that. Didn't work.
>>>>>>>>>>>> 
>>>>>>>>>>>> I found a section about needing to use bootstrapping if you want to install tomli. Followed the information for creating a wheel file for flit_core and then extracting that file into the site-packages directory. Tried that. Didn't work.
>>>>>>>>>>>> 
>>>>>>>>>>>> Rather than doing a complete build every time I changed to trying to build flit_core followed by tomli in a ./make.sh shell environment.
>>>>>>>>>>>> 
>>>>>>>>>>>> Doing that I found that I had a pip install command that was creating the flit_core directory in site-packages and also including the .dist-info directory. The info I found said that the .dist-info directory had to be there for pip to be able to find the flit_core package.
>>>>>>>>>>>> 
>>>>>>>>>>>> However the pip install of tomli still ends up not finding any flit_core package. In the bootstrapping information it mentioned about changing the PYTHONPATH environment variable for find the flit_core package so I tried that in the shell but it also made no difference.
>>>>>>>>>>>> 
>>>>>>>>>>>> I then tried running python3 -c 'import flit_core' in the tomli build shell before doing the tomli pip install. The import command completed with no error but tomli still could not find any flit_core package.
>>>>>>>>>>>> 
>>>>>>>>>>>> End result of all my work is that flit_core has been built and installed into python3.10/site-packages but nothing seems to want to make the tomli build find it.
>>>>>>>>>>>> 
>>>>>>>>>>>> Does anyone have any ideas on what command(s) I need to use in the tomli pip install to have the installed flit_core package visible.
>>>>>>>>>>>> 
>>>>>>>>>>>> Regards and a Happy and Healthy New Year to all,
>>>>>>>>>>>> 
>>>>>>>>>>>> Adolf.
>>>>>>>>>>>> 
>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> PS:  I hope everyone had an enjoyable Christmas.
>>>>>>>>>>>>> I hope you did, too!
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>> 
>>>>>>>>>>>> -- 
>>>>>>>>>>>> Sent from my laptop
>>>>>>>> 
>>>>>>>> -- 
>>>>>>>> Sent from my laptop
>>>>>> 
>>>>>> -- 
>>>>>> Sent from my laptop
>>>> 
>>>> -- 
>>>> Sent from my laptop

Re: Question about updating python to 3.10

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 20101 bytes --]

Hi Michael,

On 18/01/2022 19:51, Michael Tremer wrote:
> Hello,
> 
>> On 18 Jan 2022, at 16:42, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>
>> Hi Michael,
>>
>> On 18/01/2022 16:02, Michael Tremer wrote:
>>> Hello Adolf,
>>> I have just been trying to build a couple of Python packages that I need and I finally arrived at needing flit-core.
>>> I suppose since you did already build this, there is no point in me doing all this again and also there is no need to build all my packages on Python 3.8 when we can have a newer release.
>> I am afraid that you will need to build flit-core.  I was building it when I was trying to use pip to install tomli but that approached failed to go further and that was when I used pyproject2setuppy, from your input, to build tomli and that did not need flit-core.
> 
> Oh, I suppose that didn’t hold for very long because I now have a package that needs it :(
flit-core also has no setup.py but you will then be able to use the pyproject2setuppy package from my python3.10 branch to build flit-core.
> 
>> In terms of the python3.10 branch that I have on my git repository at home, I presume I could push it onto my personal git repository that you set up for me - https://git.ipfire.org/?p=people/bonnietwin/ipfire-2.x.git;a=summary
> 
> Yes, it is always a good idea to backup things to the server.
> 
>> The only thing is I am not sure what command I should use to push the branch from my system here to the IPFire one.
> 
> If you have your repository set up with “git remote add …” then you only need to run “git push bonnietwin python3.10”.
Aaah. I think when I tried to experiment with this before I created a local new clone of bonnietwin which was then separate from my running local repository that I used for building and I struggled to figure out how to get branches between them.

Your mention of git remote add indicates that I can add my bonnietwin repository to my running build repository. Thais is what I was missing before.
Thanks for the clue, will give it a try.

Regards,

Adolf.
> 
> -Michael
> 
>>
>> I can have a go at trying it and see how it goes.
>>
>> Regards,
>>
>> Adolf.
>>> Could you push this branch somewhere for me so that I can base my work on top of it?
>>> Best,
>>> -Michael
>>>> On 13 Jan 2022, at 09:32, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>>
>>>> Hello,
>>>>
>>>>> On 12 Jan 2022, at 21:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> On 12/01/2022 19:10, Michael Tremer wrote:
>>>>>> Hey,
>>>>>>> On 11 Jan 2022, at 14:32, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>
>>>>>>> Hi Michael and all,
>>>>>>>
>>>>>>> python3-six has to stay as python3-dateutil explicitly specifies it in its dependencies.
>>>>>>> So that solves that question.
>>>>>> Aww :(
>>>>>>> I have, with quite a bit of work, eventually been able to get all existing python modules updated that had updates and built successfully with python3.10
>>>>>> Yay!
>>>>>>> However later in the IPFire build libvirt failed with a problem parsing some documentation files.
>>>>>>> Stashing my python3.10 changes and re-running the build had libvirt building without problems. So one of the python changes has caused it. libvirt uses python3-docutils and that was updated but reverting that back to original on its own did not cause libvirt to build successfully.
>>>>>>>
>>>>>>> I looked in the libvirt .configure for an option to not build the docs but it is not mentioned under ./configure --help.
>>>>>>>
>>>>>>> I tried --without-docs and --disable-docs but both were not recognised in the build.
>>>>>>>
>>>>>>> The current libvirt is from mid 2020 (6.5.0) so it is likely that it is not compatible in some way with the updated python/modules.
>>>>>>>
>>>>>>> I tried updating libvirt to the current 7.10.0
>>>>>>> libvirt has replaced autotools with meson. I found meson options available for all existing ./configure options except three. There is nothing in meson options for:-
>>>>>>> --with-virtualport
>>>>>>> --with-macvtap
>>>>>>> --without-dbus
>>>>>> Yay, another project that is using meson :(
>>>>>> We no longer need macvtap; we have dubs, so that should not hurt; and I have no idea what virtualport is.
>>>>>>> I tried building it with all the other options and it successfully built with the updated python3.10 and modules. All other packages after libvirt built successfully without any problems.
>>>>>>>
>>>>>>> Normally I would then use the updated libvirt, as that is needed to work with the updated python packages, but I am not sure if those missing options are now built into libvirt or are no longer available and what impact that might have for the use of libvirt on IPFire.
>>>>>> I don’t think this would hurt us much. I have CCed Jonatan who maintains this, so maybe he knows more about this.
>>>>>>> There is nothing in the libvirt changelog about virtualport, macvtap or dbus.
>>>>>>>
>>>>>>> Searching on the internet, I was also not able to find anything related to libvirt and these options.
>>>>>>>
>>>>>>> Any suggestions/proposals on how to move forward on this? Should I just build libvirt without those three options or ...?
>>>>>> Yes, and I think this could be submitted independently of the Python branch because it is an add-on. We should be able to ship this at any time and might want to delay Python until we have enough space in a core update.
>>>>> Okay, then I will stash my python3.10 changes and submit the libvirt update on its own. Then Jonaton can test it out. I don't use libvirt or qemu so I can't test it easily.
>>>>>
>>>>> Then when you think the time is right, let me know and I will submit the python3.10 series of patches.
>>>>
>>>> You can submit this at any time. I was just thinking about the conversation between Peter and Arne during the last call regarding an updated kernel and updated firmware. Depending on the size of Python, all of this might make an update a little bit too large - however, this should not be a patchset that we cannot merge within two merge windows.
>>>>
>>>>>> Thank you for looking into this. Sounds like a not so nice riddle to solve :)
>>>>> There were certainly some challenges, like updating python3-daemon which then wanted to have twine as a fixed dependency which had nine dependencies some of which had more. I had got to 19 additional python packages and still going and then I said enough was enough and looked at twine. It is required for uploading python packages to PyPI which IPFire doesn't need. So I tried patching the twine requirement out and that worked and then I found others that had done the same because they had no response to the request to not have twine as a mandatory build requirement.
>>>>
>>>> Python seems to become the hell that Perl used to be. In order to add one perl package, you had 20 dependencies. And those dependencies had their own dependencies too… It would be great to disable or patch out whatever is possible so that we do not create more work for ourselves in the future and have to update packages that are barely used - we already have enough of these :)
>>>>
>>>>> I found it all challenging but rewarding when I made progress and eventually succeeded in building everything without errors.
>>>>
>>>> I am so happy that you are putting all this time and effort into this because it certainly requires patience - a trade I don’t have.
>>>>
>>>> And Python is being used in all sorts of places in the distribution, so it really isn’t anything we should not be up to date on.
>>>>
>>>> -Michael
>>>>
>>>>>
>>>>> Regards,
>>>>> Adolf.
>>>>>> -Michael
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Adolf.
>>>>>>>
>>>>>>> On 07/01/2022 17:49, Michael Tremer wrote:
>>>>>>>> Hello,
>>>>>>>>> On 4 Jan 2022, at 12:49, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>
>>>>>>>>> Hi Michael,
>>>>>>>>>
>>>>>>>>> Following the suggested Arch Linux approach I have got python3-setuptools-scm building successfully now. Yaaah.
>>>>>>>>> I had to install python3-toml, followed by python3-pyproject2setuppy followed by python3-tomli.
>>>>>>>>>
>>>>>>>>> However then python3-six which installs after python3-setuptools-scm fails due to missing the python package python3-packaging.
>>>>>>>>>
>>>>>>>>> I can install that package, and will do so as I continue testing out the python 3.10 build but I have the following question about python3-six.
>>>>>>>>>
>>>>>>>>> Is six still needed. It is designed to ensure that programs can work with either python2 or python3 but IPFire only has python3 now.
>>>>>>>>> I can find python3-six referenced only by aws-cli. In the aws-cli build python3-six is not listed at all in the build requirements. I tried to search for aws-cli and python3-six being needed in run mode but I could not find anything one way or the other.
>>>>>>>> If aws-cli no longer requires python3-six we can drop it. Last time we touched it I had to bring the package back because aws-cli was still depending on it.
>>>>>>>>> If python3-six is really needed for running aws-cli then I can install it and the new dependency but if it is not needed we could remove two packages which would partially offset the three new dependencies required to be able to build the latest python3-setuptools-scm.
>>>>>>>> Yes, that would be great :)
>>>>>>>> -Michael
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Adolf
>>>>>>>>>
>>>>>>>>> On 03/01/2022 12:32, Michael Tremer wrote:
>>>>>>>>>> Hello,
>>>>>>>>>>> On 3 Jan 2022, at 11:31, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>
>>>>>>>>>>> On 03/01/2022 11:52, Michael Tremer wrote:
>>>>>>>>>>>> Hello,
>>>>>>>>>>>> I am not sure if you already looked at what Arch is doing, but as far as I understand it works like this:
>>>>>>>>>>> No I didn't look at Arch, which is surprising as I have in the past with other packages but I didn't. So thanks very much for the pointer.
>>>>>>>>>>>> They build Python 3 without pip and setuptools (https://github.com/archlinux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/trunk/PKGBUILD#L59).
>>>>>>>>>>>> At this stage we might need to build setuptools separately.
>>>>>>>>>>>> They use a tool called pyproject2setuppy (great name!) to build tomli (https://github.com/archlinux/svntogit-packages/blob/packages/python-tomli/trunk/PKGBUILD).
>>>>>>>>>>>> Setuptools-scm can then be built with tomli: https://github.com/archlinux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD
>>>>>>>>>>>> Would this work or did I overlook anything?
>>>>>>>>>>> I will have a look at the links and give it a go.
>>>>>>>>>>>
>>>>>>>>>>> Thanks very much, I feel re-motivated now.
>>>>>>>>>> Very good! I got a little bit lost in your last email and I think I could tell. But following a solution that somebody else has come up with can be a good thing to follow and of course we would customise it to our needs.
>>>>>>>>>> Best,
>>>>>>>>>> -Michael
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Adolf.
>>>>>>>>>>>> -Michael
>>>>>>>>>>>>> On 2 Jan 2022, at 13:44, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Michael and All,
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 26/12/2021 21:55, Michael Tremer wrote:
>>>>>>>>>>>>>> Merry Christmas,
>>>>>>>>>>>>>>> On 26 Dec 2021, at 15:38, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 17/12/2021 12:41, Michael Tremer wrote:
>>>>>>>>>>>>>>>> Hello Adolf,
>>>>>>>>>>>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I am working on updating python from 3.8 to 3.10. Python itself is okay after some other python related programs were also updated due to their older versions not working with 3.10
>>>>>>>>>>>>>>>> Thank you for working on this.
>>>>>>>>>>>>>>>>> I am working through all the programs that reference python-3.8 in their rootfiles, or wherever, to change them to 3.10
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> My question comes with regard to the changes to the rootfiles to ensure they will be included properly in an update.
>>>>>>>>>>>>>>>> We will have to ship everything that puts anything into /usr/lib*/python3.*/site-packages again with the release of Python 3.10.
>>>>>>>>>>>>>>>> This isn’t fun, but not as much of a pain in the rear as shipping a new Perl release.
>>>>>>>>>>>>>>>>> For the addons then when I change the rootfile I am also incrementing the PAK_VER number in the lfs files and that ensures that they will be properly updated.
>>>>>>>>>>>>>>>> Yes. That is required for Pakfire to find a new version.
>>>>>>>>>>>>>>>>> I don't know what to do for the core programs where I have updated the rootfile. The lfs files for these programs don't have a PAK_VER number to increment and the actual package version number is not being changed so I don't know what I need to change in the lfs to ensure that the rootfile changes will be properly included in the update?
>>>>>>>>>>>>>>>> I would say committing everything into a “python-3.10” branch and submitting it all as a patchset is fine. All packages that have been touched in that branch will have to be shipped again which should be easy to identify if it all comes in one large branch.
>>>>>>>>>>>>>>> Okay, that is clear then.Will follow that approach.
>>>>>>>>>>>>>>>>> I would appreciate any help on what I need to do for these programs that have had changes only in their rootfiles.
>>>>>>>>>>>>>>>> Absolutely. Did you find anything that didn’t build yet?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have not had any problem with python3.10 itself but I found some of the older modules would not work with 3.10 so I decided to do a combined update to 3.10 plus update modules to latest versions.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I found that after python3-setuptools-scm had run that the next module python3-six failed to run due to missing the tomli library. I commented out six and reran the build and then python3-dateutil had the same problem  and then python3-jmespath
>>>>>>>>>>>>>> I completely lost track of what the status of setuptools/distutils and so on is. This has changed multiple times and become independent/part of the distribution again and so on.
>>>>>>>>>>>>>>> After some investigation I have found that since version 6.0.0 of python3-setuptools-scm, tomli has been a required library when it is being used by other modules. So running version 6.0.0 of setuptools-scm worked with python3.10 and subsequent modules built successfully as tomli not needed.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> So the obvious fix is to install tomli but that is when I ran into a problem. tomli has no setup.py or setup.cfg files in its tarball. The only way to build it is using pip.
>>>>>>>>>>>>>> Err. Great.
>>>>>>>>>>>>>>> Apparently, after some searching, setup.py has been deprecated as the recommended install method and if setup.py doesn't work the usual fix suggestion is to run pip install.
>>>>>>>>>>>>>> Yes. But apparently the new system is deprecated, too. I really dislike the state of packaging systems that come with any language these days.
>>>>>>>>>>>>>>> Having searched on pip for a while, it looks like builds using it can still be done from tarballs locally without needing to access the internet to download the modules.
>>>>>>>>>>>>>> That is good!
>>>>>>>>>>>>>>> As pip is installed as part of python3 i tried using that.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I used "python3 -m pip install --root=/ ."  in place of
>>>>>>>>>>>>>>> "python3 setup.py build
>>>>>>>>>>>>>>> python3 setup.py install --root=/"
>>>>>>>>>>>>>>> for the tomli source.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This failed because tomli requires flit_core to be available.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> flit_core also has no setup.py and so I tried to build it with pip install. This failed with the error message that tomli was required as a dependency. So I added --no-deps to the flit_core pip install line and this time flit_core successfully built but the tomli build still came back with not finding flit_core.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have tried a variety of changes to the pip install line but none of them has been successful and it has become clear that I am out of my depth here and need guidance rather than just trying anything I find from a google search. I am not even really clear if pip install is the correct approach, except if not then I have no idea how to progress.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Help urgently needed on how I should be approaching this.
>>>>>>>>>>>>>> Circular dependencies can happen. This won’t be the first time and it always is an utter nightmare with our buildsystem.
>>>>>>>>>>>>>> This page states that pip comes by default with all binary packages:
>>>>>>>>>>>>>>   https://docs.python.org/3/installing/index.html
>>>>>>>>>>>>>> Is there a chance that it is part of the distribution and can be enabled with a configure switch?
>>>>>>>>>>>>> I wasn't able to find any way to get it enabled that way.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I did find that pip install will not install if the package is already present. You have to add --upgrade to make it update any package already present. So I tried that. Didn't work.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I found a section about needing to use bootstrapping if you want to install tomli. Followed the information for creating a wheel file for flit_core and then extracting that file into the site-packages directory. Tried that. Didn't work.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Rather than doing a complete build every time I changed to trying to build flit_core followed by tomli in a ./make.sh shell environment.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Doing that I found that I had a pip install command that was creating the flit_core directory in site-packages and also including the .dist-info directory. The info I found said that the .dist-info directory had to be there for pip to be able to find the flit_core package.
>>>>>>>>>>>>>
>>>>>>>>>>>>> However the pip install of tomli still ends up not finding any flit_core package. In the bootstrapping information it mentioned about changing the PYTHONPATH environment variable for find the flit_core package so I tried that in the shell but it also made no difference.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I then tried running python3 -c 'import flit_core' in the tomli build shell before doing the tomli pip install. The import command completed with no error but tomli still could not find any flit_core package.
>>>>>>>>>>>>>
>>>>>>>>>>>>> End result of all my work is that flit_core has been built and installed into python3.10/site-packages but nothing seems to want to make the tomli build find it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Does anyone have any ideas on what command(s) I need to use in the tomli pip install to have the installed flit_core package visible.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards and a Happy and Healthy New Year to all,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> PS:  I hope everyone had an enjoyable Christmas.
>>>>>>>>>>>>>> I hope you did, too!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>>
>>>>>>>>>>>>> -- 
>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>
>>>>>>>>> -- 
>>>>>>>>> Sent from my laptop
>>>>>>>
>>>>>>> -- 
>>>>>>> Sent from my laptop
>>>>>
>>>>> -- 
>>>>> Sent from my laptop
> 

Re: Question about updating python to 3.10

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 20695 bytes --]

Hello,

I am sure this was documented in the wiki somewhere. For some reason I cannot find it any more.

Please let me know if you need any assistance from me.

-Michael

> On 18 Jan 2022, at 21:21, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi Michael,
> 
> On 18/01/2022 19:51, Michael Tremer wrote:
>> Hello,
>>> On 18 Jan 2022, at 16:42, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>> 
>>> Hi Michael,
>>> 
>>> On 18/01/2022 16:02, Michael Tremer wrote:
>>>> Hello Adolf,
>>>> I have just been trying to build a couple of Python packages that I need and I finally arrived at needing flit-core.
>>>> I suppose since you did already build this, there is no point in me doing all this again and also there is no need to build all my packages on Python 3.8 when we can have a newer release.
>>> I am afraid that you will need to build flit-core.  I was building it when I was trying to use pip to install tomli but that approached failed to go further and that was when I used pyproject2setuppy, from your input, to build tomli and that did not need flit-core.
>> Oh, I suppose that didn’t hold for very long because I now have a package that needs it :(
> flit-core also has no setup.py but you will then be able to use the pyproject2setuppy package from my python3.10 branch to build flit-core.
>>> In terms of the python3.10 branch that I have on my git repository at home, I presume I could push it onto my personal git repository that you set up for me - https://git.ipfire.org/?p=people/bonnietwin/ipfire-2.x.git;a=summary
>> Yes, it is always a good idea to backup things to the server.
>>> The only thing is I am not sure what command I should use to push the branch from my system here to the IPFire one.
>> If you have your repository set up with “git remote add …” then you only need to run “git push bonnietwin python3.10”.
> Aaah. I think when I tried to experiment with this before I created a local new clone of bonnietwin which was then separate from my running local repository that I used for building and I struggled to figure out how to get branches between them.
> 
> Your mention of git remote add indicates that I can add my bonnietwin repository to my running build repository. Thais is what I was missing before.
> Thanks for the clue, will give it a try.
> 
> Regards,
> 
> Adolf.
>> -Michael
>>> 
>>> I can have a go at trying it and see how it goes.
>>> 
>>> Regards,
>>> 
>>> Adolf.
>>>> Could you push this branch somewhere for me so that I can base my work on top of it?
>>>> Best,
>>>> -Michael
>>>>> On 13 Jan 2022, at 09:32, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>>> 
>>>>> Hello,
>>>>> 
>>>>>> On 12 Jan 2022, at 21:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>> 
>>>>>> Hi all,
>>>>>> 
>>>>>> On 12/01/2022 19:10, Michael Tremer wrote:
>>>>>>> Hey,
>>>>>>>> On 11 Jan 2022, at 14:32, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>> 
>>>>>>>> Hi Michael and all,
>>>>>>>> 
>>>>>>>> python3-six has to stay as python3-dateutil explicitly specifies it in its dependencies.
>>>>>>>> So that solves that question.
>>>>>>> Aww :(
>>>>>>>> I have, with quite a bit of work, eventually been able to get all existing python modules updated that had updates and built successfully with python3.10
>>>>>>> Yay!
>>>>>>>> However later in the IPFire build libvirt failed with a problem parsing some documentation files.
>>>>>>>> Stashing my python3.10 changes and re-running the build had libvirt building without problems. So one of the python changes has caused it. libvirt uses python3-docutils and that was updated but reverting that back to original on its own did not cause libvirt to build successfully.
>>>>>>>> 
>>>>>>>> I looked in the libvirt .configure for an option to not build the docs but it is not mentioned under ./configure --help.
>>>>>>>> 
>>>>>>>> I tried --without-docs and --disable-docs but both were not recognised in the build.
>>>>>>>> 
>>>>>>>> The current libvirt is from mid 2020 (6.5.0) so it is likely that it is not compatible in some way with the updated python/modules.
>>>>>>>> 
>>>>>>>> I tried updating libvirt to the current 7.10.0
>>>>>>>> libvirt has replaced autotools with meson. I found meson options available for all existing ./configure options except three. There is nothing in meson options for:-
>>>>>>>> --with-virtualport
>>>>>>>> --with-macvtap
>>>>>>>> --without-dbus
>>>>>>> Yay, another project that is using meson :(
>>>>>>> We no longer need macvtap; we have dubs, so that should not hurt; and I have no idea what virtualport is.
>>>>>>>> I tried building it with all the other options and it successfully built with the updated python3.10 and modules. All other packages after libvirt built successfully without any problems.
>>>>>>>> 
>>>>>>>> Normally I would then use the updated libvirt, as that is needed to work with the updated python packages, but I am not sure if those missing options are now built into libvirt or are no longer available and what impact that might have for the use of libvirt on IPFire.
>>>>>>> I don’t think this would hurt us much. I have CCed Jonatan who maintains this, so maybe he knows more about this.
>>>>>>>> There is nothing in the libvirt changelog about virtualport, macvtap or dbus.
>>>>>>>> 
>>>>>>>> Searching on the internet, I was also not able to find anything related to libvirt and these options.
>>>>>>>> 
>>>>>>>> Any suggestions/proposals on how to move forward on this? Should I just build libvirt without those three options or ...?
>>>>>>> Yes, and I think this could be submitted independently of the Python branch because it is an add-on. We should be able to ship this at any time and might want to delay Python until we have enough space in a core update.
>>>>>> Okay, then I will stash my python3.10 changes and submit the libvirt update on its own. Then Jonaton can test it out. I don't use libvirt or qemu so I can't test it easily.
>>>>>> 
>>>>>> Then when you think the time is right, let me know and I will submit the python3.10 series of patches.
>>>>> 
>>>>> You can submit this at any time. I was just thinking about the conversation between Peter and Arne during the last call regarding an updated kernel and updated firmware. Depending on the size of Python, all of this might make an update a little bit too large - however, this should not be a patchset that we cannot merge within two merge windows.
>>>>> 
>>>>>>> Thank you for looking into this. Sounds like a not so nice riddle to solve :)
>>>>>> There were certainly some challenges, like updating python3-daemon which then wanted to have twine as a fixed dependency which had nine dependencies some of which had more. I had got to 19 additional python packages and still going and then I said enough was enough and looked at twine. It is required for uploading python packages to PyPI which IPFire doesn't need. So I tried patching the twine requirement out and that worked and then I found others that had done the same because they had no response to the request to not have twine as a mandatory build requirement.
>>>>> 
>>>>> Python seems to become the hell that Perl used to be. In order to add one perl package, you had 20 dependencies. And those dependencies had their own dependencies too… It would be great to disable or patch out whatever is possible so that we do not create more work for ourselves in the future and have to update packages that are barely used - we already have enough of these :)
>>>>> 
>>>>>> I found it all challenging but rewarding when I made progress and eventually succeeded in building everything without errors.
>>>>> 
>>>>> I am so happy that you are putting all this time and effort into this because it certainly requires patience - a trade I don’t have.
>>>>> 
>>>>> And Python is being used in all sorts of places in the distribution, so it really isn’t anything we should not be up to date on.
>>>>> 
>>>>> -Michael
>>>>> 
>>>>>> 
>>>>>> Regards,
>>>>>> Adolf.
>>>>>>> -Michael
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> 
>>>>>>>> Adolf.
>>>>>>>> 
>>>>>>>> On 07/01/2022 17:49, Michael Tremer wrote:
>>>>>>>>> Hello,
>>>>>>>>>> On 4 Jan 2022, at 12:49, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hi Michael,
>>>>>>>>>> 
>>>>>>>>>> Following the suggested Arch Linux approach I have got python3-setuptools-scm building successfully now. Yaaah.
>>>>>>>>>> I had to install python3-toml, followed by python3-pyproject2setuppy followed by python3-tomli.
>>>>>>>>>> 
>>>>>>>>>> However then python3-six which installs after python3-setuptools-scm fails due to missing the python package python3-packaging.
>>>>>>>>>> 
>>>>>>>>>> I can install that package, and will do so as I continue testing out the python 3.10 build but I have the following question about python3-six.
>>>>>>>>>> 
>>>>>>>>>> Is six still needed. It is designed to ensure that programs can work with either python2 or python3 but IPFire only has python3 now.
>>>>>>>>>> I can find python3-six referenced only by aws-cli. In the aws-cli build python3-six is not listed at all in the build requirements. I tried to search for aws-cli and python3-six being needed in run mode but I could not find anything one way or the other.
>>>>>>>>> If aws-cli no longer requires python3-six we can drop it. Last time we touched it I had to bring the package back because aws-cli was still depending on it.
>>>>>>>>>> If python3-six is really needed for running aws-cli then I can install it and the new dependency but if it is not needed we could remove two packages which would partially offset the three new dependencies required to be able to build the latest python3-setuptools-scm.
>>>>>>>>> Yes, that would be great :)
>>>>>>>>> -Michael
>>>>>>>>>> 
>>>>>>>>>> Regards,
>>>>>>>>>> Adolf
>>>>>>>>>> 
>>>>>>>>>> On 03/01/2022 12:32, Michael Tremer wrote:
>>>>>>>>>>> Hello,
>>>>>>>>>>>> On 3 Jan 2022, at 11:31, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>> 
>>>>>>>>>>>> On 03/01/2022 11:52, Michael Tremer wrote:
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>> I am not sure if you already looked at what Arch is doing, but as far as I understand it works like this:
>>>>>>>>>>>> No I didn't look at Arch, which is surprising as I have in the past with other packages but I didn't. So thanks very much for the pointer.
>>>>>>>>>>>>> They build Python 3 without pip and setuptools (https://github.com/archlinux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/trunk/PKGBUILD#L59).
>>>>>>>>>>>>> At this stage we might need to build setuptools separately.
>>>>>>>>>>>>> They use a tool called pyproject2setuppy (great name!) to build tomli (https://github.com/archlinux/svntogit-packages/blob/packages/python-tomli/trunk/PKGBUILD).
>>>>>>>>>>>>> Setuptools-scm can then be built with tomli: https://github.com/archlinux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD
>>>>>>>>>>>>> Would this work or did I overlook anything?
>>>>>>>>>>>> I will have a look at the links and give it a go.
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks very much, I feel re-motivated now.
>>>>>>>>>>> Very good! I got a little bit lost in your last email and I think I could tell. But following a solution that somebody else has come up with can be a good thing to follow and of course we would customise it to our needs.
>>>>>>>>>>> Best,
>>>>>>>>>>> -Michael
>>>>>>>>>>>> 
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>> On 2 Jan 2022, at 13:44, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi Michael and All,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 26/12/2021 21:55, Michael Tremer wrote:
>>>>>>>>>>>>>>> Merry Christmas,
>>>>>>>>>>>>>>>> On 26 Dec 2021, at 15:38, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On 17/12/2021 12:41, Michael Tremer wrote:
>>>>>>>>>>>>>>>>> Hello Adolf,
>>>>>>>>>>>>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I am working on updating python from 3.8 to 3.10. Python itself is okay after some other python related programs were also updated due to their older versions not working with 3.10
>>>>>>>>>>>>>>>>> Thank you for working on this.
>>>>>>>>>>>>>>>>>> I am working through all the programs that reference python-3.8 in their rootfiles, or wherever, to change them to 3.10
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> My question comes with regard to the changes to the rootfiles to ensure they will be included properly in an update.
>>>>>>>>>>>>>>>>> We will have to ship everything that puts anything into /usr/lib*/python3.*/site-packages again with the release of Python 3.10.
>>>>>>>>>>>>>>>>> This isn’t fun, but not as much of a pain in the rear as shipping a new Perl release.
>>>>>>>>>>>>>>>>>> For the addons then when I change the rootfile I am also incrementing the PAK_VER number in the lfs files and that ensures that they will be properly updated.
>>>>>>>>>>>>>>>>> Yes. That is required for Pakfire to find a new version.
>>>>>>>>>>>>>>>>>> I don't know what to do for the core programs where I have updated the rootfile. The lfs files for these programs don't have a PAK_VER number to increment and the actual package version number is not being changed so I don't know what I need to change in the lfs to ensure that the rootfile changes will be properly included in the update?
>>>>>>>>>>>>>>>>> I would say committing everything into a “python-3.10” branch and submitting it all as a patchset is fine. All packages that have been touched in that branch will have to be shipped again which should be easy to identify if it all comes in one large branch.
>>>>>>>>>>>>>>>> Okay, that is clear then.Will follow that approach.
>>>>>>>>>>>>>>>>>> I would appreciate any help on what I need to do for these programs that have had changes only in their rootfiles.
>>>>>>>>>>>>>>>>> Absolutely. Did you find anything that didn’t build yet?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I have not had any problem with python3.10 itself but I found some of the older modules would not work with 3.10 so I decided to do a combined update to 3.10 plus update modules to latest versions.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I found that after python3-setuptools-scm had run that the next module python3-six failed to run due to missing the tomli library. I commented out six and reran the build and then python3-dateutil had the same problem  and then python3-jmespath
>>>>>>>>>>>>>>> I completely lost track of what the status of setuptools/distutils and so on is. This has changed multiple times and become independent/part of the distribution again and so on.
>>>>>>>>>>>>>>>> After some investigation I have found that since version 6.0.0 of python3-setuptools-scm, tomli has been a required library when it is being used by other modules. So running version 6.0.0 of setuptools-scm worked with python3.10 and subsequent modules built successfully as tomli not needed.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> So the obvious fix is to install tomli but that is when I ran into a problem. tomli has no setup.py or setup.cfg files in its tarball. The only way to build it is using pip.
>>>>>>>>>>>>>>> Err. Great.
>>>>>>>>>>>>>>>> Apparently, after some searching, setup.py has been deprecated as the recommended install method and if setup.py doesn't work the usual fix suggestion is to run pip install.
>>>>>>>>>>>>>>> Yes. But apparently the new system is deprecated, too. I really dislike the state of packaging systems that come with any language these days.
>>>>>>>>>>>>>>>> Having searched on pip for a while, it looks like builds using it can still be done from tarballs locally without needing to access the internet to download the modules.
>>>>>>>>>>>>>>> That is good!
>>>>>>>>>>>>>>>> As pip is installed as part of python3 i tried using that.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I used "python3 -m pip install --root=/ ."  in place of
>>>>>>>>>>>>>>>> "python3 setup.py build
>>>>>>>>>>>>>>>> python3 setup.py install --root=/"
>>>>>>>>>>>>>>>> for the tomli source.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> This failed because tomli requires flit_core to be available.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> flit_core also has no setup.py and so I tried to build it with pip install. This failed with the error message that tomli was required as a dependency. So I added --no-deps to the flit_core pip install line and this time flit_core successfully built but the tomli build still came back with not finding flit_core.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I have tried a variety of changes to the pip install line but none of them has been successful and it has become clear that I am out of my depth here and need guidance rather than just trying anything I find from a google search. I am not even really clear if pip install is the correct approach, except if not then I have no idea how to progress.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Help urgently needed on how I should be approaching this.
>>>>>>>>>>>>>>> Circular dependencies can happen. This won’t be the first time and it always is an utter nightmare with our buildsystem.
>>>>>>>>>>>>>>> This page states that pip comes by default with all binary packages:
>>>>>>>>>>>>>>>  https://docs.python.org/3/installing/index.html
>>>>>>>>>>>>>>> Is there a chance that it is part of the distribution and can be enabled with a configure switch?
>>>>>>>>>>>>>> I wasn't able to find any way to get it enabled that way.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I did find that pip install will not install if the package is already present. You have to add --upgrade to make it update any package already present. So I tried that. Didn't work.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I found a section about needing to use bootstrapping if you want to install tomli. Followed the information for creating a wheel file for flit_core and then extracting that file into the site-packages directory. Tried that. Didn't work.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Rather than doing a complete build every time I changed to trying to build flit_core followed by tomli in a ./make.sh shell environment.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Doing that I found that I had a pip install command that was creating the flit_core directory in site-packages and also including the .dist-info directory. The info I found said that the .dist-info directory had to be there for pip to be able to find the flit_core package.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> However the pip install of tomli still ends up not finding any flit_core package. In the bootstrapping information it mentioned about changing the PYTHONPATH environment variable for find the flit_core package so I tried that in the shell but it also made no difference.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I then tried running python3 -c 'import flit_core' in the tomli build shell before doing the tomli pip install. The import command completed with no error but tomli still could not find any flit_core package.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> End result of all my work is that flit_core has been built and installed into python3.10/site-packages but nothing seems to want to make the tomli build find it.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Does anyone have any ideas on what command(s) I need to use in the tomli pip install to have the installed flit_core package visible.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Regards and a Happy and Healthy New Year to all,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> PS:  I hope everyone had an enjoyable Christmas.
>>>>>>>>>>>>>>> I hope you did, too!
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>> 
>>>>>>>>>> -- 
>>>>>>>>>> Sent from my laptop
>>>>>>>> 
>>>>>>>> -- 
>>>>>>>> Sent from my laptop
>>>>>> 
>>>>>> -- 
>>>>>> Sent from my laptop

Re: Question about updating python to 3.10

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 21418 bytes --]

Hi Michael,

On 19/01/2022 09:25, Michael Tremer wrote:
> Hello,
> 
> I am sure this was documented in the wiki somewhere. For some reason I cannot find it any more.
> 
> Please let me know if you need any assistance from me.
I got it to work thanks very much. There is now a python3.10 branch in my bonnietwin repository but it only has a few of the changes so far but I wanted to confirm that it worked.

I need to finish doing the remaining commits and then push them. Hopefully I will be able to complete those later today or tomorrow.

I will email when I have completed it and the full branch of my python3.10 changes is pushed into my repository.

Regards,

Adolf.
> 
> -Michael
> 
>> On 18 Jan 2022, at 21:21, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>
>> Hi Michael,
>>
>> On 18/01/2022 19:51, Michael Tremer wrote:
>>> Hello,
>>>> On 18 Jan 2022, at 16:42, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>
>>>> Hi Michael,
>>>>
>>>> On 18/01/2022 16:02, Michael Tremer wrote:
>>>>> Hello Adolf,
>>>>> I have just been trying to build a couple of Python packages that I need and I finally arrived at needing flit-core.
>>>>> I suppose since you did already build this, there is no point in me doing all this again and also there is no need to build all my packages on Python 3.8 when we can have a newer release.
>>>> I am afraid that you will need to build flit-core.  I was building it when I was trying to use pip to install tomli but that approached failed to go further and that was when I used pyproject2setuppy, from your input, to build tomli and that did not need flit-core.
>>> Oh, I suppose that didn’t hold for very long because I now have a package that needs it :(
>> flit-core also has no setup.py but you will then be able to use the pyproject2setuppy package from my python3.10 branch to build flit-core.
>>>> In terms of the python3.10 branch that I have on my git repository at home, I presume I could push it onto my personal git repository that you set up for me - https://git.ipfire.org/?p=people/bonnietwin/ipfire-2.x.git;a=summary
>>> Yes, it is always a good idea to backup things to the server.
>>>> The only thing is I am not sure what command I should use to push the branch from my system here to the IPFire one.
>>> If you have your repository set up with “git remote add …” then you only need to run “git push bonnietwin python3.10”.
>> Aaah. I think when I tried to experiment with this before I created a local new clone of bonnietwin which was then separate from my running local repository that I used for building and I struggled to figure out how to get branches between them.
>>
>> Your mention of git remote add indicates that I can add my bonnietwin repository to my running build repository. Thais is what I was missing before.
>> Thanks for the clue, will give it a try.
>>
>> Regards,
>>
>> Adolf.
>>> -Michael
>>>>
>>>> I can have a go at trying it and see how it goes.
>>>>
>>>> Regards,
>>>>
>>>> Adolf.
>>>>> Could you push this branch somewhere for me so that I can base my work on top of it?
>>>>> Best,
>>>>> -Michael
>>>>>> On 13 Jan 2022, at 09:32, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>>> On 12 Jan 2022, at 21:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> On 12/01/2022 19:10, Michael Tremer wrote:
>>>>>>>> Hey,
>>>>>>>>> On 11 Jan 2022, at 14:32, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>
>>>>>>>>> Hi Michael and all,
>>>>>>>>>
>>>>>>>>> python3-six has to stay as python3-dateutil explicitly specifies it in its dependencies.
>>>>>>>>> So that solves that question.
>>>>>>>> Aww :(
>>>>>>>>> I have, with quite a bit of work, eventually been able to get all existing python modules updated that had updates and built successfully with python3.10
>>>>>>>> Yay!
>>>>>>>>> However later in the IPFire build libvirt failed with a problem parsing some documentation files.
>>>>>>>>> Stashing my python3.10 changes and re-running the build had libvirt building without problems. So one of the python changes has caused it. libvirt uses python3-docutils and that was updated but reverting that back to original on its own did not cause libvirt to build successfully.
>>>>>>>>>
>>>>>>>>> I looked in the libvirt .configure for an option to not build the docs but it is not mentioned under ./configure --help.
>>>>>>>>>
>>>>>>>>> I tried --without-docs and --disable-docs but both were not recognised in the build.
>>>>>>>>>
>>>>>>>>> The current libvirt is from mid 2020 (6.5.0) so it is likely that it is not compatible in some way with the updated python/modules.
>>>>>>>>>
>>>>>>>>> I tried updating libvirt to the current 7.10.0
>>>>>>>>> libvirt has replaced autotools with meson. I found meson options available for all existing ./configure options except three. There is nothing in meson options for:-
>>>>>>>>> --with-virtualport
>>>>>>>>> --with-macvtap
>>>>>>>>> --without-dbus
>>>>>>>> Yay, another project that is using meson :(
>>>>>>>> We no longer need macvtap; we have dubs, so that should not hurt; and I have no idea what virtualport is.
>>>>>>>>> I tried building it with all the other options and it successfully built with the updated python3.10 and modules. All other packages after libvirt built successfully without any problems.
>>>>>>>>>
>>>>>>>>> Normally I would then use the updated libvirt, as that is needed to work with the updated python packages, but I am not sure if those missing options are now built into libvirt or are no longer available and what impact that might have for the use of libvirt on IPFire.
>>>>>>>> I don’t think this would hurt us much. I have CCed Jonatan who maintains this, so maybe he knows more about this.
>>>>>>>>> There is nothing in the libvirt changelog about virtualport, macvtap or dbus.
>>>>>>>>>
>>>>>>>>> Searching on the internet, I was also not able to find anything related to libvirt and these options.
>>>>>>>>>
>>>>>>>>> Any suggestions/proposals on how to move forward on this? Should I just build libvirt without those three options or ...?
>>>>>>>> Yes, and I think this could be submitted independently of the Python branch because it is an add-on. We should be able to ship this at any time and might want to delay Python until we have enough space in a core update.
>>>>>>> Okay, then I will stash my python3.10 changes and submit the libvirt update on its own. Then Jonaton can test it out. I don't use libvirt or qemu so I can't test it easily.
>>>>>>>
>>>>>>> Then when you think the time is right, let me know and I will submit the python3.10 series of patches.
>>>>>>
>>>>>> You can submit this at any time. I was just thinking about the conversation between Peter and Arne during the last call regarding an updated kernel and updated firmware. Depending on the size of Python, all of this might make an update a little bit too large - however, this should not be a patchset that we cannot merge within two merge windows.
>>>>>>
>>>>>>>> Thank you for looking into this. Sounds like a not so nice riddle to solve :)
>>>>>>> There were certainly some challenges, like updating python3-daemon which then wanted to have twine as a fixed dependency which had nine dependencies some of which had more. I had got to 19 additional python packages and still going and then I said enough was enough and looked at twine. It is required for uploading python packages to PyPI which IPFire doesn't need. So I tried patching the twine requirement out and that worked and then I found others that had done the same because they had no response to the request to not have twine as a mandatory build requirement.
>>>>>>
>>>>>> Python seems to become the hell that Perl used to be. In order to add one perl package, you had 20 dependencies. And those dependencies had their own dependencies too… It would be great to disable or patch out whatever is possible so that we do not create more work for ourselves in the future and have to update packages that are barely used - we already have enough of these :)
>>>>>>
>>>>>>> I found it all challenging but rewarding when I made progress and eventually succeeded in building everything without errors.
>>>>>>
>>>>>> I am so happy that you are putting all this time and effort into this because it certainly requires patience - a trade I don’t have.
>>>>>>
>>>>>> And Python is being used in all sorts of places in the distribution, so it really isn’t anything we should not be up to date on.
>>>>>>
>>>>>> -Michael
>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>> Adolf.
>>>>>>>> -Michael
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Adolf.
>>>>>>>>>
>>>>>>>>> On 07/01/2022 17:49, Michael Tremer wrote:
>>>>>>>>>> Hello,
>>>>>>>>>>> On 4 Jan 2022, at 12:49, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>
>>>>>>>>>>> Following the suggested Arch Linux approach I have got python3-setuptools-scm building successfully now. Yaaah.
>>>>>>>>>>> I had to install python3-toml, followed by python3-pyproject2setuppy followed by python3-tomli.
>>>>>>>>>>>
>>>>>>>>>>> However then python3-six which installs after python3-setuptools-scm fails due to missing the python package python3-packaging.
>>>>>>>>>>>
>>>>>>>>>>> I can install that package, and will do so as I continue testing out the python 3.10 build but I have the following question about python3-six.
>>>>>>>>>>>
>>>>>>>>>>> Is six still needed. It is designed to ensure that programs can work with either python2 or python3 but IPFire only has python3 now.
>>>>>>>>>>> I can find python3-six referenced only by aws-cli. In the aws-cli build python3-six is not listed at all in the build requirements. I tried to search for aws-cli and python3-six being needed in run mode but I could not find anything one way or the other.
>>>>>>>>>> If aws-cli no longer requires python3-six we can drop it. Last time we touched it I had to bring the package back because aws-cli was still depending on it.
>>>>>>>>>>> If python3-six is really needed for running aws-cli then I can install it and the new dependency but if it is not needed we could remove two packages which would partially offset the three new dependencies required to be able to build the latest python3-setuptools-scm.
>>>>>>>>>> Yes, that would be great :)
>>>>>>>>>> -Michael
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Adolf
>>>>>>>>>>>
>>>>>>>>>>> On 03/01/2022 12:32, Michael Tremer wrote:
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>> On 3 Jan 2022, at 11:31, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 03/01/2022 11:52, Michael Tremer wrote:
>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>> I am not sure if you already looked at what Arch is doing, but as far as I understand it works like this:
>>>>>>>>>>>>> No I didn't look at Arch, which is surprising as I have in the past with other packages but I didn't. So thanks very much for the pointer.
>>>>>>>>>>>>>> They build Python 3 without pip and setuptools (https://github.com/archlinux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/trunk/PKGBUILD#L59).
>>>>>>>>>>>>>> At this stage we might need to build setuptools separately.
>>>>>>>>>>>>>> They use a tool called pyproject2setuppy (great name!) to build tomli (https://github.com/archlinux/svntogit-packages/blob/packages/python-tomli/trunk/PKGBUILD).
>>>>>>>>>>>>>> Setuptools-scm can then be built with tomli: https://github.com/archlinux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD
>>>>>>>>>>>>>> Would this work or did I overlook anything?
>>>>>>>>>>>>> I will have a look at the links and give it a go.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks very much, I feel re-motivated now.
>>>>>>>>>>>> Very good! I got a little bit lost in your last email and I think I could tell. But following a solution that somebody else has come up with can be a good thing to follow and of course we would customise it to our needs.
>>>>>>>>>>>> Best,
>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>> On 2 Jan 2022, at 13:44, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Michael and All,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 26/12/2021 21:55, Michael Tremer wrote:
>>>>>>>>>>>>>>>> Merry Christmas,
>>>>>>>>>>>>>>>>> On 26 Dec 2021, at 15:38, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 17/12/2021 12:41, Michael Tremer wrote:
>>>>>>>>>>>>>>>>>> Hello Adolf,
>>>>>>>>>>>>>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I am working on updating python from 3.8 to 3.10. Python itself is okay after some other python related programs were also updated due to their older versions not working with 3.10
>>>>>>>>>>>>>>>>>> Thank you for working on this.
>>>>>>>>>>>>>>>>>>> I am working through all the programs that reference python-3.8 in their rootfiles, or wherever, to change them to 3.10
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> My question comes with regard to the changes to the rootfiles to ensure they will be included properly in an update.
>>>>>>>>>>>>>>>>>> We will have to ship everything that puts anything into /usr/lib*/python3.*/site-packages again with the release of Python 3.10.
>>>>>>>>>>>>>>>>>> This isn’t fun, but not as much of a pain in the rear as shipping a new Perl release.
>>>>>>>>>>>>>>>>>>> For the addons then when I change the rootfile I am also incrementing the PAK_VER number in the lfs files and that ensures that they will be properly updated.
>>>>>>>>>>>>>>>>>> Yes. That is required for Pakfire to find a new version.
>>>>>>>>>>>>>>>>>>> I don't know what to do for the core programs where I have updated the rootfile. The lfs files for these programs don't have a PAK_VER number to increment and the actual package version number is not being changed so I don't know what I need to change in the lfs to ensure that the rootfile changes will be properly included in the update?
>>>>>>>>>>>>>>>>>> I would say committing everything into a “python-3.10” branch and submitting it all as a patchset is fine. All packages that have been touched in that branch will have to be shipped again which should be easy to identify if it all comes in one large branch.
>>>>>>>>>>>>>>>>> Okay, that is clear then.Will follow that approach.
>>>>>>>>>>>>>>>>>>> I would appreciate any help on what I need to do for these programs that have had changes only in their rootfiles.
>>>>>>>>>>>>>>>>>> Absolutely. Did you find anything that didn’t build yet?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I have not had any problem with python3.10 itself but I found some of the older modules would not work with 3.10 so I decided to do a combined update to 3.10 plus update modules to latest versions.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I found that after python3-setuptools-scm had run that the next module python3-six failed to run due to missing the tomli library. I commented out six and reran the build and then python3-dateutil had the same problem  and then python3-jmespath
>>>>>>>>>>>>>>>> I completely lost track of what the status of setuptools/distutils and so on is. This has changed multiple times and become independent/part of the distribution again and so on.
>>>>>>>>>>>>>>>>> After some investigation I have found that since version 6.0.0 of python3-setuptools-scm, tomli has been a required library when it is being used by other modules. So running version 6.0.0 of setuptools-scm worked with python3.10 and subsequent modules built successfully as tomli not needed.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> So the obvious fix is to install tomli but that is when I ran into a problem. tomli has no setup.py or setup.cfg files in its tarball. The only way to build it is using pip.
>>>>>>>>>>>>>>>> Err. Great.
>>>>>>>>>>>>>>>>> Apparently, after some searching, setup.py has been deprecated as the recommended install method and if setup.py doesn't work the usual fix suggestion is to run pip install.
>>>>>>>>>>>>>>>> Yes. But apparently the new system is deprecated, too. I really dislike the state of packaging systems that come with any language these days.
>>>>>>>>>>>>>>>>> Having searched on pip for a while, it looks like builds using it can still be done from tarballs locally without needing to access the internet to download the modules.
>>>>>>>>>>>>>>>> That is good!
>>>>>>>>>>>>>>>>> As pip is installed as part of python3 i tried using that.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I used "python3 -m pip install --root=/ ."  in place of
>>>>>>>>>>>>>>>>> "python3 setup.py build
>>>>>>>>>>>>>>>>> python3 setup.py install --root=/"
>>>>>>>>>>>>>>>>> for the tomli source.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This failed because tomli requires flit_core to be available.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> flit_core also has no setup.py and so I tried to build it with pip install. This failed with the error message that tomli was required as a dependency. So I added --no-deps to the flit_core pip install line and this time flit_core successfully built but the tomli build still came back with not finding flit_core.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I have tried a variety of changes to the pip install line but none of them has been successful and it has become clear that I am out of my depth here and need guidance rather than just trying anything I find from a google search. I am not even really clear if pip install is the correct approach, except if not then I have no idea how to progress.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Help urgently needed on how I should be approaching this.
>>>>>>>>>>>>>>>> Circular dependencies can happen. This won’t be the first time and it always is an utter nightmare with our buildsystem.
>>>>>>>>>>>>>>>> This page states that pip comes by default with all binary packages:
>>>>>>>>>>>>>>>>   https://docs.python.org/3/installing/index.html
>>>>>>>>>>>>>>>> Is there a chance that it is part of the distribution and can be enabled with a configure switch?
>>>>>>>>>>>>>>> I wasn't able to find any way to get it enabled that way.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I did find that pip install will not install if the package is already present. You have to add --upgrade to make it update any package already present. So I tried that. Didn't work.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I found a section about needing to use bootstrapping if you want to install tomli. Followed the information for creating a wheel file for flit_core and then extracting that file into the site-packages directory. Tried that. Didn't work.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Rather than doing a complete build every time I changed to trying to build flit_core followed by tomli in a ./make.sh shell environment.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Doing that I found that I had a pip install command that was creating the flit_core directory in site-packages and also including the .dist-info directory. The info I found said that the .dist-info directory had to be there for pip to be able to find the flit_core package.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> However the pip install of tomli still ends up not finding any flit_core package. In the bootstrapping information it mentioned about changing the PYTHONPATH environment variable for find the flit_core package so I tried that in the shell but it also made no difference.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I then tried running python3 -c 'import flit_core' in the tomli build shell before doing the tomli pip install. The import command completed with no error but tomli still could not find any flit_core package.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> End result of all my work is that flit_core has been built and installed into python3.10/site-packages but nothing seems to want to make the tomli build find it.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Does anyone have any ideas on what command(s) I need to use in the tomli pip install to have the installed flit_core package visible.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards and a Happy and Healthy New Year to all,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> PS:  I hope everyone had an enjoyable Christmas.
>>>>>>>>>>>>>>>> I hope you did, too!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>
>>>>>>>>>>> -- 
>>>>>>>>>>> Sent from my laptop
>>>>>>>>>
>>>>>>>>> -- 
>>>>>>>>> Sent from my laptop
>>>>>>>
>>>>>>> -- 
>>>>>>> Sent from my laptop
> 

Re: Question about updating python to 3.10

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 22408 bytes --]

Hi Michael,

There will be a bit more time before I have the python3.10 branch available.

I made a mistake with one of the commits and didn't notice till some while after. So I did a git reset --soft "commit hash" which I have been successful with in the past but this time something went wrong and messed things up.

I have had to re delete my local python3.10 branch with the commits done so far and am now doing a full build again to make sure that my restart doesn't have any errors in it.

I will be able to restart doing the commits again tomorrow so probably branch will be available on Friday.

Sorry.

Regards,

Adolf.

On 19/01/2022 13:33, Adolf Belka wrote:
> Hi Michael,
>
> On 19/01/2022 09:25, Michael Tremer wrote:
>> Hello,
>>
>> I am sure this was documented in the wiki somewhere. For some reason I cannot find it any more.
>>
>> Please let me know if you need any assistance from me.
> I got it to work thanks very much. There is now a python3.10 branch in my bonnietwin repository but it only has a few of the changes so far but I wanted to confirm that it worked.
>
> I need to finish doing the remaining commits and then push them. Hopefully I will be able to complete those later today or tomorrow.
>
> I will email when I have completed it and the full branch of my python3.10 changes is pushed into my repository.
>
> Regards,
>
> Adolf.
>>
>> -Michael
>>
>>> On 18 Jan 2022, at 21:21, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>
>>> Hi Michael,
>>>
>>> On 18/01/2022 19:51, Michael Tremer wrote:
>>>> Hello,
>>>>> On 18 Jan 2022, at 16:42, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>
>>>>> Hi Michael,
>>>>>
>>>>> On 18/01/2022 16:02, Michael Tremer wrote:
>>>>>> Hello Adolf,
>>>>>> I have just been trying to build a couple of Python packages that I need and I finally arrived at needing flit-core.
>>>>>> I suppose since you did already build this, there is no point in me doing all this again and also there is no need to build all my packages on Python 3.8 when we can have a newer release.
>>>>> I am afraid that you will need to build flit-core.  I was building it when I was trying to use pip to install tomli but that approached failed to go further and that was when I used pyproject2setuppy, from your input, to build tomli and that did not need flit-core.
>>>> Oh, I suppose that didn’t hold for very long because I now have a package that needs it :(
>>> flit-core also has no setup.py but you will then be able to use the pyproject2setuppy package from my python3.10 branch to build flit-core.
>>>>> In terms of the python3.10 branch that I have on my git repository at home, I presume I could push it onto my personal git repository that you set up for me - https://git.ipfire.org/?p=people/bonnietwin/ipfire-2.x.git;a=summary
>>>> Yes, it is always a good idea to backup things to the server.
>>>>> The only thing is I am not sure what command I should use to push the branch from my system here to the IPFire one.
>>>> If you have your repository set up with “git remote add …” then you only need to run “git push bonnietwin python3.10”.
>>> Aaah. I think when I tried to experiment with this before I created a local new clone of bonnietwin which was then separate from my running local repository that I used for building and I struggled to figure out how to get branches between them.
>>>
>>> Your mention of git remote add indicates that I can add my bonnietwin repository to my running build repository. Thais is what I was missing before.
>>> Thanks for the clue, will give it a try.
>>>
>>> Regards,
>>>
>>> Adolf.
>>>> -Michael
>>>>>
>>>>> I can have a go at trying it and see how it goes.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Adolf.
>>>>>> Could you push this branch somewhere for me so that I can base my work on top of it?
>>>>>> Best,
>>>>>> -Michael
>>>>>>> On 13 Jan 2022, at 09:32, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>>> On 12 Jan 2022, at 21:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> On 12/01/2022 19:10, Michael Tremer wrote:
>>>>>>>>> Hey,
>>>>>>>>>> On 11 Jan 2022, at 14:32, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Michael and all,
>>>>>>>>>>
>>>>>>>>>> python3-six has to stay as python3-dateutil explicitly specifies it in its dependencies.
>>>>>>>>>> So that solves that question.
>>>>>>>>> Aww :(
>>>>>>>>>> I have, with quite a bit of work, eventually been able to get all existing python modules updated that had updates and built successfully with python3.10
>>>>>>>>> Yay!
>>>>>>>>>> However later in the IPFire build libvirt failed with a problem parsing some documentation files.
>>>>>>>>>> Stashing my python3.10 changes and re-running the build had libvirt building without problems. So one of the python changes has caused it. libvirt uses python3-docutils and that was updated but reverting that back to original on its own did not cause libvirt to build successfully.
>>>>>>>>>>
>>>>>>>>>> I looked in the libvirt .configure for an option to not build the docs but it is not mentioned under ./configure --help.
>>>>>>>>>>
>>>>>>>>>> I tried --without-docs and --disable-docs but both were not recognised in the build.
>>>>>>>>>>
>>>>>>>>>> The current libvirt is from mid 2020 (6.5.0) so it is likely that it is not compatible in some way with the updated python/modules.
>>>>>>>>>>
>>>>>>>>>> I tried updating libvirt to the current 7.10.0
>>>>>>>>>> libvirt has replaced autotools with meson. I found meson options available for all existing ./configure options except three. There is nothing in meson options for:-
>>>>>>>>>> --with-virtualport
>>>>>>>>>> --with-macvtap
>>>>>>>>>> --without-dbus
>>>>>>>>> Yay, another project that is using meson :(
>>>>>>>>> We no longer need macvtap; we have dubs, so that should not hurt; and I have no idea what virtualport is.
>>>>>>>>>> I tried building it with all the other options and it successfully built with the updated python3.10 and modules. All other packages after libvirt built successfully without any problems.
>>>>>>>>>>
>>>>>>>>>> Normally I would then use the updated libvirt, as that is needed to work with the updated python packages, but I am not sure if those missing options are now built into libvirt or are no longer available and what impact that might have for the use of libvirt on IPFire.
>>>>>>>>> I don’t think this would hurt us much. I have CCed Jonatan who maintains this, so maybe he knows more about this.
>>>>>>>>>> There is nothing in the libvirt changelog about virtualport, macvtap or dbus.
>>>>>>>>>>
>>>>>>>>>> Searching on the internet, I was also not able to find anything related to libvirt and these options.
>>>>>>>>>>
>>>>>>>>>> Any suggestions/proposals on how to move forward on this? Should I just build libvirt without those three options or ...?
>>>>>>>>> Yes, and I think this could be submitted independently of the Python branch because it is an add-on. We should be able to ship this at any time and might want to delay Python until we have enough space in a core update.
>>>>>>>> Okay, then I will stash my python3.10 changes and submit the libvirt update on its own. Then Jonaton can test it out. I don't use libvirt or qemu so I can't test it easily.
>>>>>>>>
>>>>>>>> Then when you think the time is right, let me know and I will submit the python3.10 series of patches.
>>>>>>>
>>>>>>> You can submit this at any time. I was just thinking about the conversation between Peter and Arne during the last call regarding an updated kernel and updated firmware. Depending on the size of Python, all of this might make an update a little bit too large - however, this should not be a patchset that we cannot merge within two merge windows.
>>>>>>>
>>>>>>>>> Thank you for looking into this. Sounds like a not so nice riddle to solve :)
>>>>>>>> There were certainly some challenges, like updating python3-daemon which then wanted to have twine as a fixed dependency which had nine dependencies some of which had more. I had got to 19 additional python packages and still going and then I said enough was enough and looked at twine. It is required for uploading python packages to PyPI which IPFire doesn't need. So I tried patching the twine requirement out and that worked and then I found others that had done the same because they had no response to the request to not have twine as a mandatory build requirement.
>>>>>>>
>>>>>>> Python seems to become the hell that Perl used to be. In order to add one perl package, you had 20 dependencies. And those dependencies had their own dependencies too… It would be great to disable or patch out whatever is possible so that we do not create more work for ourselves in the future and have to update packages that are barely used - we already have enough of these :)
>>>>>>>
>>>>>>>> I found it all challenging but rewarding when I made progress and eventually succeeded in building everything without errors.
>>>>>>>
>>>>>>> I am so happy that you are putting all this time and effort into this because it certainly requires patience - a trade I don’t have.
>>>>>>>
>>>>>>> And Python is being used in all sorts of places in the distribution, so it really isn’t anything we should not be up to date on.
>>>>>>>
>>>>>>> -Michael
>>>>>>>
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Adolf.
>>>>>>>>> -Michael
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>> Adolf.
>>>>>>>>>>
>>>>>>>>>> On 07/01/2022 17:49, Michael Tremer wrote:
>>>>>>>>>>> Hello,
>>>>>>>>>>>> On 4 Jan 2022, at 12:49, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>
>>>>>>>>>>>> Following the suggested Arch Linux approach I have got python3-setuptools-scm building successfully now. Yaaah.
>>>>>>>>>>>> I had to install python3-toml, followed by python3-pyproject2setuppy followed by python3-tomli.
>>>>>>>>>>>>
>>>>>>>>>>>> However then python3-six which installs after python3-setuptools-scm fails due to missing the python package python3-packaging.
>>>>>>>>>>>>
>>>>>>>>>>>> I can install that package, and will do so as I continue testing out the python 3.10 build but I have the following question about python3-six.
>>>>>>>>>>>>
>>>>>>>>>>>> Is six still needed. It is designed to ensure that programs can work with either python2 or python3 but IPFire only has python3 now.
>>>>>>>>>>>> I can find python3-six referenced only by aws-cli. In the aws-cli build python3-six is not listed at all in the build requirements. I tried to search for aws-cli and python3-six being needed in run mode but I could not find anything one way or the other.
>>>>>>>>>>> If aws-cli no longer requires python3-six we can drop it. Last time we touched it I had to bring the package back because aws-cli was still depending on it.
>>>>>>>>>>>> If python3-six is really needed for running aws-cli then I can install it and the new dependency but if it is not needed we could remove two packages which would partially offset the three new dependencies required to be able to build the latest python3-setuptools-scm.
>>>>>>>>>>> Yes, that would be great :)
>>>>>>>>>>> -Michael
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Adolf
>>>>>>>>>>>>
>>>>>>>>>>>> On 03/01/2022 12:32, Michael Tremer wrote:
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>> On 3 Jan 2022, at 11:31, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 03/01/2022 11:52, Michael Tremer wrote:
>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>> I am not sure if you already looked at what Arch is doing, but as far as I understand it works like this:
>>>>>>>>>>>>>> No I didn't look at Arch, which is surprising as I have in the past with other packages but I didn't. So thanks very much for the pointer.
>>>>>>>>>>>>>>> They build Python 3 without pip and setuptools (https://github.com/archlinux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/trunk/PKGBUILD#L59).
>>>>>>>>>>>>>>> At this stage we might need to build setuptools separately.
>>>>>>>>>>>>>>> They use a tool called pyproject2setuppy (great name!) to build tomli (https://github.com/archlinux/svntogit-packages/blob/packages/python-tomli/trunk/PKGBUILD).
>>>>>>>>>>>>>>> Setuptools-scm can then be built with tomli: https://github.com/archlinux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD
>>>>>>>>>>>>>>> Would this work or did I overlook anything?
>>>>>>>>>>>>>> I will have a look at the links and give it a go.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks very much, I feel re-motivated now.
>>>>>>>>>>>>> Very good! I got a little bit lost in your last email and I think I could tell. But following a solution that somebody else has come up with can be a good thing to follow and of course we would customise it to our needs.
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>> On 2 Jan 2022, at 13:44, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Michael and All,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 26/12/2021 21:55, Michael Tremer wrote:
>>>>>>>>>>>>>>>>> Merry Christmas,
>>>>>>>>>>>>>>>>>> On 26 Dec 2021, at 15:38, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 17/12/2021 12:41, Michael Tremer wrote:
>>>>>>>>>>>>>>>>>>> Hello Adolf,
>>>>>>>>>>>>>>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I am working on updating python from 3.8 to 3.10. Python itself is okay after some other python related programs were also updated due to their older versions not working with 3.10
>>>>>>>>>>>>>>>>>>> Thank you for working on this.
>>>>>>>>>>>>>>>>>>>> I am working through all the programs that reference python-3.8 in their rootfiles, or wherever, to change them to 3.10
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> My question comes with regard to the changes to the rootfiles to ensure they will be included properly in an update.
>>>>>>>>>>>>>>>>>>> We will have to ship everything that puts anything into /usr/lib*/python3.*/site-packages again with the release of Python 3.10.
>>>>>>>>>>>>>>>>>>> This isn’t fun, but not as much of a pain in the rear as shipping a new Perl release.
>>>>>>>>>>>>>>>>>>>> For the addons then when I change the rootfile I am also incrementing the PAK_VER number in the lfs files and that ensures that they will be properly updated.
>>>>>>>>>>>>>>>>>>> Yes. That is required for Pakfire to find a new version.
>>>>>>>>>>>>>>>>>>>> I don't know what to do for the core programs where I have updated the rootfile. The lfs files for these programs don't have a PAK_VER number to increment and the actual package version number is not being changed so I don't know what I need to change in the lfs to ensure that the rootfile changes will be properly included in the update?
>>>>>>>>>>>>>>>>>>> I would say committing everything into a “python-3.10” branch and submitting it all as a patchset is fine. All packages that have been touched in that branch will have to be shipped again which should be easy to identify if it all comes in one large branch.
>>>>>>>>>>>>>>>>>> Okay, that is clear then.Will follow that approach.
>>>>>>>>>>>>>>>>>>>> I would appreciate any help on what I need to do for these programs that have had changes only in their rootfiles.
>>>>>>>>>>>>>>>>>>> Absolutely. Did you find anything that didn’t build yet?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I have not had any problem with python3.10 itself but I found some of the older modules would not work with 3.10 so I decided to do a combined update to 3.10 plus update modules to latest versions.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I found that after python3-setuptools-scm had run that the next module python3-six failed to run due to missing the tomli library. I commented out six and reran the build and then python3-dateutil had the same problem  and then python3-jmespath
>>>>>>>>>>>>>>>>> I completely lost track of what the status of setuptools/distutils and so on is. This has changed multiple times and become independent/part of the distribution again and so on.
>>>>>>>>>>>>>>>>>> After some investigation I have found that since version 6.0.0 of python3-setuptools-scm, tomli has been a required library when it is being used by other modules. So running version 6.0.0 of setuptools-scm worked with python3.10 and subsequent modules built successfully as tomli not needed.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> So the obvious fix is to install tomli but that is when I ran into a problem. tomli has no setup.py or setup.cfg files in its tarball. The only way to build it is using pip.
>>>>>>>>>>>>>>>>> Err. Great.
>>>>>>>>>>>>>>>>>> Apparently, after some searching, setup.py has been deprecated as the recommended install method and if setup.py doesn't work the usual fix suggestion is to run pip install.
>>>>>>>>>>>>>>>>> Yes. But apparently the new system is deprecated, too. I really dislike the state of packaging systems that come with any language these days.
>>>>>>>>>>>>>>>>>> Having searched on pip for a while, it looks like builds using it can still be done from tarballs locally without needing to access the internet to download the modules.
>>>>>>>>>>>>>>>>> That is good!
>>>>>>>>>>>>>>>>>> As pip is installed as part of python3 i tried using that.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I used "python3 -m pip install --root=/ ."  in place of
>>>>>>>>>>>>>>>>>> "python3 setup.py build
>>>>>>>>>>>>>>>>>> python3 setup.py install --root=/"
>>>>>>>>>>>>>>>>>> for the tomli source.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This failed because tomli requires flit_core to be available.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> flit_core also has no setup.py and so I tried to build it with pip install. This failed with the error message that tomli was required as a dependency. So I added --no-deps to the flit_core pip install line and this time flit_core successfully built but the tomli build still came back with not finding flit_core.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I have tried a variety of changes to the pip install line but none of them has been successful and it has become clear that I am out of my depth here and need guidance rather than just trying anything I find from a google search. I am not even really clear if pip install is the correct approach, except if not then I have no idea how to progress.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Help urgently needed on how I should be approaching this.
>>>>>>>>>>>>>>>>> Circular dependencies can happen. This won’t be the first time and it always is an utter nightmare with our buildsystem.
>>>>>>>>>>>>>>>>> This page states that pip comes by default with all binary packages:
>>>>>>>>>>>>>>>>> https://docs.python.org/3/installing/index.html
>>>>>>>>>>>>>>>>> Is there a chance that it is part of the distribution and can be enabled with a configure switch?
>>>>>>>>>>>>>>>> I wasn't able to find any way to get it enabled that way.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I did find that pip install will not install if the package is already present. You have to add --upgrade to make it update any package already present. So I tried that. Didn't work.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I found a section about needing to use bootstrapping if you want to install tomli. Followed the information for creating a wheel file for flit_core and then extracting that file into the site-packages directory. Tried that. Didn't work.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Rather than doing a complete build every time I changed to trying to build flit_core followed by tomli in a ./make.sh shell environment.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Doing that I found that I had a pip install command that was creating the flit_core directory in site-packages and also including the .dist-info directory. The info I found said that the .dist-info directory had to be there for pip to be able to find the flit_core package.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> However the pip install of tomli still ends up not finding any flit_core package. In the bootstrapping information it mentioned about changing the PYTHONPATH environment variable for find the flit_core package so I tried that in the shell but it also made no difference.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I then tried running python3 -c 'import flit_core' in the tomli build shell before doing the tomli pip install. The import command completed with no error but tomli still could not find any flit_core package.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> End result of all my work is that flit_core has been built and installed into python3.10/site-packages but nothing seems to want to make the tomli build find it.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Does anyone have any ideas on what command(s) I need to use in the tomli pip install to have the installed flit_core package visible.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regards and a Happy and Healthy New Year to all,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> PS:  I hope everyone had an enjoyable Christmas.
>>>>>>>>>>>>>>>>> I hope you did, too!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>
>>>>>>>>>>>> -- 
>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>
>>>>>>>>>> -- 
>>>>>>>>>> Sent from my laptop
>>>>>>>>
>>>>>>>> -- 
>>>>>>>> Sent from my laptop
>>

Re: Question about updating python to 3.10

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 23149 bytes --]

Hello,

> On 19 Jan 2022, at 19:52, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi Michael,
> 
> There will be a bit more time before I have the python3.10 branch available.
> 
> I made a mistake with one of the commits and didn't notice till some while after. So I did a git reset --soft "commit hash" which I have been successful with in the past but this time something went wrong and messed things up.

This happens. Branches like this don’t have to be perfect. They are work-in-progress and often that has its problems :)

> I have had to re delete my local python3.10 branch with the commits done so far and am now doing a full build again to make sure that my restart doesn't have any errors in it.
> 
> I will be able to restart doing the commits again tomorrow so probably branch will be available on Friday.

Thank you, I will have a look tomorrow and rebase my branch based on yours.

Best,
-Michael

> 
> Sorry.
> 
> Regards,
> 
> Adolf.
> 
> On 19/01/2022 13:33, Adolf Belka wrote:
>> Hi Michael,
>> 
>> On 19/01/2022 09:25, Michael Tremer wrote:
>>> Hello,
>>> 
>>> I am sure this was documented in the wiki somewhere. For some reason I cannot find it any more.
>>> 
>>> Please let me know if you need any assistance from me.
>> I got it to work thanks very much. There is now a python3.10 branch in my bonnietwin repository but it only has a few of the changes so far but I wanted to confirm that it worked.
>> 
>> I need to finish doing the remaining commits and then push them. Hopefully I will be able to complete those later today or tomorrow.
>> 
>> I will email when I have completed it and the full branch of my python3.10 changes is pushed into my repository.
>> 
>> Regards,
>> 
>> Adolf.
>>> 
>>> -Michael
>>> 
>>>> On 18 Jan 2022, at 21:21, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>> 
>>>> Hi Michael,
>>>> 
>>>> On 18/01/2022 19:51, Michael Tremer wrote:
>>>>> Hello,
>>>>>> On 18 Jan 2022, at 16:42, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>> 
>>>>>> Hi Michael,
>>>>>> 
>>>>>> On 18/01/2022 16:02, Michael Tremer wrote:
>>>>>>> Hello Adolf,
>>>>>>> I have just been trying to build a couple of Python packages that I need and I finally arrived at needing flit-core.
>>>>>>> I suppose since you did already build this, there is no point in me doing all this again and also there is no need to build all my packages on Python 3.8 when we can have a newer release.
>>>>>> I am afraid that you will need to build flit-core.  I was building it when I was trying to use pip to install tomli but that approached failed to go further and that was when I used pyproject2setuppy, from your input, to build tomli and that did not need flit-core.
>>>>> Oh, I suppose that didn’t hold for very long because I now have a package that needs it :(
>>>> flit-core also has no setup.py but you will then be able to use the pyproject2setuppy package from my python3.10 branch to build flit-core.
>>>>>> In terms of the python3.10 branch that I have on my git repository at home, I presume I could push it onto my personal git repository that you set up for me - https://git.ipfire.org/?p=people/bonnietwin/ipfire-2.x.git;a=summary
>>>>> Yes, it is always a good idea to backup things to the server.
>>>>>> The only thing is I am not sure what command I should use to push the branch from my system here to the IPFire one.
>>>>> If you have your repository set up with “git remote add …” then you only need to run “git push bonnietwin python3.10”.
>>>> Aaah. I think when I tried to experiment with this before I created a local new clone of bonnietwin which was then separate from my running local repository that I used for building and I struggled to figure out how to get branches between them.
>>>> 
>>>> Your mention of git remote add indicates that I can add my bonnietwin repository to my running build repository. Thais is what I was missing before.
>>>> Thanks for the clue, will give it a try.
>>>> 
>>>> Regards,
>>>> 
>>>> Adolf.
>>>>> -Michael
>>>>>> 
>>>>>> I can have a go at trying it and see how it goes.
>>>>>> 
>>>>>> Regards,
>>>>>> 
>>>>>> Adolf.
>>>>>>> Could you push this branch somewhere for me so that I can base my work on top of it?
>>>>>>> Best,
>>>>>>> -Michael
>>>>>>>> On 13 Jan 2022, at 09:32, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>>>>>> 
>>>>>>>> Hello,
>>>>>>>> 
>>>>>>>>> On 12 Jan 2022, at 21:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>> 
>>>>>>>>> Hi all,
>>>>>>>>> 
>>>>>>>>> On 12/01/2022 19:10, Michael Tremer wrote:
>>>>>>>>>> Hey,
>>>>>>>>>>> On 11 Jan 2022, at 14:32, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Hi Michael and all,
>>>>>>>>>>> 
>>>>>>>>>>> python3-six has to stay as python3-dateutil explicitly specifies it in its dependencies.
>>>>>>>>>>> So that solves that question.
>>>>>>>>>> Aww :(
>>>>>>>>>>> I have, with quite a bit of work, eventually been able to get all existing python modules updated that had updates and built successfully with python3.10
>>>>>>>>>> Yay!
>>>>>>>>>>> However later in the IPFire build libvirt failed with a problem parsing some documentation files.
>>>>>>>>>>> Stashing my python3.10 changes and re-running the build had libvirt building without problems. So one of the python changes has caused it. libvirt uses python3-docutils and that was updated but reverting that back to original on its own did not cause libvirt to build successfully.
>>>>>>>>>>> 
>>>>>>>>>>> I looked in the libvirt .configure for an option to not build the docs but it is not mentioned under ./configure --help.
>>>>>>>>>>> 
>>>>>>>>>>> I tried --without-docs and --disable-docs but both were not recognised in the build.
>>>>>>>>>>> 
>>>>>>>>>>> The current libvirt is from mid 2020 (6.5.0) so it is likely that it is not compatible in some way with the updated python/modules.
>>>>>>>>>>> 
>>>>>>>>>>> I tried updating libvirt to the current 7.10.0
>>>>>>>>>>> libvirt has replaced autotools with meson. I found meson options available for all existing ./configure options except three. There is nothing in meson options for:-
>>>>>>>>>>> --with-virtualport
>>>>>>>>>>> --with-macvtap
>>>>>>>>>>> --without-dbus
>>>>>>>>>> Yay, another project that is using meson :(
>>>>>>>>>> We no longer need macvtap; we have dubs, so that should not hurt; and I have no idea what virtualport is.
>>>>>>>>>>> I tried building it with all the other options and it successfully built with the updated python3.10 and modules. All other packages after libvirt built successfully without any problems.
>>>>>>>>>>> 
>>>>>>>>>>> Normally I would then use the updated libvirt, as that is needed to work with the updated python packages, but I am not sure if those missing options are now built into libvirt or are no longer available and what impact that might have for the use of libvirt on IPFire.
>>>>>>>>>> I don’t think this would hurt us much. I have CCed Jonatan who maintains this, so maybe he knows more about this.
>>>>>>>>>>> There is nothing in the libvirt changelog about virtualport, macvtap or dbus.
>>>>>>>>>>> 
>>>>>>>>>>> Searching on the internet, I was also not able to find anything related to libvirt and these options.
>>>>>>>>>>> 
>>>>>>>>>>> Any suggestions/proposals on how to move forward on this? Should I just build libvirt without those three options or ...?
>>>>>>>>>> Yes, and I think this could be submitted independently of the Python branch because it is an add-on. We should be able to ship this at any time and might want to delay Python until we have enough space in a core update.
>>>>>>>>> Okay, then I will stash my python3.10 changes and submit the libvirt update on its own. Then Jonaton can test it out. I don't use libvirt or qemu so I can't test it easily.
>>>>>>>>> 
>>>>>>>>> Then when you think the time is right, let me know and I will submit the python3.10 series of patches.
>>>>>>>> 
>>>>>>>> You can submit this at any time. I was just thinking about the conversation between Peter and Arne during the last call regarding an updated kernel and updated firmware. Depending on the size of Python, all of this might make an update a little bit too large - however, this should not be a patchset that we cannot merge within two merge windows.
>>>>>>>> 
>>>>>>>>>> Thank you for looking into this. Sounds like a not so nice riddle to solve :)
>>>>>>>>> There were certainly some challenges, like updating python3-daemon which then wanted to have twine as a fixed dependency which had nine dependencies some of which had more. I had got to 19 additional python packages and still going and then I said enough was enough and looked at twine. It is required for uploading python packages to PyPI which IPFire doesn't need. So I tried patching the twine requirement out and that worked and then I found others that had done the same because they had no response to the request to not have twine as a mandatory build requirement.
>>>>>>>> 
>>>>>>>> Python seems to become the hell that Perl used to be. In order to add one perl package, you had 20 dependencies. And those dependencies had their own dependencies too… It would be great to disable or patch out whatever is possible so that we do not create more work for ourselves in the future and have to update packages that are barely used - we already have enough of these :)
>>>>>>>> 
>>>>>>>>> I found it all challenging but rewarding when I made progress and eventually succeeded in building everything without errors.
>>>>>>>> 
>>>>>>>> I am so happy that you are putting all this time and effort into this because it certainly requires patience - a trade I don’t have.
>>>>>>>> 
>>>>>>>> And Python is being used in all sorts of places in the distribution, so it really isn’t anything we should not be up to date on.
>>>>>>>> 
>>>>>>>> -Michael
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Regards,
>>>>>>>>> Adolf.
>>>>>>>>>> -Michael
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Regards,
>>>>>>>>>>> 
>>>>>>>>>>> Adolf.
>>>>>>>>>>> 
>>>>>>>>>>> On 07/01/2022 17:49, Michael Tremer wrote:
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>> On 4 Jan 2022, at 12:49, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Following the suggested Arch Linux approach I have got python3-setuptools-scm building successfully now. Yaaah.
>>>>>>>>>>>>> I had to install python3-toml, followed by python3-pyproject2setuppy followed by python3-tomli.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> However then python3-six which installs after python3-setuptools-scm fails due to missing the python package python3-packaging.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I can install that package, and will do so as I continue testing out the python 3.10 build but I have the following question about python3-six.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Is six still needed. It is designed to ensure that programs can work with either python2 or python3 but IPFire only has python3 now.
>>>>>>>>>>>>> I can find python3-six referenced only by aws-cli. In the aws-cli build python3-six is not listed at all in the build requirements. I tried to search for aws-cli and python3-six being needed in run mode but I could not find anything one way or the other.
>>>>>>>>>>>> If aws-cli no longer requires python3-six we can drop it. Last time we touched it I had to bring the package back because aws-cli was still depending on it.
>>>>>>>>>>>>> If python3-six is really needed for running aws-cli then I can install it and the new dependency but if it is not needed we could remove two packages which would partially offset the three new dependencies required to be able to build the latest python3-setuptools-scm.
>>>>>>>>>>>> Yes, that would be great :)
>>>>>>>>>>>> -Michael
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Adolf
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On 03/01/2022 12:32, Michael Tremer wrote:
>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>> On 3 Jan 2022, at 11:31, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 03/01/2022 11:52, Michael Tremer wrote:
>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>> I am not sure if you already looked at what Arch is doing, but as far as I understand it works like this:
>>>>>>>>>>>>>>> No I didn't look at Arch, which is surprising as I have in the past with other packages but I didn't. So thanks very much for the pointer.
>>>>>>>>>>>>>>>> They build Python 3 without pip and setuptools (https://github.com/archlinux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/trunk/PKGBUILD#L59).
>>>>>>>>>>>>>>>> At this stage we might need to build setuptools separately.
>>>>>>>>>>>>>>>> They use a tool called pyproject2setuppy (great name!) to build tomli (https://github.com/archlinux/svntogit-packages/blob/packages/python-tomli/trunk/PKGBUILD).
>>>>>>>>>>>>>>>> Setuptools-scm can then be built with tomli: https://github.com/archlinux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD
>>>>>>>>>>>>>>>> Would this work or did I overlook anything?
>>>>>>>>>>>>>>> I will have a look at the links and give it a go.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks very much, I feel re-motivated now.
>>>>>>>>>>>>>> Very good! I got a little bit lost in your last email and I think I could tell. But following a solution that somebody else has come up with can be a good thing to follow and of course we would customise it to our needs.
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>>> On 2 Jan 2022, at 13:44, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi Michael and All,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On 26/12/2021 21:55, Michael Tremer wrote:
>>>>>>>>>>>>>>>>>> Merry Christmas,
>>>>>>>>>>>>>>>>>>> On 26 Dec 2021, at 15:38, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On 17/12/2021 12:41, Michael Tremer wrote:
>>>>>>>>>>>>>>>>>>>> Hello Adolf,
>>>>>>>>>>>>>>>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I am working on updating python from 3.8 to 3.10. Python itself is okay after some other python related programs were also updated due to their older versions not working with 3.10
>>>>>>>>>>>>>>>>>>>> Thank you for working on this.
>>>>>>>>>>>>>>>>>>>>> I am working through all the programs that reference python-3.8 in their rootfiles, or wherever, to change them to 3.10
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> My question comes with regard to the changes to the rootfiles to ensure they will be included properly in an update.
>>>>>>>>>>>>>>>>>>>> We will have to ship everything that puts anything into /usr/lib*/python3.*/site-packages again with the release of Python 3.10.
>>>>>>>>>>>>>>>>>>>> This isn’t fun, but not as much of a pain in the rear as shipping a new Perl release.
>>>>>>>>>>>>>>>>>>>>> For the addons then when I change the rootfile I am also incrementing the PAK_VER number in the lfs files and that ensures that they will be properly updated.
>>>>>>>>>>>>>>>>>>>> Yes. That is required for Pakfire to find a new version.
>>>>>>>>>>>>>>>>>>>>> I don't know what to do for the core programs where I have updated the rootfile. The lfs files for these programs don't have a PAK_VER number to increment and the actual package version number is not being changed so I don't know what I need to change in the lfs to ensure that the rootfile changes will be properly included in the update?
>>>>>>>>>>>>>>>>>>>> I would say committing everything into a “python-3.10” branch and submitting it all as a patchset is fine. All packages that have been touched in that branch will have to be shipped again which should be easy to identify if it all comes in one large branch.
>>>>>>>>>>>>>>>>>>> Okay, that is clear then.Will follow that approach.
>>>>>>>>>>>>>>>>>>>>> I would appreciate any help on what I need to do for these programs that have had changes only in their rootfiles.
>>>>>>>>>>>>>>>>>>>> Absolutely. Did you find anything that didn’t build yet?
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I have not had any problem with python3.10 itself but I found some of the older modules would not work with 3.10 so I decided to do a combined update to 3.10 plus update modules to latest versions.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I found that after python3-setuptools-scm had run that the next module python3-six failed to run due to missing the tomli library. I commented out six and reran the build and then python3-dateutil had the same problem  and then python3-jmespath
>>>>>>>>>>>>>>>>>> I completely lost track of what the status of setuptools/distutils and so on is. This has changed multiple times and become independent/part of the distribution again and so on.
>>>>>>>>>>>>>>>>>>> After some investigation I have found that since version 6.0.0 of python3-setuptools-scm, tomli has been a required library when it is being used by other modules. So running version 6.0.0 of setuptools-scm worked with python3.10 and subsequent modules built successfully as tomli not needed.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> So the obvious fix is to install tomli but that is when I ran into a problem. tomli has no setup.py or setup.cfg files in its tarball. The only way to build it is using pip.
>>>>>>>>>>>>>>>>>> Err. Great.
>>>>>>>>>>>>>>>>>>> Apparently, after some searching, setup.py has been deprecated as the recommended install method and if setup.py doesn't work the usual fix suggestion is to run pip install.
>>>>>>>>>>>>>>>>>> Yes. But apparently the new system is deprecated, too. I really dislike the state of packaging systems that come with any language these days.
>>>>>>>>>>>>>>>>>>> Having searched on pip for a while, it looks like builds using it can still be done from tarballs locally without needing to access the internet to download the modules.
>>>>>>>>>>>>>>>>>> That is good!
>>>>>>>>>>>>>>>>>>> As pip is installed as part of python3 i tried using that.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I used "python3 -m pip install --root=/ ."  in place of
>>>>>>>>>>>>>>>>>>> "python3 setup.py build
>>>>>>>>>>>>>>>>>>> python3 setup.py install --root=/"
>>>>>>>>>>>>>>>>>>> for the tomli source.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> This failed because tomli requires flit_core to be available.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> flit_core also has no setup.py and so I tried to build it with pip install. This failed with the error message that tomli was required as a dependency. So I added --no-deps to the flit_core pip install line and this time flit_core successfully built but the tomli build still came back with not finding flit_core.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I have tried a variety of changes to the pip install line but none of them has been successful and it has become clear that I am out of my depth here and need guidance rather than just trying anything I find from a google search. I am not even really clear if pip install is the correct approach, except if not then I have no idea how to progress.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Help urgently needed on how I should be approaching this.
>>>>>>>>>>>>>>>>>> Circular dependencies can happen. This won’t be the first time and it always is an utter nightmare with our buildsystem.
>>>>>>>>>>>>>>>>>> This page states that pip comes by default with all binary packages:
>>>>>>>>>>>>>>>>>> https://docs.python.org/3/installing/index.html
>>>>>>>>>>>>>>>>>> Is there a chance that it is part of the distribution and can be enabled with a configure switch?
>>>>>>>>>>>>>>>>> I wasn't able to find any way to get it enabled that way.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I did find that pip install will not install if the package is already present. You have to add --upgrade to make it update any package already present. So I tried that. Didn't work.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I found a section about needing to use bootstrapping if you want to install tomli. Followed the information for creating a wheel file for flit_core and then extracting that file into the site-packages directory. Tried that. Didn't work.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Rather than doing a complete build every time I changed to trying to build flit_core followed by tomli in a ./make.sh shell environment.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Doing that I found that I had a pip install command that was creating the flit_core directory in site-packages and also including the .dist-info directory. The info I found said that the .dist-info directory had to be there for pip to be able to find the flit_core package.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> However the pip install of tomli still ends up not finding any flit_core package. In the bootstrapping information it mentioned about changing the PYTHONPATH environment variable for find the flit_core package so I tried that in the shell but it also made no difference.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I then tried running python3 -c 'import flit_core' in the tomli build shell before doing the tomli pip install. The import command completed with no error but tomli still could not find any flit_core package.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> End result of all my work is that flit_core has been built and installed into python3.10/site-packages but nothing seems to want to make the tomli build find it.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Does anyone have any ideas on what command(s) I need to use in the tomli pip install to have the installed flit_core package visible.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Regards and a Happy and Healthy New Year to all,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> PS:  I hope everyone had an enjoyable Christmas.
>>>>>>>>>>>>>>>>>> I hope you did, too!
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>> 
>>>>>>>>>>>>> -- 
>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>> 
>>>>>>>>>>> -- 
>>>>>>>>>>> Sent from my laptop
>>>>>>>>> 
>>>>>>>>> -- 
>>>>>>>>> Sent from my laptop
>>> 

Re: Question about updating python to 3.10

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 23688 bytes --]

Hi Michael,

On 20/01/2022 17:51, Michael Tremer wrote:
> Hello,
> 
>> On 19 Jan 2022, at 19:52, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>
>> Hi Michael,
>>
>> There will be a bit more time before I have the python3.10 branch available.
>>
>> I made a mistake with one of the commits and didn't notice till some while after. So I did a git reset --soft "commit hash" which I have been successful with in the past but this time something went wrong and messed things up.
> 
> This happens. Branches like this don’t have to be perfect. They are work-in-progress and often that has its problems :)
> 
>> I have had to re delete my local python3.10 branch with the commits done so far and am now doing a full build again to make sure that my restart doesn't have any errors in it.
>>
>> I will be able to restart doing the commits again tomorrow so probably branch will be available on Friday.
> 
> Thank you, I will have a look tomorrow and rebase my branch based on yours.
It took a bit longer to get it done as I had a problem with my built-in freezer dying so I had to sort that out instead of working on the python-3.10 branch

It is now completed and pushed into my bonnietwin repository.

Regards,

Adolf.
> 
> Best,
> -Michael
> 
>>
>> Sorry.
>>
>> Regards,
>>
>> Adolf.
>>
>> On 19/01/2022 13:33, Adolf Belka wrote:
>>> Hi Michael,
>>>
>>> On 19/01/2022 09:25, Michael Tremer wrote:
>>>> Hello,
>>>>
>>>> I am sure this was documented in the wiki somewhere. For some reason I cannot find it any more.
>>>>
>>>> Please let me know if you need any assistance from me.
>>> I got it to work thanks very much. There is now a python3.10 branch in my bonnietwin repository but it only has a few of the changes so far but I wanted to confirm that it worked.
>>>
>>> I need to finish doing the remaining commits and then push them. Hopefully I will be able to complete those later today or tomorrow.
>>>
>>> I will email when I have completed it and the full branch of my python3.10 changes is pushed into my repository.
>>>
>>> Regards,
>>>
>>> Adolf.
>>>>
>>>> -Michael
>>>>
>>>>> On 18 Jan 2022, at 21:21, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>
>>>>> Hi Michael,
>>>>>
>>>>> On 18/01/2022 19:51, Michael Tremer wrote:
>>>>>> Hello,
>>>>>>> On 18 Jan 2022, at 16:42, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>
>>>>>>> Hi Michael,
>>>>>>>
>>>>>>> On 18/01/2022 16:02, Michael Tremer wrote:
>>>>>>>> Hello Adolf,
>>>>>>>> I have just been trying to build a couple of Python packages that I need and I finally arrived at needing flit-core.
>>>>>>>> I suppose since you did already build this, there is no point in me doing all this again and also there is no need to build all my packages on Python 3.8 when we can have a newer release.
>>>>>>> I am afraid that you will need to build flit-core.  I was building it when I was trying to use pip to install tomli but that approached failed to go further and that was when I used pyproject2setuppy, from your input, to build tomli and that did not need flit-core.
>>>>>> Oh, I suppose that didn’t hold for very long because I now have a package that needs it :(
>>>>> flit-core also has no setup.py but you will then be able to use the pyproject2setuppy package from my python3.10 branch to build flit-core.
>>>>>>> In terms of the python3.10 branch that I have on my git repository at home, I presume I could push it onto my personal git repository that you set up for me - https://git.ipfire.org/?p=people/bonnietwin/ipfire-2.x.git;a=summary
>>>>>> Yes, it is always a good idea to backup things to the server.
>>>>>>> The only thing is I am not sure what command I should use to push the branch from my system here to the IPFire one.
>>>>>> If you have your repository set up with “git remote add …” then you only need to run “git push bonnietwin python3.10”.
>>>>> Aaah. I think when I tried to experiment with this before I created a local new clone of bonnietwin which was then separate from my running local repository that I used for building and I struggled to figure out how to get branches between them.
>>>>>
>>>>> Your mention of git remote add indicates that I can add my bonnietwin repository to my running build repository. Thais is what I was missing before.
>>>>> Thanks for the clue, will give it a try.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Adolf.
>>>>>> -Michael
>>>>>>>
>>>>>>> I can have a go at trying it and see how it goes.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Adolf.
>>>>>>>> Could you push this branch somewhere for me so that I can base my work on top of it?
>>>>>>>> Best,
>>>>>>>> -Michael
>>>>>>>>> On 13 Jan 2022, at 09:32, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>>> On 12 Jan 2022, at 21:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> On 12/01/2022 19:10, Michael Tremer wrote:
>>>>>>>>>>> Hey,
>>>>>>>>>>>> On 11 Jan 2022, at 14:32, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Michael and all,
>>>>>>>>>>>>
>>>>>>>>>>>> python3-six has to stay as python3-dateutil explicitly specifies it in its dependencies.
>>>>>>>>>>>> So that solves that question.
>>>>>>>>>>> Aww :(
>>>>>>>>>>>> I have, with quite a bit of work, eventually been able to get all existing python modules updated that had updates and built successfully with python3.10
>>>>>>>>>>> Yay!
>>>>>>>>>>>> However later in the IPFire build libvirt failed with a problem parsing some documentation files.
>>>>>>>>>>>> Stashing my python3.10 changes and re-running the build had libvirt building without problems. So one of the python changes has caused it. libvirt uses python3-docutils and that was updated but reverting that back to original on its own did not cause libvirt to build successfully.
>>>>>>>>>>>>
>>>>>>>>>>>> I looked in the libvirt .configure for an option to not build the docs but it is not mentioned under ./configure --help.
>>>>>>>>>>>>
>>>>>>>>>>>> I tried --without-docs and --disable-docs but both were not recognised in the build.
>>>>>>>>>>>>
>>>>>>>>>>>> The current libvirt is from mid 2020 (6.5.0) so it is likely that it is not compatible in some way with the updated python/modules.
>>>>>>>>>>>>
>>>>>>>>>>>> I tried updating libvirt to the current 7.10.0
>>>>>>>>>>>> libvirt has replaced autotools with meson. I found meson options available for all existing ./configure options except three. There is nothing in meson options for:-
>>>>>>>>>>>> --with-virtualport
>>>>>>>>>>>> --with-macvtap
>>>>>>>>>>>> --without-dbus
>>>>>>>>>>> Yay, another project that is using meson :(
>>>>>>>>>>> We no longer need macvtap; we have dubs, so that should not hurt; and I have no idea what virtualport is.
>>>>>>>>>>>> I tried building it with all the other options and it successfully built with the updated python3.10 and modules. All other packages after libvirt built successfully without any problems.
>>>>>>>>>>>>
>>>>>>>>>>>> Normally I would then use the updated libvirt, as that is needed to work with the updated python packages, but I am not sure if those missing options are now built into libvirt or are no longer available and what impact that might have for the use of libvirt on IPFire.
>>>>>>>>>>> I don’t think this would hurt us much. I have CCed Jonatan who maintains this, so maybe he knows more about this.
>>>>>>>>>>>> There is nothing in the libvirt changelog about virtualport, macvtap or dbus.
>>>>>>>>>>>>
>>>>>>>>>>>> Searching on the internet, I was also not able to find anything related to libvirt and these options.
>>>>>>>>>>>>
>>>>>>>>>>>> Any suggestions/proposals on how to move forward on this? Should I just build libvirt without those three options or ...?
>>>>>>>>>>> Yes, and I think this could be submitted independently of the Python branch because it is an add-on. We should be able to ship this at any time and might want to delay Python until we have enough space in a core update.
>>>>>>>>>> Okay, then I will stash my python3.10 changes and submit the libvirt update on its own. Then Jonaton can test it out. I don't use libvirt or qemu so I can't test it easily.
>>>>>>>>>>
>>>>>>>>>> Then when you think the time is right, let me know and I will submit the python3.10 series of patches.
>>>>>>>>>
>>>>>>>>> You can submit this at any time. I was just thinking about the conversation between Peter and Arne during the last call regarding an updated kernel and updated firmware. Depending on the size of Python, all of this might make an update a little bit too large - however, this should not be a patchset that we cannot merge within two merge windows.
>>>>>>>>>
>>>>>>>>>>> Thank you for looking into this. Sounds like a not so nice riddle to solve :)
>>>>>>>>>> There were certainly some challenges, like updating python3-daemon which then wanted to have twine as a fixed dependency which had nine dependencies some of which had more. I had got to 19 additional python packages and still going and then I said enough was enough and looked at twine. It is required for uploading python packages to PyPI which IPFire doesn't need. So I tried patching the twine requirement out and that worked and then I found others that had done the same because they had no response to the request to not have twine as a mandatory build requirement.
>>>>>>>>>
>>>>>>>>> Python seems to become the hell that Perl used to be. In order to add one perl package, you had 20 dependencies. And those dependencies had their own dependencies too… It would be great to disable or patch out whatever is possible so that we do not create more work for ourselves in the future and have to update packages that are barely used - we already have enough of these :)
>>>>>>>>>
>>>>>>>>>> I found it all challenging but rewarding when I made progress and eventually succeeded in building everything without errors.
>>>>>>>>>
>>>>>>>>> I am so happy that you are putting all this time and effort into this because it certainly requires patience - a trade I don’t have.
>>>>>>>>>
>>>>>>>>> And Python is being used in all sorts of places in the distribution, so it really isn’t anything we should not be up to date on.
>>>>>>>>>
>>>>>>>>> -Michael
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Adolf.
>>>>>>>>>>> -Michael
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>
>>>>>>>>>>>> On 07/01/2022 17:49, Michael Tremer wrote:
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>> On 4 Jan 2022, at 12:49, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Following the suggested Arch Linux approach I have got python3-setuptools-scm building successfully now. Yaaah.
>>>>>>>>>>>>>> I had to install python3-toml, followed by python3-pyproject2setuppy followed by python3-tomli.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> However then python3-six which installs after python3-setuptools-scm fails due to missing the python package python3-packaging.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I can install that package, and will do so as I continue testing out the python 3.10 build but I have the following question about python3-six.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is six still needed. It is designed to ensure that programs can work with either python2 or python3 but IPFire only has python3 now.
>>>>>>>>>>>>>> I can find python3-six referenced only by aws-cli. In the aws-cli build python3-six is not listed at all in the build requirements. I tried to search for aws-cli and python3-six being needed in run mode but I could not find anything one way or the other.
>>>>>>>>>>>>> If aws-cli no longer requires python3-six we can drop it. Last time we touched it I had to bring the package back because aws-cli was still depending on it.
>>>>>>>>>>>>>> If python3-six is really needed for running aws-cli then I can install it and the new dependency but if it is not needed we could remove two packages which would partially offset the three new dependencies required to be able to build the latest python3-setuptools-scm.
>>>>>>>>>>>>> Yes, that would be great :)
>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Adolf
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 03/01/2022 12:32, Michael Tremer wrote:
>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>> On 3 Jan 2022, at 11:31, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 03/01/2022 11:52, Michael Tremer wrote:
>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>> I am not sure if you already looked at what Arch is doing, but as far as I understand it works like this:
>>>>>>>>>>>>>>>> No I didn't look at Arch, which is surprising as I have in the past with other packages but I didn't. So thanks very much for the pointer.
>>>>>>>>>>>>>>>>> They build Python 3 without pip and setuptools (https://github.com/archlinux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/trunk/PKGBUILD#L59).
>>>>>>>>>>>>>>>>> At this stage we might need to build setuptools separately.
>>>>>>>>>>>>>>>>> They use a tool called pyproject2setuppy (great name!) to build tomli (https://github.com/archlinux/svntogit-packages/blob/packages/python-tomli/trunk/PKGBUILD).
>>>>>>>>>>>>>>>>> Setuptools-scm can then be built with tomli: https://github.com/archlinux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD
>>>>>>>>>>>>>>>>> Would this work or did I overlook anything?
>>>>>>>>>>>>>>>> I will have a look at the links and give it a go.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks very much, I feel re-motivated now.
>>>>>>>>>>>>>>> Very good! I got a little bit lost in your last email and I think I could tell. But following a solution that somebody else has come up with can be a good thing to follow and of course we would customise it to our needs.
>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>>>> On 2 Jan 2022, at 13:44, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Michael and All,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 26/12/2021 21:55, Michael Tremer wrote:
>>>>>>>>>>>>>>>>>>> Merry Christmas,
>>>>>>>>>>>>>>>>>>>> On 26 Dec 2021, at 15:38, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On 17/12/2021 12:41, Michael Tremer wrote:
>>>>>>>>>>>>>>>>>>>>> Hello Adolf,
>>>>>>>>>>>>>>>>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I am working on updating python from 3.8 to 3.10. Python itself is okay after some other python related programs were also updated due to their older versions not working with 3.10
>>>>>>>>>>>>>>>>>>>>> Thank you for working on this.
>>>>>>>>>>>>>>>>>>>>>> I am working through all the programs that reference python-3.8 in their rootfiles, or wherever, to change them to 3.10
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> My question comes with regard to the changes to the rootfiles to ensure they will be included properly in an update.
>>>>>>>>>>>>>>>>>>>>> We will have to ship everything that puts anything into /usr/lib*/python3.*/site-packages again with the release of Python 3.10.
>>>>>>>>>>>>>>>>>>>>> This isn’t fun, but not as much of a pain in the rear as shipping a new Perl release.
>>>>>>>>>>>>>>>>>>>>>> For the addons then when I change the rootfile I am also incrementing the PAK_VER number in the lfs files and that ensures that they will be properly updated.
>>>>>>>>>>>>>>>>>>>>> Yes. That is required for Pakfire to find a new version.
>>>>>>>>>>>>>>>>>>>>>> I don't know what to do for the core programs where I have updated the rootfile. The lfs files for these programs don't have a PAK_VER number to increment and the actual package version number is not being changed so I don't know what I need to change in the lfs to ensure that the rootfile changes will be properly included in the update?
>>>>>>>>>>>>>>>>>>>>> I would say committing everything into a “python-3.10” branch and submitting it all as a patchset is fine. All packages that have been touched in that branch will have to be shipped again which should be easy to identify if it all comes in one large branch.
>>>>>>>>>>>>>>>>>>>> Okay, that is clear then.Will follow that approach.
>>>>>>>>>>>>>>>>>>>>>> I would appreciate any help on what I need to do for these programs that have had changes only in their rootfiles.
>>>>>>>>>>>>>>>>>>>>> Absolutely. Did you find anything that didn’t build yet?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I have not had any problem with python3.10 itself but I found some of the older modules would not work with 3.10 so I decided to do a combined update to 3.10 plus update modules to latest versions.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I found that after python3-setuptools-scm had run that the next module python3-six failed to run due to missing the tomli library. I commented out six and reran the build and then python3-dateutil had the same problem  and then python3-jmespath
>>>>>>>>>>>>>>>>>>> I completely lost track of what the status of setuptools/distutils and so on is. This has changed multiple times and become independent/part of the distribution again and so on.
>>>>>>>>>>>>>>>>>>>> After some investigation I have found that since version 6.0.0 of python3-setuptools-scm, tomli has been a required library when it is being used by other modules. So running version 6.0.0 of setuptools-scm worked with python3.10 and subsequent modules built successfully as tomli not needed.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> So the obvious fix is to install tomli but that is when I ran into a problem. tomli has no setup.py or setup.cfg files in its tarball. The only way to build it is using pip.
>>>>>>>>>>>>>>>>>>> Err. Great.
>>>>>>>>>>>>>>>>>>>> Apparently, after some searching, setup.py has been deprecated as the recommended install method and if setup.py doesn't work the usual fix suggestion is to run pip install.
>>>>>>>>>>>>>>>>>>> Yes. But apparently the new system is deprecated, too. I really dislike the state of packaging systems that come with any language these days.
>>>>>>>>>>>>>>>>>>>> Having searched on pip for a while, it looks like builds using it can still be done from tarballs locally without needing to access the internet to download the modules.
>>>>>>>>>>>>>>>>>>> That is good!
>>>>>>>>>>>>>>>>>>>> As pip is installed as part of python3 i tried using that.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I used "python3 -m pip install --root=/ ."  in place of
>>>>>>>>>>>>>>>>>>>> "python3 setup.py build
>>>>>>>>>>>>>>>>>>>> python3 setup.py install --root=/"
>>>>>>>>>>>>>>>>>>>> for the tomli source.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> This failed because tomli requires flit_core to be available.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> flit_core also has no setup.py and so I tried to build it with pip install. This failed with the error message that tomli was required as a dependency. So I added --no-deps to the flit_core pip install line and this time flit_core successfully built but the tomli build still came back with not finding flit_core.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I have tried a variety of changes to the pip install line but none of them has been successful and it has become clear that I am out of my depth here and need guidance rather than just trying anything I find from a google search. I am not even really clear if pip install is the correct approach, except if not then I have no idea how to progress.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Help urgently needed on how I should be approaching this.
>>>>>>>>>>>>>>>>>>> Circular dependencies can happen. This won’t be the first time and it always is an utter nightmare with our buildsystem.
>>>>>>>>>>>>>>>>>>> This page states that pip comes by default with all binary packages:
>>>>>>>>>>>>>>>>>>> https://docs.python.org/3/installing/index.html
>>>>>>>>>>>>>>>>>>> Is there a chance that it is part of the distribution and can be enabled with a configure switch?
>>>>>>>>>>>>>>>>>> I wasn't able to find any way to get it enabled that way.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I did find that pip install will not install if the package is already present. You have to add --upgrade to make it update any package already present. So I tried that. Didn't work.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I found a section about needing to use bootstrapping if you want to install tomli. Followed the information for creating a wheel file for flit_core and then extracting that file into the site-packages directory. Tried that. Didn't work.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Rather than doing a complete build every time I changed to trying to build flit_core followed by tomli in a ./make.sh shell environment.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Doing that I found that I had a pip install command that was creating the flit_core directory in site-packages and also including the .dist-info directory. The info I found said that the .dist-info directory had to be there for pip to be able to find the flit_core package.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> However the pip install of tomli still ends up not finding any flit_core package. In the bootstrapping information it mentioned about changing the PYTHONPATH environment variable for find the flit_core package so I tried that in the shell but it also made no difference.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I then tried running python3 -c 'import flit_core' in the tomli build shell before doing the tomli pip install. The import command completed with no error but tomli still could not find any flit_core package.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> End result of all my work is that flit_core has been built and installed into python3.10/site-packages but nothing seems to want to make the tomli build find it.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Does anyone have any ideas on what command(s) I need to use in the tomli pip install to have the installed flit_core package visible.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Regards and a Happy and Healthy New Year to all,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> PS:  I hope everyone had an enjoyable Christmas.
>>>>>>>>>>>>>>>>>>> I hope you did, too!
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>
>>>>>>>>>>>> -- 
>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>
>>>>>>>>>> -- 
>>>>>>>>>> Sent from my laptop
>>>>
> 

Re: Question about updating python to 3.10

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 24302 bytes --]

Hello,

And thank you. I am building the branch right now and will base my own changes on this.

-Michael

> On 24 Jan 2022, at 12:46, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi Michael,
> 
> On 20/01/2022 17:51, Michael Tremer wrote:
>> Hello,
>>> On 19 Jan 2022, at 19:52, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>> 
>>> Hi Michael,
>>> 
>>> There will be a bit more time before I have the python3.10 branch available.
>>> 
>>> I made a mistake with one of the commits and didn't notice till some while after. So I did a git reset --soft "commit hash" which I have been successful with in the past but this time something went wrong and messed things up.
>> This happens. Branches like this don’t have to be perfect. They are work-in-progress and often that has its problems :)
>>> I have had to re delete my local python3.10 branch with the commits done so far and am now doing a full build again to make sure that my restart doesn't have any errors in it.
>>> 
>>> I will be able to restart doing the commits again tomorrow so probably branch will be available on Friday.
>> Thank you, I will have a look tomorrow and rebase my branch based on yours.
> It took a bit longer to get it done as I had a problem with my built-in freezer dying so I had to sort that out instead of working on the python-3.10 branch
> 
> It is now completed and pushed into my bonnietwin repository.
> 
> Regards,
> 
> Adolf.
>> Best,
>> -Michael
>>> 
>>> Sorry.
>>> 
>>> Regards,
>>> 
>>> Adolf.
>>> 
>>> On 19/01/2022 13:33, Adolf Belka wrote:
>>>> Hi Michael,
>>>> 
>>>> On 19/01/2022 09:25, Michael Tremer wrote:
>>>>> Hello,
>>>>> 
>>>>> I am sure this was documented in the wiki somewhere. For some reason I cannot find it any more.
>>>>> 
>>>>> Please let me know if you need any assistance from me.
>>>> I got it to work thanks very much. There is now a python3.10 branch in my bonnietwin repository but it only has a few of the changes so far but I wanted to confirm that it worked.
>>>> 
>>>> I need to finish doing the remaining commits and then push them. Hopefully I will be able to complete those later today or tomorrow.
>>>> 
>>>> I will email when I have completed it and the full branch of my python3.10 changes is pushed into my repository.
>>>> 
>>>> Regards,
>>>> 
>>>> Adolf.
>>>>> 
>>>>> -Michael
>>>>> 
>>>>>> On 18 Jan 2022, at 21:21, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>> 
>>>>>> Hi Michael,
>>>>>> 
>>>>>> On 18/01/2022 19:51, Michael Tremer wrote:
>>>>>>> Hello,
>>>>>>>> On 18 Jan 2022, at 16:42, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>> 
>>>>>>>> Hi Michael,
>>>>>>>> 
>>>>>>>> On 18/01/2022 16:02, Michael Tremer wrote:
>>>>>>>>> Hello Adolf,
>>>>>>>>> I have just been trying to build a couple of Python packages that I need and I finally arrived at needing flit-core.
>>>>>>>>> I suppose since you did already build this, there is no point in me doing all this again and also there is no need to build all my packages on Python 3.8 when we can have a newer release.
>>>>>>>> I am afraid that you will need to build flit-core.  I was building it when I was trying to use pip to install tomli but that approached failed to go further and that was when I used pyproject2setuppy, from your input, to build tomli and that did not need flit-core.
>>>>>>> Oh, I suppose that didn’t hold for very long because I now have a package that needs it :(
>>>>>> flit-core also has no setup.py but you will then be able to use the pyproject2setuppy package from my python3.10 branch to build flit-core.
>>>>>>>> In terms of the python3.10 branch that I have on my git repository at home, I presume I could push it onto my personal git repository that you set up for me - https://git.ipfire.org/?p=people/bonnietwin/ipfire-2.x.git;a=summary
>>>>>>> Yes, it is always a good idea to backup things to the server.
>>>>>>>> The only thing is I am not sure what command I should use to push the branch from my system here to the IPFire one.
>>>>>>> If you have your repository set up with “git remote add …” then you only need to run “git push bonnietwin python3.10”.
>>>>>> Aaah. I think when I tried to experiment with this before I created a local new clone of bonnietwin which was then separate from my running local repository that I used for building and I struggled to figure out how to get branches between them.
>>>>>> 
>>>>>> Your mention of git remote add indicates that I can add my bonnietwin repository to my running build repository. Thais is what I was missing before.
>>>>>> Thanks for the clue, will give it a try.
>>>>>> 
>>>>>> Regards,
>>>>>> 
>>>>>> Adolf.
>>>>>>> -Michael
>>>>>>>> 
>>>>>>>> I can have a go at trying it and see how it goes.
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> 
>>>>>>>> Adolf.
>>>>>>>>> Could you push this branch somewhere for me so that I can base my work on top of it?
>>>>>>>>> Best,
>>>>>>>>> -Michael
>>>>>>>>>> On 13 Jan 2022, at 09:32, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hello,
>>>>>>>>>> 
>>>>>>>>>>> On 12 Jan 2022, at 21:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Hi all,
>>>>>>>>>>> 
>>>>>>>>>>> On 12/01/2022 19:10, Michael Tremer wrote:
>>>>>>>>>>>> Hey,
>>>>>>>>>>>>> On 11 Jan 2022, at 14:32, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi Michael and all,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> python3-six has to stay as python3-dateutil explicitly specifies it in its dependencies.
>>>>>>>>>>>>> So that solves that question.
>>>>>>>>>>>> Aww :(
>>>>>>>>>>>>> I have, with quite a bit of work, eventually been able to get all existing python modules updated that had updates and built successfully with python3.10
>>>>>>>>>>>> Yay!
>>>>>>>>>>>>> However later in the IPFire build libvirt failed with a problem parsing some documentation files.
>>>>>>>>>>>>> Stashing my python3.10 changes and re-running the build had libvirt building without problems. So one of the python changes has caused it. libvirt uses python3-docutils and that was updated but reverting that back to original on its own did not cause libvirt to build successfully.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I looked in the libvirt .configure for an option to not build the docs but it is not mentioned under ./configure --help.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I tried --without-docs and --disable-docs but both were not recognised in the build.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The current libvirt is from mid 2020 (6.5.0) so it is likely that it is not compatible in some way with the updated python/modules.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I tried updating libvirt to the current 7.10.0
>>>>>>>>>>>>> libvirt has replaced autotools with meson. I found meson options available for all existing ./configure options except three. There is nothing in meson options for:-
>>>>>>>>>>>>> --with-virtualport
>>>>>>>>>>>>> --with-macvtap
>>>>>>>>>>>>> --without-dbus
>>>>>>>>>>>> Yay, another project that is using meson :(
>>>>>>>>>>>> We no longer need macvtap; we have dubs, so that should not hurt; and I have no idea what virtualport is.
>>>>>>>>>>>>> I tried building it with all the other options and it successfully built with the updated python3.10 and modules. All other packages after libvirt built successfully without any problems.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Normally I would then use the updated libvirt, as that is needed to work with the updated python packages, but I am not sure if those missing options are now built into libvirt or are no longer available and what impact that might have for the use of libvirt on IPFire.
>>>>>>>>>>>> I don’t think this would hurt us much. I have CCed Jonatan who maintains this, so maybe he knows more about this.
>>>>>>>>>>>>> There is nothing in the libvirt changelog about virtualport, macvtap or dbus.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Searching on the internet, I was also not able to find anything related to libvirt and these options.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Any suggestions/proposals on how to move forward on this? Should I just build libvirt without those three options or ...?
>>>>>>>>>>>> Yes, and I think this could be submitted independently of the Python branch because it is an add-on. We should be able to ship this at any time and might want to delay Python until we have enough space in a core update.
>>>>>>>>>>> Okay, then I will stash my python3.10 changes and submit the libvirt update on its own. Then Jonaton can test it out. I don't use libvirt or qemu so I can't test it easily.
>>>>>>>>>>> 
>>>>>>>>>>> Then when you think the time is right, let me know and I will submit the python3.10 series of patches.
>>>>>>>>>> 
>>>>>>>>>> You can submit this at any time. I was just thinking about the conversation between Peter and Arne during the last call regarding an updated kernel and updated firmware. Depending on the size of Python, all of this might make an update a little bit too large - however, this should not be a patchset that we cannot merge within two merge windows.
>>>>>>>>>> 
>>>>>>>>>>>> Thank you for looking into this. Sounds like a not so nice riddle to solve :)
>>>>>>>>>>> There were certainly some challenges, like updating python3-daemon which then wanted to have twine as a fixed dependency which had nine dependencies some of which had more. I had got to 19 additional python packages and still going and then I said enough was enough and looked at twine. It is required for uploading python packages to PyPI which IPFire doesn't need. So I tried patching the twine requirement out and that worked and then I found others that had done the same because they had no response to the request to not have twine as a mandatory build requirement.
>>>>>>>>>> 
>>>>>>>>>> Python seems to become the hell that Perl used to be. In order to add one perl package, you had 20 dependencies. And those dependencies had their own dependencies too… It would be great to disable or patch out whatever is possible so that we do not create more work for ourselves in the future and have to update packages that are barely used - we already have enough of these :)
>>>>>>>>>> 
>>>>>>>>>>> I found it all challenging but rewarding when I made progress and eventually succeeded in building everything without errors.
>>>>>>>>>> 
>>>>>>>>>> I am so happy that you are putting all this time and effort into this because it certainly requires patience - a trade I don’t have.
>>>>>>>>>> 
>>>>>>>>>> And Python is being used in all sorts of places in the distribution, so it really isn’t anything we should not be up to date on.
>>>>>>>>>> 
>>>>>>>>>> -Michael
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Regards,
>>>>>>>>>>> Adolf.
>>>>>>>>>>>> -Michael
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On 07/01/2022 17:49, Michael Tremer wrote:
>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>> On 4 Jan 2022, at 12:49, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Following the suggested Arch Linux approach I have got python3-setuptools-scm building successfully now. Yaaah.
>>>>>>>>>>>>>>> I had to install python3-toml, followed by python3-pyproject2setuppy followed by python3-tomli.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> However then python3-six which installs after python3-setuptools-scm fails due to missing the python package python3-packaging.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I can install that package, and will do so as I continue testing out the python 3.10 build but I have the following question about python3-six.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Is six still needed. It is designed to ensure that programs can work with either python2 or python3 but IPFire only has python3 now.
>>>>>>>>>>>>>>> I can find python3-six referenced only by aws-cli. In the aws-cli build python3-six is not listed at all in the build requirements. I tried to search for aws-cli and python3-six being needed in run mode but I could not find anything one way or the other.
>>>>>>>>>>>>>> If aws-cli no longer requires python3-six we can drop it. Last time we touched it I had to bring the package back because aws-cli was still depending on it.
>>>>>>>>>>>>>>> If python3-six is really needed for running aws-cli then I can install it and the new dependency but if it is not needed we could remove two packages which would partially offset the three new dependencies required to be able to build the latest python3-setuptools-scm.
>>>>>>>>>>>>>> Yes, that would be great :)
>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Adolf
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 03/01/2022 12:32, Michael Tremer wrote:
>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>> On 3 Jan 2022, at 11:31, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On 03/01/2022 11:52, Michael Tremer wrote:
>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>> I am not sure if you already looked at what Arch is doing, but as far as I understand it works like this:
>>>>>>>>>>>>>>>>> No I didn't look at Arch, which is surprising as I have in the past with other packages but I didn't. So thanks very much for the pointer.
>>>>>>>>>>>>>>>>>> They build Python 3 without pip and setuptools (https://github.com/archlinux/svntogit-packages/blob/912617035af94b1466595eedd6e6637676e2e733/trunk/PKGBUILD#L59).
>>>>>>>>>>>>>>>>>> At this stage we might need to build setuptools separately.
>>>>>>>>>>>>>>>>>> They use a tool called pyproject2setuppy (great name!) to build tomli (https://github.com/archlinux/svntogit-packages/blob/packages/python-tomli/trunk/PKGBUILD).
>>>>>>>>>>>>>>>>>> Setuptools-scm can then be built with tomli: https://github.com/archlinux/svntogit-community/blob/packages/python-setuptools-scm/trunk/PKGBUILD
>>>>>>>>>>>>>>>>>> Would this work or did I overlook anything?
>>>>>>>>>>>>>>>>> I will have a look at the links and give it a go.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks very much, I feel re-motivated now.
>>>>>>>>>>>>>>>> Very good! I got a little bit lost in your last email and I think I could tell. But following a solution that somebody else has come up with can be a good thing to follow and of course we would customise it to our needs.
>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>>>>> On 2 Jan 2022, at 13:44, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hi Michael and All,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On 26/12/2021 21:55, Michael Tremer wrote:
>>>>>>>>>>>>>>>>>>>> Merry Christmas,
>>>>>>>>>>>>>>>>>>>>> On 26 Dec 2021, at 15:38, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On 17/12/2021 12:41, Michael Tremer wrote:
>>>>>>>>>>>>>>>>>>>>>> Hello Adolf,
>>>>>>>>>>>>>>>>>>>>>>> On 17 Dec 2021, at 11:55, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> I am working on updating python from 3.8 to 3.10. Python itself is okay after some other python related programs were also updated due to their older versions not working with 3.10
>>>>>>>>>>>>>>>>>>>>>> Thank you for working on this.
>>>>>>>>>>>>>>>>>>>>>>> I am working through all the programs that reference python-3.8 in their rootfiles, or wherever, to change them to 3.10
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> My question comes with regard to the changes to the rootfiles to ensure they will be included properly in an update.
>>>>>>>>>>>>>>>>>>>>>> We will have to ship everything that puts anything into /usr/lib*/python3.*/site-packages again with the release of Python 3.10.
>>>>>>>>>>>>>>>>>>>>>> This isn’t fun, but not as much of a pain in the rear as shipping a new Perl release.
>>>>>>>>>>>>>>>>>>>>>>> For the addons then when I change the rootfile I am also incrementing the PAK_VER number in the lfs files and that ensures that they will be properly updated.
>>>>>>>>>>>>>>>>>>>>>> Yes. That is required for Pakfire to find a new version.
>>>>>>>>>>>>>>>>>>>>>>> I don't know what to do for the core programs where I have updated the rootfile. The lfs files for these programs don't have a PAK_VER number to increment and the actual package version number is not being changed so I don't know what I need to change in the lfs to ensure that the rootfile changes will be properly included in the update?
>>>>>>>>>>>>>>>>>>>>>> I would say committing everything into a “python-3.10” branch and submitting it all as a patchset is fine. All packages that have been touched in that branch will have to be shipped again which should be easy to identify if it all comes in one large branch.
>>>>>>>>>>>>>>>>>>>>> Okay, that is clear then.Will follow that approach.
>>>>>>>>>>>>>>>>>>>>>>> I would appreciate any help on what I need to do for these programs that have had changes only in their rootfiles.
>>>>>>>>>>>>>>>>>>>>>> Absolutely. Did you find anything that didn’t build yet?
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I have not had any problem with python3.10 itself but I found some of the older modules would not work with 3.10 so I decided to do a combined update to 3.10 plus update modules to latest versions.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I found that after python3-setuptools-scm had run that the next module python3-six failed to run due to missing the tomli library. I commented out six and reran the build and then python3-dateutil had the same problem  and then python3-jmespath
>>>>>>>>>>>>>>>>>>>> I completely lost track of what the status of setuptools/distutils and so on is. This has changed multiple times and become independent/part of the distribution again and so on.
>>>>>>>>>>>>>>>>>>>>> After some investigation I have found that since version 6.0.0 of python3-setuptools-scm, tomli has been a required library when it is being used by other modules. So running version 6.0.0 of setuptools-scm worked with python3.10 and subsequent modules built successfully as tomli not needed.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> So the obvious fix is to install tomli but that is when I ran into a problem. tomli has no setup.py or setup.cfg files in its tarball. The only way to build it is using pip.
>>>>>>>>>>>>>>>>>>>> Err. Great.
>>>>>>>>>>>>>>>>>>>>> Apparently, after some searching, setup.py has been deprecated as the recommended install method and if setup.py doesn't work the usual fix suggestion is to run pip install.
>>>>>>>>>>>>>>>>>>>> Yes. But apparently the new system is deprecated, too. I really dislike the state of packaging systems that come with any language these days.
>>>>>>>>>>>>>>>>>>>>> Having searched on pip for a while, it looks like builds using it can still be done from tarballs locally without needing to access the internet to download the modules.
>>>>>>>>>>>>>>>>>>>> That is good!
>>>>>>>>>>>>>>>>>>>>> As pip is installed as part of python3 i tried using that.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I used "python3 -m pip install --root=/ ."  in place of
>>>>>>>>>>>>>>>>>>>>> "python3 setup.py build
>>>>>>>>>>>>>>>>>>>>> python3 setup.py install --root=/"
>>>>>>>>>>>>>>>>>>>>> for the tomli source.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> This failed because tomli requires flit_core to be available.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> flit_core also has no setup.py and so I tried to build it with pip install. This failed with the error message that tomli was required as a dependency. So I added --no-deps to the flit_core pip install line and this time flit_core successfully built but the tomli build still came back with not finding flit_core.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I have tried a variety of changes to the pip install line but none of them has been successful and it has become clear that I am out of my depth here and need guidance rather than just trying anything I find from a google search. I am not even really clear if pip install is the correct approach, except if not then I have no idea how to progress.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Help urgently needed on how I should be approaching this.
>>>>>>>>>>>>>>>>>>>> Circular dependencies can happen. This won’t be the first time and it always is an utter nightmare with our buildsystem.
>>>>>>>>>>>>>>>>>>>> This page states that pip comes by default with all binary packages:
>>>>>>>>>>>>>>>>>>>> https://docs.python.org/3/installing/index.html
>>>>>>>>>>>>>>>>>>>> Is there a chance that it is part of the distribution and can be enabled with a configure switch?
>>>>>>>>>>>>>>>>>>> I wasn't able to find any way to get it enabled that way.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I did find that pip install will not install if the package is already present. You have to add --upgrade to make it update any package already present. So I tried that. Didn't work.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I found a section about needing to use bootstrapping if you want to install tomli. Followed the information for creating a wheel file for flit_core and then extracting that file into the site-packages directory. Tried that. Didn't work.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Rather than doing a complete build every time I changed to trying to build flit_core followed by tomli in a ./make.sh shell environment.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Doing that I found that I had a pip install command that was creating the flit_core directory in site-packages and also including the .dist-info directory. The info I found said that the .dist-info directory had to be there for pip to be able to find the flit_core package.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> However the pip install of tomli still ends up not finding any flit_core package. In the bootstrapping information it mentioned about changing the PYTHONPATH environment variable for find the flit_core package so I tried that in the shell but it also made no difference.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I then tried running python3 -c 'import flit_core' in the tomli build shell before doing the tomli pip install. The import command completed with no error but tomli still could not find any flit_core package.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> End result of all my work is that flit_core has been built and installed into python3.10/site-packages but nothing seems to want to make the tomli build find it.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Does anyone have any ideas on what command(s) I need to use in the tomli pip install to have the installed flit_core package visible.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Regards and a Happy and Healthy New Year to all,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> PS:  I hope everyone had an enjoyable Christmas.
>>>>>>>>>>>>>>>>>>>> I hope you did, too!
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>> 
>>>>>>>>>>>>> -- 
>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>> 
>>>>>>>>>>> -- 
>>>>>>>>>>> Sent from my laptop
>>>>>