qemu-user-static build of other architectures

qemu-user-static build of other architectures

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 1036 bytes --]

Hi All,

I tried to run a aarch64 build of samba so that I could get the correct rootfile but I had a few problems.

I installed qemu-user-static on Arch Linux but when I tried to run the gettoolchain command I got an error message about not finding the binfmt_misc helper.

Figured out that I needed to also install qemu-user-static-binfmt as well. Then the error message changed to the fact that it could not find /usr/bin/qemu-aarch64-static but it was in that directory.

Eventually used the find static command from ./make.sh and found that on Arch Linux it does not have "statically linked" but "static-pie linked". Changing the ./make.sh find static command to look for "static-pie linked" then allowed the build for aarch64 to proceed.

Of course having changed ./make.sh means that it now shows up as a modified file in the git status. I will look at changing the find static command to look for either "statically linked" or "static-pie linked" and submit it as a patch for consideration.


Regards,

Adolf.

Re: qemu-user-static build of other architectures

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 1701 bytes --]

Hello Adolf,

> On 26 Nov 2022, at 22:24, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi All,
> 
> I tried to run a aarch64 build of samba so that I could get the correct rootfile but I had a few problems.
> 
> I installed qemu-user-static on Arch Linux but when I tried to run the gettoolchain command I got an error message about not finding the binfmt_misc helper.
> 
> Figured out that I needed to also install qemu-user-static-binfmt as well. Then the error message changed to the fact that it could not find /usr/bin/qemu-aarch64-static but it was in that directory.
> 
> Eventually used the find static command from ./make.sh and found that on Arch Linux it does not have "statically linked" but "static-pie linked". Changing the ./make.sh find static command to look for "static-pie linked" then allowed the build for aarch64 to proceed.

It is required that QEMU is statically linked.

The reason for that is that we copy the binary into the build environment where it has no access to any libraries of the host - and to make those available, they need to be compiled into the binary.

PIE does not have anything to do with a statically linked binary (well kind of, but I wouldn’t know why that is relevant in this context).

So what did you change exactly for this to work on your system? make.sh would normally automatically try to find the correct binary.

> Of course having changed ./make.sh means that it now shows up as a modified file in the git status. I will look at changing the find static command to look for either "statically linked" or "static-pie linked" and submit it as a patch for consideration.

> Regards,
> 
> Adolf.
> 

-Michael

Re: qemu-user-static build of other architectures

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 2963 bytes --]

Hi Michael,

On 27/11/2022 13:09, Michael Tremer wrote:
> Hello Adolf,
> 
>> On 26 Nov 2022, at 22:24, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>
>> Hi All,
>>
>> I tried to run a aarch64 build of samba so that I could get the correct rootfile but I had a few problems.
>>
>> I installed qemu-user-static on Arch Linux but when I tried to run the gettoolchain command I got an error message about not finding the binfmt_misc helper.
>>
>> Figured out that I needed to also install qemu-user-static-binfmt as well. Then the error message changed to the fact that it could not find /usr/bin/qemu-aarch64-static but it was in that directory.
>>
>> Eventually used the find static command from ./make.sh and found that on Arch Linux it does not have "statically linked" but "static-pie linked". Changing the ./make.sh find static command to look for "static-pie linked" then allowed the build for aarch64 to proceed.
> 
> It is required that QEMU is statically linked.
> 
> The reason for that is that we copy the binary into the build environment where it has no access to any libraries of the host - and to make those available, they need to be compiled into the binary.
> 
> PIE does not have anything to do with a statically linked binary (well kind of, but I wouldn’t know why that is relevant in this context).
> 
> So what did you change exactly for this to work on your system? make.sh would normally automatically try to find the correct binary.
I changed line 913 in make.sh from

	file -L "${file}" 2>/dev/null | grep -q "statically linked"
to
	file -L "${file}" 2>/dev/null | grep -q -e "statically linked" -e "static-pie linked"

I did this based on running file -L on the qemu-aarch64-static file in /usr/bin on my Arch Linux system after installing qemu-user-static.

I got

/usr/bin/qemu-aarch64-static: ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), static-pie linked, BuildID[sha1]=92b6a9c54239756a47441b87bed1dfab884cfdc8, for GNU/Linux 4.4.0, stripped

as the response which has static-pie linked instead of statically linked.

Changing the grep to look for static-pie linked may have got the build running but glibc took nearly three hors to build. I know it would be slower with the emulation and probably also slower the first time round as there would be nothing in the ccache but nearly three hours for one package seems excessive, especially when the native build is taking around 1.5 hours for the whole IPFire build.

Could the fact that my qemu-user-static program has the files static-pie linked be the cause of this very long run time?

Regards,

Adolf.
> 
>> Of course having changed ./make.sh means that it now shows up as a modified file in the git status. I will look at changing the find static command to look for either "statically linked" or "static-pie linked" and submit it as a patch for consideration.
> 
>> Regards,
>>
>> Adolf.
>>
> 
> -Michael

Re: qemu-user-static build of other architectures

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 3385 bytes --]

Hello Adolf,

> On 27 Nov 2022, at 12:26, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi Michael,
> 
> On 27/11/2022 13:09, Michael Tremer wrote:
>> Hello Adolf,
>>> On 26 Nov 2022, at 22:24, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>> 
>>> Hi All,
>>> 
>>> I tried to run a aarch64 build of samba so that I could get the correct rootfile but I had a few problems.
>>> 
>>> I installed qemu-user-static on Arch Linux but when I tried to run the gettoolchain command I got an error message about not finding the binfmt_misc helper.
>>> 
>>> Figured out that I needed to also install qemu-user-static-binfmt as well. Then the error message changed to the fact that it could not find /usr/bin/qemu-aarch64-static but it was in that directory.
>>> 
>>> Eventually used the find static command from ./make.sh and found that on Arch Linux it does not have "statically linked" but "static-pie linked". Changing the ./make.sh find static command to look for "static-pie linked" then allowed the build for aarch64 to proceed.
>> It is required that QEMU is statically linked.
>> The reason for that is that we copy the binary into the build environment where it has no access to any libraries of the host - and to make those available, they need to be compiled into the binary.
>> PIE does not have anything to do with a statically linked binary (well kind of, but I wouldn’t know why that is relevant in this context).
>> So what did you change exactly for this to work on your system? make.sh would normally automatically try to find the correct binary.
> I changed line 913 in make.sh from
> 
> file -L "${file}" 2>/dev/null | grep -q "statically linked"
> to
> file -L "${file}" 2>/dev/null | grep -q -e "statically linked" -e "static-pie linked"

Oh yes, this is absolutely fine. It mainly looks like the output of file has been changed and includes whether something has been compiled as PIC or not. Cool. Please send a patch.

> I did this based on running file -L on the qemu-aarch64-static file in /usr/bin on my Arch Linux system after installing qemu-user-static.
> 
> I got
> 
> /usr/bin/qemu-aarch64-static: ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), static-pie linked, BuildID[sha1]=92b6a9c54239756a47441b87bed1dfab884cfdc8, for GNU/Linux 4.4.0, stripped
> 
> as the response which has static-pie linked instead of statically linked.
> 
> Changing the grep to look for static-pie linked may have got the build running but glibc took nearly three hors to build. I know it would be slower with the emulation and probably also slower the first time round as there would be nothing in the ccache but nearly three hours for one package seems excessive, especially when the native build is taking around 1.5 hours for the whole IPFire build.

Yes, the emulation is indeed *that* slow.

> Could the fact that my qemu-user-static program has the files static-pie linked be the cause of this very long run time?

No, this is just how long it takes sadly.

-Michael

> Regards,
> 
> Adolf.
>>> Of course having changed ./make.sh means that it now shows up as a modified file in the git status. I will look at changing the find static command to look for either "statically linked" or "static-pie linked" and submit it as a patch for consideration.
>>> Regards,
>>> 
>>> Adolf.
>>> 
>> -Michael

Re: qemu-user-static build of other architectures

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 4032 bytes --]

Hi Michael,

On 27/11/2022 13:49, Michael Tremer wrote:
> Hello Adolf,
> 
>> On 27 Nov 2022, at 12:26, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>
>> Hi Michael,
>>
>> On 27/11/2022 13:09, Michael Tremer wrote:
>>> Hello Adolf,
>>>> On 26 Nov 2022, at 22:24, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>
>>>> Hi All,
>>>>
>>>> I tried to run a aarch64 build of samba so that I could get the correct rootfile but I had a few problems.
>>>>
>>>> I installed qemu-user-static on Arch Linux but when I tried to run the gettoolchain command I got an error message about not finding the binfmt_misc helper.
>>>>
>>>> Figured out that I needed to also install qemu-user-static-binfmt as well. Then the error message changed to the fact that it could not find /usr/bin/qemu-aarch64-static but it was in that directory.
>>>>
>>>> Eventually used the find static command from ./make.sh and found that on Arch Linux it does not have "statically linked" but "static-pie linked". Changing the ./make.sh find static command to look for "static-pie linked" then allowed the build for aarch64 to proceed.
>>> It is required that QEMU is statically linked.
>>> The reason for that is that we copy the binary into the build environment where it has no access to any libraries of the host - and to make those available, they need to be compiled into the binary.
>>> PIE does not have anything to do with a statically linked binary (well kind of, but I wouldn’t know why that is relevant in this context).
>>> So what did you change exactly for this to work on your system? make.sh would normally automatically try to find the correct binary.
>> I changed line 913 in make.sh from
>>
>> file -L "${file}" 2>/dev/null | grep -q "statically linked"
>> to
>> file -L "${file}" 2>/dev/null | grep -q -e "statically linked" -e "static-pie linked"
> 
> Oh yes, this is absolutely fine. It mainly looks like the output of file has been changed and includes whether something has been compiled as PIC or not. Cool. Please send a patch.
Will do so.
> 
>> I did this based on running file -L on the qemu-aarch64-static file in /usr/bin on my Arch Linux system after installing qemu-user-static.
>>
>> I got
>>
>> /usr/bin/qemu-aarch64-static: ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), static-pie linked, BuildID[sha1]=92b6a9c54239756a47441b87bed1dfab884cfdc8, for GNU/Linux 4.4.0, stripped
>>
>> as the response which has static-pie linked instead of statically linked.
>>
>> Changing the grep to look for static-pie linked may have got the build running but glibc took nearly three hors to build. I know it would be slower with the emulation and probably also slower the first time round as there would be nothing in the ccache but nearly three hours for one package seems excessive, especially when the native build is taking around 1.5 hours for the whole IPFire build.
> 
> Yes, the emulation is indeed *that* slow.
> 
>> Could the fact that my qemu-user-static program has the files static-pie linked be the cause of this very long run time?
> 
> No, this is just how long it takes sadly.
So after 24 hours there are still 15 packages left to compile on the base section. Based on this a complete build would be likely to take somewhere between 48 and 72 hours.

Based on my experience I am sorry but I won't be doing any cross compile builds for packages like samba that have different rootfiles for each architecture.
Presumably this will be easier in IPFire3 as a full build of everything won't be required so an aarch64 build could be done just of samba and its dependencies.

Regards,
Adolf.
> 
> -Michael
> 
>> Regards,
>>
>> Adolf.
>>>> Of course having changed ./make.sh means that it now shows up as a modified file in the git status. I will look at changing the find static command to look for either "statically linked" or "static-pie linked" and submit it as a patch for consideration.
>>>> Regards,
>>>>
>>>> Adolf.
>>>>
>>> -Michael
> 
>