Problem with trying to build qemu-9.0.2 - it requires dtc for all architectures

Problem with trying to build qemu-9.0.2 - it requires dtc for all architectures

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 836 bytes --]

Hi All,

I am trying to do an update of the qemu package.

It is failing for a missing fdt library which it says is required by targets x86_64-softmmu, aarch64-softmmu and riscv64-softmmu.

The previous version that we currently have, 8.1.2 has a bundled dtc which includes the fdt library. That has been removed, probably with the change to the 9.x branch as it is not in 9.0.2

We have the dtc package in make but it is after qemu.

I could move it to before qemu, however I also note that the dtc package is specified only for aarch64 but if we need to have the above softmmu targets specified then qemu is requiring the fdt library for all our architectures.

Before I move dtc and change it to be for all architectures, I just want to flag this up and confirm that it is the right thing to do.

Regards,

Adolf.

Re: Problem with trying to build qemu-9.0.2 - it requires dtc for all architectures

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 1281 bytes --]

Hello Adolf,

Yes, you can make those changes.

dtc is the device tree compiler and it should be quick to build. We just didn’t do that since we didn’t have any use for it apart from aarch64.

Move it wherever you want. It should not have any further dependencies. We can build it for all architectures.

-Michael

> On 2 Sep 2024, at 19:46, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi All,
> 
> I am trying to do an update of the qemu package.
> 
> It is failing for a missing fdt library which it says is required by targets x86_64-softmmu, aarch64-softmmu and riscv64-softmmu.
> 
> The previous version that we currently have, 8.1.2 has a bundled dtc which includes the fdt library. That has been removed, probably with the change to the 9.x branch as it is not in 9.0.2
> 
> We have the dtc package in make but it is after qemu.
> 
> I could move it to before qemu, however I also note that the dtc package is specified only for aarch64 but if we need to have the above softmmu targets specified then qemu is requiring the fdt library for all our architectures.
> 
> Before I move dtc and change it to be for all architectures, I just want to flag this up and confirm that it is the right thing to do.
> 
> Regards,
> 
> Adolf.
> 

Re: Problem with trying to build qemu-9.0.2 - it requires dtc for all architectures

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 2025 bytes --]

Hi Michael,

On 03/09/2024 09:24, Michael Tremer wrote:
> Hello Adolf,
> 
> Yes, you can make those changes.
> 
> dtc is the device tree compiler and it should be quick to build. We just didn’t do that since we didn’t have any use for it apart from aarch64.
> 
> Move it wherever you want. It should not have any further dependencies. We can build it for all architectures.

I did that and qemu found libfdt but then said that it was too old and that a min of 1.5.1 was required but the updated version of dtc I had built was version 1.7.1

It looks like qemu can correctly detect the presence of libfdt but not the version number.

I have raised this as an issue in the qemu gitlab site.

Version 9.0.0 still had the dtc subproject included in the source tarball, so I am currently building that and have left the IPFire dtc where it was for the moment.

This will still update qemu by 11 version numbers.

I will wait to see what response I get back from the qemu team for updating to 9.0.2

Regards,
Adolf.

> 
> -Michael
> 
>> On 2 Sep 2024, at 19:46, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>
>> Hi All,
>>
>> I am trying to do an update of the qemu package.
>>
>> It is failing for a missing fdt library which it says is required by targets x86_64-softmmu, aarch64-softmmu and riscv64-softmmu.
>>
>> The previous version that we currently have, 8.1.2 has a bundled dtc which includes the fdt library. That has been removed, probably with the change to the 9.x branch as it is not in 9.0.2
>>
>> We have the dtc package in make but it is after qemu.
>>
>> I could move it to before qemu, however I also note that the dtc package is specified only for aarch64 but if we need to have the above softmmu targets specified then qemu is requiring the fdt library for all our architectures.
>>
>> Before I move dtc and change it to be for all architectures, I just want to flag this up and confirm that it is the right thing to do.
>>
>> Regards,
>>
>> Adolf.
>>
> 

Re: Problem with trying to build qemu-9.0.2 - it requires dtc for all architectures

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 4100 bytes --]

Hi All,

On 03/09/2024 15:25, Adolf Belka wrote:
> Hi Michael,
>
> On 03/09/2024 09:24, Michael Tremer wrote:
>> Hello Adolf,
>>
>> Yes, you can make those changes.
>>
>> dtc is the device tree compiler and it should be quick to build. We just didn’t do that since we didn’t have any use for it apart from aarch64.
>>
>> Move it wherever you want. It should not have any further dependencies. We can build it for all architectures.
>
> I did that and qemu found libfdt but then said that it was too old and that a min of 1.5.1 was required but the updated version of dtc I had built was version 1.7.1
>
I have found the section in meson.build where it checks for the fdt library and it seems to me that the code says that if the system fdt is being used then say that it is too old. I can't see any actual check of the version number of libfdt.

Can you have a check through this following code and see if my interpretation is correct or if I have just misunderstood how the code is supposed to be working.

It also looks like the check for the fdt code being internal has not been removed from it even though dtc is no longer bundled with qemu.


fdt = not_found
fdt_opt = get_option('fdt')
if fdt_required.length() > 0 or fdt_opt == 'enabled'
   if fdt_opt == 'disabled'
     error('fdt disabled but required by targets ' + ', '.join(fdt_required))
   endif

   if fdt_opt in ['enabled', 'auto', 'system']
     if get_option('wrap_mode') == 'nodownload'
       fdt_opt = 'system'
     endif
     fdt = cc.find_library('fdt', required: fdt_opt == 'system')
     if fdt.found() and cc.links('''
        #include <libfdt.h>
        #include <libfdt_env.h>
        int main(void) { fdt_find_max_phandle(NULL, NULL); return 0; }''',
          dependencies: fdt)
       fdt_opt = 'system'
     elif fdt_opt == 'system'
        error('system libfdt requested, but it is too old (1.5.1 or newer required)')
     else
       fdt_opt = 'internal'
       fdt = not_found
     endif
   endif
   if not fdt.found()
     assert(fdt_opt == 'internal')
     libfdt_proj = subproject('dtc', required: true,
                              default_options: ['tools=false', 'yaml=disabled',
                                                'python=disabled', 'default_library=static'])
     fdt = libfdt_proj.get_variable('libfdt_dep')
   endif
else
   fdt_opt = 'disabled'
endif


> It looks like qemu can correctly detect the presence of libfdt but not the version number.
>
> I have raised this as an issue in the qemu gitlab site.
>
> Version 9.0.0 still had the dtc subproject included in the source tarball, so I am currently building that and have left the IPFire dtc where it was for the moment.
>
> This will still update qemu by 11 version numbers.
>
> I will wait to see what response I get back from the qemu team for updating to 9.0.2
>
> Regards,
> Adolf.
>
>>
>> -Michael
>>
>>> On 2 Sep 2024, at 19:46, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>
>>> Hi All,
>>>
>>> I am trying to do an update of the qemu package.
>>>
>>> It is failing for a missing fdt library which it says is required by targets x86_64-softmmu, aarch64-softmmu and riscv64-softmmu.
>>>
>>> The previous version that we currently have, 8.1.2 has a bundled dtc which includes the fdt library. That has been removed, probably with the change to the 9.x branch as it is not in 9.0.2
>>>
>>> We have the dtc package in make but it is after qemu.
>>>
>>> I could move it to before qemu, however I also note that the dtc package is specified only for aarch64 but if we need to have the above softmmu targets specified then qemu is requiring the fdt library for all our architectures.
>>>
>>> Before I move dtc and change it to be for all architectures, I just want to flag this up and confirm that it is the right thing to do.
>>>
>>> Regards,
>>>
>>> Adolf.
>>>
>>

Re: Problem with trying to build qemu-9.0.2 - it requires dtc for all architectures

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 6242 bytes --]

Hello,

> On 3 Sep 2024, at 21:06, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi All,
> 
> On 03/09/2024 15:25, Adolf Belka wrote:
>> Hi Michael,
>> 
>> On 03/09/2024 09:24, Michael Tremer wrote:
>>> Hello Adolf,
>>> 
>>> Yes, you can make those changes.
>>> 
>>> dtc is the device tree compiler and it should be quick to build. We just didn’t do that since we didn’t have any use for it apart from aarch64.
>>> 
>>> Move it wherever you want. It should not have any further dependencies. We can build it for all architectures.
>> 
>> I did that and qemu found libfdt but then said that it was too old and that a min of 1.5.1 was required but the updated version of dtc I had built was version 1.7.1
>> 
> I have found the section in meson.build where it checks for the fdt library and it seems to me that the code says that if the system fdt is being used then say that it is too old. I can't see any actual check of the version number of libfdt.

Hmm, the code is not the cleanest, but it should perform a few checks before it throws that error. The crucial part is here:

>     if fdt.found() and cc.links(''’
>        ...''',
>          dependencies: fdt)
>       fdt_opt = 'system'
>     elif fdt_opt == 'system'
>        error('system libfdt requested, but it is too old (1.5.1 or newer required)')
>     else
>       fdt_opt = 'internal'
>       fdt = not_found
>     endif


It first of all checks if fdt.found() returns true, it so, it will check to compile and build the program in cc.links(…). I assume that the function fdt_find_max_phandle() does not exitt before version 1.5.1 and therefore the program should not build with older versions. Then it will move to the “elif fdt_opt == ’system’” part and complain that the library that is already installed is incompatible.

So the check should work and actually do something.

Whenever a program bundles a library, we should remove it. Bundles libraries are evil because lets say there is a problem in libfdt, we will patch the package, but we don’t know what other programs bring their own version which won’t receive the fix. This is particularly bad for anything security-related as parts of the distribution will still be vulnerable.

Shared libraries (like libfdt.so <http://libfdt.so/>.N) are nicer because they can be pulled in by any program that needs the functionality and can be replaced with an improved version without any regard of what programs link against it - for as long as there are no breaking changes.

Rust for example is going the opposite way where everything is statically linked which is a security nightmare as described above. Every program that uses a vulnerable crate will have to be rebuilt. But how do you find them all?

There is more about bundles libraries here: https://fedoraproject.org/wiki/Bundled_Libraries

> Can you have a check through this following code and see if my interpretation is correct or if I have just misunderstood how the code is supposed to be working.
> 
> It also looks like the check for the fdt code being internal has not been removed from it even though dtc is no longer bundled with qemu.

In that case, we should set fdt=system just to make sure that QEMU does the right thing during the build.

Did I not merge any patches yesterday? Is that version not affected?

-Michael

> fdt = not_found
> fdt_opt = get_option('fdt')
> if fdt_required.length() > 0 or fdt_opt == 'enabled'
>   if fdt_opt == 'disabled'
>     error('fdt disabled but required by targets ' + ', '.join(fdt_required))
>   endif
> 
>   if fdt_opt in ['enabled', 'auto', 'system']
>     if get_option('wrap_mode') == 'nodownload'
>       fdt_opt = 'system'
>     endif
>     fdt = cc.find_library('fdt', required: fdt_opt == 'system')
>     if fdt.found() and cc.links('''
>        #include <libfdt.h>
>        #include <libfdt_env.h>
>        int main(void) { fdt_find_max_phandle(NULL, NULL); return 0; }''',
>          dependencies: fdt)
>       fdt_opt = 'system'
>     elif fdt_opt == 'system'
>        error('system libfdt requested, but it is too old (1.5.1 or newer required)')
>     else
>       fdt_opt = 'internal'
>       fdt = not_found
>     endif
>   endif
>   if not fdt.found()
>     assert(fdt_opt == 'internal')
>     libfdt_proj = subproject('dtc', required: true,
>                              default_options: ['tools=false', 'yaml=disabled',
>                                                'python=disabled', 'default_library=static'])
>     fdt = libfdt_proj.get_variable('libfdt_dep')
>   endif
> else
>   fdt_opt = 'disabled'
> endif
> 
> 
>> It looks like qemu can correctly detect the presence of libfdt but not the version number.
>> 
>> I have raised this as an issue in the qemu gitlab site.
>> 
>> Version 9.0.0 still had the dtc subproject included in the source tarball, so I am currently building that and have left the IPFire dtc where it was for the moment.
>> 
>> This will still update qemu by 11 version numbers.
>> 
>> I will wait to see what response I get back from the qemu team for updating to 9.0.2
>> 
>> Regards,
>> Adolf.
>> 
>>> 
>>> -Michael
>>> 
>>>> On 2 Sep 2024, at 19:46, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>> 
>>>> Hi All,
>>>> 
>>>> I am trying to do an update of the qemu package.
>>>> 
>>>> It is failing for a missing fdt library which it says is required by targets x86_64-softmmu, aarch64-softmmu and riscv64-softmmu.
>>>> 
>>>> The previous version that we currently have, 8.1.2 has a bundled dtc which includes the fdt library. That has been removed, probably with the change to the 9.x branch as it is not in 9.0.2
>>>> 
>>>> We have the dtc package in make but it is after qemu.
>>>> 
>>>> I could move it to before qemu, however I also note that the dtc package is specified only for aarch64 but if we need to have the above softmmu targets specified then qemu is requiring the fdt library for all our architectures.
>>>> 
>>>> Before I move dtc and change it to be for all architectures, I just want to flag this up and confirm that it is the right thing to do.
>>>> 
>>>> Regards,
>>>> 
>>>> Adolf.

Re: Problem with trying to build qemu-9.0.2 - it requires dtc for all architectures

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 7000 bytes --]

Hi Michael,

On 04/09/2024 08:58, Michael Tremer wrote:
> Hello,
> 
>> On 3 Sep 2024, at 21:06, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>
>> Hi All,
>>
>> On 03/09/2024 15:25, Adolf Belka wrote:
>>> Hi Michael,
>>>
>>> On 03/09/2024 09:24, Michael Tremer wrote:
>>>> Hello Adolf,
>>>>
>>>> Yes, you can make those changes.
>>>>
>>>> dtc is the device tree compiler and it should be quick to build. We just didn’t do that since we didn’t have any use for it apart from aarch64.
>>>>
>>>> Move it wherever you want. It should not have any further dependencies. We can build it for all architectures.
>>>
>>> I did that and qemu found libfdt but then said that it was too old and that a min of 1.5.1 was required but the updated version of dtc I had built was version 1.7.1
>>>
>> I have found the section in meson.build where it checks for the fdt library and it seems to me that the code says that if the system fdt is being used then say that it is too old. I can't see any actual check of the version number of libfdt.
> 
> Hmm, the code is not the cleanest, but it should perform a few checks before it throws that error. The crucial part is here:
> 
>>      if fdt.found() and cc.links(''’
>>         ...''',
>>           dependencies: fdt)
>>        fdt_opt = 'system'
>>      elif fdt_opt == 'system'
>>         error('system libfdt requested, but it is too old (1.5.1 or newer required)')
>>      else
>>        fdt_opt = 'internal'
>>        fdt = not_found
>>      endif
> 
> 
> It first of all checks if fdt.found() returns true, it so, it will check to compile and build the program in cc.links(…). I assume that the function fdt_find_max_phandle() does not exitt before version 1.5.1 and therefore the program should not build with older versions. Then it will move to the “elif fdt_opt == ’system’” part and complain that the library that is already installed is incompatible.
> 
> So the check should work and actually do something.
> 
> Whenever a program bundles a library, we should remove it. Bundles libraries are evil because lets say there is a problem in libfdt, we will patch the package, but we don’t know what other programs bring their own version which won’t receive the fix. This is particularly bad for anything security-related as parts of the distribution will still be vulnerable.

Then when I am doing my package updates if I see any bundled stuff then I will check how to use a system version instead.
> 
> Shared libraries (like libfdt.so <http://libfdt.so/>.N) are nicer because they can be pulled in by any program that needs the functionality and can be replaced with an improved version without any regard of what programs link against it - for as long as there are no breaking changes.
> 
> Rust for example is going the opposite way where everything is statically linked which is a security nightmare as described above. Every program that uses a vulnerable crate will have to be rebuilt. But how do you find them all?
> 
> There is more about bundles libraries here: https://fedoraproject.org/wiki/Bundled_Libraries
> 
>> Can you have a check through this following code and see if my interpretation is correct or if I have just misunderstood how the code is supposed to be working.
>>
>> It also looks like the check for the fdt code being internal has not been removed from it even though dtc is no longer bundled with qemu.
> 
> In that case, we should set fdt=system just to make sure that QEMU does the right thing during the build.

Okay, I will try that and see what happens.
> 
> Did I not merge any patches yesterday? Is that version not affected?

Yes, you did but that was for version 9.0.0 which still had the bundled dtc. I thought it was still worth updating as there have been 11 update versions since the 8.2.1 version we currently have in stable.

I will also have a look at the other packages they have in the subprojects directory to see if our build uses any of them.

Regards,

Adolf.
> 
> -Michael
> 
>> fdt = not_found
>> fdt_opt = get_option('fdt')
>> if fdt_required.length() > 0 or fdt_opt == 'enabled'
>>    if fdt_opt == 'disabled'
>>      error('fdt disabled but required by targets ' + ', '.join(fdt_required))
>>    endif
>>
>>    if fdt_opt in ['enabled', 'auto', 'system']
>>      if get_option('wrap_mode') == 'nodownload'
>>        fdt_opt = 'system'
>>      endif
>>      fdt = cc.find_library('fdt', required: fdt_opt == 'system')
>>      if fdt.found() and cc.links('''
>>         #include <libfdt.h>
>>         #include <libfdt_env.h>
>>         int main(void) { fdt_find_max_phandle(NULL, NULL); return 0; }''',
>>           dependencies: fdt)
>>        fdt_opt = 'system'
>>      elif fdt_opt == 'system'
>>         error('system libfdt requested, but it is too old (1.5.1 or newer required)')
>>      else
>>        fdt_opt = 'internal'
>>        fdt = not_found
>>      endif
>>    endif
>>    if not fdt.found()
>>      assert(fdt_opt == 'internal')
>>      libfdt_proj = subproject('dtc', required: true,
>>                               default_options: ['tools=false', 'yaml=disabled',
>>                                                 'python=disabled', 'default_library=static'])
>>      fdt = libfdt_proj.get_variable('libfdt_dep')
>>    endif
>> else
>>    fdt_opt = 'disabled'
>> endif
>>
>>
>>> It looks like qemu can correctly detect the presence of libfdt but not the version number.
>>>
>>> I have raised this as an issue in the qemu gitlab site.
>>>
>>> Version 9.0.0 still had the dtc subproject included in the source tarball, so I am currently building that and have left the IPFire dtc where it was for the moment.
>>>
>>> This will still update qemu by 11 version numbers.
>>>
>>> I will wait to see what response I get back from the qemu team for updating to 9.0.2
>>>
>>> Regards,
>>> Adolf.
>>>
>>>>
>>>> -Michael
>>>>
>>>>> On 2 Sep 2024, at 19:46, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>
>>>>> Hi All,
>>>>>
>>>>> I am trying to do an update of the qemu package.
>>>>>
>>>>> It is failing for a missing fdt library which it says is required by targets x86_64-softmmu, aarch64-softmmu and riscv64-softmmu.
>>>>>
>>>>> The previous version that we currently have, 8.1.2 has a bundled dtc which includes the fdt library. That has been removed, probably with the change to the 9.x branch as it is not in 9.0.2
>>>>>
>>>>> We have the dtc package in make but it is after qemu.
>>>>>
>>>>> I could move it to before qemu, however I also note that the dtc package is specified only for aarch64 but if we need to have the above softmmu targets specified then qemu is requiring the fdt library for all our architectures.
>>>>>
>>>>> Before I move dtc and change it to be for all architectures, I just want to flag this up and confirm that it is the right thing to do.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Adolf.
> 
> 

Re: Problem with trying to build qemu-9.0.2 - it requires dtc for all architectures

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 7280 bytes --]

Hello,

> On 4 Sep 2024, at 09:22, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi Michael,
> 
> On 04/09/2024 08:58, Michael Tremer wrote:
>> Hello,
>>> On 3 Sep 2024, at 21:06, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>> 
>>> Hi All,
>>> 
>>> On 03/09/2024 15:25, Adolf Belka wrote:
>>>> Hi Michael,
>>>> 
>>>> On 03/09/2024 09:24, Michael Tremer wrote:
>>>>> Hello Adolf,
>>>>> 
>>>>> Yes, you can make those changes.
>>>>> 
>>>>> dtc is the device tree compiler and it should be quick to build. We just didn’t do that since we didn’t have any use for it apart from aarch64.
>>>>> 
>>>>> Move it wherever you want. It should not have any further dependencies. We can build it for all architectures.
>>>> 
>>>> I did that and qemu found libfdt but then said that it was too old and that a min of 1.5.1 was required but the updated version of dtc I had built was version 1.7.1
>>>> 
>>> I have found the section in meson.build where it checks for the fdt library and it seems to me that the code says that if the system fdt is being used then say that it is too old. I can't see any actual check of the version number of libfdt.
>> Hmm, the code is not the cleanest, but it should perform a few checks before it throws that error. The crucial part is here:
>>>     if fdt.found() and cc.links(''’
>>>        ...''',
>>>          dependencies: fdt)
>>>       fdt_opt = 'system'
>>>     elif fdt_opt == 'system'
>>>        error('system libfdt requested, but it is too old (1.5.1 or newer required)')
>>>     else
>>>       fdt_opt = 'internal'
>>>       fdt = not_found
>>>     endif
>> It first of all checks if fdt.found() returns true, it so, it will check to compile and build the program in cc.links(…). I assume that the function fdt_find_max_phandle() does not exitt before version 1.5.1 and therefore the program should not build with older versions. Then it will move to the “elif fdt_opt == ’system’” part and complain that the library that is already installed is incompatible.
>> So the check should work and actually do something.
>> Whenever a program bundles a library, we should remove it. Bundles libraries are evil because lets say there is a problem in libfdt, we will patch the package, but we don’t know what other programs bring their own version which won’t receive the fix. This is particularly bad for anything security-related as parts of the distribution will still be vulnerable.
> 
> Then when I am doing my package updates if I see any bundled stuff then I will check how to use a system version instead.
>> Shared libraries (like libfdt.so <http://libfdt.so/>.N) are nicer because they can be pulled in by any program that needs the functionality and can be replaced with an improved version without any regard of what programs link against it - for as long as there are no breaking changes.
>> Rust for example is going the opposite way where everything is statically linked which is a security nightmare as described above. Every program that uses a vulnerable crate will have to be rebuilt. But how do you find them all?
>> There is more about bundles libraries here: https://fedoraproject.org/wiki/Bundled_Libraries
>>> Can you have a check through this following code and see if my interpretation is correct or if I have just misunderstood how the code is supposed to be working.
>>> 
>>> It also looks like the check for the fdt code being internal has not been removed from it even though dtc is no longer bundled with qemu.
>> In that case, we should set fdt=system just to make sure that QEMU does the right thing during the build.
> 
> Okay, I will try that and see what happens.

For as long as you have built dtc before QEMU this should work fine.

>> Did I not merge any patches yesterday? Is that version not affected?
> 
> Yes, you did but that was for version 9.0.0 which still had the bundled dtc. I thought it was still worth updating as there have been 11 update versions since the 8.2.1 version we currently have in stable.

Ah, that makes sense.

> I will also have a look at the other packages they have in the subprojects directory to see if our build uses any of them.

Thank you!

-Michael

> Regards,
> 
> Adolf.
>> -Michael
>>> fdt = not_found
>>> fdt_opt = get_option('fdt')
>>> if fdt_required.length() > 0 or fdt_opt == 'enabled'
>>>   if fdt_opt == 'disabled'
>>>     error('fdt disabled but required by targets ' + ', '.join(fdt_required))
>>>   endif
>>> 
>>>   if fdt_opt in ['enabled', 'auto', 'system']
>>>     if get_option('wrap_mode') == 'nodownload'
>>>       fdt_opt = 'system'
>>>     endif
>>>     fdt = cc.find_library('fdt', required: fdt_opt == 'system')
>>>     if fdt.found() and cc.links('''
>>>        #include <libfdt.h>
>>>        #include <libfdt_env.h>
>>>        int main(void) { fdt_find_max_phandle(NULL, NULL); return 0; }''',
>>>          dependencies: fdt)
>>>       fdt_opt = 'system'
>>>     elif fdt_opt == 'system'
>>>        error('system libfdt requested, but it is too old (1.5.1 or newer required)')
>>>     else
>>>       fdt_opt = 'internal'
>>>       fdt = not_found
>>>     endif
>>>   endif
>>>   if not fdt.found()
>>>     assert(fdt_opt == 'internal')
>>>     libfdt_proj = subproject('dtc', required: true,
>>>                              default_options: ['tools=false', 'yaml=disabled',
>>>                                                'python=disabled', 'default_library=static'])
>>>     fdt = libfdt_proj.get_variable('libfdt_dep')
>>>   endif
>>> else
>>>   fdt_opt = 'disabled'
>>> endif
>>> 
>>> 
>>>> It looks like qemu can correctly detect the presence of libfdt but not the version number.
>>>> 
>>>> I have raised this as an issue in the qemu gitlab site.
>>>> 
>>>> Version 9.0.0 still had the dtc subproject included in the source tarball, so I am currently building that and have left the IPFire dtc where it was for the moment.
>>>> 
>>>> This will still update qemu by 11 version numbers.
>>>> 
>>>> I will wait to see what response I get back from the qemu team for updating to 9.0.2
>>>> 
>>>> Regards,
>>>> Adolf.
>>>> 
>>>>> 
>>>>> -Michael
>>>>> 
>>>>>> On 2 Sep 2024, at 19:46, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>> 
>>>>>> Hi All,
>>>>>> 
>>>>>> I am trying to do an update of the qemu package.
>>>>>> 
>>>>>> It is failing for a missing fdt library which it says is required by targets x86_64-softmmu, aarch64-softmmu and riscv64-softmmu.
>>>>>> 
>>>>>> The previous version that we currently have, 8.1.2 has a bundled dtc which includes the fdt library. That has been removed, probably with the change to the 9.x branch as it is not in 9.0.2
>>>>>> 
>>>>>> We have the dtc package in make but it is after qemu.
>>>>>> 
>>>>>> I could move it to before qemu, however I also note that the dtc package is specified only for aarch64 but if we need to have the above softmmu targets specified then qemu is requiring the fdt library for all our architectures.
>>>>>> 
>>>>>> Before I move dtc and change it to be for all architectures, I just want to flag this up and confirm that it is the right thing to do.
>>>>>> 
>>>>>> Regards,
>>>>>> 
>>>>>> Adolf.

Re: Problem with trying to build qemu-9.0.2 - it requires dtc for all architectures

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 8650 bytes --]

Hi Michael,

On 04/09/2024 11:34, Michael Tremer wrote:
> Hello,
> 
>> On 4 Sep 2024, at 09:22, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>
>> Hi Michael,
>>
>> On 04/09/2024 08:58, Michael Tremer wrote:
>>> Hello,
>>>> On 3 Sep 2024, at 21:06, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>
>>>> Hi All,
>>>>
>>>> On 03/09/2024 15:25, Adolf Belka wrote:
>>>>> Hi Michael,
>>>>>
>>>>> On 03/09/2024 09:24, Michael Tremer wrote:
>>>>>> Hello Adolf,
>>>>>>
>>>>>> Yes, you can make those changes.
>>>>>>
>>>>>> dtc is the device tree compiler and it should be quick to build. We just didn’t do that since we didn’t have any use for it apart from aarch64.
>>>>>>
>>>>>> Move it wherever you want. It should not have any further dependencies. We can build it for all architectures.
>>>>>
>>>>> I did that and qemu found libfdt but then said that it was too old and that a min of 1.5.1 was required but the updated version of dtc I had built was version 1.7.1
>>>>>
>>>> I have found the section in meson.build where it checks for the fdt library and it seems to me that the code says that if the system fdt is being used then say that it is too old. I can't see any actual check of the version number of libfdt.
>>> Hmm, the code is not the cleanest, but it should perform a few checks before it throws that error. The crucial part is here:
>>>>      if fdt.found() and cc.links(''’
>>>>         ...''',
>>>>           dependencies: fdt)
>>>>        fdt_opt = 'system'
>>>>      elif fdt_opt == 'system'
>>>>         error('system libfdt requested, but it is too old (1.5.1 or newer required)')
>>>>      else
>>>>        fdt_opt = 'internal'
>>>>        fdt = not_found
>>>>      endif
>>> It first of all checks if fdt.found() returns true, it so, it will check to compile and build the program in cc.links(…). I assume that the function fdt_find_max_phandle() does not exitt before version 1.5.1 and therefore the program should not build with older versions. Then it will move to the “elif fdt_opt == ’system’” part and complain that the library that is already installed is incompatible.
>>> So the check should work and actually do something.
>>> Whenever a program bundles a library, we should remove it. Bundles libraries are evil because lets say there is a problem in libfdt, we will patch the package, but we don’t know what other programs bring their own version which won’t receive the fix. This is particularly bad for anything security-related as parts of the distribution will still be vulnerable.
>>
>> Then when I am doing my package updates if I see any bundled stuff then I will check how to use a system version instead.
>>> Shared libraries (like libfdt.so <http://libfdt.so/>.N) are nicer because they can be pulled in by any program that needs the functionality and can be replaced with an improved version without any regard of what programs link against it - for as long as there are no breaking changes.
>>> Rust for example is going the opposite way where everything is statically linked which is a security nightmare as described above. Every program that uses a vulnerable crate will have to be rebuilt. But how do you find them all?
>>> There is more about bundles libraries here: https://fedoraproject.org/wiki/Bundled_Libraries
>>>> Can you have a check through this following code and see if my interpretation is correct or if I have just misunderstood how the code is supposed to be working.
>>>>
>>>> It also looks like the check for the fdt code being internal has not been removed from it even though dtc is no longer bundled with qemu.
>>> In that case, we should set fdt=system just to make sure that QEMU does the right thing during the build.
>>
>> Okay, I will try that and see what happens.
> 
> For as long as you have built dtc before QEMU this should work fine.

That change was not enough on its own to make the libfdt detection work.

I got feedback from Thomas Huth on the qemu gitlab site saying that the error message might be a bit misleading and that the issue was more likely that the libfdt was simply not usable. He said that more info should be in the meson-logs/meson-log.txt file

Commenting out the rm -rf $(DIR_APP) command in the qemu lfs meant that I could read the build/meson-logs/meson-log.txt and that showed that the #include <libfdt.h> command was not finding the libfdt.h file.

This turned out to be that the dtc build did not have PREFIX=/usr and so those files were being put in /include/ and not /usr/include/

So I added PREFIX=/usr to the end of the dtc make install line and qemu then successfully built.

I will now do a clean and fresh build to confirm that everything stays working but I might be able to submit a patch for qemu-9.0.2 now.

Is there any problem with the dtc command being changed from

cd $(DIR_APP) && make HOME= install

to

cd $(DIR_APP) && make HOME= install PREFIX=/usr

in terms of the aarch64 requirements?

Regards,
Adolf.

> 
>>> Did I not merge any patches yesterday? Is that version not affected?
>>
>> Yes, you did but that was for version 9.0.0 which still had the bundled dtc. I thought it was still worth updating as there have been 11 update versions since the 8.2.1 version we currently have in stable.
> 
> Ah, that makes sense.
> 
>> I will also have a look at the other packages they have in the subprojects directory to see if our build uses any of them.
> 
> Thank you!
> 
> -Michael
> 
>> Regards,
>>
>> Adolf.
>>> -Michael
>>>> fdt = not_found
>>>> fdt_opt = get_option('fdt')
>>>> if fdt_required.length() > 0 or fdt_opt == 'enabled'
>>>>    if fdt_opt == 'disabled'
>>>>      error('fdt disabled but required by targets ' + ', '.join(fdt_required))
>>>>    endif
>>>>
>>>>    if fdt_opt in ['enabled', 'auto', 'system']
>>>>      if get_option('wrap_mode') == 'nodownload'
>>>>        fdt_opt = 'system'
>>>>      endif
>>>>      fdt = cc.find_library('fdt', required: fdt_opt == 'system')
>>>>      if fdt.found() and cc.links('''
>>>>         #include <libfdt.h>
>>>>         #include <libfdt_env.h>
>>>>         int main(void) { fdt_find_max_phandle(NULL, NULL); return 0; }''',
>>>>           dependencies: fdt)
>>>>        fdt_opt = 'system'
>>>>      elif fdt_opt == 'system'
>>>>         error('system libfdt requested, but it is too old (1.5.1 or newer required)')
>>>>      else
>>>>        fdt_opt = 'internal'
>>>>        fdt = not_found
>>>>      endif
>>>>    endif
>>>>    if not fdt.found()
>>>>      assert(fdt_opt == 'internal')
>>>>      libfdt_proj = subproject('dtc', required: true,
>>>>                               default_options: ['tools=false', 'yaml=disabled',
>>>>                                                 'python=disabled', 'default_library=static'])
>>>>      fdt = libfdt_proj.get_variable('libfdt_dep')
>>>>    endif
>>>> else
>>>>    fdt_opt = 'disabled'
>>>> endif
>>>>
>>>>
>>>>> It looks like qemu can correctly detect the presence of libfdt but not the version number.
>>>>>
>>>>> I have raised this as an issue in the qemu gitlab site.
>>>>>
>>>>> Version 9.0.0 still had the dtc subproject included in the source tarball, so I am currently building that and have left the IPFire dtc where it was for the moment.
>>>>>
>>>>> This will still update qemu by 11 version numbers.
>>>>>
>>>>> I will wait to see what response I get back from the qemu team for updating to 9.0.2
>>>>>
>>>>> Regards,
>>>>> Adolf.
>>>>>
>>>>>>
>>>>>> -Michael
>>>>>>
>>>>>>> On 2 Sep 2024, at 19:46, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> I am trying to do an update of the qemu package.
>>>>>>>
>>>>>>> It is failing for a missing fdt library which it says is required by targets x86_64-softmmu, aarch64-softmmu and riscv64-softmmu.
>>>>>>>
>>>>>>> The previous version that we currently have, 8.1.2 has a bundled dtc which includes the fdt library. That has been removed, probably with the change to the 9.x branch as it is not in 9.0.2
>>>>>>>
>>>>>>> We have the dtc package in make but it is after qemu.
>>>>>>>
>>>>>>> I could move it to before qemu, however I also note that the dtc package is specified only for aarch64 but if we need to have the above softmmu targets specified then qemu is requiring the fdt library for all our architectures.
>>>>>>>
>>>>>>> Before I move dtc and change it to be for all architectures, I just want to flag this up and confirm that it is the right thing to do.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Adolf.
> 
> 

Re: Problem with trying to build qemu-9.0.2 - it requires dtc for all architectures

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 9308 bytes --]

Hello,

> On 4 Sep 2024, at 15:37, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi Michael,
> 
> On 04/09/2024 11:34, Michael Tremer wrote:
>> Hello,
>>> On 4 Sep 2024, at 09:22, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>> 
>>> Hi Michael,
>>> 
>>> On 04/09/2024 08:58, Michael Tremer wrote:
>>>> Hello,
>>>>> On 3 Sep 2024, at 21:06, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>> 
>>>>> Hi All,
>>>>> 
>>>>> On 03/09/2024 15:25, Adolf Belka wrote:
>>>>>> Hi Michael,
>>>>>> 
>>>>>> On 03/09/2024 09:24, Michael Tremer wrote:
>>>>>>> Hello Adolf,
>>>>>>> 
>>>>>>> Yes, you can make those changes.
>>>>>>> 
>>>>>>> dtc is the device tree compiler and it should be quick to build. We just didn’t do that since we didn’t have any use for it apart from aarch64.
>>>>>>> 
>>>>>>> Move it wherever you want. It should not have any further dependencies. We can build it for all architectures.
>>>>>> 
>>>>>> I did that and qemu found libfdt but then said that it was too old and that a min of 1.5.1 was required but the updated version of dtc I had built was version 1.7.1
>>>>>> 
>>>>> I have found the section in meson.build where it checks for the fdt library and it seems to me that the code says that if the system fdt is being used then say that it is too old. I can't see any actual check of the version number of libfdt.
>>>> Hmm, the code is not the cleanest, but it should perform a few checks before it throws that error. The crucial part is here:
>>>>>     if fdt.found() and cc.links(''’
>>>>>        ...''',
>>>>>          dependencies: fdt)
>>>>>       fdt_opt = 'system'
>>>>>     elif fdt_opt == 'system'
>>>>>        error('system libfdt requested, but it is too old (1.5.1 or newer required)')
>>>>>     else
>>>>>       fdt_opt = 'internal'
>>>>>       fdt = not_found
>>>>>     endif
>>>> It first of all checks if fdt.found() returns true, it so, it will check to compile and build the program in cc.links(…). I assume that the function fdt_find_max_phandle() does not exitt before version 1.5.1 and therefore the program should not build with older versions. Then it will move to the “elif fdt_opt == ’system’” part and complain that the library that is already installed is incompatible.
>>>> So the check should work and actually do something.
>>>> Whenever a program bundles a library, we should remove it. Bundles libraries are evil because lets say there is a problem in libfdt, we will patch the package, but we don’t know what other programs bring their own version which won’t receive the fix. This is particularly bad for anything security-related as parts of the distribution will still be vulnerable.
>>> 
>>> Then when I am doing my package updates if I see any bundled stuff then I will check how to use a system version instead.
>>>> Shared libraries (like libfdt.so <http://libfdt.so/>.N) are nicer because they can be pulled in by any program that needs the functionality and can be replaced with an improved version without any regard of what programs link against it - for as long as there are no breaking changes.
>>>> Rust for example is going the opposite way where everything is statically linked which is a security nightmare as described above. Every program that uses a vulnerable crate will have to be rebuilt. But how do you find them all?
>>>> There is more about bundles libraries here: https://fedoraproject.org/wiki/Bundled_Libraries
>>>>> Can you have a check through this following code and see if my interpretation is correct or if I have just misunderstood how the code is supposed to be working.
>>>>> 
>>>>> It also looks like the check for the fdt code being internal has not been removed from it even though dtc is no longer bundled with qemu.
>>>> In that case, we should set fdt=system just to make sure that QEMU does the right thing during the build.
>>> 
>>> Okay, I will try that and see what happens.
>> For as long as you have built dtc before QEMU this should work fine.
> 
> That change was not enough on its own to make the libfdt detection work.
> 
> I got feedback from Thomas Huth on the qemu gitlab site saying that the error message might be a bit misleading and that the issue was more likely that the libfdt was simply not usable. He said that more info should be in the meson-logs/meson-log.txt file
> 
> Commenting out the rm -rf $(DIR_APP) command in the qemu lfs meant that I could read the build/meson-logs/meson-log.txt and that showed that the #include <libfdt.h> command was not finding the libfdt.h file.

If the build aborts, you can always run ./make.sh shell, cd into /usr/src and you should find the source when it aborted. You don’t need to remove the rm command, as it will never be reached when the build aborts.

> This turned out to be that the dtc build did not have PREFIX=/usr and so those files were being put in /include/ and not /usr/include/
> 
> So I added PREFIX=/usr to the end of the dtc make install line and qemu then successfully built.

Should this not be specified with the first build command - i.e. the equivalent of ./configure?

> I will now do a clean and fresh build to confirm that everything stays working but I might be able to submit a patch for qemu-9.0.2 now.
> 
> Is there any problem with the dtc command being changed from
> 
> cd $(DIR_APP) && make HOME= install
> 
> to
> 
> cd $(DIR_APP) && make HOME= install PREFIX=/usr

This is an acceptable change.

> in terms of the aarch64 requirements?

And aarch64 does not differ from x86 and the others…

-Michael

> 
> Regards,
> Adolf.
> 
>>>> Did I not merge any patches yesterday? Is that version not affected?
>>> 
>>> Yes, you did but that was for version 9.0.0 which still had the bundled dtc. I thought it was still worth updating as there have been 11 update versions since the 8.2.1 version we currently have in stable.
>> Ah, that makes sense.
>>> I will also have a look at the other packages they have in the subprojects directory to see if our build uses any of them.
>> Thank you!
>> -Michael
>>> Regards,
>>> 
>>> Adolf.
>>>> -Michael
>>>>> fdt = not_found
>>>>> fdt_opt = get_option('fdt')
>>>>> if fdt_required.length() > 0 or fdt_opt == 'enabled'
>>>>>   if fdt_opt == 'disabled'
>>>>>     error('fdt disabled but required by targets ' + ', '.join(fdt_required))
>>>>>   endif
>>>>> 
>>>>>   if fdt_opt in ['enabled', 'auto', 'system']
>>>>>     if get_option('wrap_mode') == 'nodownload'
>>>>>       fdt_opt = 'system'
>>>>>     endif
>>>>>     fdt = cc.find_library('fdt', required: fdt_opt == 'system')
>>>>>     if fdt.found() and cc.links('''
>>>>>        #include <libfdt.h>
>>>>>        #include <libfdt_env.h>
>>>>>        int main(void) { fdt_find_max_phandle(NULL, NULL); return 0; }''',
>>>>>          dependencies: fdt)
>>>>>       fdt_opt = 'system'
>>>>>     elif fdt_opt == 'system'
>>>>>        error('system libfdt requested, but it is too old (1.5.1 or newer required)')
>>>>>     else
>>>>>       fdt_opt = 'internal'
>>>>>       fdt = not_found
>>>>>     endif
>>>>>   endif
>>>>>   if not fdt.found()
>>>>>     assert(fdt_opt == 'internal')
>>>>>     libfdt_proj = subproject('dtc', required: true,
>>>>>                              default_options: ['tools=false', 'yaml=disabled',
>>>>>                                                'python=disabled', 'default_library=static'])
>>>>>     fdt = libfdt_proj.get_variable('libfdt_dep')
>>>>>   endif
>>>>> else
>>>>>   fdt_opt = 'disabled'
>>>>> endif
>>>>> 
>>>>> 
>>>>>> It looks like qemu can correctly detect the presence of libfdt but not the version number.
>>>>>> 
>>>>>> I have raised this as an issue in the qemu gitlab site.
>>>>>> 
>>>>>> Version 9.0.0 still had the dtc subproject included in the source tarball, so I am currently building that and have left the IPFire dtc where it was for the moment.
>>>>>> 
>>>>>> This will still update qemu by 11 version numbers.
>>>>>> 
>>>>>> I will wait to see what response I get back from the qemu team for updating to 9.0.2
>>>>>> 
>>>>>> Regards,
>>>>>> Adolf.
>>>>>> 
>>>>>>> 
>>>>>>> -Michael
>>>>>>> 
>>>>>>>> On 2 Sep 2024, at 19:46, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>> 
>>>>>>>> Hi All,
>>>>>>>> 
>>>>>>>> I am trying to do an update of the qemu package.
>>>>>>>> 
>>>>>>>> It is failing for a missing fdt library which it says is required by targets x86_64-softmmu, aarch64-softmmu and riscv64-softmmu.
>>>>>>>> 
>>>>>>>> The previous version that we currently have, 8.1.2 has a bundled dtc which includes the fdt library. That has been removed, probably with the change to the 9.x branch as it is not in 9.0.2
>>>>>>>> 
>>>>>>>> We have the dtc package in make but it is after qemu.
>>>>>>>> 
>>>>>>>> I could move it to before qemu, however I also note that the dtc package is specified only for aarch64 but if we need to have the above softmmu targets specified then qemu is requiring the fdt library for all our architectures.
>>>>>>>> 
>>>>>>>> Before I move dtc and change it to be for all architectures, I just want to flag this up and confirm that it is the right thing to do.
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> 
>>>>>>>> Adolf.