Hi Michael and all,
I ran the arm builder with the 4.20.2 version of samba to test it out.
The build got to building gdb and then failed.
Interestingly, the nightly build of arm was successful with the same version of gdb.
The build log for gdb is attached. The actual error is at line 618.
Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message.
------------------------------------------
ssh bonnietwin@arm64-01.zrh.ipfire.org PTY allocation request failed on channel 0 Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64
The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied
------------------------------------------
Regards,
Adolf.
Hello Adolf,
This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more.
I rebooted the machine and it is back up again.
-Michael
On 2 Jul 2024, at 15:42, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael and all,
I ran the arm builder with the 4.20.2 version of samba to test it out.
The build got to building gdb and then failed.
Interestingly, the nightly build of arm was successful with the same version of gdb.
The build log for gdb is attached. The actual error is at line 618.
Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message.
ssh bonnietwin@arm64-01.zrh.ipfire.org PTY allocation request failed on channel 0 Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64
The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied
Regards,
Adolf.
<_build.ipfire.gdb.log>
Worked okay this time round.
Samba build took 3.5 hours.
Regards,
Adolf.
On 03/07/2024 11:58, Michael Tremer wrote:
Hello Adolf,
This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more.
I rebooted the machine and it is back up again.
-Michael
On 2 Jul 2024, at 15:42, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael and all,
I ran the arm builder with the 4.20.2 version of samba to test it out.
The build got to building gdb and then failed.
Interestingly, the nightly build of arm was successful with the same version of gdb.
The build log for gdb is attached. The actual error is at line 618.
Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message.
ssh bonnietwin@arm64-01.zrh.ipfire.org PTY allocation request failed on channel 0 Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64
The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied
Regards,
Adolf.
<_build.ipfire.gdb.log>
Hello,
I have been spending a lot of time on this problem, because it has been bothering me for a long time. I also saw an opportunity to make more changes to the build system.
Currently this is all a little bit WIP, but I hope that we can merge this into next as soon as the current update has been moved to master.
I am referring to this branch which is currently based on next: https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/u...
It makes use of the unshare command which creates new namespaces in Linux. That way, we can isolate the build system better from the host system and in case something goes wrong, there is less damage. We can also enforce some more rules…
So, what has changed?
* The make.sh script might re-execute itself into a new mount namespace when it is suitable. This happens for “make.sh build” and “make.sh shell”, but it does not happen for “make.sh downloadsrc” for example.
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262...
* The new mount namespace means that we will no longer see any bind-mounts in the host system and we no longer need to umount anything ourselves which is where we occasionally wiped the entire hard drive of the host system. When the last process exits, the namespace is being cleaned up and everything is being umounted.
* The function that prepares the build environment has been almost entirely rewritten:
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262...
It used to mount parts of the host system into the build environment which are needed to run anything. Those were /dev, /proc, /sys, etc…
Instead, it now creates a new /dev mount point and creates a minimal amount of device nodes and symlinks. That way, we detach from the host system and no longer allow the build system access to the host’s filesystem and block devices. We also bind-mount the sources in read-only mode now, so that the build system cannot change anything in the source tree. On top of that, cache is read-only, too. ccache and the log directory are the only places that are writable.
We mount a separate /tmp directory.
* When we then build a package, we create more namespaces for each package. These isolate each build process from each other.
Mostly, this is to detach from the host system. A new UTS namespace allows to change the hostname in the build system without affecting the host and so on. We do the same thing with a new time namespace.
We do however create a new PID namespace which means that the build system no longer will see any processes running on the host system. That requires to mount a new instance of /proc with each package. This also has the effect that if the shell that we launched terminates (because the build is done) any background processes will be killed immediately.
Last, we clone the mount namespace that we have created before so that no build command can modify what we set up earlier.
* Since everything is now so decoupled, we gain a couple of new (maybe minor?) features:
It is now possible to run “make.sh shell” while a build is running. That does not happen a lot, but we can do this now :)
If the build crashes or the host system is being shut down while a build is running, there is nothing to clean up afterwards.
* I have garnished this all with a lot of code cleanup and I suppose I might have introduced some new bugs here or there :)
* This is probably mostly around a new implementation of the timer that updates the build time. It has been annoying me a lot that it takes a long time to walk through all packages that have been built before to finally get to a package that we want to rebuild. Mostly this was all help up by a call of “sleep 0.1”
Since bash does not really do any concurrency, I had to be creative and replaced the busy-loop with a background process that is launched whenever it is needed and which will “ping” the main make.sh script once a second. That way, we can just run as usual, but regularly get interrupted to update the runtime.
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262...
We now only fork one extra sub shell and we have to handle the timer events which is a lot cheaper as well as more straight-forward to code.
* As there is no difference between the different stages any more (those stages that we inherited from LFS), I have merged them all into one.
* Last but not least, I have create the option to build for multiple architectures on the same system. Since we can now mount the entire source tree into (many independent) build environments, we might as well… As discussed on the last call, this might not be the best option for ARM, but RISV-C builds at a decent speed even when emulated.
The only thing that I needed to do for this is to suffix the build and log directories which are now called build_${ARCH}, i.e. build_aarch64, build_x86_64, and so on. The packages/ directory is not changed yet, but that will have to happen as well. Most likely I want to merge this with the generated images, but I am not sure what to call this, yet. Happy to hear suggestions. result_x86_64? Just images_x86_64?
---
I have run a build and this seems to be working just fine on my Debian machine. I am writing to all of you to first of all let you know what I am up to; and secondly to ask to give this a go on your systems. I think it should run just fine, as all the tools that I require should be available everywhere. However, there might be some older kernels that might not support all of this, yet or any other problems I cannot think of yet. Please give me some feedback and send me all the bugs :)
Thank you for listening to this brain-dump.
All the best, -Michael
On 3 Jul 2024, at 10:58, Michael Tremer michael.tremer@ipfire.org wrote:
Hello Adolf,
This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more.
I rebooted the machine and it is back up again.
-Michael
On 2 Jul 2024, at 15:42, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael and all,
I ran the arm builder with the 4.20.2 version of samba to test it out.
The build got to building gdb and then failed.
Interestingly, the nightly build of arm was successful with the same version of gdb.
The build log for gdb is attached. The actual error is at line 618.
Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message.
ssh bonnietwin@arm64-01.zrh.ipfire.org PTY allocation request failed on channel 0 Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64
The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied
Regards,
Adolf.
<_build.ipfire.gdb.log>
Hi Michael,
On 08/07/2024 18:11, Michael Tremer wrote:
Hello,
I have been spending a lot of time on this problem, because it has been bothering me for a long time. I also saw an opportunity to make more changes to the build system.
Currently this is all a little bit WIP, but I hope that we can merge this into next as soon as the current update has been moved to master.
I am referring to this branch which is currently based on next: https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/u...
It makes use of the unshare command which creates new namespaces in Linux. That way, we can isolate the build system better from the host system and in case something goes wrong, there is less damage. We can also enforce some more rules…
So, what has changed?
The make.sh script might re-execute itself into a new mount namespace when it is suitable. This happens for “make.sh build” and “make.sh shell”, but it does not happen for “make.sh downloadsrc” for example.
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262...
The new mount namespace means that we will no longer see any bind-mounts in the host system and we no longer need to umount anything ourselves which is where we occasionally wiped the entire hard drive of the host system. When the last process exits, the namespace is being cleaned up and everything is being umounted.
The function that prepares the build environment has been almost entirely rewritten:
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262...
It used to mount parts of the host system into the build environment which are needed to run anything. Those were /dev, /proc, /sys, etc…
Instead, it now creates a new /dev mount point and creates a minimal amount of device nodes and symlinks. That way, we detach from the host system and no longer allow the build system access to the host’s filesystem and block devices. We also bind-mount the sources in read-only mode now, so that the build system cannot change anything in the source tree. On top of that, cache is read-only, too. ccache and the log directory are the only places that are writable.
We mount a separate /tmp directory.
- When we then build a package, we create more namespaces for each package. These isolate each build process from each other.
Mostly, this is to detach from the host system. A new UTS namespace allows to change the hostname in the build system without affecting the host and so on. We do the same thing with a new time namespace.
We do however create a new PID namespace which means that the build system no longer will see any processes running on the host system. That requires to mount a new instance of /proc with each package. This also has the effect that if the shell that we launched terminates (because the build is done) any background processes will be killed immediately.
Last, we clone the mount namespace that we have created before so that no build command can modify what we set up earlier.
Since everything is now so decoupled, we gain a couple of new (maybe minor?) features:
It is now possible to run “make.sh shell” while a build is running. That does not happen a lot, but we can do this now :)
If the build crashes or the host system is being shut down while a build is running, there is nothing to clean up afterwards.
I have garnished this all with a lot of code cleanup and I suppose I might have introduced some new bugs here or there :)
This is probably mostly around a new implementation of the timer that updates the build time. It has been annoying me a lot that it takes a long time to walk through all packages that have been built before to finally get to a package that we want to rebuild. Mostly this was all help up by a call of “sleep 0.1”
Since bash does not really do any concurrency, I had to be creative and replaced the busy-loop with a background process that is launched whenever it is needed and which will “ping” the main make.sh script once a second. That way, we can just run as usual, but regularly get interrupted to update the runtime.
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262...
We now only fork one extra sub shell and we have to handle the timer events which is a lot cheaper as well as more straight-forward to code.
As there is no difference between the different stages any more (those stages that we inherited from LFS), I have merged them all into one.
Last but not least, I have create the option to build for multiple architectures on the same system. Since we can now mount the entire source tree into (many independent) build environments, we might as well… As discussed on the last call, this might not be the best option for ARM, but RISV-C builds at a decent speed even when emulated.
The only thing that I needed to do for this is to suffix the build and log directories which are now called build_${ARCH}, i.e. build_aarch64, build_x86_64, and so on. The packages/ directory is not changed yet, but that will have to happen as well. Most likely I want to merge this with the generated images, but I am not sure what to call this, yet. Happy to hear suggestions. result_x86_64? Just images_x86_64?
I have run a build and this seems to be working just fine on my Debian machine. I am writing to all of you to first of all let you know what I am up to; and secondly to ask to give this a go on your systems. I think it should run just fine, as all the tools that I require should be available everywhere. However, there might be some older kernels that might not support all of this, yet or any other problems I cannot think of yet. Please give me some feedback and send me all the bugs :)
I gave this a go but it didn't work.
Not sure if I should have run the ./make.sh clean command on the old version before I pulled the unshare branch into my clone of your repo.
Should I have started with a complete new clone of your repo? I might try that anyway just to see.
So I ran ./make.sh gettoolchain first, as I usually would.
./make.sh gettoolchain b2sum: cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: No such file or directory cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: FAILED open or read b2sum: WARNING: 1 listed file could not be read
ipfire-2.29-toolchain-20240210-x86_64.tar.zst is present together with its b2 file.
Then ran ./make.sh downloadsrc
Previous version ends with
***Verifying BLAKE2 checksum all files BLAKE2 checksum match [ DONE]
after zstd has been checked.
New version stops at zstd entry.
./make.sh clean gave the message Cleaning Build directory... but was completed very quickly. Log and Build directories have not been cleaned out. The img and iso files are still present.
./make.sh build gave message
chroot: failed to run command ‘env’: No such file or directory
and then did a full toolchain compilation which failed with gcc but log is >9000 lines.
Regards, Adolf.
Thank you for listening to this brain-dump.
All the best, -Michael
On 3 Jul 2024, at 10:58, Michael Tremer michael.tremer@ipfire.org wrote:
Hello Adolf,
This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more.
I rebooted the machine and it is back up again.
-Michael
On 2 Jul 2024, at 15:42, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael and all,
I ran the arm builder with the 4.20.2 version of samba to test it out.
The build got to building gdb and then failed.
Interestingly, the nightly build of arm was successful with the same version of gdb.
The build log for gdb is attached. The actual error is at line 618.
Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message.
ssh bonnietwin@arm64-01.zrh.ipfire.org PTY allocation request failed on channel 0 Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64
The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied
Regards,
Adolf.
<_build.ipfire.gdb.log>
Hi Michael,
On 08/07/2024 21:15, Adolf Belka wrote:
Hi Michael,
On 08/07/2024 18:11, Michael Tremer wrote:
Hello,
I have been spending a lot of time on this problem, because it has been bothering me for a long time. I also saw an opportunity to make more changes to the build system.
Currently this is all a little bit WIP, but I hope that we can merge this into next as soon as the current update has been moved to master.
I am referring to this branch which is currently based on next: https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/u...
It makes use of the unshare command which creates new namespaces in Linux. That way, we can isolate the build system better from the host system and in case something goes wrong, there is less damage. We can also enforce some more rules…
So, what has changed?
- The make.sh script might re-execute itself into a new mount namespace when it is suitable. This happens for “make.sh build” and “make.sh shell”, but it does not happen for “make.sh downloadsrc” for example.
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262...
The new mount namespace means that we will no longer see any bind-mounts in the host system and we no longer need to umount anything ourselves which is where we occasionally wiped the entire hard drive of the host system. When the last process exits, the namespace is being cleaned up and everything is being umounted.
The function that prepares the build environment has been almost entirely rewritten:
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262...
It used to mount parts of the host system into the build environment which are needed to run anything. Those were /dev, /proc, /sys, etc…
Instead, it now creates a new /dev mount point and creates a minimal amount of device nodes and symlinks. That way, we detach from the host system and no longer allow the build system access to the host’s filesystem and block devices. We also bind-mount the sources in read-only mode now, so that the build system cannot change anything in the source tree. On top of that, cache is read-only, too. ccache and the log directory are the only places that are writable.
We mount a separate /tmp directory.
- When we then build a package, we create more namespaces for each package. These isolate each build process from each other.
Mostly, this is to detach from the host system. A new UTS namespace allows to change the hostname in the build system without affecting the host and so on. We do the same thing with a new time namespace.
We do however create a new PID namespace which means that the build system no longer will see any processes running on the host system. That requires to mount a new instance of /proc with each package. This also has the effect that if the shell that we launched terminates (because the build is done) any background processes will be killed immediately.
Last, we clone the mount namespace that we have created before so that no build command can modify what we set up earlier.
- Since everything is now so decoupled, we gain a couple of new (maybe minor?) features:
It is now possible to run “make.sh shell” while a build is running. That does not happen a lot, but we can do this now :)
If the build crashes or the host system is being shut down while a build is running, there is nothing to clean up afterwards.
I have garnished this all with a lot of code cleanup and I suppose I might have introduced some new bugs here or there :)
This is probably mostly around a new implementation of the timer that updates the build time. It has been annoying me a lot that it takes a long time to walk through all packages that have been built before to finally get to a package that we want to rebuild. Mostly this was all help up by a call of “sleep 0.1”
Since bash does not really do any concurrency, I had to be creative and replaced the busy-loop with a background process that is launched whenever it is needed and which will “ping” the main make.sh script once a second. That way, we can just run as usual, but regularly get interrupted to update the runtime.
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262...
We now only fork one extra sub shell and we have to handle the timer events which is a lot cheaper as well as more straight-forward to code.
As there is no difference between the different stages any more (those stages that we inherited from LFS), I have merged them all into one.
Last but not least, I have create the option to build for multiple architectures on the same system. Since we can now mount the entire source tree into (many independent) build environments, we might as well… As discussed on the last call, this might not be the best option for ARM, but RISV-C builds at a decent speed even when emulated.
The only thing that I needed to do for this is to suffix the build and log directories which are now called build_${ARCH}, i.e. build_aarch64, build_x86_64, and so on. The packages/ directory is not changed yet, but that will have to happen as well. Most likely I want to merge this with the generated images, but I am not sure what to call this, yet. Happy to hear suggestions. result_x86_64? Just images_x86_64?
I have run a build and this seems to be working just fine on my Debian machine. I am writing to all of you to first of all let you know what I am up to; and secondly to ask to give this a go on your systems. I think it should run just fine, as all the tools that I require should be available everywhere. However, there might be some older kernels that might not support all of this, yet or any other problems I cannot think of yet. Please give me some feedback and send me all the bugs :)
I gave this a go but it didn't work.
Not sure if I should have run the ./make.sh clean command on the old version before I pulled the unshare branch into my clone of your repo.
Should I have started with a complete new clone of your repo? I might try that anyway just to see.
I created a completely new clone of you ipfire-2.x repor and then checked out the unshare branch to a branch called unshare in my local repo clone.
gettoolchain gave the same issue, except that this time the toolchain directory ended up completely empty.
downloadsrc had the same result.
clean had nothing to clean up as it was a fresh clone.
build then tried to build the toolchain and came up with this error, different from before.
./make.sh build chroot: failed to run command ‘env’: No such file or directory Full toolchain compilation stage1 [ FAIL ]
Jul 8 19:26:39: Building stage1 ====================================== Installing stage1 ... mkdir -pv /tools_x86_64/lib mkdir: cannot create directory '/tools_x86_64': File exists make: *** [stage1:50: /home/ahb/sandbox/ms/ipfire-2.x/log/stage1] Error 1
ERROR: Building stage1 [ FAIL ] Check /home/ahb/sandbox/ms/ipfire-2.x/log_x86_64/_build.toolchain.log for errors if applicable [ FAIL ]
so it wasn't as simple as doing a fresh git clone.
Regards,
Adolf.
So I ran ./make.sh gettoolchain first, as I usually would.
./make.sh gettoolchain b2sum: cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: No such file or directory cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: FAILED open or read b2sum: WARNING: 1 listed file could not be read
ipfire-2.29-toolchain-20240210-x86_64.tar.zst is present together with its b2 file.
Then ran ./make.sh downloadsrc
Previous version ends with
***Verifying BLAKE2 checksum all files BLAKE2 checksum match [ DONE]
after zstd has been checked.
New version stops at zstd entry.
./make.sh clean gave the message Cleaning Build directory... but was completed very quickly. Log and Build directories have not been cleaned out. The img and iso files are still present.
./make.sh build gave message
chroot: failed to run command ‘env’: No such file or directory
and then did a full toolchain compilation which failed with gcc but log is >9000 lines.
Regards, Adolf.
Thank you for listening to this brain-dump.
All the best, -Michael
On 3 Jul 2024, at 10:58, Michael Tremer michael.tremer@ipfire.org wrote:
Hello Adolf,
This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more.
I rebooted the machine and it is back up again.
-Michael
On 2 Jul 2024, at 15:42, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael and all,
I ran the arm builder with the 4.20.2 version of samba to test it out.
The build got to building gdb and then failed.
Interestingly, the nightly build of arm was successful with the same version of gdb.
The build log for gdb is attached. The actual error is at line 618.
Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message.
ssh bonnietwin@arm64-01.zrh.ipfire.org PTY allocation request failed on channel 0 Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64
The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied
Regards,
Adolf.
<_build.ipfire.gdb.log>
Hello Adolf,
Thank you for testing this.
There have indeed been plenty of problems there… I spent a lot of time on this today and hopefully fixed most of them.
I cannot build the toolchain on my machine and I am not sure why yet, but a build with the packaged toolchain runs through.
I have also spent some time on getting rid of the strip stage because it annoyed me how long it takes and creating the disk images as well as packages should be significantly faster now, too. I hope I didn’t introduce too many new bugs.
Please let me know if you have more success now.
Best, -Michael
On 8 Jul 2024, at 20:34, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 08/07/2024 21:15, Adolf Belka wrote:
Hi Michael,
On 08/07/2024 18:11, Michael Tremer wrote:
Hello,
I have been spending a lot of time on this problem, because it has been bothering me for a long time. I also saw an opportunity to make more changes to the build system.
Currently this is all a little bit WIP, but I hope that we can merge this into next as soon as the current update has been moved to master.
I am referring to this branch which is currently based on next: https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/u...
It makes use of the unshare command which creates new namespaces in Linux. That way, we can isolate the build system better from the host system and in case something goes wrong, there is less damage. We can also enforce some more rules…
So, what has changed?
- The make.sh script might re-execute itself into a new mount namespace when it is suitable. This happens for “make.sh build” and “make.sh shell”, but it does not happen for “make.sh downloadsrc” for example.
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262...
The new mount namespace means that we will no longer see any bind-mounts in the host system and we no longer need to umount anything ourselves which is where we occasionally wiped the entire hard drive of the host system. When the last process exits, the namespace is being cleaned up and everything is being umounted.
The function that prepares the build environment has been almost entirely rewritten:
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262...
It used to mount parts of the host system into the build environment which are needed to run anything. Those were /dev, /proc, /sys, etc…
Instead, it now creates a new /dev mount point and creates a minimal amount of device nodes and symlinks. That way, we detach from the host system and no longer allow the build system access to the host’s filesystem and block devices. We also bind-mount the sources in read-only mode now, so that the build system cannot change anything in the source tree. On top of that, cache is read-only, too. ccache and the log directory are the only places that are writable.
We mount a separate /tmp directory.
- When we then build a package, we create more namespaces for each package. These isolate each build process from each other.
Mostly, this is to detach from the host system. A new UTS namespace allows to change the hostname in the build system without affecting the host and so on. We do the same thing with a new time namespace.
We do however create a new PID namespace which means that the build system no longer will see any processes running on the host system. That requires to mount a new instance of /proc with each package. This also has the effect that if the shell that we launched terminates (because the build is done) any background processes will be killed immediately.
Last, we clone the mount namespace that we have created before so that no build command can modify what we set up earlier.
Since everything is now so decoupled, we gain a couple of new (maybe minor?) features:
It is now possible to run “make.sh shell” while a build is running. That does not happen a lot, but we can do this now :)
If the build crashes or the host system is being shut down while a build is running, there is nothing to clean up afterwards.
I have garnished this all with a lot of code cleanup and I suppose I might have introduced some new bugs here or there :)
This is probably mostly around a new implementation of the timer that updates the build time. It has been annoying me a lot that it takes a long time to walk through all packages that have been built before to finally get to a package that we want to rebuild. Mostly this was all help up by a call of “sleep 0.1”
Since bash does not really do any concurrency, I had to be creative and replaced the busy-loop with a background process that is launched whenever it is needed and which will “ping” the main make.sh script once a second. That way, we can just run as usual, but regularly get interrupted to update the runtime.
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262...
We now only fork one extra sub shell and we have to handle the timer events which is a lot cheaper as well as more straight-forward to code.
As there is no difference between the different stages any more (those stages that we inherited from LFS), I have merged them all into one.
Last but not least, I have create the option to build for multiple architectures on the same system. Since we can now mount the entire source tree into (many independent) build environments, we might as well… As discussed on the last call, this might not be the best option for ARM, but RISV-C builds at a decent speed even when emulated.
The only thing that I needed to do for this is to suffix the build and log directories which are now called build_${ARCH}, i.e. build_aarch64, build_x86_64, and so on. The packages/ directory is not changed yet, but that will have to happen as well. Most likely I want to merge this with the generated images, but I am not sure what to call this, yet. Happy to hear suggestions. result_x86_64? Just images_x86_64?
I have run a build and this seems to be working just fine on my Debian machine. I am writing to all of you to first of all let you know what I am up to; and secondly to ask to give this a go on your systems. I think it should run just fine, as all the tools that I require should be available everywhere. However, there might be some older kernels that might not support all of this, yet or any other problems I cannot think of yet. Please give me some feedback and send me all the bugs :)
I gave this a go but it didn't work.
Not sure if I should have run the ./make.sh clean command on the old version before I pulled the unshare branch into my clone of your repo.
Should I have started with a complete new clone of your repo? I might try that anyway just to see.
I created a completely new clone of you ipfire-2.x repor and then checked out the unshare branch to a branch called unshare in my local repo clone.
gettoolchain gave the same issue, except that this time the toolchain directory ended up completely empty.
downloadsrc had the same result.
clean had nothing to clean up as it was a fresh clone.
build then tried to build the toolchain and came up with this error, different from before.
./make.sh build chroot: failed to run command ‘env’: No such file or directory Full toolchain compilation stage1 [ FAIL ]
Jul 8 19:26:39: Building stage1 ====================================== Installing stage1 ... mkdir -pv /tools_x86_64/lib mkdir: cannot create directory '/tools_x86_64': File exists make: *** [stage1:50: /home/ahb/sandbox/ms/ipfire-2.x/log/stage1] Error 1ERROR: Building stage1 [ FAIL ] Check /home/ahb/sandbox/ms/ipfire-2.x/log_x86_64/_build.toolchain.log for errors if applicable [ FAIL ]
so it wasn't as simple as doing a fresh git clone.
Regards,
Adolf.
So I ran ./make.sh gettoolchain first, as I usually would.
./make.sh gettoolchain b2sum: cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: No such file or directory cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: FAILED open or read b2sum: WARNING: 1 listed file could not be read
ipfire-2.29-toolchain-20240210-x86_64.tar.zst is present together with its b2 file.
Then ran ./make.sh downloadsrc
Previous version ends with
***Verifying BLAKE2 checksum all files BLAKE2 checksum match [ DONE]
after zstd has been checked.
New version stops at zstd entry.
./make.sh clean gave the message Cleaning Build directory... but was completed very quickly. Log and Build directories have not been cleaned out. The img and iso files are still present.
./make.sh build gave message
chroot: failed to run command ‘env’: No such file or directory
and then did a full toolchain compilation which failed with gcc but log is >9000 lines.
Regards, Adolf.
Thank you for listening to this brain-dump.
All the best, -Michael
On 3 Jul 2024, at 10:58, Michael Tremer michael.tremer@ipfire.org wrote:
Hello Adolf,
This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more.
I rebooted the machine and it is back up again.
-Michael
On 2 Jul 2024, at 15:42, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael and all,
I ran the arm builder with the 4.20.2 version of samba to test it out.
The build got to building gdb and then failed.
Interestingly, the nightly build of arm was successful with the same version of gdb.
The build log for gdb is attached. The actual error is at line 618.
Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message.
ssh bonnietwin@arm64-01.zrh.ipfire.org PTY allocation request failed on channel 0 Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64
The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied
Regards,
Adolf.
<_build.ipfire.gdb.log>
Hello again,
I managed to (finally) build the toolchain with the updated system. So hopefully there should not be any more outstanding problems that I know of so far.
Best, -Michael
On 9 Jul 2024, at 22:29, Michael Tremer michael.tremer@ipfire.org wrote:
Hello Adolf,
Thank you for testing this.
There have indeed been plenty of problems there… I spent a lot of time on this today and hopefully fixed most of them.
I cannot build the toolchain on my machine and I am not sure why yet, but a build with the packaged toolchain runs through.
I have also spent some time on getting rid of the strip stage because it annoyed me how long it takes and creating the disk images as well as packages should be significantly faster now, too. I hope I didn’t introduce too many new bugs.
Please let me know if you have more success now.
Best, -Michael
On 8 Jul 2024, at 20:34, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 08/07/2024 21:15, Adolf Belka wrote:
Hi Michael,
On 08/07/2024 18:11, Michael Tremer wrote:
Hello,
I have been spending a lot of time on this problem, because it has been bothering me for a long time. I also saw an opportunity to make more changes to the build system.
Currently this is all a little bit WIP, but I hope that we can merge this into next as soon as the current update has been moved to master.
I am referring to this branch which is currently based on next: https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/u...
It makes use of the unshare command which creates new namespaces in Linux. That way, we can isolate the build system better from the host system and in case something goes wrong, there is less damage. We can also enforce some more rules…
So, what has changed?
- The make.sh script might re-execute itself into a new mount namespace when it is suitable. This happens for “make.sh build” and “make.sh shell”, but it does not happen for “make.sh downloadsrc” for example.
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262...
The new mount namespace means that we will no longer see any bind-mounts in the host system and we no longer need to umount anything ourselves which is where we occasionally wiped the entire hard drive of the host system. When the last process exits, the namespace is being cleaned up and everything is being umounted.
The function that prepares the build environment has been almost entirely rewritten:
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262...
It used to mount parts of the host system into the build environment which are needed to run anything. Those were /dev, /proc, /sys, etc…
Instead, it now creates a new /dev mount point and creates a minimal amount of device nodes and symlinks. That way, we detach from the host system and no longer allow the build system access to the host’s filesystem and block devices. We also bind-mount the sources in read-only mode now, so that the build system cannot change anything in the source tree. On top of that, cache is read-only, too. ccache and the log directory are the only places that are writable.
We mount a separate /tmp directory.
- When we then build a package, we create more namespaces for each package. These isolate each build process from each other.
Mostly, this is to detach from the host system. A new UTS namespace allows to change the hostname in the build system without affecting the host and so on. We do the same thing with a new time namespace.
We do however create a new PID namespace which means that the build system no longer will see any processes running on the host system. That requires to mount a new instance of /proc with each package. This also has the effect that if the shell that we launched terminates (because the build is done) any background processes will be killed immediately.
Last, we clone the mount namespace that we have created before so that no build command can modify what we set up earlier.
Since everything is now so decoupled, we gain a couple of new (maybe minor?) features:
It is now possible to run “make.sh shell” while a build is running. That does not happen a lot, but we can do this now :)
If the build crashes or the host system is being shut down while a build is running, there is nothing to clean up afterwards.
I have garnished this all with a lot of code cleanup and I suppose I might have introduced some new bugs here or there :)
This is probably mostly around a new implementation of the timer that updates the build time. It has been annoying me a lot that it takes a long time to walk through all packages that have been built before to finally get to a package that we want to rebuild. Mostly this was all help up by a call of “sleep 0.1”
Since bash does not really do any concurrency, I had to be creative and replaced the busy-loop with a background process that is launched whenever it is needed and which will “ping” the main make.sh script once a second. That way, we can just run as usual, but regularly get interrupted to update the runtime.
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262...
We now only fork one extra sub shell and we have to handle the timer events which is a lot cheaper as well as more straight-forward to code.
As there is no difference between the different stages any more (those stages that we inherited from LFS), I have merged them all into one.
Last but not least, I have create the option to build for multiple architectures on the same system. Since we can now mount the entire source tree into (many independent) build environments, we might as well… As discussed on the last call, this might not be the best option for ARM, but RISV-C builds at a decent speed even when emulated.
The only thing that I needed to do for this is to suffix the build and log directories which are now called build_${ARCH}, i.e. build_aarch64, build_x86_64, and so on. The packages/ directory is not changed yet, but that will have to happen as well. Most likely I want to merge this with the generated images, but I am not sure what to call this, yet. Happy to hear suggestions. result_x86_64? Just images_x86_64?
I have run a build and this seems to be working just fine on my Debian machine. I am writing to all of you to first of all let you know what I am up to; and secondly to ask to give this a go on your systems. I think it should run just fine, as all the tools that I require should be available everywhere. However, there might be some older kernels that might not support all of this, yet or any other problems I cannot think of yet. Please give me some feedback and send me all the bugs :)
I gave this a go but it didn't work.
Not sure if I should have run the ./make.sh clean command on the old version before I pulled the unshare branch into my clone of your repo.
Should I have started with a complete new clone of your repo? I might try that anyway just to see.
I created a completely new clone of you ipfire-2.x repor and then checked out the unshare branch to a branch called unshare in my local repo clone.
gettoolchain gave the same issue, except that this time the toolchain directory ended up completely empty.
downloadsrc had the same result.
clean had nothing to clean up as it was a fresh clone.
build then tried to build the toolchain and came up with this error, different from before.
./make.sh build chroot: failed to run command ‘env’: No such file or directory Full toolchain compilation stage1 [ FAIL ]
Jul 8 19:26:39: Building stage1 ====================================== Installing stage1 ... mkdir -pv /tools_x86_64/lib mkdir: cannot create directory '/tools_x86_64': File exists make: *** [stage1:50: /home/ahb/sandbox/ms/ipfire-2.x/log/stage1] Error 1
ERROR: Building stage1 [ FAIL ] Check /home/ahb/sandbox/ms/ipfire-2.x/log_x86_64/_build.toolchain.log for errors if applicable [ FAIL ]
so it wasn't as simple as doing a fresh git clone.
Regards,
Adolf.
So I ran ./make.sh gettoolchain first, as I usually would.
./make.sh gettoolchain b2sum: cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: No such file or directory cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: FAILED open or read b2sum: WARNING: 1 listed file could not be read
ipfire-2.29-toolchain-20240210-x86_64.tar.zst is present together with its b2 file.
Then ran ./make.sh downloadsrc
Previous version ends with
***Verifying BLAKE2 checksum all files BLAKE2 checksum match [ DONE]
after zstd has been checked.
New version stops at zstd entry.
./make.sh clean gave the message Cleaning Build directory... but was completed very quickly. Log and Build directories have not been cleaned out. The img and iso files are still present.
./make.sh build gave message
chroot: failed to run command ‘env’: No such file or directory
and then did a full toolchain compilation which failed with gcc but log is >9000 lines.
Regards, Adolf.
Thank you for listening to this brain-dump.
All the best, -Michael
On 3 Jul 2024, at 10:58, Michael Tremer michael.tremer@ipfire.org wrote:
Hello Adolf,
This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more.
I rebooted the machine and it is back up again.
-Michael
On 2 Jul 2024, at 15:42, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael and all,
I ran the arm builder with the 4.20.2 version of samba to test it out.
The build got to building gdb and then failed.
Interestingly, the nightly build of arm was successful with the same version of gdb.
The build log for gdb is attached. The actual error is at line 618.
Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message.
ssh bonnietwin@arm64-01.zrh.ipfire.org PTY allocation request failed on channel 0 Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64
The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied
Regards,
Adolf.
<_build.ipfire.gdb.log>
Hi Michael,
On 10/07/2024 11:57, Michael Tremer wrote:
Hello again,
I managed to (finally) build the toolchain with the updated system. So hopefully there should not be any more outstanding problems that I know of so far.
I just did a git pull on your repo to my clone.
Ran ./make.sh gettoolchain and it successfully downloaded the toolchain.
Ran ./make.sh downloadsrc and it successfully tested everything.
Ran ./make.sh clean and build and log directories were cleared out and removed. As far as I can tell it was successful.
Currently running ./make.sh build. So up to this point everything going well. Will let you know how it goes.
Regards, Adolf.
Best, -Michael
On 9 Jul 2024, at 22:29, Michael Tremer michael.tremer@ipfire.org wrote:
Hello Adolf,
Thank you for testing this.
There have indeed been plenty of problems there… I spent a lot of time on this today and hopefully fixed most of them.
I cannot build the toolchain on my machine and I am not sure why yet, but a build with the packaged toolchain runs through.
I have also spent some time on getting rid of the strip stage because it annoyed me how long it takes and creating the disk images as well as packages should be significantly faster now, too. I hope I didn’t introduce too many new bugs.
Please let me know if you have more success now.
Best, -Michael
On 8 Jul 2024, at 20:34, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 08/07/2024 21:15, Adolf Belka wrote:
Hi Michael,
On 08/07/2024 18:11, Michael Tremer wrote:
Hello,
I have been spending a lot of time on this problem, because it has been bothering me for a long time. I also saw an opportunity to make more changes to the build system.
Currently this is all a little bit WIP, but I hope that we can merge this into next as soon as the current update has been moved to master.
I am referring to this branch which is currently based on next: https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/u...
It makes use of the unshare command which creates new namespaces in Linux. That way, we can isolate the build system better from the host system and in case something goes wrong, there is less damage. We can also enforce some more rules…
So, what has changed?
- The make.sh script might re-execute itself into a new mount namespace when it is suitable. This happens for “make.sh build” and “make.sh shell”, but it does not happen for “make.sh downloadsrc” for example.
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262...
The new mount namespace means that we will no longer see any bind-mounts in the host system and we no longer need to umount anything ourselves which is where we occasionally wiped the entire hard drive of the host system. When the last process exits, the namespace is being cleaned up and everything is being umounted.
The function that prepares the build environment has been almost entirely rewritten:
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262...
It used to mount parts of the host system into the build environment which are needed to run anything. Those were /dev, /proc, /sys, etc…
Instead, it now creates a new /dev mount point and creates a minimal amount of device nodes and symlinks. That way, we detach from the host system and no longer allow the build system access to the host’s filesystem and block devices. We also bind-mount the sources in read-only mode now, so that the build system cannot change anything in the source tree. On top of that, cache is read-only, too. ccache and the log directory are the only places that are writable.
We mount a separate /tmp directory.
- When we then build a package, we create more namespaces for each package. These isolate each build process from each other.
Mostly, this is to detach from the host system. A new UTS namespace allows to change the hostname in the build system without affecting the host and so on. We do the same thing with a new time namespace.
We do however create a new PID namespace which means that the build system no longer will see any processes running on the host system. That requires to mount a new instance of /proc with each package. This also has the effect that if the shell that we launched terminates (because the build is done) any background processes will be killed immediately.
Last, we clone the mount namespace that we have created before so that no build command can modify what we set up earlier.
Since everything is now so decoupled, we gain a couple of new (maybe minor?) features:
It is now possible to run “make.sh shell” while a build is running. That does not happen a lot, but we can do this now :)
If the build crashes or the host system is being shut down while a build is running, there is nothing to clean up afterwards.
I have garnished this all with a lot of code cleanup and I suppose I might have introduced some new bugs here or there :)
This is probably mostly around a new implementation of the timer that updates the build time. It has been annoying me a lot that it takes a long time to walk through all packages that have been built before to finally get to a package that we want to rebuild. Mostly this was all help up by a call of “sleep 0.1”
Since bash does not really do any concurrency, I had to be creative and replaced the busy-loop with a background process that is launched whenever it is needed and which will “ping” the main make.sh script once a second. That way, we can just run as usual, but regularly get interrupted to update the runtime.
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262...
We now only fork one extra sub shell and we have to handle the timer events which is a lot cheaper as well as more straight-forward to code.
As there is no difference between the different stages any more (those stages that we inherited from LFS), I have merged them all into one.
Last but not least, I have create the option to build for multiple architectures on the same system. Since we can now mount the entire source tree into (many independent) build environments, we might as well… As discussed on the last call, this might not be the best option for ARM, but RISV-C builds at a decent speed even when emulated.
The only thing that I needed to do for this is to suffix the build and log directories which are now called build_${ARCH}, i.e. build_aarch64, build_x86_64, and so on. The packages/ directory is not changed yet, but that will have to happen as well. Most likely I want to merge this with the generated images, but I am not sure what to call this, yet. Happy to hear suggestions. result_x86_64? Just images_x86_64?
I have run a build and this seems to be working just fine on my Debian machine. I am writing to all of you to first of all let you know what I am up to; and secondly to ask to give this a go on your systems. I think it should run just fine, as all the tools that I require should be available everywhere. However, there might be some older kernels that might not support all of this, yet or any other problems I cannot think of yet. Please give me some feedback and send me all the bugs :)
I gave this a go but it didn't work.
Not sure if I should have run the ./make.sh clean command on the old version before I pulled the unshare branch into my clone of your repo.
Should I have started with a complete new clone of your repo? I might try that anyway just to see.
I created a completely new clone of you ipfire-2.x repor and then checked out the unshare branch to a branch called unshare in my local repo clone.
gettoolchain gave the same issue, except that this time the toolchain directory ended up completely empty.
downloadsrc had the same result.
clean had nothing to clean up as it was a fresh clone.
build then tried to build the toolchain and came up with this error, different from before.
./make.sh build chroot: failed to run command ‘env’: No such file or directory Full toolchain compilation stage1 [ FAIL ]
Jul 8 19:26:39: Building stage1 ====================================== Installing stage1 ... mkdir -pv /tools_x86_64/lib mkdir: cannot create directory '/tools_x86_64': File exists make: *** [stage1:50: /home/ahb/sandbox/ms/ipfire-2.x/log/stage1] Error 1ERROR: Building stage1 [ FAIL ] Check /home/ahb/sandbox/ms/ipfire-2.x/log_x86_64/_build.toolchain.log for errors if applicable [ FAIL ]
so it wasn't as simple as doing a fresh git clone.
Regards,
Adolf.
So I ran ./make.sh gettoolchain first, as I usually would.
./make.sh gettoolchain b2sum: cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: No such file or directory cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: FAILED open or read b2sum: WARNING: 1 listed file could not be read
ipfire-2.29-toolchain-20240210-x86_64.tar.zst is present together with its b2 file.
Then ran ./make.sh downloadsrc
Previous version ends with
***Verifying BLAKE2 checksum all files BLAKE2 checksum match [ DONE]
after zstd has been checked.
New version stops at zstd entry.
./make.sh clean gave the message Cleaning Build directory... but was completed very quickly. Log and Build directories have not been cleaned out. The img and iso files are still present.
./make.sh build gave message
chroot: failed to run command ‘env’: No such file or directory
and then did a full toolchain compilation which failed with gcc but log is >9000 lines.
Regards, Adolf.
Thank you for listening to this brain-dump.
All the best, -Michael
On 3 Jul 2024, at 10:58, Michael Tremer michael.tremer@ipfire.org wrote:
Hello Adolf,
This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more.
I rebooted the machine and it is back up again.
-Michael
> On 2 Jul 2024, at 15:42, Adolf Belka adolf.belka@ipfire.org wrote: > > Hi Michael and all, > > > I ran the arm builder with the 4.20.2 version of samba to test it out. > > The build got to building gdb and then failed. > > Interestingly, the nightly build of arm was successful with the same version of gdb. > > The build log for gdb is attached. The actual error is at line 618. > > Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message. > > ------------------------------------------ > > ssh bonnietwin@arm64-01.zrh.ipfire.org > PTY allocation request failed on channel 0 > Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64 > > The programs included with the Debian GNU/Linux system are free software; > the exact distribution terms for each program are described in the > individual files in /usr/share/doc/*/copyright. > > Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent > permitted by applicable law. > /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied > > ------------------------------------------ > > Regards, > > Adolf. > > <_build.ipfire.gdb.log>
Hi Michael,
On 10/07/2024 12:33, Adolf Belka wrote:
Hi Michael,
On 10/07/2024 11:57, Michael Tremer wrote:
Hello again,
I managed to (finally) build the toolchain with the updated system. So hopefully there should not be any more outstanding problems that I know of so far.
I just did a git pull on your repo to my clone.
Ran ./make.sh gettoolchain and it successfully downloaded the toolchain.
Ran ./make.sh downloadsrc and it successfully tested everything.
Ran ./make.sh clean and build and log directories were cleared out and removed. As far as I can tell it was successful.
Currently running ./make.sh build. So up to this point everything going well. Will let you know how it goes.
It has got to building popt. In the normal build system this takes around 3 secs. Currently in the new build it is at nearly 2 hours. Even with an empty cache, that seems a long build time for popt, unless I am being too optimistic.
I will let it keep going.
One thing I found. I am running the new build system while I have been running some package updates with the old system with its mount points. The two have each run without any impact on the other.
Regards,
Adolf.
Regards, Adolf.
Best, -Michael
On 9 Jul 2024, at 22:29, Michael Tremer michael.tremer@ipfire.org wrote:
Hello Adolf,
Thank you for testing this.
There have indeed been plenty of problems there… I spent a lot of time on this today and hopefully fixed most of them.
I cannot build the toolchain on my machine and I am not sure why yet, but a build with the packaged toolchain runs through.
I have also spent some time on getting rid of the strip stage because it annoyed me how long it takes and creating the disk images as well as packages should be significantly faster now, too. I hope I didn’t introduce too many new bugs.
Please let me know if you have more success now.
Best, -Michael
On 8 Jul 2024, at 20:34, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 08/07/2024 21:15, Adolf Belka wrote:
Hi Michael,
On 08/07/2024 18:11, Michael Tremer wrote:
Hello,
I have been spending a lot of time on this problem, because it has been bothering me for a long time. I also saw an opportunity to make more changes to the build system.
Currently this is all a little bit WIP, but I hope that we can merge this into next as soon as the current update has been moved to master.
I am referring to this branch which is currently based on next: https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/u...
It makes use of the unshare command which creates new namespaces in Linux. That way, we can isolate the build system better from the host system and in case something goes wrong, there is less damage. We can also enforce some more rules…
So, what has changed?
- The make.sh script might re-execute itself into a new mount namespace when it is suitable. This happens for “make.sh build” and “make.sh shell”, but it does not happen for “make.sh downloadsrc” for example.
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262...
The new mount namespace means that we will no longer see any bind-mounts in the host system and we no longer need to umount anything ourselves which is where we occasionally wiped the entire hard drive of the host system. When the last process exits, the namespace is being cleaned up and everything is being umounted.
The function that prepares the build environment has been almost entirely rewritten:
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262...
It used to mount parts of the host system into the build environment which are needed to run anything. Those were /dev, /proc, /sys, etc…
Instead, it now creates a new /dev mount point and creates a minimal amount of device nodes and symlinks. That way, we detach from the host system and no longer allow the build system access to the host’s filesystem and block devices. We also bind-mount the sources in read-only mode now, so that the build system cannot change anything in the source tree. On top of that, cache is read-only, too. ccache and the log directory are the only places that are writable.
We mount a separate /tmp directory.
- When we then build a package, we create more namespaces for each package. These isolate each build process from each other.
Mostly, this is to detach from the host system. A new UTS namespace allows to change the hostname in the build system without affecting the host and so on. We do the same thing with a new time namespace.
We do however create a new PID namespace which means that the build system no longer will see any processes running on the host system. That requires to mount a new instance of /proc with each package. This also has the effect that if the shell that we launched terminates (because the build is done) any background processes will be killed immediately.
Last, we clone the mount namespace that we have created before so that no build command can modify what we set up earlier.
- Since everything is now so decoupled, we gain a couple of new (maybe minor?) features:
It is now possible to run “make.sh shell” while a build is running. That does not happen a lot, but we can do this now :)
If the build crashes or the host system is being shut down while a build is running, there is nothing to clean up afterwards.
I have garnished this all with a lot of code cleanup and I suppose I might have introduced some new bugs here or there :)
This is probably mostly around a new implementation of the timer that updates the build time. It has been annoying me a lot that it takes a long time to walk through all packages that have been built before to finally get to a package that we want to rebuild. Mostly this was all help up by a call of “sleep 0.1”
Since bash does not really do any concurrency, I had to be creative and replaced the busy-loop with a background process that is launched whenever it is needed and which will “ping” the main make.sh script once a second. That way, we can just run as usual, but regularly get interrupted to update the runtime.
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262...
We now only fork one extra sub shell and we have to handle the timer events which is a lot cheaper as well as more straight-forward to code.
As there is no difference between the different stages any more (those stages that we inherited from LFS), I have merged them all into one.
Last but not least, I have create the option to build for multiple architectures on the same system. Since we can now mount the entire source tree into (many independent) build environments, we might as well… As discussed on the last call, this might not be the best option for ARM, but RISV-C builds at a decent speed even when emulated.
The only thing that I needed to do for this is to suffix the build and log directories which are now called build_${ARCH}, i.e. build_aarch64, build_x86_64, and so on. The packages/ directory is not changed yet, but that will have to happen as well. Most likely I want to merge this with the generated images, but I am not sure what to call this, yet. Happy to hear suggestions. result_x86_64? Just images_x86_64?
I have run a build and this seems to be working just fine on my Debian machine. I am writing to all of you to first of all let you know what I am up to; and secondly to ask to give this a go on your systems. I think it should run just fine, as all the tools that I require should be available everywhere. However, there might be some older kernels that might not support all of this, yet or any other problems I cannot think of yet. Please give me some feedback and send me all the bugs :)
I gave this a go but it didn't work.
Not sure if I should have run the ./make.sh clean command on the old version before I pulled the unshare branch into my clone of your repo.
Should I have started with a complete new clone of your repo? I might try that anyway just to see.
I created a completely new clone of you ipfire-2.x repor and then checked out the unshare branch to a branch called unshare in my local repo clone.
gettoolchain gave the same issue, except that this time the toolchain directory ended up completely empty.
downloadsrc had the same result.
clean had nothing to clean up as it was a fresh clone.
build then tried to build the toolchain and came up with this error, different from before.
./make.sh build chroot: failed to run command ‘env’: No such file or directory Full toolchain compilation stage1 [ FAIL ]
Jul 8 19:26:39: Building stage1 ====================================== Installing stage1 ... mkdir -pv /tools_x86_64/lib mkdir: cannot create directory '/tools_x86_64': File exists make: *** [stage1:50: /home/ahb/sandbox/ms/ipfire-2.x/log/stage1] Error 1
ERROR: Building stage1 [ FAIL ] Check /home/ahb/sandbox/ms/ipfire-2.x/log_x86_64/_build.toolchain.log for errors if applicable [ FAIL ]
so it wasn't as simple as doing a fresh git clone.
Regards,
Adolf.
So I ran ./make.sh gettoolchain first, as I usually would.
./make.sh gettoolchain b2sum: cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: No such file or directory cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: FAILED open or read b2sum: WARNING: 1 listed file could not be read
ipfire-2.29-toolchain-20240210-x86_64.tar.zst is present together with its b2 file.
Then ran ./make.sh downloadsrc
Previous version ends with
***Verifying BLAKE2 checksum all files BLAKE2 checksum match [ DONE]
after zstd has been checked.
New version stops at zstd entry.
./make.sh clean gave the message Cleaning Build directory... but was completed very quickly. Log and Build directories have not been cleaned out. The img and iso files are still present.
./make.sh build gave message
chroot: failed to run command ‘env’: No such file or directory
and then did a full toolchain compilation which failed with gcc but log is >9000 lines.
Regards, Adolf.
Thank you for listening to this brain-dump.
All the best, -Michael
> On 3 Jul 2024, at 10:58, Michael Tremer michael.tremer@ipfire.org wrote: > > Hello Adolf, > > This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more. > > I rebooted the machine and it is back up again. > > -Michael > >> On 2 Jul 2024, at 15:42, Adolf Belka adolf.belka@ipfire.org wrote: >> >> Hi Michael and all, >> >> >> I ran the arm builder with the 4.20.2 version of samba to test it out. >> >> The build got to building gdb and then failed. >> >> Interestingly, the nightly build of arm was successful with the same version of gdb. >> >> The build log for gdb is attached. The actual error is at line 618. >> >> Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message. >> >> ------------------------------------------ >> >> ssh bonnietwin@arm64-01.zrh.ipfire.org >> PTY allocation request failed on channel 0 >> Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64 >> >> The programs included with the Debian GNU/Linux system are free software; >> the exact distribution terms for each program are described in the >> individual files in /usr/share/doc/*/copyright. >> >> Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent >> permitted by applicable law. >> /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied >> >> ------------------------------------------ >> >> Regards, >> >> Adolf. >> >> <_build.ipfire.gdb.log>
Hi Michael,
On 10/07/2024 14:59, Adolf Belka wrote:
Hi Michael,
On 10/07/2024 12:33, Adolf Belka wrote:
Hi Michael,
On 10/07/2024 11:57, Michael Tremer wrote:
Hello again,
I managed to (finally) build the toolchain with the updated system. So hopefully there should not be any more outstanding problems that I know of so far.
I just did a git pull on your repo to my clone.
Ran ./make.sh gettoolchain and it successfully downloaded the toolchain.
Ran ./make.sh downloadsrc and it successfully tested everything.
Ran ./make.sh clean and build and log directories were cleared out and removed. As far as I can tell it was successful.
Currently running ./make.sh build. So up to this point everything going well. Will let you know how it goes.
It has got to building popt. In the normal build system this takes around 3 secs. Currently in the new build it is at nearly 2 hours. Even with an empty cache, that seems a long build time for popt, unless I am being too optimistic.
I will let it keep going.
I just had a look at the log file and it looks like it completed popt but then is stuck on trying to leave the directory /usr/src/lfs. Here is the output from the log, nothing new is getting written to the log.
make[3]: Leaving directory '/usr/src/popt-1.19/po' Making install in tests make[3]: Entering directory '/usr/src/popt-1.19/tests' make[4]: Entering directory '/usr/src/popt-1.19/tests' make[4]: Nothing to be done for 'install-exec-am'. make[4]: Nothing to be done for 'install-data-am'. make[4]: Leaving directory '/usr/src/popt-1.19/tests' make[3]: Leaving directory '/usr/src/popt-1.19/tests' make[3]: Entering directory '/usr/src/popt-1.19' make[4]: Entering directory '/usr/src/popt-1.19' make[4]: Nothing to be done for 'install-exec-am'. /bin/mkdir -p '/usr/share/man/man3' /usr/bin/install -c -m 644 popt.3 '/usr/share/man/man3' /bin/mkdir -p '/usr/lib/pkgconfig' /usr/bin/install -c -m 644 popt.pc '/usr/lib/pkgconfig' make[4]: Leaving directory '/usr/src/popt-1.19' make[3]: Leaving directory '/usr/src/popt-1.19' make[2]: Leaving directory '/usr/src/popt-1.19' make[1]: Leaving directory '/usr/src/popt-1.19' Updating linker cache... Install done; saving file list to /usr/src/log/popt-1.19 ...
make: Leaving directory '/usr/src/lfs'
Regards,
Adolf.
One thing I found. I am running the new build system while I have been running some package updates with the old system with its mount points. The two have each run without any impact on the other.
Regards,
Adolf.
Regards, Adolf.
Best, -Michael
On 9 Jul 2024, at 22:29, Michael Tremer michael.tremer@ipfire.org wrote:
Hello Adolf,
Thank you for testing this.
There have indeed been plenty of problems there… I spent a lot of time on this today and hopefully fixed most of them.
I cannot build the toolchain on my machine and I am not sure why yet, but a build with the packaged toolchain runs through.
I have also spent some time on getting rid of the strip stage because it annoyed me how long it takes and creating the disk images as well as packages should be significantly faster now, too. I hope I didn’t introduce too many new bugs.
Please let me know if you have more success now.
Best, -Michael
On 8 Jul 2024, at 20:34, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 08/07/2024 21:15, Adolf Belka wrote:
Hi Michael,
On 08/07/2024 18:11, Michael Tremer wrote: > Hello, > > I have been spending a lot of time on this problem, because it has been bothering me for a long time. I also saw an opportunity to make more changes to the build system. > > Currently this is all a little bit WIP, but I hope that we can merge this into next as soon as the current update has been moved to master. > > I am referring to this branch which is currently based on next: https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/u... > > It makes use of the unshare command which creates new namespaces in Linux. That way, we can isolate the build system better from the host system and in case something goes wrong, there is less damage. We can also enforce some more rules… > > So, what has changed? > > * The make.sh script might re-execute itself into a new mount namespace when it is suitable. This happens for “make.sh build” and “make.sh shell”, but it does not happen for “make.sh downloadsrc” for example. > > https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... > https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... > > * The new mount namespace means that we will no longer see any bind-mounts in the host system and we no longer need to umount anything ourselves which is where we occasionally wiped the entire hard drive of the host system. When the last process exits, the namespace is being cleaned up and everything is being umounted. > > * The function that prepares the build environment has been almost entirely rewritten: > > https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... > > It used to mount parts of the host system into the build environment which are needed to run anything. Those were /dev, /proc, /sys, etc… > > Instead, it now creates a new /dev mount point and creates a minimal amount of device nodes and symlinks. That way, we detach from the host system and no longer allow the build system access to the host’s filesystem and block devices. We also bind-mount the sources in read-only mode now, so that the build system cannot change anything in the source tree. On top of that, cache is read-only, too. ccache and the log directory are the only places that are writable. > > We mount a separate /tmp directory. > > * When we then build a package, we create more namespaces for each package. These isolate each build process from each other. > > Mostly, this is to detach from the host system. A new UTS namespace allows to change the hostname in the build system without affecting the host and so on. We do the same thing with a new time namespace. > > We do however create a new PID namespace which means that the build system no longer will see any processes running on the host system. That requires to mount a new instance of /proc with each package. This also has the effect that if the shell that we launched terminates (because the build is done) any background processes will be killed immediately. > > Last, we clone the mount namespace that we have created before so that no build command can modify what we set up earlier. > > * Since everything is now so decoupled, we gain a couple of new (maybe minor?) features: > > It is now possible to run “make.sh shell” while a build is running. That does not happen a lot, but we can do this now :) > > If the build crashes or the host system is being shut down while a build is running, there is nothing to clean up afterwards. > > * I have garnished this all with a lot of code cleanup and I suppose I might have introduced some new bugs here or there :) > > * This is probably mostly around a new implementation of the timer that updates the build time. It has been annoying me a lot that it takes a long time to walk through all packages that have been built before to finally get to a package that we want to rebuild. Mostly this was all help up by a call of “sleep 0.1” > > Since bash does not really do any concurrency, I had to be creative and replaced the busy-loop with a background process that is launched whenever it is needed and which will “ping” the main make.sh script once a second. That way, we can just run as usual, but regularly get interrupted to update the runtime. > > https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... > https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... > > We now only fork one extra sub shell and we have to handle the timer events which is a lot cheaper as well as more straight-forward to code. > > * As there is no difference between the different stages any more (those stages that we inherited from LFS), I have merged them all into one. > > * Last but not least, I have create the option to build for multiple architectures on the same system. Since we can now mount the entire source tree into (many independent) build environments, we might as well… As discussed on the last call, this might not be the best option for ARM, but RISV-C builds at a decent speed even when emulated. > > The only thing that I needed to do for this is to suffix the build and log directories which are now called build_${ARCH}, i.e. build_aarch64, build_x86_64, and so on. The packages/ directory is not changed yet, but that will have to happen as well. Most likely I want to merge this with the generated images, but I am not sure what to call this, yet. Happy to hear suggestions. result_x86_64? Just images_x86_64? > > --- > > I have run a build and this seems to be working just fine on my Debian machine. I am writing to all of you to first of all let you know what I am up to; and secondly to ask to give this a go on your systems. I think it should run just fine, as all the tools that I require should be available everywhere. However, there might be some older kernels that might not support all of this, yet or any other problems I cannot think of yet. Please give me some feedback and send me all the bugs :) I gave this a go but it didn't work.
Not sure if I should have run the ./make.sh clean command on the old version before I pulled the unshare branch into my clone of your repo.
Should I have started with a complete new clone of your repo? I might try that anyway just to see.
I created a completely new clone of you ipfire-2.x repor and then checked out the unshare branch to a branch called unshare in my local repo clone.
gettoolchain gave the same issue, except that this time the toolchain directory ended up completely empty.
downloadsrc had the same result.
clean had nothing to clean up as it was a fresh clone.
build then tried to build the toolchain and came up with this error, different from before.
./make.sh build chroot: failed to run command ‘env’: No such file or directory Full toolchain compilation stage1 [ FAIL ]
Jul 8 19:26:39: Building stage1 ====================================== Installing stage1 ... mkdir -pv /tools_x86_64/lib mkdir: cannot create directory '/tools_x86_64': File exists make: *** [stage1:50: /home/ahb/sandbox/ms/ipfire-2.x/log/stage1] Error 1
ERROR: Building stage1 [ FAIL ] Check /home/ahb/sandbox/ms/ipfire-2.x/log_x86_64/_build.toolchain.log for errors if applicable [ FAIL ]
so it wasn't as simple as doing a fresh git clone.
Regards,
Adolf.
So I ran ./make.sh gettoolchain first, as I usually would.
./make.sh gettoolchain b2sum: cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: No such file or directory cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: FAILED open or read b2sum: WARNING: 1 listed file could not be read
ipfire-2.29-toolchain-20240210-x86_64.tar.zst is present together with its b2 file.
Then ran ./make.sh downloadsrc
Previous version ends with
***Verifying BLAKE2 checksum all files BLAKE2 checksum match [ DONE]
after zstd has been checked.
New version stops at zstd entry.
./make.sh clean gave the message Cleaning Build directory... but was completed very quickly. Log and Build directories have not been cleaned out. The img and iso files are still present.
./make.sh build gave message
chroot: failed to run command ‘env’: No such file or directory
and then did a full toolchain compilation which failed with gcc but log is >9000 lines.
Regards, Adolf.
> > Thank you for listening to this brain-dump. > > All the best, > -Michael > >> On 3 Jul 2024, at 10:58, Michael Tremer michael.tremer@ipfire.org wrote: >> >> Hello Adolf, >> >> This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more. >> >> I rebooted the machine and it is back up again. >> >> -Michael >> >>> On 2 Jul 2024, at 15:42, Adolf Belka adolf.belka@ipfire.org wrote: >>> >>> Hi Michael and all, >>> >>> >>> I ran the arm builder with the 4.20.2 version of samba to test it out. >>> >>> The build got to building gdb and then failed. >>> >>> Interestingly, the nightly build of arm was successful with the same version of gdb. >>> >>> The build log for gdb is attached. The actual error is at line 618. >>> >>> Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message. >>> >>> ------------------------------------------ >>> >>> ssh bonnietwin@arm64-01.zrh.ipfire.org >>> PTY allocation request failed on channel 0 >>> Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64 >>> >>> The programs included with the Debian GNU/Linux system are free software; >>> the exact distribution terms for each program are described in the >>> individual files in /usr/share/doc/*/copyright. >>> >>> Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent >>> permitted by applicable law. >>> /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied >>> >>> ------------------------------------------ >>> >>> Regards, >>> >>> Adolf. >>> >>> <_build.ipfire.gdb.log>
Hi Michael,
On 10/07/2024 15:05, Adolf Belka wrote:
Hi Michael,
On 10/07/2024 14:59, Adolf Belka wrote:
Hi Michael,
On 10/07/2024 12:33, Adolf Belka wrote:
Hi Michael,
On 10/07/2024 11:57, Michael Tremer wrote:
Hello again,
I managed to (finally) build the toolchain with the updated system. So hopefully there should not be any more outstanding problems that I know of so far.
I just did a git pull on your repo to my clone.
Ran ./make.sh gettoolchain and it successfully downloaded the toolchain.
Ran ./make.sh downloadsrc and it successfully tested everything.
Ran ./make.sh clean and build and log directories were cleared out and removed. As far as I can tell it was successful.
Currently running ./make.sh build. So up to this point everything going well. Will let you know how it goes.
It has got to building popt. In the normal build system this takes around 3 secs. Currently in the new build it is at nearly 2 hours. Even with an empty cache, that seems a long build time for popt, unless I am being too optimistic.
I will let it keep going.
I just had a look at the log file and it looks like it completed popt but then is stuck on trying to leave the directory /usr/src/lfs. Here is the output from the log, nothing new is getting written to the log.
make[3]: Leaving directory '/usr/src/popt-1.19/po' Making install in tests make[3]: Entering directory '/usr/src/popt-1.19/tests' make[4]: Entering directory '/usr/src/popt-1.19/tests' make[4]: Nothing to be done for 'install-exec-am'. make[4]: Nothing to be done for 'install-data-am'. make[4]: Leaving directory '/usr/src/popt-1.19/tests' make[3]: Leaving directory '/usr/src/popt-1.19/tests' make[3]: Entering directory '/usr/src/popt-1.19' make[4]: Entering directory '/usr/src/popt-1.19' make[4]: Nothing to be done for 'install-exec-am'. /bin/mkdir -p '/usr/share/man/man3' /usr/bin/install -c -m 644 popt.3 '/usr/share/man/man3' /bin/mkdir -p '/usr/lib/pkgconfig' /usr/bin/install -c -m 644 popt.pc '/usr/lib/pkgconfig' make[4]: Leaving directory '/usr/src/popt-1.19' make[3]: Leaving directory '/usr/src/popt-1.19' make[2]: Leaving directory '/usr/src/popt-1.19' make[1]: Leaving directory '/usr/src/popt-1.19' Updating linker cache... Install done; saving file list to /usr/src/log/popt-1.19 ...
make: Leaving directory '/usr/src/lfs'
I stopped the build with Ctrl-C and then ran ./make.sh build again without doing a clean. It got to popt and went straight to libedit, the next package, and started to build it.
Something must have just put the build into a loop of some sort. It was not writing anything to the log file.
Regards,
Adolf.
Regards,
Adolf.
One thing I found. I am running the new build system while I have been running some package updates with the old system with its mount points. The two have each run without any impact on the other.
Regards,
Adolf.
Regards, Adolf.
Best, -Michael
On 9 Jul 2024, at 22:29, Michael Tremer michael.tremer@ipfire.org wrote:
Hello Adolf,
Thank you for testing this.
There have indeed been plenty of problems there… I spent a lot of time on this today and hopefully fixed most of them.
I cannot build the toolchain on my machine and I am not sure why yet, but a build with the packaged toolchain runs through.
I have also spent some time on getting rid of the strip stage because it annoyed me how long it takes and creating the disk images as well as packages should be significantly faster now, too. I hope I didn’t introduce too many new bugs.
Please let me know if you have more success now.
Best, -Michael
On 8 Jul 2024, at 20:34, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 08/07/2024 21:15, Adolf Belka wrote: > Hi Michael, > > On 08/07/2024 18:11, Michael Tremer wrote: >> Hello, >> >> I have been spending a lot of time on this problem, because it has been bothering me for a long time. I also saw an opportunity to make more changes to the build system. >> >> Currently this is all a little bit WIP, but I hope that we can merge this into next as soon as the current update has been moved to master. >> >> I am referring to this branch which is currently based on next: https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/u... >> >> It makes use of the unshare command which creates new namespaces in Linux. That way, we can isolate the build system better from the host system and in case something goes wrong, there is less damage. We can also enforce some more rules… >> >> So, what has changed? >> >> * The make.sh script might re-execute itself into a new mount namespace when it is suitable. This happens for “make.sh build” and “make.sh shell”, but it does not happen for “make.sh downloadsrc” for example. >> >> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... >> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... >> >> * The new mount namespace means that we will no longer see any bind-mounts in the host system and we no longer need to umount anything ourselves which is where we occasionally wiped the entire hard drive of the host system. When the last process exits, the namespace is being cleaned up and everything is being umounted. >> >> * The function that prepares the build environment has been almost entirely rewritten: >> >> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... >> >> It used to mount parts of the host system into the build environment which are needed to run anything. Those were /dev, /proc, /sys, etc… >> >> Instead, it now creates a new /dev mount point and creates a minimal amount of device nodes and symlinks. That way, we detach from the host system and no longer allow the build system access to the host’s filesystem and block devices. We also bind-mount the sources in read-only mode now, so that the build system cannot change anything in the source tree. On top of that, cache is read-only, too. ccache and the log directory are the only places that are writable. >> >> We mount a separate /tmp directory. >> >> * When we then build a package, we create more namespaces for each package. These isolate each build process from each other. >> >> Mostly, this is to detach from the host system. A new UTS namespace allows to change the hostname in the build system without affecting the host and so on. We do the same thing with a new time namespace. >> >> We do however create a new PID namespace which means that the build system no longer will see any processes running on the host system. That requires to mount a new instance of /proc with each package. This also has the effect that if the shell that we launched terminates (because the build is done) any background processes will be killed immediately. >> >> Last, we clone the mount namespace that we have created before so that no build command can modify what we set up earlier. >> >> * Since everything is now so decoupled, we gain a couple of new (maybe minor?) features: >> >> It is now possible to run “make.sh shell” while a build is running. That does not happen a lot, but we can do this now :) >> >> If the build crashes or the host system is being shut down while a build is running, there is nothing to clean up afterwards. >> >> * I have garnished this all with a lot of code cleanup and I suppose I might have introduced some new bugs here or there :) >> >> * This is probably mostly around a new implementation of the timer that updates the build time. It has been annoying me a lot that it takes a long time to walk through all packages that have been built before to finally get to a package that we want to rebuild. Mostly this was all help up by a call of “sleep 0.1” >> >> Since bash does not really do any concurrency, I had to be creative and replaced the busy-loop with a background process that is launched whenever it is needed and which will “ping” the main make.sh script once a second. That way, we can just run as usual, but regularly get interrupted to update the runtime. >> >> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... >> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... >> >> We now only fork one extra sub shell and we have to handle the timer events which is a lot cheaper as well as more straight-forward to code. >> >> * As there is no difference between the different stages any more (those stages that we inherited from LFS), I have merged them all into one. >> >> * Last but not least, I have create the option to build for multiple architectures on the same system. Since we can now mount the entire source tree into (many independent) build environments, we might as well… As discussed on the last call, this might not be the best option for ARM, but RISV-C builds at a decent speed even when emulated. >> >> The only thing that I needed to do for this is to suffix the build and log directories which are now called build_${ARCH}, i.e. build_aarch64, build_x86_64, and so on. The packages/ directory is not changed yet, but that will have to happen as well. Most likely I want to merge this with the generated images, but I am not sure what to call this, yet. Happy to hear suggestions. result_x86_64? Just images_x86_64? >> >> --- >> >> I have run a build and this seems to be working just fine on my Debian machine. I am writing to all of you to first of all let you know what I am up to; and secondly to ask to give this a go on your systems. I think it should run just fine, as all the tools that I require should be available everywhere. However, there might be some older kernels that might not support all of this, yet or any other problems I cannot think of yet. Please give me some feedback and send me all the bugs :) > I gave this a go but it didn't work. > > Not sure if I should have run the ./make.sh clean command on the old version before I pulled the unshare branch into my clone of your repo. > > Should I have started with a complete new clone of your repo? I might try that anyway just to see. > I created a completely new clone of you ipfire-2.x repor and then checked out the unshare branch to a branch called unshare in my local repo clone.
gettoolchain gave the same issue, except that this time the toolchain directory ended up completely empty.
downloadsrc had the same result.
clean had nothing to clean up as it was a fresh clone.
build then tried to build the toolchain and came up with this error, different from before.
./make.sh build chroot: failed to run command ‘env’: No such file or directory Full toolchain compilation stage1 [ FAIL ]
Jul 8 19:26:39: Building stage1 ====================================== Installing stage1 ... mkdir -pv /tools_x86_64/lib mkdir: cannot create directory '/tools_x86_64': File exists make: *** [stage1:50: /home/ahb/sandbox/ms/ipfire-2.x/log/stage1] Error 1
ERROR: Building stage1 [ FAIL ] Check /home/ahb/sandbox/ms/ipfire-2.x/log_x86_64/_build.toolchain.log for errors if applicable [ FAIL ]
so it wasn't as simple as doing a fresh git clone.
Regards,
Adolf.
> So I ran ./make.sh gettoolchain first, as I usually would. > > ./make.sh gettoolchain > b2sum: cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: No such file or directory > cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: FAILED open or read > b2sum: WARNING: 1 listed file could not be read > > ipfire-2.29-toolchain-20240210-x86_64.tar.zst is present together with its b2 file. > > > Then ran ./make.sh downloadsrc > > Previous version ends with > > ***Verifying BLAKE2 checksum > all files BLAKE2 checksum match [ DONE] > > after zstd has been checked. > > New version stops at zstd entry. > > > ./make.sh clean gave the message Cleaning Build directory... but was completed very quickly. > Log and Build directories have not been cleaned out. The img and iso files are still present. > > > ./make.sh build gave message > > chroot: failed to run command ‘env’: No such file or directory > > and then did a full toolchain compilation which failed with gcc but log is >9000 lines. > > > Regards, > Adolf. > >> >> Thank you for listening to this brain-dump. >> >> All the best, >> -Michael >> >>> On 3 Jul 2024, at 10:58, Michael Tremer michael.tremer@ipfire.org wrote: >>> >>> Hello Adolf, >>> >>> This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more. >>> >>> I rebooted the machine and it is back up again. >>> >>> -Michael >>> >>>> On 2 Jul 2024, at 15:42, Adolf Belka adolf.belka@ipfire.org wrote: >>>> >>>> Hi Michael and all, >>>> >>>> >>>> I ran the arm builder with the 4.20.2 version of samba to test it out. >>>> >>>> The build got to building gdb and then failed. >>>> >>>> Interestingly, the nightly build of arm was successful with the same version of gdb. >>>> >>>> The build log for gdb is attached. The actual error is at line 618. >>>> >>>> Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message. >>>> >>>> ------------------------------------------ >>>> >>>> ssh bonnietwin@arm64-01.zrh.ipfire.org >>>> PTY allocation request failed on channel 0 >>>> Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64 >>>> >>>> The programs included with the Debian GNU/Linux system are free software; >>>> the exact distribution terms for each program are described in the >>>> individual files in /usr/share/doc/*/copyright. >>>> >>>> Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent >>>> permitted by applicable law. >>>> /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied >>>> >>>> ------------------------------------------ >>>> >>>> Regards, >>>> >>>> Adolf. >>>> >>>> <_build.ipfire.gdb.log>
Hello,
On 10 Jul 2024, at 14:21, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 10/07/2024 15:05, Adolf Belka wrote:
Hi Michael,
On 10/07/2024 14:59, Adolf Belka wrote:
Hi Michael,
On 10/07/2024 12:33, Adolf Belka wrote:
Hi Michael,
On 10/07/2024 11:57, Michael Tremer wrote:
Hello again,
I managed to (finally) build the toolchain with the updated system. So hopefully there should not be any more outstanding problems that I know of so far.
I just did a git pull on your repo to my clone.
Ran ./make.sh gettoolchain and it successfully downloaded the toolchain.
Ran ./make.sh downloadsrc and it successfully tested everything.
Ran ./make.sh clean and build and log directories were cleared out and removed. As far as I can tell it was successful.
Currently running ./make.sh build. So up to this point everything going well. Will let you know how it goes.
It has got to building popt. In the normal build system this takes around 3 secs. Currently in the new build it is at nearly 2 hours. Even with an empty cache, that seems a long build time for popt, unless I am being too optimistic.
I will let it keep going.
I just had a look at the log file and it looks like it completed popt but then is stuck on trying to leave the directory /usr/src/lfs. Here is the output from the log, nothing new is getting written to the log.
make[3]: Leaving directory '/usr/src/popt-1.19/po' Making install in tests make[3]: Entering directory '/usr/src/popt-1.19/tests' make[4]: Entering directory '/usr/src/popt-1.19/tests' make[4]: Nothing to be done for 'install-exec-am'. make[4]: Nothing to be done for 'install-data-am'. make[4]: Leaving directory '/usr/src/popt-1.19/tests' make[3]: Leaving directory '/usr/src/popt-1.19/tests' make[3]: Entering directory '/usr/src/popt-1.19' make[4]: Entering directory '/usr/src/popt-1.19' make[4]: Nothing to be done for 'install-exec-am'. /bin/mkdir -p '/usr/share/man/man3' /usr/bin/install -c -m 644 popt.3 '/usr/share/man/man3' /bin/mkdir -p '/usr/lib/pkgconfig' /usr/bin/install -c -m 644 popt.pc '/usr/lib/pkgconfig' make[4]: Leaving directory '/usr/src/popt-1.19' make[3]: Leaving directory '/usr/src/popt-1.19' make[2]: Leaving directory '/usr/src/popt-1.19' make[1]: Leaving directory '/usr/src/popt-1.19' Updating linker cache... Install done; saving file list to /usr/src/log/popt-1.19 ...
make: Leaving directory '/usr/src/lfs'
I stopped the build with Ctrl-C and then ran ./make.sh build again without doing a clean. It got to popt and went straight to libedit, the next package, and started to build it.
Something must have just put the build into a loop of some sort. It was not writing anything to the log file.
The log file says that popt was done. It might be that you interrupted the build just after it has finished, but the root file was not created, yet. That should however not directly be a problem…
If this is however the only problem that you are seeing now than I would be happy :)
-Michael
Regards,
Adolf.
Regards,
Adolf.
One thing I found. I am running the new build system while I have been running some package updates with the old system with its mount points. The two have each run without any impact on the other.
Regards,
Adolf.
Regards, Adolf.
Best, -Michael
On 9 Jul 2024, at 22:29, Michael Tremer michael.tremer@ipfire.org wrote:
Hello Adolf,
Thank you for testing this.
There have indeed been plenty of problems there… I spent a lot of time on this today and hopefully fixed most of them.
I cannot build the toolchain on my machine and I am not sure why yet, but a build with the packaged toolchain runs through.
I have also spent some time on getting rid of the strip stage because it annoyed me how long it takes and creating the disk images as well as packages should be significantly faster now, too. I hope I didn’t introduce too many new bugs.
Please let me know if you have more success now.
Best, -Michael
> On 8 Jul 2024, at 20:34, Adolf Belka adolf.belka@ipfire.org wrote: > > Hi Michael, > > On 08/07/2024 21:15, Adolf Belka wrote: >> Hi Michael, >> >> On 08/07/2024 18:11, Michael Tremer wrote: >>> Hello, >>> >>> I have been spending a lot of time on this problem, because it has been bothering me for a long time. I also saw an opportunity to make more changes to the build system. >>> >>> Currently this is all a little bit WIP, but I hope that we can merge this into next as soon as the current update has been moved to master. >>> >>> I am referring to this branch which is currently based on next: https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/u... >>> >>> It makes use of the unshare command which creates new namespaces in Linux. That way, we can isolate the build system better from the host system and in case something goes wrong, there is less damage. We can also enforce some more rules… >>> >>> So, what has changed? >>> >>> * The make.sh script might re-execute itself into a new mount namespace when it is suitable. This happens for “make.sh build” and “make.sh shell”, but it does not happen for “make.sh downloadsrc” for example. >>> >>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... >>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... >>> >>> * The new mount namespace means that we will no longer see any bind-mounts in the host system and we no longer need to umount anything ourselves which is where we occasionally wiped the entire hard drive of the host system. When the last process exits, the namespace is being cleaned up and everything is being umounted. >>> >>> * The function that prepares the build environment has been almost entirely rewritten: >>> >>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... >>> >>> It used to mount parts of the host system into the build environment which are needed to run anything. Those were /dev, /proc, /sys, etc… >>> >>> Instead, it now creates a new /dev mount point and creates a minimal amount of device nodes and symlinks. That way, we detach from the host system and no longer allow the build system access to the host’s filesystem and block devices. We also bind-mount the sources in read-only mode now, so that the build system cannot change anything in the source tree. On top of that, cache is read-only, too. ccache and the log directory are the only places that are writable. >>> >>> We mount a separate /tmp directory. >>> >>> * When we then build a package, we create more namespaces for each package. These isolate each build process from each other. >>> >>> Mostly, this is to detach from the host system. A new UTS namespace allows to change the hostname in the build system without affecting the host and so on. We do the same thing with a new time namespace. >>> >>> We do however create a new PID namespace which means that the build system no longer will see any processes running on the host system. That requires to mount a new instance of /proc with each package. This also has the effect that if the shell that we launched terminates (because the build is done) any background processes will be killed immediately. >>> >>> Last, we clone the mount namespace that we have created before so that no build command can modify what we set up earlier. >>> >>> * Since everything is now so decoupled, we gain a couple of new (maybe minor?) features: >>> >>> It is now possible to run “make.sh shell” while a build is running. That does not happen a lot, but we can do this now :) >>> >>> If the build crashes or the host system is being shut down while a build is running, there is nothing to clean up afterwards. >>> >>> * I have garnished this all with a lot of code cleanup and I suppose I might have introduced some new bugs here or there :) >>> >>> * This is probably mostly around a new implementation of the timer that updates the build time. It has been annoying me a lot that it takes a long time to walk through all packages that have been built before to finally get to a package that we want to rebuild. Mostly this was all help up by a call of “sleep 0.1” >>> >>> Since bash does not really do any concurrency, I had to be creative and replaced the busy-loop with a background process that is launched whenever it is needed and which will “ping” the main make.sh script once a second. That way, we can just run as usual, but regularly get interrupted to update the runtime. >>> >>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... >>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... >>> >>> We now only fork one extra sub shell and we have to handle the timer events which is a lot cheaper as well as more straight-forward to code. >>> >>> * As there is no difference between the different stages any more (those stages that we inherited from LFS), I have merged them all into one. >>> >>> * Last but not least, I have create the option to build for multiple architectures on the same system. Since we can now mount the entire source tree into (many independent) build environments, we might as well… As discussed on the last call, this might not be the best option for ARM, but RISV-C builds at a decent speed even when emulated. >>> >>> The only thing that I needed to do for this is to suffix the build and log directories which are now called build_${ARCH}, i.e. build_aarch64, build_x86_64, and so on. The packages/ directory is not changed yet, but that will have to happen as well. Most likely I want to merge this with the generated images, but I am not sure what to call this, yet. Happy to hear suggestions. result_x86_64? Just images_x86_64? >>> >>> --- >>> >>> I have run a build and this seems to be working just fine on my Debian machine. I am writing to all of you to first of all let you know what I am up to; and secondly to ask to give this a go on your systems. I think it should run just fine, as all the tools that I require should be available everywhere. However, there might be some older kernels that might not support all of this, yet or any other problems I cannot think of yet. Please give me some feedback and send me all the bugs :) >> I gave this a go but it didn't work. >> >> Not sure if I should have run the ./make.sh clean command on the old version before I pulled the unshare branch into my clone of your repo. >> >> Should I have started with a complete new clone of your repo? I might try that anyway just to see. >> > I created a completely new clone of you ipfire-2.x repor and then checked out the unshare branch to a branch called unshare in my local repo clone. > > gettoolchain gave the same issue, except that this time the toolchain directory ended up completely empty. > > downloadsrc had the same result. > > clean had nothing to clean up as it was a fresh clone. > > build then tried to build the toolchain and came up with this error, different from before. > > ./make.sh build > chroot: failed to run command ‘env’: No such file or directory > Full toolchain compilation > stage1 [ FAIL ] > > Jul 8 19:26:39: Building stage1 ====================================== Installing stage1 ... > mkdir -pv /tools_x86_64/lib > mkdir: cannot create directory '/tools_x86_64': File exists > make: *** [stage1:50: /home/ahb/sandbox/ms/ipfire-2.x/log/stage1] Error 1 > > ERROR: Building stage1 [ FAIL ] > Check /home/ahb/sandbox/ms/ipfire-2.x/log_x86_64/_build.toolchain.log for errors if applicable [ FAIL ] > > so it wasn't as simple as doing a fresh git clone. > > Regards, > > Adolf. > > >> So I ran ./make.sh gettoolchain first, as I usually would. >> >> ./make.sh gettoolchain >> b2sum: cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: No such file or directory >> cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: FAILED open or read >> b2sum: WARNING: 1 listed file could not be read >> >> ipfire-2.29-toolchain-20240210-x86_64.tar.zst is present together with its b2 file. >> >> >> Then ran ./make.sh downloadsrc >> >> Previous version ends with >> >> ***Verifying BLAKE2 checksum >> all files BLAKE2 checksum match [ DONE] >> >> after zstd has been checked. >> >> New version stops at zstd entry. >> >> >> ./make.sh clean gave the message Cleaning Build directory... but was completed very quickly. >> Log and Build directories have not been cleaned out. The img and iso files are still present. >> >> >> ./make.sh build gave message >> >> chroot: failed to run command ‘env’: No such file or directory >> >> and then did a full toolchain compilation which failed with gcc but log is >9000 lines. >> >> >> Regards, >> Adolf. >> >>> >>> Thank you for listening to this brain-dump. >>> >>> All the best, >>> -Michael >>> >>>> On 3 Jul 2024, at 10:58, Michael Tremer michael.tremer@ipfire.org wrote: >>>> >>>> Hello Adolf, >>>> >>>> This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more. >>>> >>>> I rebooted the machine and it is back up again. >>>> >>>> -Michael >>>> >>>>> On 2 Jul 2024, at 15:42, Adolf Belka adolf.belka@ipfire.org wrote: >>>>> >>>>> Hi Michael and all, >>>>> >>>>> >>>>> I ran the arm builder with the 4.20.2 version of samba to test it out. >>>>> >>>>> The build got to building gdb and then failed. >>>>> >>>>> Interestingly, the nightly build of arm was successful with the same version of gdb. >>>>> >>>>> The build log for gdb is attached. The actual error is at line 618. >>>>> >>>>> Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message. >>>>> >>>>> ------------------------------------------ >>>>> >>>>> ssh bonnietwin@arm64-01.zrh.ipfire.org >>>>> PTY allocation request failed on channel 0 >>>>> Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64 >>>>> >>>>> The programs included with the Debian GNU/Linux system are free software; >>>>> the exact distribution terms for each program are described in the >>>>> individual files in /usr/share/doc/*/copyright. >>>>> >>>>> Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent >>>>> permitted by applicable law. >>>>> /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied >>>>> >>>>> ------------------------------------------ >>>>> >>>>> Regards, >>>>> >>>>> Adolf. >>>>> >>>>> <_build.ipfire.gdb.log>
Hi Michael,
On 10/07/2024 16:24, Michael Tremer wrote:
Hello,
On 10 Jul 2024, at 14:21, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 10/07/2024 15:05, Adolf Belka wrote:
Hi Michael,
On 10/07/2024 14:59, Adolf Belka wrote:
Hi Michael,
On 10/07/2024 12:33, Adolf Belka wrote:
Hi Michael,
On 10/07/2024 11:57, Michael Tremer wrote:
Hello again,
I managed to (finally) build the toolchain with the updated system. So hopefully there should not be any more outstanding problems that I know of so far.
I just did a git pull on your repo to my clone.
Ran ./make.sh gettoolchain and it successfully downloaded the toolchain.
Ran ./make.sh downloadsrc and it successfully tested everything.
Ran ./make.sh clean and build and log directories were cleared out and removed. As far as I can tell it was successful.
Currently running ./make.sh build. So up to this point everything going well. Will let you know how it goes.
It has got to building popt. In the normal build system this takes around 3 secs. Currently in the new build it is at nearly 2 hours. Even with an empty cache, that seems a long build time for popt, unless I am being too optimistic.
I will let it keep going.
I just had a look at the log file and it looks like it completed popt but then is stuck on trying to leave the directory /usr/src/lfs. Here is the output from the log, nothing new is getting written to the log.
make[3]: Leaving directory '/usr/src/popt-1.19/po' Making install in tests make[3]: Entering directory '/usr/src/popt-1.19/tests' make[4]: Entering directory '/usr/src/popt-1.19/tests' make[4]: Nothing to be done for 'install-exec-am'. make[4]: Nothing to be done for 'install-data-am'. make[4]: Leaving directory '/usr/src/popt-1.19/tests' make[3]: Leaving directory '/usr/src/popt-1.19/tests' make[3]: Entering directory '/usr/src/popt-1.19' make[4]: Entering directory '/usr/src/popt-1.19' make[4]: Nothing to be done for 'install-exec-am'. /bin/mkdir -p '/usr/share/man/man3' /usr/bin/install -c -m 644 popt.3 '/usr/share/man/man3' /bin/mkdir -p '/usr/lib/pkgconfig' /usr/bin/install -c -m 644 popt.pc '/usr/lib/pkgconfig' make[4]: Leaving directory '/usr/src/popt-1.19' make[3]: Leaving directory '/usr/src/popt-1.19' make[2]: Leaving directory '/usr/src/popt-1.19' make[1]: Leaving directory '/usr/src/popt-1.19' Updating linker cache... Install done; saving file list to /usr/src/log/popt-1.19 ...
make: Leaving directory '/usr/src/lfs'
I stopped the build with Ctrl-C and then ran ./make.sh build again without doing a clean. It got to popt and went straight to libedit, the next package, and started to build it.
Something must have just put the build into a loop of some sort. It was not writing anything to the log file.
The log file says that popt was done. It might be that you interrupted the build just after it has finished, but the root file was not created, yet. That should however not directly be a problem…
2 hours had already gone by when I checked the log for that message. I then left the system running for a further 20 minutes with no new info in the log file and that is when I interrupted the build.
If this is however the only problem that you are seeing now than I would be happy :)
I have found another problem when it got to the flash-images section.
The flash-images log stops with the following:-
# Insert the UUID because grub-mkconfig often fails to # detect that correctly sed -i /tmp/harddisk/boot/grub/grub.cfg \ -e "s/root=[A-Za-z0-9/=-]*/root=UUID=$(blkid -o value -s UUID /dev/mapper/loop0p3)/g" # Install GRUB grub-install --force --recheck --no-floppy --target=i386-pc \ --root-directory=/tmp/harddisk /dev/loop0 Installing for i386-pc platform. Installation finished. No error reported. # Install GRUB for EFI grub-install --target=x86_64-efi --removable --no-nvram \ --boot-directory=/tmp/harddisk/boot --efi-directory=/tmp/harddisk/boot/efi Installing for x86_64-efi platform. Installation finished. No error reported. # restore orginal defaults mv -f /tmp/harddisk/etc/default/grub.backup /tmp/harddisk/etc/default/grub rm -f /tmp/harddisk/etc/grub.d/11_linux_scon # Set ramdisk mode to automatic echo RAMDISK_MODE=2 > /tmp/harddisk/etc/sysconfig/ramdisk # Automatically resize the root partition to its maximum size at first boot touch /tmp/harddisk/.partresize # Unmount umount /tmp/harddisk/proc umount /tmp/harddisk/sys umount /tmp/harddisk/dev umount /tmp/harddisk/boot/efi umount /tmp/harddisk/boot umount /tmp/harddisk # zerofree the ext2 images to get better compression zerofree /dev/mapper/loop0p1 fsck.ext2 -f -y /dev/mapper/loop0p1 e2fsck 1.47.0 (5-Feb-2023) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/mapper/loop0p1: 631/130048 files (22.2% non-contiguous), 124148/520192 blocks fsck.ext2 -f -y /dev/mapper/loop0p1 e2fsck 1.47.0 (5-Feb-2023) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/mapper/loop0p1: 631/130048 files (22.2% non-contiguous), 124148/520192 blocks zerofree /dev/mapper/loop0p3 fsck.ext4 -f -y /dev/mapper/loop0p3 e2fsck 1.47.0 (5-Feb-2023) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/mapper/loop0p3: 25477/118080 files (0.1% non-contiguous), 418098/471661 blocks fsck.ext4 -f -y /dev/mapper/loop0p3 e2fsck 1.47.0 (5-Feb-2023) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/mapper/loop0p3: 25477/118080 files (0.1% non-contiguous), 418098/471661 blocks kpartx -d -v /dev/loop0 del devmap : loop0p1 del devmap : loop0p2 del devmap : loop0p3 losetup -d /dev/loop0 # Add padding at the end of the image (to fix alignment issues if the image is # not copied to a block device) dd if=/dev/zero bs=1M count=100 >> /tmp/image.img 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.0387438 s, 2.7 GB/s # Compress Image xz -T0 -8 < /tmp/image.img > /usr/src/images/ipfire-2.29-core187-x86_64.img.xz xz: Reduced the number of threads from 16 to 11 to not exceed the memory usage limit of 7860 MiB rm -rf /tmp/image.img /tmp/harddisk /tmp/cdrom # Create checksum file cd /usr/src/images && b2sum "ipfire-2.29-core187-x86_64.img.xz" > "ipfire-2.29-core187-x86_64.img.xz".b2" /bin/sh: -c: line 1: unexpected EOF while looking for matching `"' make: *** [flash-images:186: /usr/src/log/flash-image] Error 2 make: Leaving directory '/usr/src/lfs'
In the images directory there is the iso and img files but only a b2 file for the iso.
Prior to the above section of the log there are about 165 entries similar saying:-
ERROR: ld.so: object '/tools_x86_64/lib/libpakfire_preload.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
Not sure that it has anything to do with the flash-images fail I am having as all those messages say the fail was ignored.
Looking in the flash-images lfs file there looks like
# Create checksum file cd $(IMAGES_DIR) && b2sum "$(notdir $(IMAGE_FILE))" > "$(notdir "$(IMAGE_FILE)").b2"
has "$(IMAGE_FILE)" for the destination while the source only has $(IMAGE_FILE)
So I removed the "" from the destination end and ran the build again.
This time the b2 file was created but now the flash-images failed with
make: Nothing to be done for 'download'. make: Leaving directory '/home/ahb/sandbox/ms/ipfire-2.x/lfs' make: Entering directory '/usr/src/lfs' rm -rf /install/packages/package /tmp/* mkdir -p /install/packages/package eval $(cat /usr/src/config/rootfiles/core/187/meta) cat: /usr/src/config/rootfiles/core/187/meta: No such file or directory #Generate ROOTFILES from filelists BUILD_ARCH=x86_64 BUILDTARGET=x86_64-pc-linux-gnu KVER=6.6.32 \ /usr/src/src/scripts/archive.files \ /usr/src/config/rootfiles/core/187/filelists \ /usr/src/config/rootfiles/core/187/files \ /usr/src/config/rootfiles/core/187/files.x86_64 \ > /tmp/ROOTFILES.tmp #remove excluded files from ROOTFILES grep -f /usr/src/config/rootfiles/core/187/exclude -v /tmp/ROOTFILES.tmp > /tmp/ROOTFILES make: *** [core-updates:60: core/187] Error 1 make: Leaving directory '/usr/src/lfs'
The above might have some relation to not doing a clean before the build. So I will leave the change in the flash-images lfs file and do a clean and a fresh new build
Regards,
Adolf.
-Michael
Regards,
Adolf.
Regards,
Adolf.
One thing I found. I am running the new build system while I have been running some package updates with the old system with its mount points. The two have each run without any impact on the other.
Regards,
Adolf.
Regards, Adolf.
Best, -Michael
> On 9 Jul 2024, at 22:29, Michael Tremer michael.tremer@ipfire.org wrote: > > Hello Adolf, > > Thank you for testing this. > > There have indeed been plenty of problems there… I spent a lot of time on this today and hopefully fixed most of them. > > I cannot build the toolchain on my machine and I am not sure why yet, but a build with the packaged toolchain runs through. > > I have also spent some time on getting rid of the strip stage because it annoyed me how long it takes and creating the disk images as well as packages should be significantly faster now, too. I hope I didn’t introduce too many new bugs. > > Please let me know if you have more success now. > > Best, > -Michael > >> On 8 Jul 2024, at 20:34, Adolf Belka adolf.belka@ipfire.org wrote: >> >> Hi Michael, >> >> On 08/07/2024 21:15, Adolf Belka wrote: >>> Hi Michael, >>> >>> On 08/07/2024 18:11, Michael Tremer wrote: >>>> Hello, >>>> >>>> I have been spending a lot of time on this problem, because it has been bothering me for a long time. I also saw an opportunity to make more changes to the build system. >>>> >>>> Currently this is all a little bit WIP, but I hope that we can merge this into next as soon as the current update has been moved to master. >>>> >>>> I am referring to this branch which is currently based on next: https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/u... >>>> >>>> It makes use of the unshare command which creates new namespaces in Linux. That way, we can isolate the build system better from the host system and in case something goes wrong, there is less damage. We can also enforce some more rules… >>>> >>>> So, what has changed? >>>> >>>> * The make.sh script might re-execute itself into a new mount namespace when it is suitable. This happens for “make.sh build” and “make.sh shell”, but it does not happen for “make.sh downloadsrc” for example. >>>> >>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... >>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... >>>> >>>> * The new mount namespace means that we will no longer see any bind-mounts in the host system and we no longer need to umount anything ourselves which is where we occasionally wiped the entire hard drive of the host system. When the last process exits, the namespace is being cleaned up and everything is being umounted. >>>> >>>> * The function that prepares the build environment has been almost entirely rewritten: >>>> >>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... >>>> >>>> It used to mount parts of the host system into the build environment which are needed to run anything. Those were /dev, /proc, /sys, etc… >>>> >>>> Instead, it now creates a new /dev mount point and creates a minimal amount of device nodes and symlinks. That way, we detach from the host system and no longer allow the build system access to the host’s filesystem and block devices. We also bind-mount the sources in read-only mode now, so that the build system cannot change anything in the source tree. On top of that, cache is read-only, too. ccache and the log directory are the only places that are writable. >>>> >>>> We mount a separate /tmp directory. >>>> >>>> * When we then build a package, we create more namespaces for each package. These isolate each build process from each other. >>>> >>>> Mostly, this is to detach from the host system. A new UTS namespace allows to change the hostname in the build system without affecting the host and so on. We do the same thing with a new time namespace. >>>> >>>> We do however create a new PID namespace which means that the build system no longer will see any processes running on the host system. That requires to mount a new instance of /proc with each package. This also has the effect that if the shell that we launched terminates (because the build is done) any background processes will be killed immediately. >>>> >>>> Last, we clone the mount namespace that we have created before so that no build command can modify what we set up earlier. >>>> >>>> * Since everything is now so decoupled, we gain a couple of new (maybe minor?) features: >>>> >>>> It is now possible to run “make.sh shell” while a build is running. That does not happen a lot, but we can do this now :) >>>> >>>> If the build crashes or the host system is being shut down while a build is running, there is nothing to clean up afterwards. >>>> >>>> * I have garnished this all with a lot of code cleanup and I suppose I might have introduced some new bugs here or there :) >>>> >>>> * This is probably mostly around a new implementation of the timer that updates the build time. It has been annoying me a lot that it takes a long time to walk through all packages that have been built before to finally get to a package that we want to rebuild. Mostly this was all help up by a call of “sleep 0.1” >>>> >>>> Since bash does not really do any concurrency, I had to be creative and replaced the busy-loop with a background process that is launched whenever it is needed and which will “ping” the main make.sh script once a second. That way, we can just run as usual, but regularly get interrupted to update the runtime. >>>> >>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... >>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... >>>> >>>> We now only fork one extra sub shell and we have to handle the timer events which is a lot cheaper as well as more straight-forward to code. >>>> >>>> * As there is no difference between the different stages any more (those stages that we inherited from LFS), I have merged them all into one. >>>> >>>> * Last but not least, I have create the option to build for multiple architectures on the same system. Since we can now mount the entire source tree into (many independent) build environments, we might as well… As discussed on the last call, this might not be the best option for ARM, but RISV-C builds at a decent speed even when emulated. >>>> >>>> The only thing that I needed to do for this is to suffix the build and log directories which are now called build_${ARCH}, i.e. build_aarch64, build_x86_64, and so on. The packages/ directory is not changed yet, but that will have to happen as well. Most likely I want to merge this with the generated images, but I am not sure what to call this, yet. Happy to hear suggestions. result_x86_64? Just images_x86_64? >>>> >>>> --- >>>> >>>> I have run a build and this seems to be working just fine on my Debian machine. I am writing to all of you to first of all let you know what I am up to; and secondly to ask to give this a go on your systems. I think it should run just fine, as all the tools that I require should be available everywhere. However, there might be some older kernels that might not support all of this, yet or any other problems I cannot think of yet. Please give me some feedback and send me all the bugs :) >>> I gave this a go but it didn't work. >>> >>> Not sure if I should have run the ./make.sh clean command on the old version before I pulled the unshare branch into my clone of your repo. >>> >>> Should I have started with a complete new clone of your repo? I might try that anyway just to see. >>> >> I created a completely new clone of you ipfire-2.x repor and then checked out the unshare branch to a branch called unshare in my local repo clone. >> >> gettoolchain gave the same issue, except that this time the toolchain directory ended up completely empty. >> >> downloadsrc had the same result. >> >> clean had nothing to clean up as it was a fresh clone. >> >> build then tried to build the toolchain and came up with this error, different from before. >> >> ./make.sh build >> chroot: failed to run command ‘env’: No such file or directory >> Full toolchain compilation >> stage1 [ FAIL ] >> >> Jul 8 19:26:39: Building stage1 ====================================== Installing stage1 ... >> mkdir -pv /tools_x86_64/lib >> mkdir: cannot create directory '/tools_x86_64': File exists >> make: *** [stage1:50: /home/ahb/sandbox/ms/ipfire-2.x/log/stage1] Error 1 >> >> ERROR: Building stage1 [ FAIL ] >> Check /home/ahb/sandbox/ms/ipfire-2.x/log_x86_64/_build.toolchain.log for errors if applicable [ FAIL ] >> >> so it wasn't as simple as doing a fresh git clone. >> >> Regards, >> >> Adolf. >> >> >>> So I ran ./make.sh gettoolchain first, as I usually would. >>> >>> ./make.sh gettoolchain >>> b2sum: cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: No such file or directory >>> cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: FAILED open or read >>> b2sum: WARNING: 1 listed file could not be read >>> >>> ipfire-2.29-toolchain-20240210-x86_64.tar.zst is present together with its b2 file. >>> >>> >>> Then ran ./make.sh downloadsrc >>> >>> Previous version ends with >>> >>> ***Verifying BLAKE2 checksum >>> all files BLAKE2 checksum match [ DONE] >>> >>> after zstd has been checked. >>> >>> New version stops at zstd entry. >>> >>> >>> ./make.sh clean gave the message Cleaning Build directory... but was completed very quickly. >>> Log and Build directories have not been cleaned out. The img and iso files are still present. >>> >>> >>> ./make.sh build gave message >>> >>> chroot: failed to run command ‘env’: No such file or directory >>> >>> and then did a full toolchain compilation which failed with gcc but log is >9000 lines. >>> >>> >>> Regards, >>> Adolf. >>> >>>> >>>> Thank you for listening to this brain-dump. >>>> >>>> All the best, >>>> -Michael >>>> >>>>> On 3 Jul 2024, at 10:58, Michael Tremer michael.tremer@ipfire.org wrote: >>>>> >>>>> Hello Adolf, >>>>> >>>>> This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more. >>>>> >>>>> I rebooted the machine and it is back up again. >>>>> >>>>> -Michael >>>>> >>>>>> On 2 Jul 2024, at 15:42, Adolf Belka adolf.belka@ipfire.org wrote: >>>>>> >>>>>> Hi Michael and all, >>>>>> >>>>>> >>>>>> I ran the arm builder with the 4.20.2 version of samba to test it out. >>>>>> >>>>>> The build got to building gdb and then failed. >>>>>> >>>>>> Interestingly, the nightly build of arm was successful with the same version of gdb. >>>>>> >>>>>> The build log for gdb is attached. The actual error is at line 618. >>>>>> >>>>>> Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message. >>>>>> >>>>>> ------------------------------------------ >>>>>> >>>>>> ssh bonnietwin@arm64-01.zrh.ipfire.org >>>>>> PTY allocation request failed on channel 0 >>>>>> Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64 >>>>>> >>>>>> The programs included with the Debian GNU/Linux system are free software; >>>>>> the exact distribution terms for each program are described in the >>>>>> individual files in /usr/share/doc/*/copyright. >>>>>> >>>>>> Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent >>>>>> permitted by applicable law. >>>>>> /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied >>>>>> >>>>>> ------------------------------------------ >>>>>> >>>>>> Regards, >>>>>> >>>>>> Adolf. >>>>>> >>>>>> <_build.ipfire.gdb.log>
Hello,
On 10 Jul 2024, at 17:53, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 10/07/2024 16:24, Michael Tremer wrote:
Hello,
On 10 Jul 2024, at 14:21, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 10/07/2024 15:05, Adolf Belka wrote:
Hi Michael,
On 10/07/2024 14:59, Adolf Belka wrote:
Hi Michael,
On 10/07/2024 12:33, Adolf Belka wrote:
Hi Michael,
On 10/07/2024 11:57, Michael Tremer wrote: > Hello again, > > I managed to (finally) build the toolchain with the updated system. So hopefully there should not be any more outstanding problems that I know of so far.
I just did a git pull on your repo to my clone.
Ran ./make.sh gettoolchain and it successfully downloaded the toolchain.
Ran ./make.sh downloadsrc and it successfully tested everything.
Ran ./make.sh clean and build and log directories were cleared out and removed. As far as I can tell it was successful.
Currently running ./make.sh build. So up to this point everything going well. Will let you know how it goes.
It has got to building popt. In the normal build system this takes around 3 secs. Currently in the new build it is at nearly 2 hours. Even with an empty cache, that seems a long build time for popt, unless I am being too optimistic.
I will let it keep going.
I just had a look at the log file and it looks like it completed popt but then is stuck on trying to leave the directory /usr/src/lfs. Here is the output from the log, nothing new is getting written to the log.
make[3]: Leaving directory '/usr/src/popt-1.19/po' Making install in tests make[3]: Entering directory '/usr/src/popt-1.19/tests' make[4]: Entering directory '/usr/src/popt-1.19/tests' make[4]: Nothing to be done for 'install-exec-am'. make[4]: Nothing to be done for 'install-data-am'. make[4]: Leaving directory '/usr/src/popt-1.19/tests' make[3]: Leaving directory '/usr/src/popt-1.19/tests' make[3]: Entering directory '/usr/src/popt-1.19' make[4]: Entering directory '/usr/src/popt-1.19' make[4]: Nothing to be done for 'install-exec-am'. /bin/mkdir -p '/usr/share/man/man3' /usr/bin/install -c -m 644 popt.3 '/usr/share/man/man3' /bin/mkdir -p '/usr/lib/pkgconfig' /usr/bin/install -c -m 644 popt.pc '/usr/lib/pkgconfig' make[4]: Leaving directory '/usr/src/popt-1.19' make[3]: Leaving directory '/usr/src/popt-1.19' make[2]: Leaving directory '/usr/src/popt-1.19' make[1]: Leaving directory '/usr/src/popt-1.19' Updating linker cache... Install done; saving file list to /usr/src/log/popt-1.19 ...
make: Leaving directory '/usr/src/lfs'
I stopped the build with Ctrl-C and then ran ./make.sh build again without doing a clean. It got to popt and went straight to libedit, the next package, and started to build it.
Something must have just put the build into a loop of some sort. It was not writing anything to the log file.
The log file says that popt was done. It might be that you interrupted the build just after it has finished, but the root file was not created, yet. That should however not directly be a problem…
2 hours had already gone by when I checked the log for that message. I then left the system running for a further 20 minutes with no new info in the log file and that is when I interrupted the build.
Oh, that. Yes, I think I have seen this too, but I am not sure why yet.
It looks like the wait call misses that the process has already been gone here:
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=69e1fd...
If this is however the only problem that you are seeing now than I would be happy :)
I have found another problem when it got to the flash-images section.
The flash-images log stops with the following:-
# Insert the UUID because grub-mkconfig often fails to # detect that correctly sed -i /tmp/harddisk/boot/grub/grub.cfg \ -e "s/root=[A-Za-z0-9/=-]*/root=UUID=$(blkid -o value -s UUID /dev/mapper/loop0p3)/g" # Install GRUB grub-install --force --recheck --no-floppy --target=i386-pc \ --root-directory=/tmp/harddisk /dev/loop0 Installing for i386-pc platform. Installation finished. No error reported. # Install GRUB for EFI grub-install --target=x86_64-efi --removable --no-nvram \ --boot-directory=/tmp/harddisk/boot --efi-directory=/tmp/harddisk/boot/efi Installing for x86_64-efi platform. Installation finished. No error reported. # restore orginal defaults mv -f /tmp/harddisk/etc/default/grub.backup /tmp/harddisk/etc/default/grub rm -f /tmp/harddisk/etc/grub.d/11_linux_scon # Set ramdisk mode to automatic echo RAMDISK_MODE=2 > /tmp/harddisk/etc/sysconfig/ramdisk # Automatically resize the root partition to its maximum size at first boot touch /tmp/harddisk/.partresize # Unmount umount /tmp/harddisk/proc umount /tmp/harddisk/sys umount /tmp/harddisk/dev umount /tmp/harddisk/boot/efi umount /tmp/harddisk/boot umount /tmp/harddisk # zerofree the ext2 images to get better compression zerofree /dev/mapper/loop0p1 fsck.ext2 -f -y /dev/mapper/loop0p1 e2fsck 1.47.0 (5-Feb-2023) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/mapper/loop0p1: 631/130048 files (22.2% non-contiguous), 124148/520192 blocks fsck.ext2 -f -y /dev/mapper/loop0p1 e2fsck 1.47.0 (5-Feb-2023) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/mapper/loop0p1: 631/130048 files (22.2% non-contiguous), 124148/520192 blocks zerofree /dev/mapper/loop0p3 fsck.ext4 -f -y /dev/mapper/loop0p3 e2fsck 1.47.0 (5-Feb-2023) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/mapper/loop0p3: 25477/118080 files (0.1% non-contiguous), 418098/471661 blocks fsck.ext4 -f -y /dev/mapper/loop0p3 e2fsck 1.47.0 (5-Feb-2023) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/mapper/loop0p3: 25477/118080 files (0.1% non-contiguous), 418098/471661 blocks kpartx -d -v /dev/loop0 del devmap : loop0p1 del devmap : loop0p2 del devmap : loop0p3 losetup -d /dev/loop0 # Add padding at the end of the image (to fix alignment issues if the image is # not copied to a block device) dd if=/dev/zero bs=1M count=100 >> /tmp/image.img 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.0387438 s, 2.7 GB/s # Compress Image xz -T0 -8 < /tmp/image.img > /usr/src/images/ipfire-2.29-core187-x86_64.img.xz xz: Reduced the number of threads from 16 to 11 to not exceed the memory usage limit of 7860 MiB rm -rf /tmp/image.img /tmp/harddisk /tmp/cdrom # Create checksum file cd /usr/src/images && b2sum "ipfire-2.29-core187-x86_64.img.xz" > "ipfire-2.29-core187-x86_64.img.xz".b2" /bin/sh: -c: line 1: unexpected EOF while looking for matching `"' make: *** [flash-images:186: /usr/src/log/flash-image] Error 2 make: Leaving directory '/usr/src/lfs'
I fixed the extra quotes:
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=d24f7382fb...
In the images directory there is the iso and img files but only a b2 file for the iso.
Prior to the above section of the log there are about 165 entries similar saying:-
ERROR: ld.so: object '/tools_x86_64/lib/libpakfire_preload.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
This comes from the commands that we run in a second, inner chroot. I will have to check whether we still need all this stuff to fake the environment. QEMU might do most of the job for us when we are emulating a different architecture, and otherwise we don’t need this any more I believe.
Not sure that it has anything to do with the flash-images fail I am having as all those messages say the fail was ignored.
Looking in the flash-images lfs file there looks like
# Create checksum file cd $(IMAGES_DIR) && b2sum "$(notdir $(IMAGE_FILE))" > "$(notdir "$(IMAGE_FILE)").b2"
has "$(IMAGE_FILE)" for the destination while the source only has $(IMAGE_FILE)
So I removed the "" from the destination end and ran the build again.
This time the b2 file was created but now the flash-images failed with
make: Nothing to be done for 'download'. make: Leaving directory '/home/ahb/sandbox/ms/ipfire-2.x/lfs' make: Entering directory '/usr/src/lfs' rm -rf /install/packages/package /tmp/* mkdir -p /install/packages/package eval $(cat /usr/src/config/rootfiles/core/187/meta) cat: /usr/src/config/rootfiles/core/187/meta: No such file or directory #Generate ROOTFILES from filelists BUILD_ARCH=x86_64 BUILDTARGET=x86_64-pc-linux-gnu KVER=6.6.32 \ /usr/src/src/scripts/archive.files \ /usr/src/config/rootfiles/core/187/filelists \ /usr/src/config/rootfiles/core/187/files \ /usr/src/config/rootfiles/core/187/files.x86_64 \
/tmp/ROOTFILES.tmp
#remove excluded files from ROOTFILES grep -f /usr/src/config/rootfiles/core/187/exclude -v /tmp/ROOTFILES.tmp > /tmp/ROOTFILES make: *** [core-updates:60: core/187] Error 1 make: Leaving directory '/usr/src/lfs'
I didn’t look at the packaging before, but this has also been dealt with. Please pull again and re-run the build.
The above might have some relation to not doing a clean before the build. So I will leave the change in the flash-images lfs file and do a clean and a fresh new build
A clean build should not be needed.
-Michael
Regards,
Adolf.
-Michael
Regards,
Adolf.
Regards,
Adolf.
One thing I found. I am running the new build system while I have been running some package updates with the old system with its mount points. The two have each run without any impact on the other.
Regards,
Adolf.
Regards, Adolf.
> > Best, > -Michael > >> On 9 Jul 2024, at 22:29, Michael Tremer michael.tremer@ipfire.org wrote: >> >> Hello Adolf, >> >> Thank you for testing this. >> >> There have indeed been plenty of problems there… I spent a lot of time on this today and hopefully fixed most of them. >> >> I cannot build the toolchain on my machine and I am not sure why yet, but a build with the packaged toolchain runs through. >> >> I have also spent some time on getting rid of the strip stage because it annoyed me how long it takes and creating the disk images as well as packages should be significantly faster now, too. I hope I didn’t introduce too many new bugs. >> >> Please let me know if you have more success now. >> >> Best, >> -Michael >> >>> On 8 Jul 2024, at 20:34, Adolf Belka adolf.belka@ipfire.org wrote: >>> >>> Hi Michael, >>> >>> On 08/07/2024 21:15, Adolf Belka wrote: >>>> Hi Michael, >>>> >>>> On 08/07/2024 18:11, Michael Tremer wrote: >>>>> Hello, >>>>> >>>>> I have been spending a lot of time on this problem, because it has been bothering me for a long time. I also saw an opportunity to make more changes to the build system. >>>>> >>>>> Currently this is all a little bit WIP, but I hope that we can merge this into next as soon as the current update has been moved to master. >>>>> >>>>> I am referring to this branch which is currently based on next: https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/u... >>>>> >>>>> It makes use of the unshare command which creates new namespaces in Linux. That way, we can isolate the build system better from the host system and in case something goes wrong, there is less damage. We can also enforce some more rules… >>>>> >>>>> So, what has changed? >>>>> >>>>> * The make.sh script might re-execute itself into a new mount namespace when it is suitable. This happens for “make.sh build” and “make.sh shell”, but it does not happen for “make.sh downloadsrc” for example. >>>>> >>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... >>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... >>>>> >>>>> * The new mount namespace means that we will no longer see any bind-mounts in the host system and we no longer need to umount anything ourselves which is where we occasionally wiped the entire hard drive of the host system. When the last process exits, the namespace is being cleaned up and everything is being umounted. >>>>> >>>>> * The function that prepares the build environment has been almost entirely rewritten: >>>>> >>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... >>>>> >>>>> It used to mount parts of the host system into the build environment which are needed to run anything. Those were /dev, /proc, /sys, etc… >>>>> >>>>> Instead, it now creates a new /dev mount point and creates a minimal amount of device nodes and symlinks. That way, we detach from the host system and no longer allow the build system access to the host’s filesystem and block devices. We also bind-mount the sources in read-only mode now, so that the build system cannot change anything in the source tree. On top of that, cache is read-only, too. ccache and the log directory are the only places that are writable. >>>>> >>>>> We mount a separate /tmp directory. >>>>> >>>>> * When we then build a package, we create more namespaces for each package. These isolate each build process from each other. >>>>> >>>>> Mostly, this is to detach from the host system. A new UTS namespace allows to change the hostname in the build system without affecting the host and so on. We do the same thing with a new time namespace. >>>>> >>>>> We do however create a new PID namespace which means that the build system no longer will see any processes running on the host system. That requires to mount a new instance of /proc with each package. This also has the effect that if the shell that we launched terminates (because the build is done) any background processes will be killed immediately. >>>>> >>>>> Last, we clone the mount namespace that we have created before so that no build command can modify what we set up earlier. >>>>> >>>>> * Since everything is now so decoupled, we gain a couple of new (maybe minor?) features: >>>>> >>>>> It is now possible to run “make.sh shell” while a build is running. That does not happen a lot, but we can do this now :) >>>>> >>>>> If the build crashes or the host system is being shut down while a build is running, there is nothing to clean up afterwards. >>>>> >>>>> * I have garnished this all with a lot of code cleanup and I suppose I might have introduced some new bugs here or there :) >>>>> >>>>> * This is probably mostly around a new implementation of the timer that updates the build time. It has been annoying me a lot that it takes a long time to walk through all packages that have been built before to finally get to a package that we want to rebuild. Mostly this was all help up by a call of “sleep 0.1” >>>>> >>>>> Since bash does not really do any concurrency, I had to be creative and replaced the busy-loop with a background process that is launched whenever it is needed and which will “ping” the main make.sh script once a second. That way, we can just run as usual, but regularly get interrupted to update the runtime. >>>>> >>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... >>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... >>>>> >>>>> We now only fork one extra sub shell and we have to handle the timer events which is a lot cheaper as well as more straight-forward to code. >>>>> >>>>> * As there is no difference between the different stages any more (those stages that we inherited from LFS), I have merged them all into one. >>>>> >>>>> * Last but not least, I have create the option to build for multiple architectures on the same system. Since we can now mount the entire source tree into (many independent) build environments, we might as well… As discussed on the last call, this might not be the best option for ARM, but RISV-C builds at a decent speed even when emulated. >>>>> >>>>> The only thing that I needed to do for this is to suffix the build and log directories which are now called build_${ARCH}, i.e. build_aarch64, build_x86_64, and so on. The packages/ directory is not changed yet, but that will have to happen as well. Most likely I want to merge this with the generated images, but I am not sure what to call this, yet. Happy to hear suggestions. result_x86_64? Just images_x86_64? >>>>> >>>>> --- >>>>> >>>>> I have run a build and this seems to be working just fine on my Debian machine. I am writing to all of you to first of all let you know what I am up to; and secondly to ask to give this a go on your systems. I think it should run just fine, as all the tools that I require should be available everywhere. However, there might be some older kernels that might not support all of this, yet or any other problems I cannot think of yet. Please give me some feedback and send me all the bugs :) >>>> I gave this a go but it didn't work. >>>> >>>> Not sure if I should have run the ./make.sh clean command on the old version before I pulled the unshare branch into my clone of your repo. >>>> >>>> Should I have started with a complete new clone of your repo? I might try that anyway just to see. >>>> >>> I created a completely new clone of you ipfire-2.x repor and then checked out the unshare branch to a branch called unshare in my local repo clone. >>> >>> gettoolchain gave the same issue, except that this time the toolchain directory ended up completely empty. >>> >>> downloadsrc had the same result. >>> >>> clean had nothing to clean up as it was a fresh clone. >>> >>> build then tried to build the toolchain and came up with this error, different from before. >>> >>> ./make.sh build >>> chroot: failed to run command ‘env’: No such file or directory >>> Full toolchain compilation >>> stage1 [ FAIL ] >>> >>> Jul 8 19:26:39: Building stage1 ====================================== Installing stage1 ... >>> mkdir -pv /tools_x86_64/lib >>> mkdir: cannot create directory '/tools_x86_64': File exists >>> make: *** [stage1:50: /home/ahb/sandbox/ms/ipfire-2.x/log/stage1] Error 1 >>> >>> ERROR: Building stage1 [ FAIL ] >>> Check /home/ahb/sandbox/ms/ipfire-2.x/log_x86_64/_build.toolchain.log for errors if applicable [ FAIL ] >>> >>> so it wasn't as simple as doing a fresh git clone. >>> >>> Regards, >>> >>> Adolf. >>> >>> >>>> So I ran ./make.sh gettoolchain first, as I usually would. >>>> >>>> ./make.sh gettoolchain >>>> b2sum: cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: No such file or directory >>>> cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: FAILED open or read >>>> b2sum: WARNING: 1 listed file could not be read >>>> >>>> ipfire-2.29-toolchain-20240210-x86_64.tar.zst is present together with its b2 file. >>>> >>>> >>>> Then ran ./make.sh downloadsrc >>>> >>>> Previous version ends with >>>> >>>> ***Verifying BLAKE2 checksum >>>> all files BLAKE2 checksum match [ DONE] >>>> >>>> after zstd has been checked. >>>> >>>> New version stops at zstd entry. >>>> >>>> >>>> ./make.sh clean gave the message Cleaning Build directory... but was completed very quickly. >>>> Log and Build directories have not been cleaned out. The img and iso files are still present. >>>> >>>> >>>> ./make.sh build gave message >>>> >>>> chroot: failed to run command ‘env’: No such file or directory >>>> >>>> and then did a full toolchain compilation which failed with gcc but log is >9000 lines. >>>> >>>> >>>> Regards, >>>> Adolf. >>>> >>>>> >>>>> Thank you for listening to this brain-dump. >>>>> >>>>> All the best, >>>>> -Michael >>>>> >>>>>> On 3 Jul 2024, at 10:58, Michael Tremer michael.tremer@ipfire.org wrote: >>>>>> >>>>>> Hello Adolf, >>>>>> >>>>>> This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more. >>>>>> >>>>>> I rebooted the machine and it is back up again. >>>>>> >>>>>> -Michael >>>>>> >>>>>>> On 2 Jul 2024, at 15:42, Adolf Belka adolf.belka@ipfire.org wrote: >>>>>>> >>>>>>> Hi Michael and all, >>>>>>> >>>>>>> >>>>>>> I ran the arm builder with the 4.20.2 version of samba to test it out. >>>>>>> >>>>>>> The build got to building gdb and then failed. >>>>>>> >>>>>>> Interestingly, the nightly build of arm was successful with the same version of gdb. >>>>>>> >>>>>>> The build log for gdb is attached. The actual error is at line 618. >>>>>>> >>>>>>> Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message. >>>>>>> >>>>>>> ------------------------------------------ >>>>>>> >>>>>>> ssh bonnietwin@arm64-01.zrh.ipfire.org >>>>>>> PTY allocation request failed on channel 0 >>>>>>> Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64 >>>>>>> >>>>>>> The programs included with the Debian GNU/Linux system are free software; >>>>>>> the exact distribution terms for each program are described in the >>>>>>> individual files in /usr/share/doc/*/copyright. >>>>>>> >>>>>>> Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent >>>>>>> permitted by applicable law. >>>>>>> /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied >>>>>>> >>>>>>> ------------------------------------------ >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Adolf. >>>>>>> >>>>>>> <_build.ipfire.gdb.log>
Hi Michael,
On 10/07/2024 20:19, Michael Tremer wrote:
Hello,
On 10 Jul 2024, at 17:53, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 10/07/2024 16:24, Michael Tremer wrote:
Hello,
On 10 Jul 2024, at 14:21, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 10/07/2024 15:05, Adolf Belka wrote:
Hi Michael,
On 10/07/2024 14:59, Adolf Belka wrote:
Hi Michael,
On 10/07/2024 12:33, Adolf Belka wrote: > Hi Michael, > > On 10/07/2024 11:57, Michael Tremer wrote: >> Hello again, >> >> I managed to (finally) build the toolchain with the updated system. So hopefully there should not be any more outstanding problems that I know of so far. > > I just did a git pull on your repo to my clone. > > Ran ./make.sh gettoolchain and it successfully downloaded the toolchain. > > Ran ./make.sh downloadsrc and it successfully tested everything. > > Ran ./make.sh clean and build and log directories were cleared out and removed. As far as I can tell it was successful. > > Currently running ./make.sh build. So up to this point everything going well. Will let you know how it goes. > It has got to building popt. In the normal build system this takes around 3 secs. Currently in the new build it is at nearly 2 hours. Even with an empty cache, that seems a long build time for popt, unless I am being too optimistic.
I will let it keep going.
I just had a look at the log file and it looks like it completed popt but then is stuck on trying to leave the directory /usr/src/lfs. Here is the output from the log, nothing new is getting written to the log.
make[3]: Leaving directory '/usr/src/popt-1.19/po' Making install in tests make[3]: Entering directory '/usr/src/popt-1.19/tests' make[4]: Entering directory '/usr/src/popt-1.19/tests' make[4]: Nothing to be done for 'install-exec-am'. make[4]: Nothing to be done for 'install-data-am'. make[4]: Leaving directory '/usr/src/popt-1.19/tests' make[3]: Leaving directory '/usr/src/popt-1.19/tests' make[3]: Entering directory '/usr/src/popt-1.19' make[4]: Entering directory '/usr/src/popt-1.19' make[4]: Nothing to be done for 'install-exec-am'. /bin/mkdir -p '/usr/share/man/man3' /usr/bin/install -c -m 644 popt.3 '/usr/share/man/man3' /bin/mkdir -p '/usr/lib/pkgconfig' /usr/bin/install -c -m 644 popt.pc '/usr/lib/pkgconfig' make[4]: Leaving directory '/usr/src/popt-1.19' make[3]: Leaving directory '/usr/src/popt-1.19' make[2]: Leaving directory '/usr/src/popt-1.19' make[1]: Leaving directory '/usr/src/popt-1.19' Updating linker cache... Install done; saving file list to /usr/src/log/popt-1.19 ...
make: Leaving directory '/usr/src/lfs'
I stopped the build with Ctrl-C and then ran ./make.sh build again without doing a clean. It got to popt and went straight to libedit, the next package, and started to build it.
Something must have just put the build into a loop of some sort. It was not writing anything to the log file.
The log file says that popt was done. It might be that you interrupted the build just after it has finished, but the root file was not created, yet. That should however not directly be a problem…
2 hours had already gone by when I checked the log for that message. I then left the system running for a further 20 minutes with no new info in the log file and that is when I interrupted the build.
Oh, that. Yes, I think I have seen this too, but I am not sure why yet.
It looks like the wait call misses that the process has already been gone here:
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=69e1fd...
If this is however the only problem that you are seeing now than I would be happy :)
I have found another problem when it got to the flash-images section.
The flash-images log stops with the following:-
# Insert the UUID because grub-mkconfig often fails to # detect that correctly sed -i /tmp/harddisk/boot/grub/grub.cfg \ -e "s/root=[A-Za-z0-9/=-]*/root=UUID=$(blkid -o value -s UUID /dev/mapper/loop0p3)/g" # Install GRUB grub-install --force --recheck --no-floppy --target=i386-pc \ --root-directory=/tmp/harddisk /dev/loop0 Installing for i386-pc platform. Installation finished. No error reported. # Install GRUB for EFI grub-install --target=x86_64-efi --removable --no-nvram \ --boot-directory=/tmp/harddisk/boot --efi-directory=/tmp/harddisk/boot/efi Installing for x86_64-efi platform. Installation finished. No error reported. # restore orginal defaults mv -f /tmp/harddisk/etc/default/grub.backup /tmp/harddisk/etc/default/grub rm -f /tmp/harddisk/etc/grub.d/11_linux_scon # Set ramdisk mode to automatic echo RAMDISK_MODE=2 > /tmp/harddisk/etc/sysconfig/ramdisk # Automatically resize the root partition to its maximum size at first boot touch /tmp/harddisk/.partresize # Unmount umount /tmp/harddisk/proc umount /tmp/harddisk/sys umount /tmp/harddisk/dev umount /tmp/harddisk/boot/efi umount /tmp/harddisk/boot umount /tmp/harddisk # zerofree the ext2 images to get better compression zerofree /dev/mapper/loop0p1 fsck.ext2 -f -y /dev/mapper/loop0p1 e2fsck 1.47.0 (5-Feb-2023) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/mapper/loop0p1: 631/130048 files (22.2% non-contiguous), 124148/520192 blocks fsck.ext2 -f -y /dev/mapper/loop0p1 e2fsck 1.47.0 (5-Feb-2023) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/mapper/loop0p1: 631/130048 files (22.2% non-contiguous), 124148/520192 blocks zerofree /dev/mapper/loop0p3 fsck.ext4 -f -y /dev/mapper/loop0p3 e2fsck 1.47.0 (5-Feb-2023) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/mapper/loop0p3: 25477/118080 files (0.1% non-contiguous), 418098/471661 blocks fsck.ext4 -f -y /dev/mapper/loop0p3 e2fsck 1.47.0 (5-Feb-2023) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/mapper/loop0p3: 25477/118080 files (0.1% non-contiguous), 418098/471661 blocks kpartx -d -v /dev/loop0 del devmap : loop0p1 del devmap : loop0p2 del devmap : loop0p3 losetup -d /dev/loop0 # Add padding at the end of the image (to fix alignment issues if the image is # not copied to a block device) dd if=/dev/zero bs=1M count=100 >> /tmp/image.img 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.0387438 s, 2.7 GB/s # Compress Image xz -T0 -8 < /tmp/image.img > /usr/src/images/ipfire-2.29-core187-x86_64.img.xz xz: Reduced the number of threads from 16 to 11 to not exceed the memory usage limit of 7860 MiB rm -rf /tmp/image.img /tmp/harddisk /tmp/cdrom # Create checksum file cd /usr/src/images && b2sum "ipfire-2.29-core187-x86_64.img.xz" > "ipfire-2.29-core187-x86_64.img.xz".b2" /bin/sh: -c: line 1: unexpected EOF while looking for matching `"' make: *** [flash-images:186: /usr/src/log/flash-image] Error 2 make: Leaving directory '/usr/src/lfs'
I fixed the extra quotes:
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=d24f7382fb...
In the images directory there is the iso and img files but only a b2 file for the iso.
Prior to the above section of the log there are about 165 entries similar saying:-
ERROR: ld.so: object '/tools_x86_64/lib/libpakfire_preload.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
This comes from the commands that we run in a second, inner chroot. I will have to check whether we still need all this stuff to fake the environment. QEMU might do most of the job for us when we are emulating a different architecture, and otherwise we don’t need this any more I believe.
Not sure that it has anything to do with the flash-images fail I am having as all those messages say the fail was ignored.
Looking in the flash-images lfs file there looks like
# Create checksum file cd $(IMAGES_DIR) && b2sum "$(notdir $(IMAGE_FILE))" > "$(notdir "$(IMAGE_FILE)").b2"
has "$(IMAGE_FILE)" for the destination while the source only has $(IMAGE_FILE)
So I removed the "" from the destination end and ran the build again.
This time the b2 file was created but now the flash-images failed with
make: Nothing to be done for 'download'. make: Leaving directory '/home/ahb/sandbox/ms/ipfire-2.x/lfs' make: Entering directory '/usr/src/lfs' rm -rf /install/packages/package /tmp/* mkdir -p /install/packages/package eval $(cat /usr/src/config/rootfiles/core/187/meta) cat: /usr/src/config/rootfiles/core/187/meta: No such file or directory #Generate ROOTFILES from filelists BUILD_ARCH=x86_64 BUILDTARGET=x86_64-pc-linux-gnu KVER=6.6.32 \ /usr/src/src/scripts/archive.files \ /usr/src/config/rootfiles/core/187/filelists \ /usr/src/config/rootfiles/core/187/files \ /usr/src/config/rootfiles/core/187/files.x86_64 \ > /tmp/ROOTFILES.tmp #remove excluded files from ROOTFILES grep -f /usr/src/config/rootfiles/core/187/exclude -v /tmp/ROOTFILES.tmp > /tmp/ROOTFILES make: *** [core-updates:60: core/187] Error 1 make: Leaving directory '/usr/src/lfs'I didn’t look at the packaging before, but this has also been dealt with. Please pull again and re-run the build.
The above might have some relation to not doing a clean before the build. So I will leave the change in the flash-images lfs file and do a clean and a fresh new build
I did a git pull and then did a full clean and build and this time the whole process was run through without any fails.
Regards,
Adolf.
A clean build should not be needed.
-Michael
Regards,
Adolf.
-Michael
Regards,
Adolf.
Regards,
Adolf.
One thing I found. I am running the new build system while I have been running some package updates with the old system with its mount points. The two have each run without any impact on the other.
Regards,
Adolf.
> Regards, > Adolf. > >> >> Best, >> -Michael >> >>> On 9 Jul 2024, at 22:29, Michael Tremer michael.tremer@ipfire.org wrote: >>> >>> Hello Adolf, >>> >>> Thank you for testing this. >>> >>> There have indeed been plenty of problems there… I spent a lot of time on this today and hopefully fixed most of them. >>> >>> I cannot build the toolchain on my machine and I am not sure why yet, but a build with the packaged toolchain runs through. >>> >>> I have also spent some time on getting rid of the strip stage because it annoyed me how long it takes and creating the disk images as well as packages should be significantly faster now, too. I hope I didn’t introduce too many new bugs. >>> >>> Please let me know if you have more success now. >>> >>> Best, >>> -Michael >>> >>>> On 8 Jul 2024, at 20:34, Adolf Belka adolf.belka@ipfire.org wrote: >>>> >>>> Hi Michael, >>>> >>>> On 08/07/2024 21:15, Adolf Belka wrote: >>>>> Hi Michael, >>>>> >>>>> On 08/07/2024 18:11, Michael Tremer wrote: >>>>>> Hello, >>>>>> >>>>>> I have been spending a lot of time on this problem, because it has been bothering me for a long time. I also saw an opportunity to make more changes to the build system. >>>>>> >>>>>> Currently this is all a little bit WIP, but I hope that we can merge this into next as soon as the current update has been moved to master. >>>>>> >>>>>> I am referring to this branch which is currently based on next: https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/u... >>>>>> >>>>>> It makes use of the unshare command which creates new namespaces in Linux. That way, we can isolate the build system better from the host system and in case something goes wrong, there is less damage. We can also enforce some more rules… >>>>>> >>>>>> So, what has changed? >>>>>> >>>>>> * The make.sh script might re-execute itself into a new mount namespace when it is suitable. This happens for “make.sh build” and “make.sh shell”, but it does not happen for “make.sh downloadsrc” for example. >>>>>> >>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... >>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... >>>>>> >>>>>> * The new mount namespace means that we will no longer see any bind-mounts in the host system and we no longer need to umount anything ourselves which is where we occasionally wiped the entire hard drive of the host system. When the last process exits, the namespace is being cleaned up and everything is being umounted. >>>>>> >>>>>> * The function that prepares the build environment has been almost entirely rewritten: >>>>>> >>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... >>>>>> >>>>>> It used to mount parts of the host system into the build environment which are needed to run anything. Those were /dev, /proc, /sys, etc… >>>>>> >>>>>> Instead, it now creates a new /dev mount point and creates a minimal amount of device nodes and symlinks. That way, we detach from the host system and no longer allow the build system access to the host’s filesystem and block devices. We also bind-mount the sources in read-only mode now, so that the build system cannot change anything in the source tree. On top of that, cache is read-only, too. ccache and the log directory are the only places that are writable. >>>>>> >>>>>> We mount a separate /tmp directory. >>>>>> >>>>>> * When we then build a package, we create more namespaces for each package. These isolate each build process from each other. >>>>>> >>>>>> Mostly, this is to detach from the host system. A new UTS namespace allows to change the hostname in the build system without affecting the host and so on. We do the same thing with a new time namespace. >>>>>> >>>>>> We do however create a new PID namespace which means that the build system no longer will see any processes running on the host system. That requires to mount a new instance of /proc with each package. This also has the effect that if the shell that we launched terminates (because the build is done) any background processes will be killed immediately. >>>>>> >>>>>> Last, we clone the mount namespace that we have created before so that no build command can modify what we set up earlier. >>>>>> >>>>>> * Since everything is now so decoupled, we gain a couple of new (maybe minor?) features: >>>>>> >>>>>> It is now possible to run “make.sh shell” while a build is running. That does not happen a lot, but we can do this now :) >>>>>> >>>>>> If the build crashes or the host system is being shut down while a build is running, there is nothing to clean up afterwards. >>>>>> >>>>>> * I have garnished this all with a lot of code cleanup and I suppose I might have introduced some new bugs here or there :) >>>>>> >>>>>> * This is probably mostly around a new implementation of the timer that updates the build time. It has been annoying me a lot that it takes a long time to walk through all packages that have been built before to finally get to a package that we want to rebuild. Mostly this was all help up by a call of “sleep 0.1” >>>>>> >>>>>> Since bash does not really do any concurrency, I had to be creative and replaced the busy-loop with a background process that is launched whenever it is needed and which will “ping” the main make.sh script once a second. That way, we can just run as usual, but regularly get interrupted to update the runtime. >>>>>> >>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... >>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b95262... >>>>>> >>>>>> We now only fork one extra sub shell and we have to handle the timer events which is a lot cheaper as well as more straight-forward to code. >>>>>> >>>>>> * As there is no difference between the different stages any more (those stages that we inherited from LFS), I have merged them all into one. >>>>>> >>>>>> * Last but not least, I have create the option to build for multiple architectures on the same system. Since we can now mount the entire source tree into (many independent) build environments, we might as well… As discussed on the last call, this might not be the best option for ARM, but RISV-C builds at a decent speed even when emulated. >>>>>> >>>>>> The only thing that I needed to do for this is to suffix the build and log directories which are now called build_${ARCH}, i.e. build_aarch64, build_x86_64, and so on. The packages/ directory is not changed yet, but that will have to happen as well. Most likely I want to merge this with the generated images, but I am not sure what to call this, yet. Happy to hear suggestions. result_x86_64? Just images_x86_64? >>>>>> >>>>>> --- >>>>>> >>>>>> I have run a build and this seems to be working just fine on my Debian machine. I am writing to all of you to first of all let you know what I am up to; and secondly to ask to give this a go on your systems. I think it should run just fine, as all the tools that I require should be available everywhere. However, there might be some older kernels that might not support all of this, yet or any other problems I cannot think of yet. Please give me some feedback and send me all the bugs :) >>>>> I gave this a go but it didn't work. >>>>> >>>>> Not sure if I should have run the ./make.sh clean command on the old version before I pulled the unshare branch into my clone of your repo. >>>>> >>>>> Should I have started with a complete new clone of your repo? I might try that anyway just to see. >>>>> >>>> I created a completely new clone of you ipfire-2.x repor and then checked out the unshare branch to a branch called unshare in my local repo clone. >>>> >>>> gettoolchain gave the same issue, except that this time the toolchain directory ended up completely empty. >>>> >>>> downloadsrc had the same result. >>>> >>>> clean had nothing to clean up as it was a fresh clone. >>>> >>>> build then tried to build the toolchain and came up with this error, different from before. >>>> >>>> ./make.sh build >>>> chroot: failed to run command ‘env’: No such file or directory >>>> Full toolchain compilation >>>> stage1 [ FAIL ] >>>> >>>> Jul 8 19:26:39: Building stage1 ====================================== Installing stage1 ... >>>> mkdir -pv /tools_x86_64/lib >>>> mkdir: cannot create directory '/tools_x86_64': File exists >>>> make: *** [stage1:50: /home/ahb/sandbox/ms/ipfire-2.x/log/stage1] Error 1 >>>> >>>> ERROR: Building stage1 [ FAIL ] >>>> Check /home/ahb/sandbox/ms/ipfire-2.x/log_x86_64/_build.toolchain.log for errors if applicable [ FAIL ] >>>> >>>> so it wasn't as simple as doing a fresh git clone. >>>> >>>> Regards, >>>> >>>> Adolf. >>>> >>>> >>>>> So I ran ./make.sh gettoolchain first, as I usually would. >>>>> >>>>> ./make.sh gettoolchain >>>>> b2sum: cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: No such file or directory >>>>> cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: FAILED open or read >>>>> b2sum: WARNING: 1 listed file could not be read >>>>> >>>>> ipfire-2.29-toolchain-20240210-x86_64.tar.zst is present together with its b2 file. >>>>> >>>>> >>>>> Then ran ./make.sh downloadsrc >>>>> >>>>> Previous version ends with >>>>> >>>>> ***Verifying BLAKE2 checksum >>>>> all files BLAKE2 checksum match [ DONE] >>>>> >>>>> after zstd has been checked. >>>>> >>>>> New version stops at zstd entry. >>>>> >>>>> >>>>> ./make.sh clean gave the message Cleaning Build directory... but was completed very quickly. >>>>> Log and Build directories have not been cleaned out. The img and iso files are still present. >>>>> >>>>> >>>>> ./make.sh build gave message >>>>> >>>>> chroot: failed to run command ‘env’: No such file or directory >>>>> >>>>> and then did a full toolchain compilation which failed with gcc but log is >9000 lines. >>>>> >>>>> >>>>> Regards, >>>>> Adolf. >>>>> >>>>>> >>>>>> Thank you for listening to this brain-dump. >>>>>> >>>>>> All the best, >>>>>> -Michael >>>>>> >>>>>>> On 3 Jul 2024, at 10:58, Michael Tremer michael.tremer@ipfire.org wrote: >>>>>>> >>>>>>> Hello Adolf, >>>>>>> >>>>>>> This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more. >>>>>>> >>>>>>> I rebooted the machine and it is back up again. >>>>>>> >>>>>>> -Michael >>>>>>> >>>>>>>> On 2 Jul 2024, at 15:42, Adolf Belka adolf.belka@ipfire.org wrote: >>>>>>>> >>>>>>>> Hi Michael and all, >>>>>>>> >>>>>>>> >>>>>>>> I ran the arm builder with the 4.20.2 version of samba to test it out. >>>>>>>> >>>>>>>> The build got to building gdb and then failed. >>>>>>>> >>>>>>>> Interestingly, the nightly build of arm was successful with the same version of gdb. >>>>>>>> >>>>>>>> The build log for gdb is attached. The actual error is at line 618. >>>>>>>> >>>>>>>> Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message. >>>>>>>> >>>>>>>> ------------------------------------------ >>>>>>>> >>>>>>>> ssh bonnietwin@arm64-01.zrh.ipfire.org >>>>>>>> PTY allocation request failed on channel 0 >>>>>>>> Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64 >>>>>>>> >>>>>>>> The programs included with the Debian GNU/Linux system are free software; >>>>>>>> the exact distribution terms for each program are described in the >>>>>>>> individual files in /usr/share/doc/*/copyright. >>>>>>>> >>>>>>>> Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent >>>>>>>> permitted by applicable law. >>>>>>>> /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied >>>>>>>> >>>>>>>> ------------------------------------------ >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Adolf. >>>>>>>> >>>>>>>> <_build.ipfire.gdb.log>
-
Adolf Belka -
Michael Tremer