Problem during building of samba on arm builder

public inbox for development@lists.ipfire.org
 help / color / mirror / Atom feed

* Problem during building of samba on arm builder
@ 2024-07-02 14:42 Adolf Belka
  2024-07-03  9:58 ` Michael Tremer
  0 siblings, 1 reply; 16+ messages in thread
From: Adolf Belka @ 2024-07-02 14:42 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 1179 bytes --]

Hi Michael and all,


I ran the arm builder with the 4.20.2 version of samba to test it out.

The build got to building gdb and then failed.

Interestingly, the nightly build of arm was successful with the same version of gdb.

The build log for gdb is attached. The actual error is at line 618.

Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message.

------------------------------------------

ssh bonnietwin(a)arm64-01.zrh.ipfire.org
PTY allocation request failed on channel 0
Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
/etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied

------------------------------------------

Regards,

Adolf.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: _build.ipfire.gdb.log --]
[-- Type: text/x-log, Size: 36943 bytes --]



Jul  2 13:58:24: Building gdb gdb-14.2.tar.xz checksum OK
+ cd /usr/src/lfs
+ make -f gdb LFS_BASEDIR=/usr/src install
====================================== Installing gdb-14.2 ...
Install started; saving file list to /usr/src/lsalr ...
cd /usr/src/gdb-14.2 && mkdir -pv build
mkdir: created directory 'build'
cd /usr/src/gdb-14.2/build && \
	../configure \
		--prefix=/usr \
		--sysconfdir=/etc \
		--disable-nls \
		--enable-tui \
		--with-system-readline \
		--with-python=/usr/bin/python3
checking build system type... aarch64-unknown-linux-gnu
checking host system type... aarch64-unknown-linux-gnu
checking target system type... aarch64-unknown-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether ln works... yes
checking whether ln -s works... yes
checking for a sed that does not truncate output... /bin/sed
checking for gawk... gawk
checking for gdbserver support... yes
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for gcc option to accept ISO C99... none needed
checking for g++... g++
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking whether g++ accepts -static-libstdc++ -static-libgcc... yes
checking for gnatbind... no
checking for gnatmake... no
checking whether compiler driver understands Ada and is recent enough... no
checking for gdc... no
checking whether the D compiler works... no
checking how to compare bootstrapped objects... cmp --ignore-initial=16 $$f1 $$f2
checking for objdir... .libs
checking for the correct version of gmp.h... yes
checking for the correct version of mpfr.h... yes
checking for the correct version of the gmp/mpfr libraries... yes
checking for isl 0.15 or later... no
required isl version is 0.15 or later
*** This configuration is not supported in the following subdirectories:
     readline
    (Any other directories should still work fine.)
checking for default BUILD_CONFIG... 
checking for --enable-vtable-verify... no
checking for bison... bison -y
checking for bison... bison
checking for gm4... no
checking for gnum4... no
checking for m4... m4
checking for flex... flex
checking for flex... flex
checking for makeinfo... makeinfo
checking for expect... expect
checking for runtest... runtest
checking for ar... ar
checking for as... as
checking for dlltool... no
checking for dsymutil... no
checking for ld... ld
checking for lipo... no
checking for nm... nm
checking for ranlib... ranlib
checking for strip... strip
checking for windres... no
checking for windmc... no
checking for objcopy... objcopy
checking for objdump... objdump
checking for otool... no
checking for readelf... readelf
checking for -plugin option... --plugin /usr/lib/gcc/aarch64-unknown-linux-gnu/13.3.0/liblto_plugin.so
checking for cc... cc
checking for c++... c++
checking for gcc... gcc
checking for gfortran... no
checking for gccgo... no
checking for gdc... no
checking for gm2... no
checking for ar... ar
checking for as... as
checking for dlltool... no
checking for dsymutil... no
checking for ld... ld
checking for lipo... no
checking for nm... nm
checking for objcopy... objcopy
checking for objdump... objdump
checking for otool... no
checking for ranlib... ranlib
checking for readelf... readelf
checking for strip... strip
checking for windres... no
checking for windmc... no
checking where to find the target ar... host tool
checking where to find the target as... host tool
checking where to find the target cc... host tool
checking where to find the target c++... host tool
checking where to find the target c++ for libstdc++... host tool
checking where to find the target dlltool... host tool
checking where to find the target dsymutil... host tool
checking where to find the target gcc... host tool
checking where to find the target gfortran... host tool
checking where to find the target gccgo... host tool
checking where to find the target gdc... host tool
checking where to find the target gm2... host tool
checking where to find the target ld... host tool
checking where to find the target lipo... host tool
checking where to find the target nm... host tool
checking where to find the target objcopy... host tool
checking where to find the target objdump... host tool
checking where to find the target otool... host tool
checking where to find the target ranlib... host tool
checking where to find the target readelf... host tool
checking where to find the target strip... host tool
checking where to find the target windres... host tool
checking where to find the target windmc... host tool
checking whether to enable maintainer-specific portions of Makefiles... no
configure: creating ./config.status
config.status: creating Makefile
cd /usr/src/gdb-14.2/build && make -j4
make[1]: Entering directory '/usr/src/gdb-14.2/build'
make[2]: Entering directory '/usr/src/gdb-14.2/build'
mkdir -p -- ./libiberty
mkdir -p -- ./intl
mkdir -p -- ./zlib
mkdir -p -- ./libsframe
Configuring in ./libiberty
Configuring in ./intl
Configuring in ./zlib
Configuring in ./libsframe
configure: creating cache ./config.cache
configure: creating cache ./config.cache
configure: creating cache ./config.cache
checking whether to enable maintainer-specific portions of Makefiles... no
checking for makeinfo... makeinfo --split-size=5000000
configure: creating cache ./config.cache
checking for aarch64-unknown-linux-gnu-gcc... gcc
checking for aarch64-unknown-linux-gnu-gcc... gcc
checking build system type... aarch64-unknown-linux-gnu
checking host system type... aarch64-unknown-linux-gnu
checking target system type... checking whether the C compiler works... aarch64-unknown-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for perl... perl
checking for a thread-safe mkdir -p... checking whether the C compiler works... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... checking build system type... yes
aarch64-unknown-linux-gnu
checking host system type... checking whether make supports nested variables... aarch64-unknown-linux-gnu
checking for aarch64-unknown-linux-gnu-ar... ar --plugin /usr/lib/gcc/aarch64-unknown-linux-gnu/13.3.0/liblto_plugin.so
yes
checking for aarch64-unknown-linux-gnu-ranlib... ranlib --plugin /usr/lib/gcc/aarch64-unknown-linux-gnu/13.3.0/liblto_plugin.so
checking for -plugin option... checking for aarch64-unknown-linux-gnu-ar... (cached) ar --plugin /usr/lib/gcc/aarch64-unknown-linux-gnu/13.3.0/liblto_plugin.so
yes
checking for C compiler default output file name... a.out
checking for suffix of executables... checking whether make supports nested variables... (cached) yes
checking whether to enable maintainer-specific portions of Makefiles... no
checking for aarch64-unknown-linux-gnu-gcc... gcc
--plugin /usr/lib/gcc/aarch64-unknown-linux-gnu/13.3.0/liblto_plugin.so
checking whether to install libiberty headers and static library... no
configure: target_header_dir = 
checking for aarch64-unknown-linux-gnu-gcc... gcc
yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... 
checking whether we are cross compiling... checking whether the C compiler works... no
checking for suffix of object files... checking whether the C compiler works... o
checking whether we are using the GNU C compiler... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... no
checking for suffix of object files... yes
checking whether gcc accepts -g... yes
checking for C compiler default output file name... a.out
yes
checking for gcc option to accept ISO C89... checking for suffix of executables... o
checking whether we are using the GNU C compiler... 
checking whether we are cross compiling... none needed
checking whether gcc understands -c and -o together... yes
checking whether gcc accepts -g... yes
checking how to run the C preprocessor... yes
checking for gcc option to accept ISO C89... 
checking whether we are cross compiling... none needed
checking how to run the C preprocessor... no
checking for suffix of object files... gcc -E
o
checking whether we are using the GNU C compiler... no
checking for suffix of object files... gcc -E
yes
checking whether gcc accepts -g... checking for grep that handles long lines and -e... o
checking whether we are using the GNU C compiler... /bin/grep
checking for egrep... yes
checking for gcc option to accept ISO C89... /bin/grep -E
checking for ANSI C header files... yes
checking whether gcc accepts -g... none needed
checking whether gcc understands -c and -o together... yes
checking for gcc option to accept ISO C89... yes
checking for grep that handles long lines and -e... checking for style of include used by make... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... GNU
checking dependency style of gcc... none needed
checking how to run the C preprocessor... gcc3
checking how to print strings... printf
checking for a sed that does not truncate output... /bin/sed
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for fgrep... /bin/grep -F
checking for ld used by gcc... ld
checking if the linker (ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... gcc -E
/tools_aarch64/bin/nm -B
checking the name lister (/tools_aarch64/bin/nm -B) interface... checking for grep that handles long lines and -e... yes
/bin/grep
checking for egrep... checking for sys/types.h... /bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... checking for sys/stat.h... yes
checking for ld option to reload object files... -r
checking for aarch64-unknown-linux-gnu-objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for aarch64-unknown-linux-gnu-ar... ar --plugin /usr/lib/gcc/aarch64-unknown-linux-gnu/13.3.0/liblto_plugin.so
yes
checking for aarch64-unknown-linux-gnu-strip... no
checking for strip... strip
checking for aarch64-unknown-linux-gnu-ranlib... ranlib --plugin /usr/lib/gcc/aarch64-unknown-linux-gnu/13.3.0/liblto_plugin.so
checking command to parse /tools_aarch64/bin/nm -B output from gcc object... checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for stdlib.h... yes
checking for string.h... yes
ok
checking how to run the C preprocessor... checking for string.h... yes
checking for sys/types.h... yes
checking for memory.h... yes
yes
gcc -E
checking for memory.h... checking for sys/stat.h... yes
checking for strings.h... yes
yes
checking for stdlib.h... yes
checking for strings.h... checking for inttypes.h... checking for ANSI C header files... yes
yes
checking for string.h... yes
checking for inttypes.h... checking for stdint.h... yes
yes
checking for memory.h... checking for stdint.h... yes
yes
yes
checking for unistd.h... checking for strings.h... checking for unistd.h... yes
yes
yes
checking for sys/types.h... checking minix/config.h usability... checking for inttypes.h... yes
checking minix/config.h usability... yes
yes
checking for stdint.h... checking for sys/stat.h... yes
no
checking minix/config.h presence... checking for unistd.h... no
checking minix/config.h presence... yes
no
checking for minix/config.h... no
checking whether it is safe to define __EXTENSIONS__... checking for stdlib.h... no
checking for minix/config.h... no
checking whether it is safe to define __EXTENSIONS__... yes
checking minix/config.h usability... yes
checking for string.h... yes
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
yes
checking whether make sets $(MAKE)... checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking for a BSD-compatible install... /usr/bin/install -c
checking whether NLS is requested... no
yes
checking for msgfmt... checking for style of include used by make... yes
GNU
/usr/bin/msgfmt
checking for gmsgfmt... /usr/bin/msgfmt
checking whether make supports nested variables... checking for xgettext... yes
checking for memory.h... checking dependency style of gcc... /usr/bin/xgettext
checking for msgmerge... /usr/bin/msgmerge
no
checking minix/config.h presence... gcc3
yes
checking whether make supports nested variables... (cached) yes
checking whether make sets $(MAKE)... checking build system type... (cached) yes
checking for aarch64-unknown-linux-gnu-gcc... (cached) gcc
checking for strings.h... aarch64-unknown-linux-gnu
checking host system type... no
checking for minix/config.h... no
checking whether it is safe to define __EXTENSIONS__... aarch64-unknown-linux-gnu
checking for aarch64-unknown-linux-gnu-ranlib... ranlib --plugin /usr/lib/gcc/aarch64-unknown-linux-gnu/13.3.0/liblto_plugin.so
checking for library containing strerror... yes
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for gcc option to accept ISO C89... (cached) none needed
checking whether gcc understands -c and -o together... (cached) yes
checking for aarch64-unknown-linux-gnu-ranlib... ranlib --plugin /usr/lib/gcc/aarch64-unknown-linux-gnu/13.3.0/liblto_plugin.so
checking for aarch64-unknown-linux-gnu-ar... ar --plugin /usr/lib/gcc/aarch64-unknown-linux-gnu/13.3.0/liblto_plugin.so
checking the archiver (ar --plugin /usr/lib/gcc/aarch64-unknown-linux-gnu/13.3.0/liblto_plugin.so) interface... checking for inttypes.h... yes
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... ar
none required
checking for an ANSI C-conforming const... no
checking build system type... checking whether gcc supports -W... yes
aarch64-unknown-linux-gnu
checking host system type... yes
checking for inline... checking for stdint.h... yes
aarch64-unknown-linux-gnu
checking how to print strings... checking whether gcc supports -Wall... printf
checking for a sed that does not truncate output... yes
/bin/sed
checking for fgrep... checking whether gcc supports -Wwrite-strings... inline
checking for off_t... /bin/grep -F
checking for ld used by gcc... ld
checking if the linker (ld) is GNU ld... yes
yes
checking for BSD- or MS-compatible name lister (nm)... yes
checking whether gcc supports -Wc++-compat... checking for unistd.h... yes
checking whether gcc supports -Wstrict-prototypes... yes
checking whether gcc supports -Wshadow=local... yes
checking for dlfcn.h... /tools_aarch64/bin/nm -B
checking the name lister (/tools_aarch64/bin/nm -B) interface... yes
checking whether gcc supports -pedantic ... yes
checking whether gcc and cc understand -c and -o together... yes
BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... checking for objdir... .libs
yes
checking for size_t... 1572864
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking for ld option to reload object files... -r
checking for aarch64-unknown-linux-gnu-objdump... objdump
checking how to recognize dependent libraries... pass_all
yes
checking for an ANSI C-conforming const... checking for aarch64-unknown-linux-gnu-ar... (cached) ar --plugin /usr/lib/gcc/aarch64-unknown-linux-gnu/13.3.0/liblto_plugin.so
checking for aarch64-unknown-linux-gnu-strip... no
checking for strip... strip
checking for aarch64-unknown-linux-gnu-ranlib... (cached) ranlib --plugin /usr/lib/gcc/aarch64-unknown-linux-gnu/13.3.0/liblto_plugin.so
checking command to parse /tools_aarch64/bin/nm -B output from gcc object... yes
checking for inline... inline
checking whether byte ordering is bigendian... checking if gcc supports -fno-rtti -fno-exceptions... yes
checking for working alloca.h... ok
no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... checking for dlfcn.h... yes
checking if gcc static flag -static works... yes
yes
checking for alloca... checking for objdir... .libs
no
checking for a BSD-compatible install... /usr/bin/install -c
checking for CET support... no
yes
checking if gcc supports -c -o file.o... checking for sys/file.h... yes
checking for stdlib.h... (cached) yes
checking for unistd.h... (cached) yes
checking if gcc supports -fno-rtti -fno-exceptions... checking for sys/param.h... yes
checking for sys/param.h... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (ld) supports shared libraries... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking whether -lc should be explicitly linked in... yes
yes
checking for limits.h... checking for getpagesize... yes
checking if gcc static flag -static works... yes
checking for stdlib.h... (cached) yes
no
checking dynamic linker characteristics... checking for malloc.h... yes
checking for working mmap... yes
checking for string.h... (cached) yes
checking for unistd.h... (cached) yes
yes
checking if gcc supports -c -o file.o... checking for strings.h... (cached) yes
checking for sys/time.h... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking how to run the C preprocessor... gcc -E
yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (ld) supports shared libraries... yes
checking for time.h... yes
checking dynamic linker characteristics... yes
checking whether we are using the GNU C Library 2.1 or newer... yes
checking for stdlib.h... (cached) yes
checking for sys/resource.h... checking for unistd.h... (cached) yes
yes
checking whether integer division by zero raises SIGFPE... checking for sys/param.h... yes
checking for sys/stat.h... (cached) yes
checking for sys/mman.h... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
yes
yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... no
checking whether to build static libraries... yes
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... checking for getpagesize... checking for fcntl.h... no
no
checking for inttypes.h... checking for aclocal... ${SHELL} /usr/src/gdb-14.2/missing aclocal-1.15
checking for autoconf... ${SHELL} /usr/src/gdb-14.2/missing autoconf
checking for autoheader... ${SHELL} /usr/src/gdb-14.2/missing autoheader
yes
checking whether gcc supports -Wall... checking for alloca.h... yes
checking whether gcc supports -Wextra... yes
yes
checking for working mmap... checking for stdint.h... yes
yes
checking whether gcc supports -Wwrite-strings... yes
checking for sys/pstat.h... checking whether gcc supports -Wmissing-format-attribute... yes
yes
checking for unsigned long long... checking whether gcc supports -Wstrict-prototypes... yes
no
checking whether gcc supports -Wmissing-prototypes... checking for sys/sysmp.h... yes
checking for incompatibility between DejaGnu and GCC... 
no
checking for sys/sysinfo.h... yes
checking for inttypes.h... no
checking for makeinfo... makeinfo --split-size=5000000
checking if using Solaris linker... no
yes
yes
checking for memcpy... yes
checking whether to enable maintainer-specific portions of Makefiles... no
checking whether to install libbfd... yes
checking for stdlib.h... (cached) yes
yes
checking whether the inttypes.h PRIxNN macros are broken... checking for machine/hal_sysinfo.h... checking for unistd.h... (cached) yes
no
checking for ld used by GCC... no
checking for sys/param.h... ld
checking if the linker (ld) is GNU ld... checking for sys/table.h... yes
checking for shared library run path origin... egrep: warning: egrep is obsolescent; using grep -E
yes
no
checking for strerror... done
checking for sys/sysctl.h... yes
checking argz.h usability... checking for getpagesize... no
yes
checking argz.h presence... checking for sys/systemcfg.h... yes
checking for unistd.h... (cached) yes
yes
checking for working mmap... yes
checking for argz.h... yes
no
checking limits.h usability... checking for stdint.h... (cached) yes
checking for stdio_ext.h... configure: updating cache ./config.cache
checking that generated files are newer than configure... done
configure: creating ./config.status
yes
checking for process.h... yes
checking limits.h presence... no
yes
checking for limits.h... yes
checking for sys/prctl.h... checking locale.h usability... yes
yes
checking byteswap.h usability... checking for sys/wait.h that is POSIX.1 compatible... yes
checking locale.h presence... yes
checking byteswap.h presence... yes
checking whether time.h and sys/time.h may both be included... yes
checking for locale.h... yes
yes
checking for byteswap.h... yes
checking nl_types.h usability... yes
checking whether errno must be declared... checking endian.h usability... yes
checking nl_types.h presence... no
checking size of int... yes
checking endian.h presence... yes
checking for nl_types.h... yes
checking malloc.h usability... yes
checking for endian.h... yes
checking whether bswap_16 is declared... yes
yes
checking malloc.h presence... checking whether bswap_32 is declared... 4
checking size of long... yes
yes
checking for malloc.h... yes
checking whether bswap_64 is declared... checking stddef.h usability... yes
yes
checking stddef.h presence... yes
checking for stddef.h... yes
configure: updating cache ./config.cache
checking that generated files are newer than configure... checking for stdlib.h... (cached) yes
done
8
configure: creating ./config.status
checking size of size_t... checking for string.h... (cached) yes
checking for unistd.h... (cached) yes
checking for sys/param.h... (cached) yes
checking for feof_unlocked... 8
checking for long long... yes
checking for fgets_unlocked... yes
checking for getc_unlocked... yes
checking size of long long... yes
checking for getcwd... 8
checking for a 64-bit type... yes
checking for getegid... uint64_t
checking for intptr_t... yes
checking for geteuid... config.status: creating Makefile
config.status: executing depfiles commands
yes
checking for getgid... yes
checking for uintptr_t... config.status: executing libtool commands
yes
checking for getuid... yes
checking for ssize_t... mkdir -p -- ./etc
yes
Configuring in ./etc
checking for mempcpy... yes
checking for munmap... yes
checking for pid_t... config.status: creating Makefile
yes
config.status: creating config.h
checking for putenv... config.status: executing depfiles commands
configure: creating cache ./config.cache
checking whether to enable maintainer-specific portions of Makefiles... no
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
yes
config.status: executing libtool commands
checking whether make supports nested variables... (cached) yes
checking for setenv... configure: updating cache ./config.cache
checking that generated files are newer than configure... yes
checking for library containing strerror... yes
checking for setlocale... mkdir -p -- ./libbacktrace
Configuring in ./libbacktrace
none required
checking for asprintf... yes
checking for stpcpy... yes
checking for atexit... yes
checking for strcasecmp... yes
checking for basename... configure: creating cache ./config.cache
checking build system type... aarch64-unknown-linux-gnu
checking host system type... aarch64-unknown-linux-gnu
checking target system type... aarch64-unknown-linux-gnu
checking for aarch64-unknown-linux-gnu-gcc... gcc
yes
checking for strdup... yes
checking for bcmp... checking whether the C compiler works... yes
yes
checking for strtoul... yes
checking for C compiler default output file name... a.out
checking for bcopy... checking for suffix of executables... 
checking whether we are cross compiling... yes
yes
checking for tsearch... checking for bsearch... no
checking for suffix of object files... yes
yes
checking for bzero... checking for __argz_count... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... yes
yes
checking for __argz_stringify... checking for calloc... none needed
checking whether gcc understands -c and -o together... yes
checking how to run the C preprocessor... yes
yes
checking for clock... checking for __argz_next... gcc -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... yes
yes
/bin/grep -E
checking for ANSI C header files... checking for __fsetlocking... checking for ffs... yes
checking for iconv... yes
checking for getcwd... yes
yes
checking for iconv declaration... checking for getpagesize... yes
checking for sys/types.h... 
         extern size_t iconv (iconv_t cd, char * *inbuf, size_t *inbytesleft, char * *outbuf, size_t *outbytesleft);
checking for nl_langinfo and CODESET... yes
checking for sys/stat.h... yes
checking for gettimeofday... yes
yes
checking for LC_MESSAGES... checking for stdlib.h... yes
checking for index... yes
checking for string.h... yes
checking for bison... bison
checking version of bison... 3.8.2, ok
checking whether NLS is requested... no
checking whether to use NLS... no
checking for aclocal... aclocal
checking for autoconf... autoconf
checking for autoheader... autoheader
checking bison 3 or later... 3.8.2, bison3
yes
checking for memory.h... done
configure: creating ./config.status
configure: updating cache ./config.cache
configure: creating ./config.status
yes
checking for insque... yes
checking for strings.h... yes
yes
checking for memchr... checking for inttypes.h... ./config.status: line 518: 0a1,126: command not found
./config.status: line 519: syntax error near unexpected token `newline'
./config.status: line 519: `> # This file is a shell script that caches the results of configure'
config.status: creating Makefile
yes
checking for stdint.h... config.status: creating config.intl
yes
make[2]: *** [Makefile:4356: configure-etc] Error 1
make[2]: *** Waiting for unfinished jobs....
checking for memcmp... config.status: creating config.h
config.status: executing default-1 commands
yes
checking for unistd.h... yes
yes
checking for memcpy... checking minix/config.h usability... yes
checking for memmem... no
checking minix/config.h presence... no
checking for minix/config.h... no
checking whether it is safe to define __EXTENSIONS__... yes
checking for memmove... yes
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether to enable maintainer-specific portions of Makefiles... no
yes
checking for aarch64-unknown-linux-gnu-gcc... (cached) gcc
checking for mempcpy... checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for gcc option to accept ISO C89... (cached) none needed
checking whether gcc understands -c and -o together... (cached) yes
checking for aarch64-unknown-linux-gnu-ranlib... ranlib --plugin /usr/lib/gcc/aarch64-unknown-linux-gnu/13.3.0/liblto_plugin.so
checking for gawk... (cached) gawk
checking for dwz... no
checking how to print strings... printf
checking for a sed that does not truncate output... /bin/sed
checking for fgrep... /bin/grep -F
checking for ld used by gcc... ld
checking if the linker (ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... yes
checking for memset... /tools_aarch64/bin/nm -B
checking the name lister (/tools_aarch64/bin/nm -B) interface... yes
checking for mkstemps... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking for ld option to reload object files... -r
checking for aarch64-unknown-linux-gnu-objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for aarch64-unknown-linux-gnu-ar... ar --plugin /usr/lib/gcc/aarch64-unknown-linux-gnu/13.3.0/liblto_plugin.so
checking for aarch64-unknown-linux-gnu-strip... no
checking for strip... strip
checking for aarch64-unknown-linux-gnu-ranlib... (cached) ranlib --plugin /usr/lib/gcc/aarch64-unknown-linux-gnu/13.3.0/liblto_plugin.so
checking command to parse /tools_aarch64/bin/nm -B output from gcc object... yes
checking for putenv... ok
checking for dlfcn.h... yes
checking for random... yes
checking for objdir... .libs
yes
checking for rename... checking if gcc supports -fno-rtti -fno-exceptions... yes
checking for rindex... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... yes
checking for setenv... yes
checking if gcc supports -c -o file.o... yes
checking for snprintf... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (ld) supports shared libraries... yes
checking whether -lc should be explicitly linked in... yes
checking for sigsetmask... no
checking dynamic linker characteristics... yes
checking for stpcpy... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking unwind.h usability... yes
checking for stpncpy... yes
checking unwind.h presence... yes
checking for unwind.h... yes
checking for _Unwind_Backtrace... yes
checking for strcasecmp... yes
checking for -funwind-tables option... yes
yes
checking for -frandom-seed=string option... checking for strchr... yes
checking whether gcc supports -W... yes
checking whether gcc supports -Wall... yes
checking whether gcc supports -Wwrite-strings... yes
yes
checking for strdup... checking whether gcc supports -Wstrict-prototypes... yes
checking whether gcc supports -Wmissing-prototypes... yes
checking whether gcc supports -Wold-style-definition... yes
checking whether gcc supports -Wmissing-format-attribute... yes
yes
checking for strncasecmp... checking whether gcc supports -Wcast-qual... yes
checking for _Unwind_GetIPInfo... yes
yes
checking for CET support... no
checking __sync extensions... checking for strndup... yes
checking __atomic extensions... yes
checking for strnlen... yes
checking output filetype... yes
checking for strrchr... elf64
looking for a compliant stdint.h in stdint.h, checking for uintmax_t... yes
checking for strstr... yes
checking for uintptr_t... yes
checking for strtod... yes
checking for int_least32_t... yes
checking for strtol... yes
yes
checking for int_fast32_t... checking for strtoul... yes
checking for strtoll... yes
checking for uint64_t... yes
checking for strtoull... yes
checking what to include in gstdint.h... stdint.h (already complete)
checking sys/mman.h usability... yes
checking for strverscmp... yes
checking sys/mman.h presence... yes
checking for sys/mman.h... yes
checking for mmap... yes
checking for tmpnam... yes
checking link.h usability... yes
checking for vasprintf... yes
checking link.h presence... yes
checking for link.h... yes
checking for dl_iterate_phdr... yes
checking for vfprintf... yes
checking mach-o/dyld.h usability... yes
checking for vprintf... yes
no
checking mach-o/dyld.h presence... checking for vsnprintf... no
checking for mach-o/dyld.h... no
checking sys/ldr.h usability... yes
checking for vsprintf... no
checking sys/ldr.h presence... yes
checking for waitpid... no
checking for sys/ldr.h... no
checking for fcntl... yes
yes
checking whether strnlen is declared... checking for setproctitle... yes
checking whether getpagesize is declared... no
checking whether alloca needs Cray hooks... no
checking stack direction for C alloca... yes
checking for lstat... yes
-1
checking for vfork.h... checking for readlink... no
checking for fork... yes
checking for getexecname... yes
checking for vfork... no
checking for KERN_PROC... yes
checking for working fork... no
checking for KERN_PROG_ARGS... no
checking for clock_gettime... yes
checking for working vfork... (cached) yes
checking for _doprnt... yes
checking whether -pthread is supported... no
checking for sys_errlist... yes
checking whether -gdwarf-5 is supported... no
checking for sys_nerr... yes
checking for compress in -lz... yes
checking whether --compress-debug-sections is supported... no
checking for sys_siglist... yes
checking for objcopy... objcopy
checking for readelf... readelf
checking whether objcopy supports debuglink... yes
checking for dsymutil... dsymutil
checking for nm... /tools_aarch64/bin/nm -B
checking for xz... xz
checking for comm... comm
checking for lzma_auto_decoder in -llzma... no
checking for external symbol _system_configuration... no
checking for __fsetlocking... yes
checking whether tests can run... yes
yes
checking for canonicalize_file_name... configure: updating cache ./config.cache
checking that generated files are newer than configure... done
configure: creating ./config.status
yes
checking for dup3... yes
checking for getrlimit... yes
checking for getrusage... yes
checking for getsysinfo... no
checking for gettimeofday... (cached) yes
checking for on_exit... yes
checking for pipe2... yes
checking for psignal... yes
checking for pstat_getdynamic... no
checking for pstat_getstatic... config.status: creating Makefile
config.status: creating backtrace-supported.h
config.status: creating install-debuginfo-for-buildid.sh
no
checking for realpath... config.status: creating config.h
config.status: executing libtool commands
config.status: executing gstdint.h commands
yes
config.status: executing default commands
checking for setrlimit... yes
checking for spawnve... no
checking for spawnvpe... no
checking for strerror... yes
checking for strsignal... yes
checking for sysconf... yes
checking for sysctl... no
checking for sysmp... no
checking for table... no
checking for times... yes
checking for wait3... yes
checking for wait4... yes
checking for sbrk... yes
checking whether basename is declared... yes
checking whether ffs is declared... yes
checking whether asprintf is declared... yes
checking whether vasprintf is declared... yes
checking whether snprintf is declared... yes
checking whether vsnprintf is declared... yes
checking whether calloc is declared... yes
checking whether getenv is declared... yes
checking whether getopt is declared... yes
checking whether malloc is declared... yes
checking whether realloc is declared... yes
checking whether sbrk is declared... yes
checking whether strtol is declared... yes
checking whether strtoul is declared... yes
checking whether strtoll is declared... yes
checking whether strtoull is declared... yes
checking whether strverscmp is declared... yes
checking whether strnlen is declared... yes
checking whether canonicalize_file_name must be declared... no
checking for stdlib.h... (cached) yes
checking for unistd.h... (cached) yes
checking for sys/param.h... (cached) yes
checking for getpagesize... (cached) yes
checking for working mmap... yes
checking for working strncmp... no
configure: updating cache ./config.cache
configure: creating ./config.status
config.status: creating Makefile
config.status: creating testsuite/Makefile
config.status: creating config.h
config.status: executing default commands
make[2]: Leaving directory '/usr/src/gdb-14.2/build'
make[1]: *** [Makefile:1021: all] Error 2
make[1]: Leaving directory '/usr/src/gdb-14.2/build'
make: *** [gdb:75: /usr/src/log/gdb-14.2] Error 2

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: Problem during building of samba on arm builder
  2024-07-02 14:42 Problem during building of samba on arm builder Adolf Belka
@ 2024-07-03  9:58 ` Michael Tremer
  2024-07-03 20:44   ` Adolf Belka
  2024-07-08 16:11   ` Michael Tremer
  0 siblings, 2 replies; 16+ messages in thread
From: Michael Tremer @ 2024-07-03  9:58 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 1538 bytes --]

Hello Adolf,

This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more.

I rebooted the machine and it is back up again.

-Michael

> On 2 Jul 2024, at 15:42, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi Michael and all,
> 
> 
> I ran the arm builder with the 4.20.2 version of samba to test it out.
> 
> The build got to building gdb and then failed.
> 
> Interestingly, the nightly build of arm was successful with the same version of gdb.
> 
> The build log for gdb is attached. The actual error is at line 618.
> 
> Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message.
> 
> ------------------------------------------
> 
> ssh bonnietwin(a)arm64-01.zrh.ipfire.org
> PTY allocation request failed on channel 0
> Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64
> 
> The programs included with the Debian GNU/Linux system are free software;
> the exact distribution terms for each program are described in the
> individual files in /usr/share/doc/*/copyright.
> 
> Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
> permitted by applicable law.
> /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied
> 
> ------------------------------------------
> 
> Regards,
> 
> Adolf.
> 
> <_build.ipfire.gdb.log>


^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: Problem during building of samba on arm builder
  2024-07-03  9:58 ` Michael Tremer
@ 2024-07-03 20:44   ` Adolf Belka
  2024-07-08 16:11   ` Michael Tremer
  1 sibling, 0 replies; 16+ messages in thread
From: Adolf Belka @ 2024-07-03 20:44 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 1708 bytes --]

Worked okay this time round.

Samba build took 3.5 hours.

Regards,

Adolf.


On 03/07/2024 11:58, Michael Tremer wrote:
> Hello Adolf,
> 
> This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more.
> 
> I rebooted the machine and it is back up again.
> 
> -Michael
> 
>> On 2 Jul 2024, at 15:42, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>
>> Hi Michael and all,
>>
>>
>> I ran the arm builder with the 4.20.2 version of samba to test it out.
>>
>> The build got to building gdb and then failed.
>>
>> Interestingly, the nightly build of arm was successful with the same version of gdb.
>>
>> The build log for gdb is attached. The actual error is at line 618.
>>
>> Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message.
>>
>> ------------------------------------------
>>
>> ssh bonnietwin(a)arm64-01.zrh.ipfire.org
>> PTY allocation request failed on channel 0
>> Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64
>>
>> The programs included with the Debian GNU/Linux system are free software;
>> the exact distribution terms for each program are described in the
>> individual files in /usr/share/doc/*/copyright.
>>
>> Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
>> permitted by applicable law.
>> /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied
>>
>> ------------------------------------------
>>
>> Regards,
>>
>> Adolf.
>>
>> <_build.ipfire.gdb.log>
> 

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: Problem during building of samba on arm builder
  2024-07-03  9:58 ` Michael Tremer
  2024-07-03 20:44   ` Adolf Belka
@ 2024-07-08 16:11   ` Michael Tremer
  2024-07-08 19:15     ` Adolf Belka
  1 sibling, 1 reply; 16+ messages in thread
From: Michael Tremer @ 2024-07-08 16:11 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 7991 bytes --]

Hello,

I have been spending a lot of time on this problem, because it has been bothering me for a long time. I also saw an opportunity to make more changes to the build system.

Currently this is all a little bit WIP, but I hope that we can merge this into next as soon as the current update has been moved to master.

I am referring to this branch which is currently based on next: https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/unshare

It makes use of the unshare command which creates new namespaces in Linux. That way, we can isolate the build system better from the host system and in case something goes wrong, there is less damage. We can also enforce some more rules…

So, what has changed?

* The make.sh script might re-execute itself into a new mount namespace when it is suitable. This happens for “make.sh build” and “make.sh shell”, but it does not happen for “make.sh downloadsrc” for example.

  https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l2129
  https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l2251

* The new mount namespace means that we will no longer see any bind-mounts in the host system and we no longer need to umount anything ourselves which is where we occasionally wiped the entire hard drive of the host system. When the last process exits, the namespace is being cleaned up and everything is being umounted.

* The function that prepares the build environment has been almost entirely rewritten:

  https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l426

It used to mount parts of the host system into the build environment which are needed to run anything. Those were /dev, /proc, /sys, etc…

Instead, it now creates a new /dev mount point and creates a minimal amount of device nodes and symlinks. That way, we detach from the host system and no longer allow the build system access to the host’s filesystem and block devices. We also bind-mount the sources in read-only mode now, so that the build system cannot change anything in the source tree. On top of that, cache is read-only, too. ccache and the log directory are the only places that are writable.

We mount a separate /tmp directory.

* When we then build a package, we create more namespaces for each package. These isolate each build process from each other.

Mostly, this is to detach from the host system. A new UTS namespace allows to change the hostname in the build system without affecting the host and so on. We do the same thing with a new time namespace.

We do however create a new PID namespace which means that the build system no longer will see any processes running on the host system. That requires to mount a new instance of /proc with each package. This also has the effect that if the shell that we launched terminates (because the build is done) any background processes will be killed immediately.

Last, we clone the mount namespace that we have created before so that no build command can modify what we set up earlier.

* Since everything is now so decoupled, we gain a couple of new (maybe minor?) features:

  It is now possible to run “make.sh shell” while a build is running. That does not happen a lot, but we can do this now :)

  If the build crashes or the host system is being shut down while a build is running, there is nothing to clean up afterwards.

* I have garnished this all with a lot of code cleanup and I suppose I might have introduced some new bugs here or there :)

* This is probably mostly around a new implementation of the timer that updates the build time. It has been annoying me a lot that it takes a long time to walk through all packages that have been built before to finally get to a package that we want to rebuild. Mostly this was all help up by a call of “sleep 0.1”

Since bash does not really do any concurrency, I had to be creative and replaced the busy-loop with a background process that is launched whenever it is needed and which will “ping” the main make.sh script once a second. That way, we can just run as usual, but regularly get interrupted to update the runtime.

  https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l361
  https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l834

We now only fork one extra sub shell and we have to handle the timer events which is a lot cheaper as well as more straight-forward to code.

* As there is no difference between the different stages any more (those stages that we inherited from LFS), I have merged them all into one.

* Last but not least, I have create the option to build for multiple architectures on the same system. Since we can now mount the entire source tree into (many independent) build environments, we might as well… As discussed on the last call, this might not be the best option for ARM, but RISV-C builds at a decent speed even when emulated.

The only thing that I needed to do for this is to suffix the build and log directories which are now called build_${ARCH}, i.e. build_aarch64, build_x86_64, and so on. The packages/ directory is not changed yet, but that will have to happen as well. Most likely I want to merge this with the generated images, but I am not sure what to call this, yet. Happy to hear suggestions. result_x86_64? Just images_x86_64?

---

I have run a build and this seems to be working just fine on my Debian machine. I am writing to all of you to first of all let you know what I am up to; and secondly to ask to give this a go on your systems. I think it should run just fine, as all the tools that I require should be available everywhere. However, there might be some older kernels that might not support all of this, yet or any other problems I cannot think of yet. Please give me some feedback and send me all the bugs :)

Thank you for listening to this brain-dump.

All the best,
-Michael

> On 3 Jul 2024, at 10:58, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
> 
> Hello Adolf,
> 
> This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more.
> 
> I rebooted the machine and it is back up again.
> 
> -Michael
> 
>> On 2 Jul 2024, at 15:42, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>> 
>> Hi Michael and all,
>> 
>> 
>> I ran the arm builder with the 4.20.2 version of samba to test it out.
>> 
>> The build got to building gdb and then failed.
>> 
>> Interestingly, the nightly build of arm was successful with the same version of gdb.
>> 
>> The build log for gdb is attached. The actual error is at line 618.
>> 
>> Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message.
>> 
>> ------------------------------------------
>> 
>> ssh bonnietwin(a)arm64-01.zrh.ipfire.org
>> PTY allocation request failed on channel 0
>> Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64
>> 
>> The programs included with the Debian GNU/Linux system are free software;
>> the exact distribution terms for each program are described in the
>> individual files in /usr/share/doc/*/copyright.
>> 
>> Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
>> permitted by applicable law.
>> /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied
>> 
>> ------------------------------------------
>> 
>> Regards,
>> 
>> Adolf.
>> 
>> <_build.ipfire.gdb.log>
> 

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: Problem during building of samba on arm builder
  2024-07-08 16:11   ` Michael Tremer
@ 2024-07-08 19:15     ` Adolf Belka
  2024-07-08 19:34       ` Adolf Belka
  0 siblings, 1 reply; 16+ messages in thread
From: Adolf Belka @ 2024-07-08 19:15 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 9586 bytes --]

Hi Michael,

On 08/07/2024 18:11, Michael Tremer wrote:
> Hello,
> 
> I have been spending a lot of time on this problem, because it has been bothering me for a long time. I also saw an opportunity to make more changes to the build system.
> 
> Currently this is all a little bit WIP, but I hope that we can merge this into next as soon as the current update has been moved to master.
> 
> I am referring to this branch which is currently based on next: https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/unshare
> 
> It makes use of the unshare command which creates new namespaces in Linux. That way, we can isolate the build system better from the host system and in case something goes wrong, there is less damage. We can also enforce some more rules…
> 
> So, what has changed?
> 
> * The make.sh script might re-execute itself into a new mount namespace when it is suitable. This happens for “make.sh build” and “make.sh shell”, but it does not happen for “make.sh downloadsrc” for example.
> 
>    https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l2129
>    https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l2251
> 
> * The new mount namespace means that we will no longer see any bind-mounts in the host system and we no longer need to umount anything ourselves which is where we occasionally wiped the entire hard drive of the host system. When the last process exits, the namespace is being cleaned up and everything is being umounted.
> 
> * The function that prepares the build environment has been almost entirely rewritten:
> 
>    https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l426
> 
> It used to mount parts of the host system into the build environment which are needed to run anything. Those were /dev, /proc, /sys, etc…
> 
> Instead, it now creates a new /dev mount point and creates a minimal amount of device nodes and symlinks. That way, we detach from the host system and no longer allow the build system access to the host’s filesystem and block devices. We also bind-mount the sources in read-only mode now, so that the build system cannot change anything in the source tree. On top of that, cache is read-only, too. ccache and the log directory are the only places that are writable.
> 
> We mount a separate /tmp directory.
> 
> * When we then build a package, we create more namespaces for each package. These isolate each build process from each other.
> 
> Mostly, this is to detach from the host system. A new UTS namespace allows to change the hostname in the build system without affecting the host and so on. We do the same thing with a new time namespace.
> 
> We do however create a new PID namespace which means that the build system no longer will see any processes running on the host system. That requires to mount a new instance of /proc with each package. This also has the effect that if the shell that we launched terminates (because the build is done) any background processes will be killed immediately.
> 
> Last, we clone the mount namespace that we have created before so that no build command can modify what we set up earlier.
> 
> * Since everything is now so decoupled, we gain a couple of new (maybe minor?) features:
> 
>    It is now possible to run “make.sh shell” while a build is running. That does not happen a lot, but we can do this now :)
> 
>    If the build crashes or the host system is being shut down while a build is running, there is nothing to clean up afterwards.
> 
> * I have garnished this all with a lot of code cleanup and I suppose I might have introduced some new bugs here or there :)
> 
> * This is probably mostly around a new implementation of the timer that updates the build time. It has been annoying me a lot that it takes a long time to walk through all packages that have been built before to finally get to a package that we want to rebuild. Mostly this was all help up by a call of “sleep 0.1”
> 
> Since bash does not really do any concurrency, I had to be creative and replaced the busy-loop with a background process that is launched whenever it is needed and which will “ping” the main make.sh script once a second. That way, we can just run as usual, but regularly get interrupted to update the runtime.
> 
>    https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l361
>    https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l834
> 
> We now only fork one extra sub shell and we have to handle the timer events which is a lot cheaper as well as more straight-forward to code.
> 
> * As there is no difference between the different stages any more (those stages that we inherited from LFS), I have merged them all into one.
> 
> * Last but not least, I have create the option to build for multiple architectures on the same system. Since we can now mount the entire source tree into (many independent) build environments, we might as well… As discussed on the last call, this might not be the best option for ARM, but RISV-C builds at a decent speed even when emulated.
> 
> The only thing that I needed to do for this is to suffix the build and log directories which are now called build_${ARCH}, i.e. build_aarch64, build_x86_64, and so on. The packages/ directory is not changed yet, but that will have to happen as well. Most likely I want to merge this with the generated images, but I am not sure what to call this, yet. Happy to hear suggestions. result_x86_64? Just images_x86_64?
> 
> ---
> 
> I have run a build and this seems to be working just fine on my Debian machine. I am writing to all of you to first of all let you know what I am up to; and secondly to ask to give this a go on your systems. I think it should run just fine, as all the tools that I require should be available everywhere. However, there might be some older kernels that might not support all of this, yet or any other problems I cannot think of yet. Please give me some feedback and send me all the bugs :)
I gave this a go but it didn't work.

Not sure if I should have run the ./make.sh clean command on the old version before I pulled the unshare branch into my clone of your repo.

Should I have started with a complete new clone of your repo? I might try that anyway just to see.

So I ran ./make.sh gettoolchain first, as I usually would.

./make.sh gettoolchain
b2sum: cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: No such file or directory
cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: FAILED open or read
b2sum: WARNING: 1 listed file could not be read

ipfire-2.29-toolchain-20240210-x86_64.tar.zst is present together with its b2 file.


Then ran ./make.sh downloadsrc

Previous version ends with

***Verifying BLAKE2 checksum
all files BLAKE2 checksum match                                                       [ DONE]

after zstd has been checked.

New version stops at zstd entry.


./make.sh clean gave the message Cleaning Build directory... but was completed very quickly.
Log and Build directories have not been cleaned out. The img and iso files are still present.


./make.sh build gave message

chroot: failed to run command ‘env’: No such file or directory

and then did a full toolchain compilation which failed with gcc but log is >9000 lines.


Regards,
Adolf.

> 
> Thank you for listening to this brain-dump.
> 
> All the best,
> -Michael
> 
>> On 3 Jul 2024, at 10:58, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>
>> Hello Adolf,
>>
>> This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more.
>>
>> I rebooted the machine and it is back up again.
>>
>> -Michael
>>
>>> On 2 Jul 2024, at 15:42, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>
>>> Hi Michael and all,
>>>
>>>
>>> I ran the arm builder with the 4.20.2 version of samba to test it out.
>>>
>>> The build got to building gdb and then failed.
>>>
>>> Interestingly, the nightly build of arm was successful with the same version of gdb.
>>>
>>> The build log for gdb is attached. The actual error is at line 618.
>>>
>>> Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message.
>>>
>>> ------------------------------------------
>>>
>>> ssh bonnietwin(a)arm64-01.zrh.ipfire.org
>>> PTY allocation request failed on channel 0
>>> Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64
>>>
>>> The programs included with the Debian GNU/Linux system are free software;
>>> the exact distribution terms for each program are described in the
>>> individual files in /usr/share/doc/*/copyright.
>>>
>>> Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
>>> permitted by applicable law.
>>> /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied
>>>
>>> ------------------------------------------
>>>
>>> Regards,
>>>
>>> Adolf.
>>>
>>> <_build.ipfire.gdb.log>
>>
> 

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: Problem during building of samba on arm builder
  2024-07-08 19:15     ` Adolf Belka
@ 2024-07-08 19:34       ` Adolf Belka
  2024-07-09 21:29         ` Michael Tremer
  0 siblings, 1 reply; 16+ messages in thread
From: Adolf Belka @ 2024-07-08 19:34 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 10940 bytes --]

Hi Michael,

On 08/07/2024 21:15, Adolf Belka wrote:
> Hi Michael,
>
> On 08/07/2024 18:11, Michael Tremer wrote:
>> Hello,
>>
>> I have been spending a lot of time on this problem, because it has been bothering me for a long time. I also saw an opportunity to make more changes to the build system.
>>
>> Currently this is all a little bit WIP, but I hope that we can merge this into next as soon as the current update has been moved to master.
>>
>> I am referring to this branch which is currently based on next: https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/unshare
>>
>> It makes use of the unshare command which creates new namespaces in Linux. That way, we can isolate the build system better from the host system and in case something goes wrong, there is less damage. We can also enforce some more rules…
>>
>> So, what has changed?
>>
>> * The make.sh script might re-execute itself into a new mount namespace when it is suitable. This happens for “make.sh build” and “make.sh shell”, but it does not happen for “make.sh downloadsrc” for example.
>>
>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l2129
>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l2251
>>
>> * The new mount namespace means that we will no longer see any bind-mounts in the host system and we no longer need to umount anything ourselves which is where we occasionally wiped the entire hard drive of the host system. When the last process exits, the namespace is being cleaned up and everything is being umounted.
>>
>> * The function that prepares the build environment has been almost entirely rewritten:
>>
>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l426
>>
>> It used to mount parts of the host system into the build environment which are needed to run anything. Those were /dev, /proc, /sys, etc…
>>
>> Instead, it now creates a new /dev mount point and creates a minimal amount of device nodes and symlinks. That way, we detach from the host system and no longer allow the build system access to the host’s filesystem and block devices. We also bind-mount the sources in read-only mode now, so that the build system cannot change anything in the source tree. On top of that, cache is read-only, too. ccache and the log directory are the only places that are writable.
>>
>> We mount a separate /tmp directory.
>>
>> * When we then build a package, we create more namespaces for each package. These isolate each build process from each other.
>>
>> Mostly, this is to detach from the host system. A new UTS namespace allows to change the hostname in the build system without affecting the host and so on. We do the same thing with a new time namespace.
>>
>> We do however create a new PID namespace which means that the build system no longer will see any processes running on the host system. That requires to mount a new instance of /proc with each package. This also has the effect that if the shell that we launched terminates (because the build is done) any background processes will be killed immediately.
>>
>> Last, we clone the mount namespace that we have created before so that no build command can modify what we set up earlier.
>>
>> * Since everything is now so decoupled, we gain a couple of new (maybe minor?) features:
>>
>>    It is now possible to run “make.sh shell” while a build is running. That does not happen a lot, but we can do this now :)
>>
>>    If the build crashes or the host system is being shut down while a build is running, there is nothing to clean up afterwards.
>>
>> * I have garnished this all with a lot of code cleanup and I suppose I might have introduced some new bugs here or there :)
>>
>> * This is probably mostly around a new implementation of the timer that updates the build time. It has been annoying me a lot that it takes a long time to walk through all packages that have been built before to finally get to a package that we want to rebuild. Mostly this was all help up by a call of “sleep 0.1”
>>
>> Since bash does not really do any concurrency, I had to be creative and replaced the busy-loop with a background process that is launched whenever it is needed and which will “ping” the main make.sh script once a second. That way, we can just run as usual, but regularly get interrupted to update the runtime.
>>
>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l361
>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l834
>>
>> We now only fork one extra sub shell and we have to handle the timer events which is a lot cheaper as well as more straight-forward to code.
>>
>> * As there is no difference between the different stages any more (those stages that we inherited from LFS), I have merged them all into one.
>>
>> * Last but not least, I have create the option to build for multiple architectures on the same system. Since we can now mount the entire source tree into (many independent) build environments, we might as well… As discussed on the last call, this might not be the best option for ARM, but RISV-C builds at a decent speed even when emulated.
>>
>> The only thing that I needed to do for this is to suffix the build and log directories which are now called build_${ARCH}, i.e. build_aarch64, build_x86_64, and so on. The packages/ directory is not changed yet, but that will have to happen as well. Most likely I want to merge this with the generated images, but I am not sure what to call this, yet. Happy to hear suggestions. result_x86_64? Just images_x86_64?
>>
>> ---
>>
>> I have run a build and this seems to be working just fine on my Debian machine. I am writing to all of you to first of all let you know what I am up to; and secondly to ask to give this a go on your systems. I think it should run just fine, as all the tools that I require should be available everywhere. However, there might be some older kernels that might not support all of this, yet or any other problems I cannot think of yet. Please give me some feedback and send me all the bugs :)
> I gave this a go but it didn't work.
>
> Not sure if I should have run the ./make.sh clean command on the old version before I pulled the unshare branch into my clone of your repo.
>
> Should I have started with a complete new clone of your repo? I might try that anyway just to see.
>
I created a completely new clone of you ipfire-2.x repor and then checked out the unshare branch to a branch called unshare in my local repo clone.

gettoolchain gave the same issue, except that this time the toolchain directory ended up completely empty.

downloadsrc had the same result.

clean had nothing to clean up as it was a fresh clone.

build then tried to build the toolchain and came up with this error, different from before.

./make.sh build
chroot: failed to run command ‘env’: No such file or directory
Full toolchain compilation
stage1 [ FAIL ]

     Jul  8 19:26:39: Building stage1 ====================================== Installing stage1 ...
     mkdir -pv /tools_x86_64/lib
     mkdir: cannot create directory '/tools_x86_64': File exists
     make: *** [stage1:50: /home/ahb/sandbox/ms/ipfire-2.x/log/stage1] Error 1

ERROR: Building stage1 [ FAIL ]
     Check /home/ahb/sandbox/ms/ipfire-2.x/log_x86_64/_build.toolchain.log for errors if applicable [ FAIL ]

so it wasn't as simple as doing a fresh git clone.

Regards,

Adolf.


> So I ran ./make.sh gettoolchain first, as I usually would.
>
> ./make.sh gettoolchain
> b2sum: cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: No such file or directory
> cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: FAILED open or read
> b2sum: WARNING: 1 listed file could not be read
>
> ipfire-2.29-toolchain-20240210-x86_64.tar.zst is present together with its b2 file.
>
>
> Then ran ./make.sh downloadsrc
>
> Previous version ends with
>
> ***Verifying BLAKE2 checksum
> all files BLAKE2 checksum match                                                       [ DONE]
>
> after zstd has been checked.
>
> New version stops at zstd entry.
>
>
> ./make.sh clean gave the message Cleaning Build directory... but was completed very quickly.
> Log and Build directories have not been cleaned out. The img and iso files are still present.
>
>
> ./make.sh build gave message
>
> chroot: failed to run command ‘env’: No such file or directory
>
> and then did a full toolchain compilation which failed with gcc but log is >9000 lines.
>
>
> Regards,
> Adolf.
>
>>
>> Thank you for listening to this brain-dump.
>>
>> All the best,
>> -Michael
>>
>>> On 3 Jul 2024, at 10:58, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>
>>> Hello Adolf,
>>>
>>> This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more.
>>>
>>> I rebooted the machine and it is back up again.
>>>
>>> -Michael
>>>
>>>> On 2 Jul 2024, at 15:42, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>
>>>> Hi Michael and all,
>>>>
>>>>
>>>> I ran the arm builder with the 4.20.2 version of samba to test it out.
>>>>
>>>> The build got to building gdb and then failed.
>>>>
>>>> Interestingly, the nightly build of arm was successful with the same version of gdb.
>>>>
>>>> The build log for gdb is attached. The actual error is at line 618.
>>>>
>>>> Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message.
>>>>
>>>> ------------------------------------------
>>>>
>>>> ssh bonnietwin(a)arm64-01.zrh.ipfire.org
>>>> PTY allocation request failed on channel 0
>>>> Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64
>>>>
>>>> The programs included with the Debian GNU/Linux system are free software;
>>>> the exact distribution terms for each program are described in the
>>>> individual files in /usr/share/doc/*/copyright.
>>>>
>>>> Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
>>>> permitted by applicable law.
>>>> /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied
>>>>
>>>> ------------------------------------------
>>>>
>>>> Regards,
>>>>
>>>> Adolf.
>>>>
>>>> <_build.ipfire.gdb.log>
>>>
>>

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: Problem during building of samba on arm builder
  2024-07-08 19:34       ` Adolf Belka
@ 2024-07-09 21:29         ` Michael Tremer
  2024-07-10  9:57           ` Michael Tremer
  0 siblings, 1 reply; 16+ messages in thread
From: Michael Tremer @ 2024-07-09 21:29 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 11845 bytes --]

Hello Adolf,

Thank you for testing this.

There have indeed been plenty of problems there… I spent a lot of time on this today and hopefully fixed most of them.

I cannot build the toolchain on my machine and I am not sure why yet, but a build with the packaged toolchain runs through.

I have also spent some time on getting rid of the strip stage because it annoyed me how long it takes and creating the disk images as well as packages should be significantly faster now, too. I hope I didn’t introduce too many new bugs.

Please let me know if you have more success now.

Best,
-Michael

> On 8 Jul 2024, at 20:34, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi Michael,
> 
> On 08/07/2024 21:15, Adolf Belka wrote:
>> Hi Michael,
>> 
>> On 08/07/2024 18:11, Michael Tremer wrote:
>>> Hello,
>>> 
>>> I have been spending a lot of time on this problem, because it has been bothering me for a long time. I also saw an opportunity to make more changes to the build system.
>>> 
>>> Currently this is all a little bit WIP, but I hope that we can merge this into next as soon as the current update has been moved to master.
>>> 
>>> I am referring to this branch which is currently based on next: https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/unshare
>>> 
>>> It makes use of the unshare command which creates new namespaces in Linux. That way, we can isolate the build system better from the host system and in case something goes wrong, there is less damage. We can also enforce some more rules…
>>> 
>>> So, what has changed?
>>> 
>>> * The make.sh script might re-execute itself into a new mount namespace when it is suitable. This happens for “make.sh build” and “make.sh shell”, but it does not happen for “make.sh downloadsrc” for example.
>>> 
>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l2129
>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l2251
>>> 
>>> * The new mount namespace means that we will no longer see any bind-mounts in the host system and we no longer need to umount anything ourselves which is where we occasionally wiped the entire hard drive of the host system. When the last process exits, the namespace is being cleaned up and everything is being umounted.
>>> 
>>> * The function that prepares the build environment has been almost entirely rewritten:
>>> 
>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l426
>>> 
>>> It used to mount parts of the host system into the build environment which are needed to run anything. Those were /dev, /proc, /sys, etc…
>>> 
>>> Instead, it now creates a new /dev mount point and creates a minimal amount of device nodes and symlinks. That way, we detach from the host system and no longer allow the build system access to the host’s filesystem and block devices. We also bind-mount the sources in read-only mode now, so that the build system cannot change anything in the source tree. On top of that, cache is read-only, too. ccache and the log directory are the only places that are writable.
>>> 
>>> We mount a separate /tmp directory.
>>> 
>>> * When we then build a package, we create more namespaces for each package. These isolate each build process from each other.
>>> 
>>> Mostly, this is to detach from the host system. A new UTS namespace allows to change the hostname in the build system without affecting the host and so on. We do the same thing with a new time namespace.
>>> 
>>> We do however create a new PID namespace which means that the build system no longer will see any processes running on the host system. That requires to mount a new instance of /proc with each package. This also has the effect that if the shell that we launched terminates (because the build is done) any background processes will be killed immediately.
>>> 
>>> Last, we clone the mount namespace that we have created before so that no build command can modify what we set up earlier.
>>> 
>>> * Since everything is now so decoupled, we gain a couple of new (maybe minor?) features:
>>> 
>>>    It is now possible to run “make.sh shell” while a build is running. That does not happen a lot, but we can do this now :)
>>> 
>>>    If the build crashes or the host system is being shut down while a build is running, there is nothing to clean up afterwards.
>>> 
>>> * I have garnished this all with a lot of code cleanup and I suppose I might have introduced some new bugs here or there :)
>>> 
>>> * This is probably mostly around a new implementation of the timer that updates the build time. It has been annoying me a lot that it takes a long time to walk through all packages that have been built before to finally get to a package that we want to rebuild. Mostly this was all help up by a call of “sleep 0.1”
>>> 
>>> Since bash does not really do any concurrency, I had to be creative and replaced the busy-loop with a background process that is launched whenever it is needed and which will “ping” the main make.sh script once a second. That way, we can just run as usual, but regularly get interrupted to update the runtime.
>>> 
>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l361
>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l834
>>> 
>>> We now only fork one extra sub shell and we have to handle the timer events which is a lot cheaper as well as more straight-forward to code.
>>> 
>>> * As there is no difference between the different stages any more (those stages that we inherited from LFS), I have merged them all into one.
>>> 
>>> * Last but not least, I have create the option to build for multiple architectures on the same system. Since we can now mount the entire source tree into (many independent) build environments, we might as well… As discussed on the last call, this might not be the best option for ARM, but RISV-C builds at a decent speed even when emulated.
>>> 
>>> The only thing that I needed to do for this is to suffix the build and log directories which are now called build_${ARCH}, i.e. build_aarch64, build_x86_64, and so on. The packages/ directory is not changed yet, but that will have to happen as well. Most likely I want to merge this with the generated images, but I am not sure what to call this, yet. Happy to hear suggestions. result_x86_64? Just images_x86_64?
>>> 
>>> ---
>>> 
>>> I have run a build and this seems to be working just fine on my Debian machine. I am writing to all of you to first of all let you know what I am up to; and secondly to ask to give this a go on your systems. I think it should run just fine, as all the tools that I require should be available everywhere. However, there might be some older kernels that might not support all of this, yet or any other problems I cannot think of yet. Please give me some feedback and send me all the bugs :)
>> I gave this a go but it didn't work.
>> 
>> Not sure if I should have run the ./make.sh clean command on the old version before I pulled the unshare branch into my clone of your repo.
>> 
>> Should I have started with a complete new clone of your repo? I might try that anyway just to see.
>> 
> I created a completely new clone of you ipfire-2.x repor and then checked out the unshare branch to a branch called unshare in my local repo clone.
> 
> gettoolchain gave the same issue, except that this time the toolchain directory ended up completely empty.
> 
> downloadsrc had the same result.
> 
> clean had nothing to clean up as it was a fresh clone.
> 
> build then tried to build the toolchain and came up with this error, different from before.
> 
> ./make.sh build
> chroot: failed to run command ‘env’: No such file or directory
> Full toolchain compilation
> stage1 [ FAIL ]
> 
>     Jul  8 19:26:39: Building stage1 ====================================== Installing stage1 ...
>     mkdir -pv /tools_x86_64/lib
>     mkdir: cannot create directory '/tools_x86_64': File exists
>     make: *** [stage1:50: /home/ahb/sandbox/ms/ipfire-2.x/log/stage1] Error 1
> 
> ERROR: Building stage1 [ FAIL ]
>     Check /home/ahb/sandbox/ms/ipfire-2.x/log_x86_64/_build.toolchain.log for errors if applicable [ FAIL ]
> 
> so it wasn't as simple as doing a fresh git clone.
> 
> Regards,
> 
> Adolf.
> 
> 
>> So I ran ./make.sh gettoolchain first, as I usually would.
>> 
>> ./make.sh gettoolchain
>> b2sum: cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: No such file or directory
>> cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: FAILED open or read
>> b2sum: WARNING: 1 listed file could not be read
>> 
>> ipfire-2.29-toolchain-20240210-x86_64.tar.zst is present together with its b2 file.
>> 
>> 
>> Then ran ./make.sh downloadsrc
>> 
>> Previous version ends with
>> 
>> ***Verifying BLAKE2 checksum
>> all files BLAKE2 checksum match                                                       [ DONE]
>> 
>> after zstd has been checked.
>> 
>> New version stops at zstd entry.
>> 
>> 
>> ./make.sh clean gave the message Cleaning Build directory... but was completed very quickly.
>> Log and Build directories have not been cleaned out. The img and iso files are still present.
>> 
>> 
>> ./make.sh build gave message
>> 
>> chroot: failed to run command ‘env’: No such file or directory
>> 
>> and then did a full toolchain compilation which failed with gcc but log is >9000 lines.
>> 
>> 
>> Regards,
>> Adolf.
>> 
>>> 
>>> Thank you for listening to this brain-dump.
>>> 
>>> All the best,
>>> -Michael
>>> 
>>>> On 3 Jul 2024, at 10:58, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>> 
>>>> Hello Adolf,
>>>> 
>>>> This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more.
>>>> 
>>>> I rebooted the machine and it is back up again.
>>>> 
>>>> -Michael
>>>> 
>>>>> On 2 Jul 2024, at 15:42, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>> 
>>>>> Hi Michael and all,
>>>>> 
>>>>> 
>>>>> I ran the arm builder with the 4.20.2 version of samba to test it out.
>>>>> 
>>>>> The build got to building gdb and then failed.
>>>>> 
>>>>> Interestingly, the nightly build of arm was successful with the same version of gdb.
>>>>> 
>>>>> The build log for gdb is attached. The actual error is at line 618.
>>>>> 
>>>>> Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message.
>>>>> 
>>>>> ------------------------------------------
>>>>> 
>>>>> ssh bonnietwin(a)arm64-01.zrh.ipfire.org
>>>>> PTY allocation request failed on channel 0
>>>>> Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64
>>>>> 
>>>>> The programs included with the Debian GNU/Linux system are free software;
>>>>> the exact distribution terms for each program are described in the
>>>>> individual files in /usr/share/doc/*/copyright.
>>>>> 
>>>>> Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
>>>>> permitted by applicable law.
>>>>> /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied
>>>>> 
>>>>> ------------------------------------------
>>>>> 
>>>>> Regards,
>>>>> 
>>>>> Adolf.
>>>>> 
>>>>> <_build.ipfire.gdb.log>



^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: Problem during building of samba on arm builder
  2024-07-09 21:29         ` Michael Tremer
@ 2024-07-10  9:57           ` Michael Tremer
  2024-07-10 10:33             ` Adolf Belka
  0 siblings, 1 reply; 16+ messages in thread
From: Michael Tremer @ 2024-07-10  9:57 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 12345 bytes --]

Hello again,

I managed to (finally) build the toolchain with the updated system. So hopefully there should not be any more outstanding problems that I know of so far.

Best,
-Michael

> On 9 Jul 2024, at 22:29, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
> 
> Hello Adolf,
> 
> Thank you for testing this.
> 
> There have indeed been plenty of problems there… I spent a lot of time on this today and hopefully fixed most of them.
> 
> I cannot build the toolchain on my machine and I am not sure why yet, but a build with the packaged toolchain runs through.
> 
> I have also spent some time on getting rid of the strip stage because it annoyed me how long it takes and creating the disk images as well as packages should be significantly faster now, too. I hope I didn’t introduce too many new bugs.
> 
> Please let me know if you have more success now.
> 
> Best,
> -Michael
> 
>> On 8 Jul 2024, at 20:34, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>> 
>> Hi Michael,
>> 
>> On 08/07/2024 21:15, Adolf Belka wrote:
>>> Hi Michael,
>>> 
>>> On 08/07/2024 18:11, Michael Tremer wrote:
>>>> Hello,
>>>> 
>>>> I have been spending a lot of time on this problem, because it has been bothering me for a long time. I also saw an opportunity to make more changes to the build system.
>>>> 
>>>> Currently this is all a little bit WIP, but I hope that we can merge this into next as soon as the current update has been moved to master.
>>>> 
>>>> I am referring to this branch which is currently based on next: https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/unshare
>>>> 
>>>> It makes use of the unshare command which creates new namespaces in Linux. That way, we can isolate the build system better from the host system and in case something goes wrong, there is less damage. We can also enforce some more rules…
>>>> 
>>>> So, what has changed?
>>>> 
>>>> * The make.sh script might re-execute itself into a new mount namespace when it is suitable. This happens for “make.sh build” and “make.sh shell”, but it does not happen for “make.sh downloadsrc” for example.
>>>> 
>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l2129
>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l2251
>>>> 
>>>> * The new mount namespace means that we will no longer see any bind-mounts in the host system and we no longer need to umount anything ourselves which is where we occasionally wiped the entire hard drive of the host system. When the last process exits, the namespace is being cleaned up and everything is being umounted.
>>>> 
>>>> * The function that prepares the build environment has been almost entirely rewritten:
>>>> 
>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l426
>>>> 
>>>> It used to mount parts of the host system into the build environment which are needed to run anything. Those were /dev, /proc, /sys, etc…
>>>> 
>>>> Instead, it now creates a new /dev mount point and creates a minimal amount of device nodes and symlinks. That way, we detach from the host system and no longer allow the build system access to the host’s filesystem and block devices. We also bind-mount the sources in read-only mode now, so that the build system cannot change anything in the source tree. On top of that, cache is read-only, too. ccache and the log directory are the only places that are writable.
>>>> 
>>>> We mount a separate /tmp directory.
>>>> 
>>>> * When we then build a package, we create more namespaces for each package. These isolate each build process from each other.
>>>> 
>>>> Mostly, this is to detach from the host system. A new UTS namespace allows to change the hostname in the build system without affecting the host and so on. We do the same thing with a new time namespace.
>>>> 
>>>> We do however create a new PID namespace which means that the build system no longer will see any processes running on the host system. That requires to mount a new instance of /proc with each package. This also has the effect that if the shell that we launched terminates (because the build is done) any background processes will be killed immediately.
>>>> 
>>>> Last, we clone the mount namespace that we have created before so that no build command can modify what we set up earlier.
>>>> 
>>>> * Since everything is now so decoupled, we gain a couple of new (maybe minor?) features:
>>>> 
>>>>   It is now possible to run “make.sh shell” while a build is running. That does not happen a lot, but we can do this now :)
>>>> 
>>>>   If the build crashes or the host system is being shut down while a build is running, there is nothing to clean up afterwards.
>>>> 
>>>> * I have garnished this all with a lot of code cleanup and I suppose I might have introduced some new bugs here or there :)
>>>> 
>>>> * This is probably mostly around a new implementation of the timer that updates the build time. It has been annoying me a lot that it takes a long time to walk through all packages that have been built before to finally get to a package that we want to rebuild. Mostly this was all help up by a call of “sleep 0.1”
>>>> 
>>>> Since bash does not really do any concurrency, I had to be creative and replaced the busy-loop with a background process that is launched whenever it is needed and which will “ping” the main make.sh script once a second. That way, we can just run as usual, but regularly get interrupted to update the runtime.
>>>> 
>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l361
>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l834
>>>> 
>>>> We now only fork one extra sub shell and we have to handle the timer events which is a lot cheaper as well as more straight-forward to code.
>>>> 
>>>> * As there is no difference between the different stages any more (those stages that we inherited from LFS), I have merged them all into one.
>>>> 
>>>> * Last but not least, I have create the option to build for multiple architectures on the same system. Since we can now mount the entire source tree into (many independent) build environments, we might as well… As discussed on the last call, this might not be the best option for ARM, but RISV-C builds at a decent speed even when emulated.
>>>> 
>>>> The only thing that I needed to do for this is to suffix the build and log directories which are now called build_${ARCH}, i.e. build_aarch64, build_x86_64, and so on. The packages/ directory is not changed yet, but that will have to happen as well. Most likely I want to merge this with the generated images, but I am not sure what to call this, yet. Happy to hear suggestions. result_x86_64? Just images_x86_64?
>>>> 
>>>> ---
>>>> 
>>>> I have run a build and this seems to be working just fine on my Debian machine. I am writing to all of you to first of all let you know what I am up to; and secondly to ask to give this a go on your systems. I think it should run just fine, as all the tools that I require should be available everywhere. However, there might be some older kernels that might not support all of this, yet or any other problems I cannot think of yet. Please give me some feedback and send me all the bugs :)
>>> I gave this a go but it didn't work.
>>> 
>>> Not sure if I should have run the ./make.sh clean command on the old version before I pulled the unshare branch into my clone of your repo.
>>> 
>>> Should I have started with a complete new clone of your repo? I might try that anyway just to see.
>>> 
>> I created a completely new clone of you ipfire-2.x repor and then checked out the unshare branch to a branch called unshare in my local repo clone.
>> 
>> gettoolchain gave the same issue, except that this time the toolchain directory ended up completely empty.
>> 
>> downloadsrc had the same result.
>> 
>> clean had nothing to clean up as it was a fresh clone.
>> 
>> build then tried to build the toolchain and came up with this error, different from before.
>> 
>> ./make.sh build
>> chroot: failed to run command ‘env’: No such file or directory
>> Full toolchain compilation
>> stage1 [ FAIL ]
>> 
>>    Jul  8 19:26:39: Building stage1 ====================================== Installing stage1 ...
>>    mkdir -pv /tools_x86_64/lib
>>    mkdir: cannot create directory '/tools_x86_64': File exists
>>    make: *** [stage1:50: /home/ahb/sandbox/ms/ipfire-2.x/log/stage1] Error 1
>> 
>> ERROR: Building stage1 [ FAIL ]
>>    Check /home/ahb/sandbox/ms/ipfire-2.x/log_x86_64/_build.toolchain.log for errors if applicable [ FAIL ]
>> 
>> so it wasn't as simple as doing a fresh git clone.
>> 
>> Regards,
>> 
>> Adolf.
>> 
>> 
>>> So I ran ./make.sh gettoolchain first, as I usually would.
>>> 
>>> ./make.sh gettoolchain
>>> b2sum: cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: No such file or directory
>>> cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: FAILED open or read
>>> b2sum: WARNING: 1 listed file could not be read
>>> 
>>> ipfire-2.29-toolchain-20240210-x86_64.tar.zst is present together with its b2 file.
>>> 
>>> 
>>> Then ran ./make.sh downloadsrc
>>> 
>>> Previous version ends with
>>> 
>>> ***Verifying BLAKE2 checksum
>>> all files BLAKE2 checksum match                                                       [ DONE]
>>> 
>>> after zstd has been checked.
>>> 
>>> New version stops at zstd entry.
>>> 
>>> 
>>> ./make.sh clean gave the message Cleaning Build directory... but was completed very quickly.
>>> Log and Build directories have not been cleaned out. The img and iso files are still present.
>>> 
>>> 
>>> ./make.sh build gave message
>>> 
>>> chroot: failed to run command ‘env’: No such file or directory
>>> 
>>> and then did a full toolchain compilation which failed with gcc but log is >9000 lines.
>>> 
>>> 
>>> Regards,
>>> Adolf.
>>> 
>>>> 
>>>> Thank you for listening to this brain-dump.
>>>> 
>>>> All the best,
>>>> -Michael
>>>> 
>>>>> On 3 Jul 2024, at 10:58, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>>> 
>>>>> Hello Adolf,
>>>>> 
>>>>> This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more.
>>>>> 
>>>>> I rebooted the machine and it is back up again.
>>>>> 
>>>>> -Michael
>>>>> 
>>>>>> On 2 Jul 2024, at 15:42, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>> 
>>>>>> Hi Michael and all,
>>>>>> 
>>>>>> 
>>>>>> I ran the arm builder with the 4.20.2 version of samba to test it out.
>>>>>> 
>>>>>> The build got to building gdb and then failed.
>>>>>> 
>>>>>> Interestingly, the nightly build of arm was successful with the same version of gdb.
>>>>>> 
>>>>>> The build log for gdb is attached. The actual error is at line 618.
>>>>>> 
>>>>>> Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message.
>>>>>> 
>>>>>> ------------------------------------------
>>>>>> 
>>>>>> ssh bonnietwin(a)arm64-01.zrh.ipfire.org
>>>>>> PTY allocation request failed on channel 0
>>>>>> Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64
>>>>>> 
>>>>>> The programs included with the Debian GNU/Linux system are free software;
>>>>>> the exact distribution terms for each program are described in the
>>>>>> individual files in /usr/share/doc/*/copyright.
>>>>>> 
>>>>>> Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
>>>>>> permitted by applicable law.
>>>>>> /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied
>>>>>> 
>>>>>> ------------------------------------------
>>>>>> 
>>>>>> Regards,
>>>>>> 
>>>>>> Adolf.
>>>>>> 
>>>>>> <_build.ipfire.gdb.log>
> 
> 


^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: Problem during building of samba on arm builder
  2024-07-10  9:57           ` Michael Tremer
@ 2024-07-10 10:33             ` Adolf Belka
  2024-07-10 12:59               ` Adolf Belka
  0 siblings, 1 reply; 16+ messages in thread
From: Adolf Belka @ 2024-07-10 10:33 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 12995 bytes --]

Hi Michael,

On 10/07/2024 11:57, Michael Tremer wrote:
> Hello again,
> 
> I managed to (finally) build the toolchain with the updated system. So hopefully there should not be any more outstanding problems that I know of so far.

I just did a git pull on your repo to my clone.

Ran ./make.sh gettoolchain and it successfully downloaded the toolchain.

Ran ./make.sh downloadsrc and it successfully tested everything.

Ran ./make.sh clean and build and log directories were cleared out and removed. As far as I can tell it was successful.

Currently running ./make.sh build. So up to this point everything going well. Will let you know how it goes.

Regards,
Adolf.

> 
> Best,
> -Michael
> 
>> On 9 Jul 2024, at 22:29, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>
>> Hello Adolf,
>>
>> Thank you for testing this.
>>
>> There have indeed been plenty of problems there… I spent a lot of time on this today and hopefully fixed most of them.
>>
>> I cannot build the toolchain on my machine and I am not sure why yet, but a build with the packaged toolchain runs through.
>>
>> I have also spent some time on getting rid of the strip stage because it annoyed me how long it takes and creating the disk images as well as packages should be significantly faster now, too. I hope I didn’t introduce too many new bugs.
>>
>> Please let me know if you have more success now.
>>
>> Best,
>> -Michael
>>
>>> On 8 Jul 2024, at 20:34, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>
>>> Hi Michael,
>>>
>>> On 08/07/2024 21:15, Adolf Belka wrote:
>>>> Hi Michael,
>>>>
>>>> On 08/07/2024 18:11, Michael Tremer wrote:
>>>>> Hello,
>>>>>
>>>>> I have been spending a lot of time on this problem, because it has been bothering me for a long time. I also saw an opportunity to make more changes to the build system.
>>>>>
>>>>> Currently this is all a little bit WIP, but I hope that we can merge this into next as soon as the current update has been moved to master.
>>>>>
>>>>> I am referring to this branch which is currently based on next: https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/unshare
>>>>>
>>>>> It makes use of the unshare command which creates new namespaces in Linux. That way, we can isolate the build system better from the host system and in case something goes wrong, there is less damage. We can also enforce some more rules…
>>>>>
>>>>> So, what has changed?
>>>>>
>>>>> * The make.sh script might re-execute itself into a new mount namespace when it is suitable. This happens for “make.sh build” and “make.sh shell”, but it does not happen for “make.sh downloadsrc” for example.
>>>>>
>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l2129
>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l2251
>>>>>
>>>>> * The new mount namespace means that we will no longer see any bind-mounts in the host system and we no longer need to umount anything ourselves which is where we occasionally wiped the entire hard drive of the host system. When the last process exits, the namespace is being cleaned up and everything is being umounted.
>>>>>
>>>>> * The function that prepares the build environment has been almost entirely rewritten:
>>>>>
>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l426
>>>>>
>>>>> It used to mount parts of the host system into the build environment which are needed to run anything. Those were /dev, /proc, /sys, etc…
>>>>>
>>>>> Instead, it now creates a new /dev mount point and creates a minimal amount of device nodes and symlinks. That way, we detach from the host system and no longer allow the build system access to the host’s filesystem and block devices. We also bind-mount the sources in read-only mode now, so that the build system cannot change anything in the source tree. On top of that, cache is read-only, too. ccache and the log directory are the only places that are writable.
>>>>>
>>>>> We mount a separate /tmp directory.
>>>>>
>>>>> * When we then build a package, we create more namespaces for each package. These isolate each build process from each other.
>>>>>
>>>>> Mostly, this is to detach from the host system. A new UTS namespace allows to change the hostname in the build system without affecting the host and so on. We do the same thing with a new time namespace.
>>>>>
>>>>> We do however create a new PID namespace which means that the build system no longer will see any processes running on the host system. That requires to mount a new instance of /proc with each package. This also has the effect that if the shell that we launched terminates (because the build is done) any background processes will be killed immediately.
>>>>>
>>>>> Last, we clone the mount namespace that we have created before so that no build command can modify what we set up earlier.
>>>>>
>>>>> * Since everything is now so decoupled, we gain a couple of new (maybe minor?) features:
>>>>>
>>>>>    It is now possible to run “make.sh shell” while a build is running. That does not happen a lot, but we can do this now :)
>>>>>
>>>>>    If the build crashes or the host system is being shut down while a build is running, there is nothing to clean up afterwards.
>>>>>
>>>>> * I have garnished this all with a lot of code cleanup and I suppose I might have introduced some new bugs here or there :)
>>>>>
>>>>> * This is probably mostly around a new implementation of the timer that updates the build time. It has been annoying me a lot that it takes a long time to walk through all packages that have been built before to finally get to a package that we want to rebuild. Mostly this was all help up by a call of “sleep 0.1”
>>>>>
>>>>> Since bash does not really do any concurrency, I had to be creative and replaced the busy-loop with a background process that is launched whenever it is needed and which will “ping” the main make.sh script once a second. That way, we can just run as usual, but regularly get interrupted to update the runtime.
>>>>>
>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l361
>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l834
>>>>>
>>>>> We now only fork one extra sub shell and we have to handle the timer events which is a lot cheaper as well as more straight-forward to code.
>>>>>
>>>>> * As there is no difference between the different stages any more (those stages that we inherited from LFS), I have merged them all into one.
>>>>>
>>>>> * Last but not least, I have create the option to build for multiple architectures on the same system. Since we can now mount the entire source tree into (many independent) build environments, we might as well… As discussed on the last call, this might not be the best option for ARM, but RISV-C builds at a decent speed even when emulated.
>>>>>
>>>>> The only thing that I needed to do for this is to suffix the build and log directories which are now called build_${ARCH}, i.e. build_aarch64, build_x86_64, and so on. The packages/ directory is not changed yet, but that will have to happen as well. Most likely I want to merge this with the generated images, but I am not sure what to call this, yet. Happy to hear suggestions. result_x86_64? Just images_x86_64?
>>>>>
>>>>> ---
>>>>>
>>>>> I have run a build and this seems to be working just fine on my Debian machine. I am writing to all of you to first of all let you know what I am up to; and secondly to ask to give this a go on your systems. I think it should run just fine, as all the tools that I require should be available everywhere. However, there might be some older kernels that might not support all of this, yet or any other problems I cannot think of yet. Please give me some feedback and send me all the bugs :)
>>>> I gave this a go but it didn't work.
>>>>
>>>> Not sure if I should have run the ./make.sh clean command on the old version before I pulled the unshare branch into my clone of your repo.
>>>>
>>>> Should I have started with a complete new clone of your repo? I might try that anyway just to see.
>>>>
>>> I created a completely new clone of you ipfire-2.x repor and then checked out the unshare branch to a branch called unshare in my local repo clone.
>>>
>>> gettoolchain gave the same issue, except that this time the toolchain directory ended up completely empty.
>>>
>>> downloadsrc had the same result.
>>>
>>> clean had nothing to clean up as it was a fresh clone.
>>>
>>> build then tried to build the toolchain and came up with this error, different from before.
>>>
>>> ./make.sh build
>>> chroot: failed to run command ‘env’: No such file or directory
>>> Full toolchain compilation
>>> stage1 [ FAIL ]
>>>
>>>     Jul  8 19:26:39: Building stage1 ====================================== Installing stage1 ...
>>>     mkdir -pv /tools_x86_64/lib
>>>     mkdir: cannot create directory '/tools_x86_64': File exists
>>>     make: *** [stage1:50: /home/ahb/sandbox/ms/ipfire-2.x/log/stage1] Error 1
>>>
>>> ERROR: Building stage1 [ FAIL ]
>>>     Check /home/ahb/sandbox/ms/ipfire-2.x/log_x86_64/_build.toolchain.log for errors if applicable [ FAIL ]
>>>
>>> so it wasn't as simple as doing a fresh git clone.
>>>
>>> Regards,
>>>
>>> Adolf.
>>>
>>>
>>>> So I ran ./make.sh gettoolchain first, as I usually would.
>>>>
>>>> ./make.sh gettoolchain
>>>> b2sum: cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: No such file or directory
>>>> cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: FAILED open or read
>>>> b2sum: WARNING: 1 listed file could not be read
>>>>
>>>> ipfire-2.29-toolchain-20240210-x86_64.tar.zst is present together with its b2 file.
>>>>
>>>>
>>>> Then ran ./make.sh downloadsrc
>>>>
>>>> Previous version ends with
>>>>
>>>> ***Verifying BLAKE2 checksum
>>>> all files BLAKE2 checksum match                                                       [ DONE]
>>>>
>>>> after zstd has been checked.
>>>>
>>>> New version stops at zstd entry.
>>>>
>>>>
>>>> ./make.sh clean gave the message Cleaning Build directory... but was completed very quickly.
>>>> Log and Build directories have not been cleaned out. The img and iso files are still present.
>>>>
>>>>
>>>> ./make.sh build gave message
>>>>
>>>> chroot: failed to run command ‘env’: No such file or directory
>>>>
>>>> and then did a full toolchain compilation which failed with gcc but log is >9000 lines.
>>>>
>>>>
>>>> Regards,
>>>> Adolf.
>>>>
>>>>>
>>>>> Thank you for listening to this brain-dump.
>>>>>
>>>>> All the best,
>>>>> -Michael
>>>>>
>>>>>> On 3 Jul 2024, at 10:58, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>>>>
>>>>>> Hello Adolf,
>>>>>>
>>>>>> This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more.
>>>>>>
>>>>>> I rebooted the machine and it is back up again.
>>>>>>
>>>>>> -Michael
>>>>>>
>>>>>>> On 2 Jul 2024, at 15:42, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>
>>>>>>> Hi Michael and all,
>>>>>>>
>>>>>>>
>>>>>>> I ran the arm builder with the 4.20.2 version of samba to test it out.
>>>>>>>
>>>>>>> The build got to building gdb and then failed.
>>>>>>>
>>>>>>> Interestingly, the nightly build of arm was successful with the same version of gdb.
>>>>>>>
>>>>>>> The build log for gdb is attached. The actual error is at line 618.
>>>>>>>
>>>>>>> Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message.
>>>>>>>
>>>>>>> ------------------------------------------
>>>>>>>
>>>>>>> ssh bonnietwin(a)arm64-01.zrh.ipfire.org
>>>>>>> PTY allocation request failed on channel 0
>>>>>>> Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64
>>>>>>>
>>>>>>> The programs included with the Debian GNU/Linux system are free software;
>>>>>>> the exact distribution terms for each program are described in the
>>>>>>> individual files in /usr/share/doc/*/copyright.
>>>>>>>
>>>>>>> Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
>>>>>>> permitted by applicable law.
>>>>>>> /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied
>>>>>>>
>>>>>>> ------------------------------------------
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Adolf.
>>>>>>>
>>>>>>> <_build.ipfire.gdb.log>
>>
>>
> 

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: Problem during building of samba on arm builder
  2024-07-10 10:33             ` Adolf Belka
@ 2024-07-10 12:59               ` Adolf Belka
  2024-07-10 13:05                 ` Adolf Belka
  0 siblings, 1 reply; 16+ messages in thread
From: Adolf Belka @ 2024-07-10 12:59 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 13751 bytes --]

Hi Michael,

On 10/07/2024 12:33, Adolf Belka wrote:
> Hi Michael,
>
> On 10/07/2024 11:57, Michael Tremer wrote:
>> Hello again,
>>
>> I managed to (finally) build the toolchain with the updated system. So hopefully there should not be any more outstanding problems that I know of so far.
>
> I just did a git pull on your repo to my clone.
>
> Ran ./make.sh gettoolchain and it successfully downloaded the toolchain.
>
> Ran ./make.sh downloadsrc and it successfully tested everything.
>
> Ran ./make.sh clean and build and log directories were cleared out and removed. As far as I can tell it was successful.
>
> Currently running ./make.sh build. So up to this point everything going well. Will let you know how it goes.
>
It has got to building popt. In the normal build system this takes around 3 secs. Currently in the new build it is at nearly 2 hours. Even with an empty cache, that seems a long build time for popt, unless I am being too optimistic.

I will let it keep going.


One thing I found. I am running the new build system while I have been running some package updates with the old system with its mount points. The two have each run without any impact on the other.

Regards,

Adolf.

> Regards,
> Adolf.
>
>>
>> Best,
>> -Michael
>>
>>> On 9 Jul 2024, at 22:29, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>
>>> Hello Adolf,
>>>
>>> Thank you for testing this.
>>>
>>> There have indeed been plenty of problems there… I spent a lot of time on this today and hopefully fixed most of them.
>>>
>>> I cannot build the toolchain on my machine and I am not sure why yet, but a build with the packaged toolchain runs through.
>>>
>>> I have also spent some time on getting rid of the strip stage because it annoyed me how long it takes and creating the disk images as well as packages should be significantly faster now, too. I hope I didn’t introduce too many new bugs.
>>>
>>> Please let me know if you have more success now.
>>>
>>> Best,
>>> -Michael
>>>
>>>> On 8 Jul 2024, at 20:34, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>
>>>> Hi Michael,
>>>>
>>>> On 08/07/2024 21:15, Adolf Belka wrote:
>>>>> Hi Michael,
>>>>>
>>>>> On 08/07/2024 18:11, Michael Tremer wrote:
>>>>>> Hello,
>>>>>>
>>>>>> I have been spending a lot of time on this problem, because it has been bothering me for a long time. I also saw an opportunity to make more changes to the build system.
>>>>>>
>>>>>> Currently this is all a little bit WIP, but I hope that we can merge this into next as soon as the current update has been moved to master.
>>>>>>
>>>>>> I am referring to this branch which is currently based on next: https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/unshare
>>>>>>
>>>>>> It makes use of the unshare command which creates new namespaces in Linux. That way, we can isolate the build system better from the host system and in case something goes wrong, there is less damage. We can also enforce some more rules…
>>>>>>
>>>>>> So, what has changed?
>>>>>>
>>>>>> * The make.sh script might re-execute itself into a new mount namespace when it is suitable. This happens for “make.sh build” and “make.sh shell”, but it does not happen for “make.sh downloadsrc” for example.
>>>>>>
>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l2129
>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l2251
>>>>>>
>>>>>> * The new mount namespace means that we will no longer see any bind-mounts in the host system and we no longer need to umount anything ourselves which is where we occasionally wiped the entire hard drive of the host system. When the last process exits, the namespace is being cleaned up and everything is being umounted.
>>>>>>
>>>>>> * The function that prepares the build environment has been almost entirely rewritten:
>>>>>>
>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l426
>>>>>>
>>>>>> It used to mount parts of the host system into the build environment which are needed to run anything. Those were /dev, /proc, /sys, etc…
>>>>>>
>>>>>> Instead, it now creates a new /dev mount point and creates a minimal amount of device nodes and symlinks. That way, we detach from the host system and no longer allow the build system access to the host’s filesystem and block devices. We also bind-mount the sources in read-only mode now, so that the build system cannot change anything in the source tree. On top of that, cache is read-only, too. ccache and the log directory are the only places that are writable.
>>>>>>
>>>>>> We mount a separate /tmp directory.
>>>>>>
>>>>>> * When we then build a package, we create more namespaces for each package. These isolate each build process from each other.
>>>>>>
>>>>>> Mostly, this is to detach from the host system. A new UTS namespace allows to change the hostname in the build system without affecting the host and so on. We do the same thing with a new time namespace.
>>>>>>
>>>>>> We do however create a new PID namespace which means that the build system no longer will see any processes running on the host system. That requires to mount a new instance of /proc with each package. This also has the effect that if the shell that we launched terminates (because the build is done) any background processes will be killed immediately.
>>>>>>
>>>>>> Last, we clone the mount namespace that we have created before so that no build command can modify what we set up earlier.
>>>>>>
>>>>>> * Since everything is now so decoupled, we gain a couple of new (maybe minor?) features:
>>>>>>
>>>>>>    It is now possible to run “make.sh shell” while a build is running. That does not happen a lot, but we can do this now :)
>>>>>>
>>>>>>    If the build crashes or the host system is being shut down while a build is running, there is nothing to clean up afterwards.
>>>>>>
>>>>>> * I have garnished this all with a lot of code cleanup and I suppose I might have introduced some new bugs here or there :)
>>>>>>
>>>>>> * This is probably mostly around a new implementation of the timer that updates the build time. It has been annoying me a lot that it takes a long time to walk through all packages that have been built before to finally get to a package that we want to rebuild. Mostly this was all help up by a call of “sleep 0.1”
>>>>>>
>>>>>> Since bash does not really do any concurrency, I had to be creative and replaced the busy-loop with a background process that is launched whenever it is needed and which will “ping” the main make.sh script once a second. That way, we can just run as usual, but regularly get interrupted to update the runtime.
>>>>>>
>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l361
>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l834
>>>>>>
>>>>>> We now only fork one extra sub shell and we have to handle the timer events which is a lot cheaper as well as more straight-forward to code.
>>>>>>
>>>>>> * As there is no difference between the different stages any more (those stages that we inherited from LFS), I have merged them all into one.
>>>>>>
>>>>>> * Last but not least, I have create the option to build for multiple architectures on the same system. Since we can now mount the entire source tree into (many independent) build environments, we might as well… As discussed on the last call, this might not be the best option for ARM, but RISV-C builds at a decent speed even when emulated.
>>>>>>
>>>>>> The only thing that I needed to do for this is to suffix the build and log directories which are now called build_${ARCH}, i.e. build_aarch64, build_x86_64, and so on. The packages/ directory is not changed yet, but that will have to happen as well. Most likely I want to merge this with the generated images, but I am not sure what to call this, yet. Happy to hear suggestions. result_x86_64? Just images_x86_64?
>>>>>>
>>>>>> ---
>>>>>>
>>>>>> I have run a build and this seems to be working just fine on my Debian machine. I am writing to all of you to first of all let you know what I am up to; and secondly to ask to give this a go on your systems. I think it should run just fine, as all the tools that I require should be available everywhere. However, there might be some older kernels that might not support all of this, yet or any other problems I cannot think of yet. Please give me some feedback and send me all the bugs :)
>>>>> I gave this a go but it didn't work.
>>>>>
>>>>> Not sure if I should have run the ./make.sh clean command on the old version before I pulled the unshare branch into my clone of your repo.
>>>>>
>>>>> Should I have started with a complete new clone of your repo? I might try that anyway just to see.
>>>>>
>>>> I created a completely new clone of you ipfire-2.x repor and then checked out the unshare branch to a branch called unshare in my local repo clone.
>>>>
>>>> gettoolchain gave the same issue, except that this time the toolchain directory ended up completely empty.
>>>>
>>>> downloadsrc had the same result.
>>>>
>>>> clean had nothing to clean up as it was a fresh clone.
>>>>
>>>> build then tried to build the toolchain and came up with this error, different from before.
>>>>
>>>> ./make.sh build
>>>> chroot: failed to run command ‘env’: No such file or directory
>>>> Full toolchain compilation
>>>> stage1 [ FAIL ]
>>>>
>>>>     Jul  8 19:26:39: Building stage1 ====================================== Installing stage1 ...
>>>>     mkdir -pv /tools_x86_64/lib
>>>>     mkdir: cannot create directory '/tools_x86_64': File exists
>>>>     make: *** [stage1:50: /home/ahb/sandbox/ms/ipfire-2.x/log/stage1] Error 1
>>>>
>>>> ERROR: Building stage1 [ FAIL ]
>>>>     Check /home/ahb/sandbox/ms/ipfire-2.x/log_x86_64/_build.toolchain.log for errors if applicable [ FAIL ]
>>>>
>>>> so it wasn't as simple as doing a fresh git clone.
>>>>
>>>> Regards,
>>>>
>>>> Adolf.
>>>>
>>>>
>>>>> So I ran ./make.sh gettoolchain first, as I usually would.
>>>>>
>>>>> ./make.sh gettoolchain
>>>>> b2sum: cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: No such file or directory
>>>>> cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: FAILED open or read
>>>>> b2sum: WARNING: 1 listed file could not be read
>>>>>
>>>>> ipfire-2.29-toolchain-20240210-x86_64.tar.zst is present together with its b2 file.
>>>>>
>>>>>
>>>>> Then ran ./make.sh downloadsrc
>>>>>
>>>>> Previous version ends with
>>>>>
>>>>> ***Verifying BLAKE2 checksum
>>>>> all files BLAKE2 checksum match [ DONE]
>>>>>
>>>>> after zstd has been checked.
>>>>>
>>>>> New version stops at zstd entry.
>>>>>
>>>>>
>>>>> ./make.sh clean gave the message Cleaning Build directory... but was completed very quickly.
>>>>> Log and Build directories have not been cleaned out. The img and iso files are still present.
>>>>>
>>>>>
>>>>> ./make.sh build gave message
>>>>>
>>>>> chroot: failed to run command ‘env’: No such file or directory
>>>>>
>>>>> and then did a full toolchain compilation which failed with gcc but log is >9000 lines.
>>>>>
>>>>>
>>>>> Regards,
>>>>> Adolf.
>>>>>
>>>>>>
>>>>>> Thank you for listening to this brain-dump.
>>>>>>
>>>>>> All the best,
>>>>>> -Michael
>>>>>>
>>>>>>> On 3 Jul 2024, at 10:58, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>>>>>
>>>>>>> Hello Adolf,
>>>>>>>
>>>>>>> This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more.
>>>>>>>
>>>>>>> I rebooted the machine and it is back up again.
>>>>>>>
>>>>>>> -Michael
>>>>>>>
>>>>>>>> On 2 Jul 2024, at 15:42, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>
>>>>>>>> Hi Michael and all,
>>>>>>>>
>>>>>>>>
>>>>>>>> I ran the arm builder with the 4.20.2 version of samba to test it out.
>>>>>>>>
>>>>>>>> The build got to building gdb and then failed.
>>>>>>>>
>>>>>>>> Interestingly, the nightly build of arm was successful with the same version of gdb.
>>>>>>>>
>>>>>>>> The build log for gdb is attached. The actual error is at line 618.
>>>>>>>>
>>>>>>>> Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message.
>>>>>>>>
>>>>>>>> ------------------------------------------
>>>>>>>>
>>>>>>>> ssh bonnietwin(a)arm64-01.zrh.ipfire.org
>>>>>>>> PTY allocation request failed on channel 0
>>>>>>>> Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64
>>>>>>>>
>>>>>>>> The programs included with the Debian GNU/Linux system are free software;
>>>>>>>> the exact distribution terms for each program are described in the
>>>>>>>> individual files in /usr/share/doc/*/copyright.
>>>>>>>>
>>>>>>>> Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
>>>>>>>> permitted by applicable law.
>>>>>>>> /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied
>>>>>>>>
>>>>>>>> ------------------------------------------
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Adolf.
>>>>>>>>
>>>>>>>> <_build.ipfire.gdb.log>
>>>
>>>
>>

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: Problem during building of samba on arm builder
  2024-07-10 12:59               ` Adolf Belka
@ 2024-07-10 13:05                 ` Adolf Belka
  2024-07-10 13:21                   ` Adolf Belka
  0 siblings, 1 reply; 16+ messages in thread
From: Adolf Belka @ 2024-07-10 13:05 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 15389 bytes --]

Hi Michael,

On 10/07/2024 14:59, Adolf Belka wrote:
> Hi Michael,
>
> On 10/07/2024 12:33, Adolf Belka wrote:
>> Hi Michael,
>>
>> On 10/07/2024 11:57, Michael Tremer wrote:
>>> Hello again,
>>>
>>> I managed to (finally) build the toolchain with the updated system. So hopefully there should not be any more outstanding problems that I know of so far.
>>
>> I just did a git pull on your repo to my clone.
>>
>> Ran ./make.sh gettoolchain and it successfully downloaded the toolchain.
>>
>> Ran ./make.sh downloadsrc and it successfully tested everything.
>>
>> Ran ./make.sh clean and build and log directories were cleared out and removed. As far as I can tell it was successful.
>>
>> Currently running ./make.sh build. So up to this point everything going well. Will let you know how it goes.
>>
> It has got to building popt. In the normal build system this takes around 3 secs. Currently in the new build it is at nearly 2 hours. Even with an empty cache, that seems a long build time for popt, unless I am being too optimistic.
>
> I will let it keep going.
>
I just had a look at the log file and it looks like it completed popt but then is stuck on trying to leave the directory /usr/src/lfs. Here is the output from the log, nothing new is getting written to the log.

make[3]: Leaving directory '/usr/src/popt-1.19/po'
Making install in tests
make[3]: Entering directory '/usr/src/popt-1.19/tests'
make[4]: Entering directory '/usr/src/popt-1.19/tests'
make[4]: Nothing to be done for 'install-exec-am'.
make[4]: Nothing to be done for 'install-data-am'.
make[4]: Leaving directory '/usr/src/popt-1.19/tests'
make[3]: Leaving directory '/usr/src/popt-1.19/tests'
make[3]: Entering directory '/usr/src/popt-1.19'
make[4]: Entering directory '/usr/src/popt-1.19'
make[4]: Nothing to be done for 'install-exec-am'.
  /bin/mkdir -p '/usr/share/man/man3'
  /usr/bin/install -c -m 644 popt.3 '/usr/share/man/man3'
  /bin/mkdir -p '/usr/lib/pkgconfig'
  /usr/bin/install -c -m 644 popt.pc '/usr/lib/pkgconfig'
make[4]: Leaving directory '/usr/src/popt-1.19'
make[3]: Leaving directory '/usr/src/popt-1.19'
make[2]: Leaving directory '/usr/src/popt-1.19'
make[1]: Leaving directory '/usr/src/popt-1.19'
Updating linker cache...
Install done; saving file list to /usr/src/log/popt-1.19 ...

make: Leaving directory '/usr/src/lfs'


Regards,

Adolf.


>
> One thing I found. I am running the new build system while I have been running some package updates with the old system with its mount points. The two have each run without any impact on the other.
>
> Regards,
>
> Adolf.
>
>> Regards,
>> Adolf.
>>
>>>
>>> Best,
>>> -Michael
>>>
>>>> On 9 Jul 2024, at 22:29, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>>
>>>> Hello Adolf,
>>>>
>>>> Thank you for testing this.
>>>>
>>>> There have indeed been plenty of problems there… I spent a lot of time on this today and hopefully fixed most of them.
>>>>
>>>> I cannot build the toolchain on my machine and I am not sure why yet, but a build with the packaged toolchain runs through.
>>>>
>>>> I have also spent some time on getting rid of the strip stage because it annoyed me how long it takes and creating the disk images as well as packages should be significantly faster now, too. I hope I didn’t introduce too many new bugs.
>>>>
>>>> Please let me know if you have more success now.
>>>>
>>>> Best,
>>>> -Michael
>>>>
>>>>> On 8 Jul 2024, at 20:34, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>
>>>>> Hi Michael,
>>>>>
>>>>> On 08/07/2024 21:15, Adolf Belka wrote:
>>>>>> Hi Michael,
>>>>>>
>>>>>> On 08/07/2024 18:11, Michael Tremer wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> I have been spending a lot of time on this problem, because it has been bothering me for a long time. I also saw an opportunity to make more changes to the build system.
>>>>>>>
>>>>>>> Currently this is all a little bit WIP, but I hope that we can merge this into next as soon as the current update has been moved to master.
>>>>>>>
>>>>>>> I am referring to this branch which is currently based on next: https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/unshare
>>>>>>>
>>>>>>> It makes use of the unshare command which creates new namespaces in Linux. That way, we can isolate the build system better from the host system and in case something goes wrong, there is less damage. We can also enforce some more rules…
>>>>>>>
>>>>>>> So, what has changed?
>>>>>>>
>>>>>>> * The make.sh script might re-execute itself into a new mount namespace when it is suitable. This happens for “make.sh build” and “make.sh shell”, but it does not happen for “make.sh downloadsrc” for example.
>>>>>>>
>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l2129
>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l2251
>>>>>>>
>>>>>>> * The new mount namespace means that we will no longer see any bind-mounts in the host system and we no longer need to umount anything ourselves which is where we occasionally wiped the entire hard drive of the host system. When the last process exits, the namespace is being cleaned up and everything is being umounted.
>>>>>>>
>>>>>>> * The function that prepares the build environment has been almost entirely rewritten:
>>>>>>>
>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l426
>>>>>>>
>>>>>>> It used to mount parts of the host system into the build environment which are needed to run anything. Those were /dev, /proc, /sys, etc…
>>>>>>>
>>>>>>> Instead, it now creates a new /dev mount point and creates a minimal amount of device nodes and symlinks. That way, we detach from the host system and no longer allow the build system access to the host’s filesystem and block devices. We also bind-mount the sources in read-only mode now, so that the build system cannot change anything in the source tree. On top of that, cache is read-only, too. ccache and the log directory are the only places that are writable.
>>>>>>>
>>>>>>> We mount a separate /tmp directory.
>>>>>>>
>>>>>>> * When we then build a package, we create more namespaces for each package. These isolate each build process from each other.
>>>>>>>
>>>>>>> Mostly, this is to detach from the host system. A new UTS namespace allows to change the hostname in the build system without affecting the host and so on. We do the same thing with a new time namespace.
>>>>>>>
>>>>>>> We do however create a new PID namespace which means that the build system no longer will see any processes running on the host system. That requires to mount a new instance of /proc with each package. This also has the effect that if the shell that we launched terminates (because the build is done) any background processes will be killed immediately.
>>>>>>>
>>>>>>> Last, we clone the mount namespace that we have created before so that no build command can modify what we set up earlier.
>>>>>>>
>>>>>>> * Since everything is now so decoupled, we gain a couple of new (maybe minor?) features:
>>>>>>>
>>>>>>>    It is now possible to run “make.sh shell” while a build is running. That does not happen a lot, but we can do this now :)
>>>>>>>
>>>>>>>    If the build crashes or the host system is being shut down while a build is running, there is nothing to clean up afterwards.
>>>>>>>
>>>>>>> * I have garnished this all with a lot of code cleanup and I suppose I might have introduced some new bugs here or there :)
>>>>>>>
>>>>>>> * This is probably mostly around a new implementation of the timer that updates the build time. It has been annoying me a lot that it takes a long time to walk through all packages that have been built before to finally get to a package that we want to rebuild. Mostly this was all help up by a call of “sleep 0.1”
>>>>>>>
>>>>>>> Since bash does not really do any concurrency, I had to be creative and replaced the busy-loop with a background process that is launched whenever it is needed and which will “ping” the main make.sh script once a second. That way, we can just run as usual, but regularly get interrupted to update the runtime.
>>>>>>>
>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l361
>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l834
>>>>>>>
>>>>>>> We now only fork one extra sub shell and we have to handle the timer events which is a lot cheaper as well as more straight-forward to code.
>>>>>>>
>>>>>>> * As there is no difference between the different stages any more (those stages that we inherited from LFS), I have merged them all into one.
>>>>>>>
>>>>>>> * Last but not least, I have create the option to build for multiple architectures on the same system. Since we can now mount the entire source tree into (many independent) build environments, we might as well… As discussed on the last call, this might not be the best option for ARM, but RISV-C builds at a decent speed even when emulated.
>>>>>>>
>>>>>>> The only thing that I needed to do for this is to suffix the build and log directories which are now called build_${ARCH}, i.e. build_aarch64, build_x86_64, and so on. The packages/ directory is not changed yet, but that will have to happen as well. Most likely I want to merge this with the generated images, but I am not sure what to call this, yet. Happy to hear suggestions. result_x86_64? Just images_x86_64?
>>>>>>>
>>>>>>> ---
>>>>>>>
>>>>>>> I have run a build and this seems to be working just fine on my Debian machine. I am writing to all of you to first of all let you know what I am up to; and secondly to ask to give this a go on your systems. I think it should run just fine, as all the tools that I require should be available everywhere. However, there might be some older kernels that might not support all of this, yet or any other problems I cannot think of yet. Please give me some feedback and send me all the bugs :)
>>>>>> I gave this a go but it didn't work.
>>>>>>
>>>>>> Not sure if I should have run the ./make.sh clean command on the old version before I pulled the unshare branch into my clone of your repo.
>>>>>>
>>>>>> Should I have started with a complete new clone of your repo? I might try that anyway just to see.
>>>>>>
>>>>> I created a completely new clone of you ipfire-2.x repor and then checked out the unshare branch to a branch called unshare in my local repo clone.
>>>>>
>>>>> gettoolchain gave the same issue, except that this time the toolchain directory ended up completely empty.
>>>>>
>>>>> downloadsrc had the same result.
>>>>>
>>>>> clean had nothing to clean up as it was a fresh clone.
>>>>>
>>>>> build then tried to build the toolchain and came up with this error, different from before.
>>>>>
>>>>> ./make.sh build
>>>>> chroot: failed to run command ‘env’: No such file or directory
>>>>> Full toolchain compilation
>>>>> stage1 [ FAIL ]
>>>>>
>>>>>     Jul  8 19:26:39: Building stage1 ====================================== Installing stage1 ...
>>>>>     mkdir -pv /tools_x86_64/lib
>>>>>     mkdir: cannot create directory '/tools_x86_64': File exists
>>>>>     make: *** [stage1:50: /home/ahb/sandbox/ms/ipfire-2.x/log/stage1] Error 1
>>>>>
>>>>> ERROR: Building stage1 [ FAIL ]
>>>>>     Check /home/ahb/sandbox/ms/ipfire-2.x/log_x86_64/_build.toolchain.log for errors if applicable [ FAIL ]
>>>>>
>>>>> so it wasn't as simple as doing a fresh git clone.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Adolf.
>>>>>
>>>>>
>>>>>> So I ran ./make.sh gettoolchain first, as I usually would.
>>>>>>
>>>>>> ./make.sh gettoolchain
>>>>>> b2sum: cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: No such file or directory
>>>>>> cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: FAILED open or read
>>>>>> b2sum: WARNING: 1 listed file could not be read
>>>>>>
>>>>>> ipfire-2.29-toolchain-20240210-x86_64.tar.zst is present together with its b2 file.
>>>>>>
>>>>>>
>>>>>> Then ran ./make.sh downloadsrc
>>>>>>
>>>>>> Previous version ends with
>>>>>>
>>>>>> ***Verifying BLAKE2 checksum
>>>>>> all files BLAKE2 checksum match [ DONE]
>>>>>>
>>>>>> after zstd has been checked.
>>>>>>
>>>>>> New version stops at zstd entry.
>>>>>>
>>>>>>
>>>>>> ./make.sh clean gave the message Cleaning Build directory... but was completed very quickly.
>>>>>> Log and Build directories have not been cleaned out. The img and iso files are still present.
>>>>>>
>>>>>>
>>>>>> ./make.sh build gave message
>>>>>>
>>>>>> chroot: failed to run command ‘env’: No such file or directory
>>>>>>
>>>>>> and then did a full toolchain compilation which failed with gcc but log is >9000 lines.
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> Adolf.
>>>>>>
>>>>>>>
>>>>>>> Thank you for listening to this brain-dump.
>>>>>>>
>>>>>>> All the best,
>>>>>>> -Michael
>>>>>>>
>>>>>>>> On 3 Jul 2024, at 10:58, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>>>>>>
>>>>>>>> Hello Adolf,
>>>>>>>>
>>>>>>>> This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more.
>>>>>>>>
>>>>>>>> I rebooted the machine and it is back up again.
>>>>>>>>
>>>>>>>> -Michael
>>>>>>>>
>>>>>>>>> On 2 Jul 2024, at 15:42, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>
>>>>>>>>> Hi Michael and all,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I ran the arm builder with the 4.20.2 version of samba to test it out.
>>>>>>>>>
>>>>>>>>> The build got to building gdb and then failed.
>>>>>>>>>
>>>>>>>>> Interestingly, the nightly build of arm was successful with the same version of gdb.
>>>>>>>>>
>>>>>>>>> The build log for gdb is attached. The actual error is at line 618.
>>>>>>>>>
>>>>>>>>> Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message.
>>>>>>>>>
>>>>>>>>> ------------------------------------------
>>>>>>>>>
>>>>>>>>> ssh bonnietwin(a)arm64-01.zrh.ipfire.org
>>>>>>>>> PTY allocation request failed on channel 0
>>>>>>>>> Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64
>>>>>>>>>
>>>>>>>>> The programs included with the Debian GNU/Linux system are free software;
>>>>>>>>> the exact distribution terms for each program are described in the
>>>>>>>>> individual files in /usr/share/doc/*/copyright.
>>>>>>>>>
>>>>>>>>> Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
>>>>>>>>> permitted by applicable law.
>>>>>>>>> /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied
>>>>>>>>>
>>>>>>>>> ------------------------------------------
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Adolf.
>>>>>>>>>
>>>>>>>>> <_build.ipfire.gdb.log>
>>>>
>>>>
>>>

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: Problem during building of samba on arm builder
  2024-07-10 13:05                 ` Adolf Belka
@ 2024-07-10 13:21                   ` Adolf Belka
  2024-07-10 14:24                     ` Michael Tremer
  0 siblings, 1 reply; 16+ messages in thread
From: Adolf Belka @ 2024-07-10 13:21 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 16074 bytes --]

Hi Michael,

On 10/07/2024 15:05, Adolf Belka wrote:
> Hi Michael,
>
> On 10/07/2024 14:59, Adolf Belka wrote:
>> Hi Michael,
>>
>> On 10/07/2024 12:33, Adolf Belka wrote:
>>> Hi Michael,
>>>
>>> On 10/07/2024 11:57, Michael Tremer wrote:
>>>> Hello again,
>>>>
>>>> I managed to (finally) build the toolchain with the updated system. So hopefully there should not be any more outstanding problems that I know of so far.
>>>
>>> I just did a git pull on your repo to my clone.
>>>
>>> Ran ./make.sh gettoolchain and it successfully downloaded the toolchain.
>>>
>>> Ran ./make.sh downloadsrc and it successfully tested everything.
>>>
>>> Ran ./make.sh clean and build and log directories were cleared out and removed. As far as I can tell it was successful.
>>>
>>> Currently running ./make.sh build. So up to this point everything going well. Will let you know how it goes.
>>>
>> It has got to building popt. In the normal build system this takes around 3 secs. Currently in the new build it is at nearly 2 hours. Even with an empty cache, that seems a long build time for popt, unless I am being too optimistic.
>>
>> I will let it keep going.
>>
> I just had a look at the log file and it looks like it completed popt but then is stuck on trying to leave the directory /usr/src/lfs. Here is the output from the log, nothing new is getting written to the log.
>
> make[3]: Leaving directory '/usr/src/popt-1.19/po'
> Making install in tests
> make[3]: Entering directory '/usr/src/popt-1.19/tests'
> make[4]: Entering directory '/usr/src/popt-1.19/tests'
> make[4]: Nothing to be done for 'install-exec-am'.
> make[4]: Nothing to be done for 'install-data-am'.
> make[4]: Leaving directory '/usr/src/popt-1.19/tests'
> make[3]: Leaving directory '/usr/src/popt-1.19/tests'
> make[3]: Entering directory '/usr/src/popt-1.19'
> make[4]: Entering directory '/usr/src/popt-1.19'
> make[4]: Nothing to be done for 'install-exec-am'.
>  /bin/mkdir -p '/usr/share/man/man3'
>  /usr/bin/install -c -m 644 popt.3 '/usr/share/man/man3'
>  /bin/mkdir -p '/usr/lib/pkgconfig'
>  /usr/bin/install -c -m 644 popt.pc '/usr/lib/pkgconfig'
> make[4]: Leaving directory '/usr/src/popt-1.19'
> make[3]: Leaving directory '/usr/src/popt-1.19'
> make[2]: Leaving directory '/usr/src/popt-1.19'
> make[1]: Leaving directory '/usr/src/popt-1.19'
> Updating linker cache...
> Install done; saving file list to /usr/src/log/popt-1.19 ...
>
> make: Leaving directory '/usr/src/lfs'
>
>
I stopped the build with Ctrl-C and then ran ./make.sh build again without doing a clean. It got to popt and went straight to libedit, the next package, and started to build it.

Something must have just put the build into a loop of some sort. It was not writing anything to the log file.

Regards,

Adolf.


> Regards,
>
> Adolf.
>
>
>>
>> One thing I found. I am running the new build system while I have been running some package updates with the old system with its mount points. The two have each run without any impact on the other.
>>
>> Regards,
>>
>> Adolf.
>>
>>> Regards,
>>> Adolf.
>>>
>>>>
>>>> Best,
>>>> -Michael
>>>>
>>>>> On 9 Jul 2024, at 22:29, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>>>
>>>>> Hello Adolf,
>>>>>
>>>>> Thank you for testing this.
>>>>>
>>>>> There have indeed been plenty of problems there… I spent a lot of time on this today and hopefully fixed most of them.
>>>>>
>>>>> I cannot build the toolchain on my machine and I am not sure why yet, but a build with the packaged toolchain runs through.
>>>>>
>>>>> I have also spent some time on getting rid of the strip stage because it annoyed me how long it takes and creating the disk images as well as packages should be significantly faster now, too. I hope I didn’t introduce too many new bugs.
>>>>>
>>>>> Please let me know if you have more success now.
>>>>>
>>>>> Best,
>>>>> -Michael
>>>>>
>>>>>> On 8 Jul 2024, at 20:34, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>
>>>>>> Hi Michael,
>>>>>>
>>>>>> On 08/07/2024 21:15, Adolf Belka wrote:
>>>>>>> Hi Michael,
>>>>>>>
>>>>>>> On 08/07/2024 18:11, Michael Tremer wrote:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I have been spending a lot of time on this problem, because it has been bothering me for a long time. I also saw an opportunity to make more changes to the build system.
>>>>>>>>
>>>>>>>> Currently this is all a little bit WIP, but I hope that we can merge this into next as soon as the current update has been moved to master.
>>>>>>>>
>>>>>>>> I am referring to this branch which is currently based on next: https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/unshare
>>>>>>>>
>>>>>>>> It makes use of the unshare command which creates new namespaces in Linux. That way, we can isolate the build system better from the host system and in case something goes wrong, there is less damage. We can also enforce some more rules…
>>>>>>>>
>>>>>>>> So, what has changed?
>>>>>>>>
>>>>>>>> * The make.sh script might re-execute itself into a new mount namespace when it is suitable. This happens for “make.sh build” and “make.sh shell”, but it does not happen for “make.sh downloadsrc” for example.
>>>>>>>>
>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l2129
>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l2251
>>>>>>>>
>>>>>>>> * The new mount namespace means that we will no longer see any bind-mounts in the host system and we no longer need to umount anything ourselves which is where we occasionally wiped the entire hard drive of the host system. When the last process exits, the namespace is being cleaned up and everything is being umounted.
>>>>>>>>
>>>>>>>> * The function that prepares the build environment has been almost entirely rewritten:
>>>>>>>>
>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l426
>>>>>>>>
>>>>>>>> It used to mount parts of the host system into the build environment which are needed to run anything. Those were /dev, /proc, /sys, etc…
>>>>>>>>
>>>>>>>> Instead, it now creates a new /dev mount point and creates a minimal amount of device nodes and symlinks. That way, we detach from the host system and no longer allow the build system access to the host’s filesystem and block devices. We also bind-mount the sources in read-only mode now, so that the build system cannot change anything in the source tree. On top of that, cache is read-only, too. ccache and the log directory are the only places that are writable.
>>>>>>>>
>>>>>>>> We mount a separate /tmp directory.
>>>>>>>>
>>>>>>>> * When we then build a package, we create more namespaces for each package. These isolate each build process from each other.
>>>>>>>>
>>>>>>>> Mostly, this is to detach from the host system. A new UTS namespace allows to change the hostname in the build system without affecting the host and so on. We do the same thing with a new time namespace.
>>>>>>>>
>>>>>>>> We do however create a new PID namespace which means that the build system no longer will see any processes running on the host system. That requires to mount a new instance of /proc with each package. This also has the effect that if the shell that we launched terminates (because the build is done) any background processes will be killed immediately.
>>>>>>>>
>>>>>>>> Last, we clone the mount namespace that we have created before so that no build command can modify what we set up earlier.
>>>>>>>>
>>>>>>>> * Since everything is now so decoupled, we gain a couple of new (maybe minor?) features:
>>>>>>>>
>>>>>>>>    It is now possible to run “make.sh shell” while a build is running. That does not happen a lot, but we can do this now :)
>>>>>>>>
>>>>>>>>    If the build crashes or the host system is being shut down while a build is running, there is nothing to clean up afterwards.
>>>>>>>>
>>>>>>>> * I have garnished this all with a lot of code cleanup and I suppose I might have introduced some new bugs here or there :)
>>>>>>>>
>>>>>>>> * This is probably mostly around a new implementation of the timer that updates the build time. It has been annoying me a lot that it takes a long time to walk through all packages that have been built before to finally get to a package that we want to rebuild. Mostly this was all help up by a call of “sleep 0.1”
>>>>>>>>
>>>>>>>> Since bash does not really do any concurrency, I had to be creative and replaced the busy-loop with a background process that is launched whenever it is needed and which will “ping” the main make.sh script once a second. That way, we can just run as usual, but regularly get interrupted to update the runtime.
>>>>>>>>
>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l361
>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l834
>>>>>>>>
>>>>>>>> We now only fork one extra sub shell and we have to handle the timer events which is a lot cheaper as well as more straight-forward to code.
>>>>>>>>
>>>>>>>> * As there is no difference between the different stages any more (those stages that we inherited from LFS), I have merged them all into one.
>>>>>>>>
>>>>>>>> * Last but not least, I have create the option to build for multiple architectures on the same system. Since we can now mount the entire source tree into (many independent) build environments, we might as well… As discussed on the last call, this might not be the best option for ARM, but RISV-C builds at a decent speed even when emulated.
>>>>>>>>
>>>>>>>> The only thing that I needed to do for this is to suffix the build and log directories which are now called build_${ARCH}, i.e. build_aarch64, build_x86_64, and so on. The packages/ directory is not changed yet, but that will have to happen as well. Most likely I want to merge this with the generated images, but I am not sure what to call this, yet. Happy to hear suggestions. result_x86_64? Just images_x86_64?
>>>>>>>>
>>>>>>>> ---
>>>>>>>>
>>>>>>>> I have run a build and this seems to be working just fine on my Debian machine. I am writing to all of you to first of all let you know what I am up to; and secondly to ask to give this a go on your systems. I think it should run just fine, as all the tools that I require should be available everywhere. However, there might be some older kernels that might not support all of this, yet or any other problems I cannot think of yet. Please give me some feedback and send me all the bugs :)
>>>>>>> I gave this a go but it didn't work.
>>>>>>>
>>>>>>> Not sure if I should have run the ./make.sh clean command on the old version before I pulled the unshare branch into my clone of your repo.
>>>>>>>
>>>>>>> Should I have started with a complete new clone of your repo? I might try that anyway just to see.
>>>>>>>
>>>>>> I created a completely new clone of you ipfire-2.x repor and then checked out the unshare branch to a branch called unshare in my local repo clone.
>>>>>>
>>>>>> gettoolchain gave the same issue, except that this time the toolchain directory ended up completely empty.
>>>>>>
>>>>>> downloadsrc had the same result.
>>>>>>
>>>>>> clean had nothing to clean up as it was a fresh clone.
>>>>>>
>>>>>> build then tried to build the toolchain and came up with this error, different from before.
>>>>>>
>>>>>> ./make.sh build
>>>>>> chroot: failed to run command ‘env’: No such file or directory
>>>>>> Full toolchain compilation
>>>>>> stage1 [ FAIL ]
>>>>>>
>>>>>>     Jul  8 19:26:39: Building stage1 ====================================== Installing stage1 ...
>>>>>>     mkdir -pv /tools_x86_64/lib
>>>>>>     mkdir: cannot create directory '/tools_x86_64': File exists
>>>>>>     make: *** [stage1:50: /home/ahb/sandbox/ms/ipfire-2.x/log/stage1] Error 1
>>>>>>
>>>>>> ERROR: Building stage1 [ FAIL ]
>>>>>>     Check /home/ahb/sandbox/ms/ipfire-2.x/log_x86_64/_build.toolchain.log for errors if applicable [ FAIL ]
>>>>>>
>>>>>> so it wasn't as simple as doing a fresh git clone.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Adolf.
>>>>>>
>>>>>>
>>>>>>> So I ran ./make.sh gettoolchain first, as I usually would.
>>>>>>>
>>>>>>> ./make.sh gettoolchain
>>>>>>> b2sum: cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: No such file or directory
>>>>>>> cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: FAILED open or read
>>>>>>> b2sum: WARNING: 1 listed file could not be read
>>>>>>>
>>>>>>> ipfire-2.29-toolchain-20240210-x86_64.tar.zst is present together with its b2 file.
>>>>>>>
>>>>>>>
>>>>>>> Then ran ./make.sh downloadsrc
>>>>>>>
>>>>>>> Previous version ends with
>>>>>>>
>>>>>>> ***Verifying BLAKE2 checksum
>>>>>>> all files BLAKE2 checksum match [ DONE]
>>>>>>>
>>>>>>> after zstd has been checked.
>>>>>>>
>>>>>>> New version stops at zstd entry.
>>>>>>>
>>>>>>>
>>>>>>> ./make.sh clean gave the message Cleaning Build directory... but was completed very quickly.
>>>>>>> Log and Build directories have not been cleaned out. The img and iso files are still present.
>>>>>>>
>>>>>>>
>>>>>>> ./make.sh build gave message
>>>>>>>
>>>>>>> chroot: failed to run command ‘env’: No such file or directory
>>>>>>>
>>>>>>> and then did a full toolchain compilation which failed with gcc but log is >9000 lines.
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>> Adolf.
>>>>>>>
>>>>>>>>
>>>>>>>> Thank you for listening to this brain-dump.
>>>>>>>>
>>>>>>>> All the best,
>>>>>>>> -Michael
>>>>>>>>
>>>>>>>>> On 3 Jul 2024, at 10:58, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>>>>>>>
>>>>>>>>> Hello Adolf,
>>>>>>>>>
>>>>>>>>> This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more.
>>>>>>>>>
>>>>>>>>> I rebooted the machine and it is back up again.
>>>>>>>>>
>>>>>>>>> -Michael
>>>>>>>>>
>>>>>>>>>> On 2 Jul 2024, at 15:42, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Michael and all,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I ran the arm builder with the 4.20.2 version of samba to test it out.
>>>>>>>>>>
>>>>>>>>>> The build got to building gdb and then failed.
>>>>>>>>>>
>>>>>>>>>> Interestingly, the nightly build of arm was successful with the same version of gdb.
>>>>>>>>>>
>>>>>>>>>> The build log for gdb is attached. The actual error is at line 618.
>>>>>>>>>>
>>>>>>>>>> Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message.
>>>>>>>>>>
>>>>>>>>>> ------------------------------------------
>>>>>>>>>>
>>>>>>>>>> ssh bonnietwin(a)arm64-01.zrh.ipfire.org
>>>>>>>>>> PTY allocation request failed on channel 0
>>>>>>>>>> Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64
>>>>>>>>>>
>>>>>>>>>> The programs included with the Debian GNU/Linux system are free software;
>>>>>>>>>> the exact distribution terms for each program are described in the
>>>>>>>>>> individual files in /usr/share/doc/*/copyright.
>>>>>>>>>>
>>>>>>>>>> Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
>>>>>>>>>> permitted by applicable law.
>>>>>>>>>> /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied
>>>>>>>>>>
>>>>>>>>>> ------------------------------------------
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>> Adolf.
>>>>>>>>>>
>>>>>>>>>> <_build.ipfire.gdb.log>
>>>>>
>>>>>
>>>>

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: Problem during building of samba on arm builder
  2024-07-10 13:21                   ` Adolf Belka
@ 2024-07-10 14:24                     ` Michael Tremer
  2024-07-10 16:53                       ` Adolf Belka
  0 siblings, 1 reply; 16+ messages in thread
From: Michael Tremer @ 2024-07-10 14:24 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 16857 bytes --]

Hello,

> On 10 Jul 2024, at 14:21, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi Michael,
> 
> On 10/07/2024 15:05, Adolf Belka wrote:
>> Hi Michael,
>> 
>> On 10/07/2024 14:59, Adolf Belka wrote:
>>> Hi Michael,
>>> 
>>> On 10/07/2024 12:33, Adolf Belka wrote:
>>>> Hi Michael,
>>>> 
>>>> On 10/07/2024 11:57, Michael Tremer wrote:
>>>>> Hello again,
>>>>> 
>>>>> I managed to (finally) build the toolchain with the updated system. So hopefully there should not be any more outstanding problems that I know of so far.
>>>> 
>>>> I just did a git pull on your repo to my clone.
>>>> 
>>>> Ran ./make.sh gettoolchain and it successfully downloaded the toolchain.
>>>> 
>>>> Ran ./make.sh downloadsrc and it successfully tested everything.
>>>> 
>>>> Ran ./make.sh clean and build and log directories were cleared out and removed. As far as I can tell it was successful.
>>>> 
>>>> Currently running ./make.sh build. So up to this point everything going well. Will let you know how it goes.
>>>> 
>>> It has got to building popt. In the normal build system this takes around 3 secs. Currently in the new build it is at nearly 2 hours. Even with an empty cache, that seems a long build time for popt, unless I am being too optimistic.
>>> 
>>> I will let it keep going.
>>> 
>> I just had a look at the log file and it looks like it completed popt but then is stuck on trying to leave the directory /usr/src/lfs. Here is the output from the log, nothing new is getting written to the log.
>> 
>> make[3]: Leaving directory '/usr/src/popt-1.19/po'
>> Making install in tests
>> make[3]: Entering directory '/usr/src/popt-1.19/tests'
>> make[4]: Entering directory '/usr/src/popt-1.19/tests'
>> make[4]: Nothing to be done for 'install-exec-am'.
>> make[4]: Nothing to be done for 'install-data-am'.
>> make[4]: Leaving directory '/usr/src/popt-1.19/tests'
>> make[3]: Leaving directory '/usr/src/popt-1.19/tests'
>> make[3]: Entering directory '/usr/src/popt-1.19'
>> make[4]: Entering directory '/usr/src/popt-1.19'
>> make[4]: Nothing to be done for 'install-exec-am'.
>>  /bin/mkdir -p '/usr/share/man/man3'
>>  /usr/bin/install -c -m 644 popt.3 '/usr/share/man/man3'
>>  /bin/mkdir -p '/usr/lib/pkgconfig'
>>  /usr/bin/install -c -m 644 popt.pc '/usr/lib/pkgconfig'
>> make[4]: Leaving directory '/usr/src/popt-1.19'
>> make[3]: Leaving directory '/usr/src/popt-1.19'
>> make[2]: Leaving directory '/usr/src/popt-1.19'
>> make[1]: Leaving directory '/usr/src/popt-1.19'
>> Updating linker cache...
>> Install done; saving file list to /usr/src/log/popt-1.19 ...
>> 
>> make: Leaving directory '/usr/src/lfs'
>> 
>> 
> I stopped the build with Ctrl-C and then ran ./make.sh build again without doing a clean. It got to popt and went straight to libedit, the next package, and started to build it.
> 
> Something must have just put the build into a loop of some sort. It was not writing anything to the log file.

The log file says that popt was done. It might be that you interrupted the build just after it has finished, but the root file was not created, yet. That should however not directly be a problem…

If this is however the only problem that you are seeing now than I would be happy :)

-Michael

> 
> Regards,
> 
> Adolf.
> 
> 
>> Regards,
>> 
>> Adolf.
>> 
>> 
>>> 
>>> One thing I found. I am running the new build system while I have been running some package updates with the old system with its mount points. The two have each run without any impact on the other.
>>> 
>>> Regards,
>>> 
>>> Adolf.
>>> 
>>>> Regards,
>>>> Adolf.
>>>> 
>>>>> 
>>>>> Best,
>>>>> -Michael
>>>>> 
>>>>>> On 9 Jul 2024, at 22:29, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>>>> 
>>>>>> Hello Adolf,
>>>>>> 
>>>>>> Thank you for testing this.
>>>>>> 
>>>>>> There have indeed been plenty of problems there… I spent a lot of time on this today and hopefully fixed most of them.
>>>>>> 
>>>>>> I cannot build the toolchain on my machine and I am not sure why yet, but a build with the packaged toolchain runs through.
>>>>>> 
>>>>>> I have also spent some time on getting rid of the strip stage because it annoyed me how long it takes and creating the disk images as well as packages should be significantly faster now, too. I hope I didn’t introduce too many new bugs.
>>>>>> 
>>>>>> Please let me know if you have more success now.
>>>>>> 
>>>>>> Best,
>>>>>> -Michael
>>>>>> 
>>>>>>> On 8 Jul 2024, at 20:34, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>> 
>>>>>>> Hi Michael,
>>>>>>> 
>>>>>>> On 08/07/2024 21:15, Adolf Belka wrote:
>>>>>>>> Hi Michael,
>>>>>>>> 
>>>>>>>> On 08/07/2024 18:11, Michael Tremer wrote:
>>>>>>>>> Hello,
>>>>>>>>> 
>>>>>>>>> I have been spending a lot of time on this problem, because it has been bothering me for a long time. I also saw an opportunity to make more changes to the build system.
>>>>>>>>> 
>>>>>>>>> Currently this is all a little bit WIP, but I hope that we can merge this into next as soon as the current update has been moved to master.
>>>>>>>>> 
>>>>>>>>> I am referring to this branch which is currently based on next: https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/unshare
>>>>>>>>> 
>>>>>>>>> It makes use of the unshare command which creates new namespaces in Linux. That way, we can isolate the build system better from the host system and in case something goes wrong, there is less damage. We can also enforce some more rules…
>>>>>>>>> 
>>>>>>>>> So, what has changed?
>>>>>>>>> 
>>>>>>>>> * The make.sh script might re-execute itself into a new mount namespace when it is suitable. This happens for “make.sh build” and “make.sh shell”, but it does not happen for “make.sh downloadsrc” for example.
>>>>>>>>> 
>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l2129
>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l2251
>>>>>>>>> 
>>>>>>>>> * The new mount namespace means that we will no longer see any bind-mounts in the host system and we no longer need to umount anything ourselves which is where we occasionally wiped the entire hard drive of the host system. When the last process exits, the namespace is being cleaned up and everything is being umounted.
>>>>>>>>> 
>>>>>>>>> * The function that prepares the build environment has been almost entirely rewritten:
>>>>>>>>> 
>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l426
>>>>>>>>> 
>>>>>>>>> It used to mount parts of the host system into the build environment which are needed to run anything. Those were /dev, /proc, /sys, etc…
>>>>>>>>> 
>>>>>>>>> Instead, it now creates a new /dev mount point and creates a minimal amount of device nodes and symlinks. That way, we detach from the host system and no longer allow the build system access to the host’s filesystem and block devices. We also bind-mount the sources in read-only mode now, so that the build system cannot change anything in the source tree. On top of that, cache is read-only, too. ccache and the log directory are the only places that are writable.
>>>>>>>>> 
>>>>>>>>> We mount a separate /tmp directory.
>>>>>>>>> 
>>>>>>>>> * When we then build a package, we create more namespaces for each package. These isolate each build process from each other.
>>>>>>>>> 
>>>>>>>>> Mostly, this is to detach from the host system. A new UTS namespace allows to change the hostname in the build system without affecting the host and so on. We do the same thing with a new time namespace.
>>>>>>>>> 
>>>>>>>>> We do however create a new PID namespace which means that the build system no longer will see any processes running on the host system. That requires to mount a new instance of /proc with each package. This also has the effect that if the shell that we launched terminates (because the build is done) any background processes will be killed immediately.
>>>>>>>>> 
>>>>>>>>> Last, we clone the mount namespace that we have created before so that no build command can modify what we set up earlier.
>>>>>>>>> 
>>>>>>>>> * Since everything is now so decoupled, we gain a couple of new (maybe minor?) features:
>>>>>>>>> 
>>>>>>>>>    It is now possible to run “make.sh shell” while a build is running. That does not happen a lot, but we can do this now :)
>>>>>>>>> 
>>>>>>>>>    If the build crashes or the host system is being shut down while a build is running, there is nothing to clean up afterwards.
>>>>>>>>> 
>>>>>>>>> * I have garnished this all with a lot of code cleanup and I suppose I might have introduced some new bugs here or there :)
>>>>>>>>> 
>>>>>>>>> * This is probably mostly around a new implementation of the timer that updates the build time. It has been annoying me a lot that it takes a long time to walk through all packages that have been built before to finally get to a package that we want to rebuild. Mostly this was all help up by a call of “sleep 0.1”
>>>>>>>>> 
>>>>>>>>> Since bash does not really do any concurrency, I had to be creative and replaced the busy-loop with a background process that is launched whenever it is needed and which will “ping” the main make.sh script once a second. That way, we can just run as usual, but regularly get interrupted to update the runtime.
>>>>>>>>> 
>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l361
>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l834
>>>>>>>>> 
>>>>>>>>> We now only fork one extra sub shell and we have to handle the timer events which is a lot cheaper as well as more straight-forward to code.
>>>>>>>>> 
>>>>>>>>> * As there is no difference between the different stages any more (those stages that we inherited from LFS), I have merged them all into one.
>>>>>>>>> 
>>>>>>>>> * Last but not least, I have create the option to build for multiple architectures on the same system. Since we can now mount the entire source tree into (many independent) build environments, we might as well… As discussed on the last call, this might not be the best option for ARM, but RISV-C builds at a decent speed even when emulated.
>>>>>>>>> 
>>>>>>>>> The only thing that I needed to do for this is to suffix the build and log directories which are now called build_${ARCH}, i.e. build_aarch64, build_x86_64, and so on. The packages/ directory is not changed yet, but that will have to happen as well. Most likely I want to merge this with the generated images, but I am not sure what to call this, yet. Happy to hear suggestions. result_x86_64? Just images_x86_64?
>>>>>>>>> 
>>>>>>>>> ---
>>>>>>>>> 
>>>>>>>>> I have run a build and this seems to be working just fine on my Debian machine. I am writing to all of you to first of all let you know what I am up to; and secondly to ask to give this a go on your systems. I think it should run just fine, as all the tools that I require should be available everywhere. However, there might be some older kernels that might not support all of this, yet or any other problems I cannot think of yet. Please give me some feedback and send me all the bugs :)
>>>>>>>> I gave this a go but it didn't work.
>>>>>>>> 
>>>>>>>> Not sure if I should have run the ./make.sh clean command on the old version before I pulled the unshare branch into my clone of your repo.
>>>>>>>> 
>>>>>>>> Should I have started with a complete new clone of your repo? I might try that anyway just to see.
>>>>>>>> 
>>>>>>> I created a completely new clone of you ipfire-2.x repor and then checked out the unshare branch to a branch called unshare in my local repo clone.
>>>>>>> 
>>>>>>> gettoolchain gave the same issue, except that this time the toolchain directory ended up completely empty.
>>>>>>> 
>>>>>>> downloadsrc had the same result.
>>>>>>> 
>>>>>>> clean had nothing to clean up as it was a fresh clone.
>>>>>>> 
>>>>>>> build then tried to build the toolchain and came up with this error, different from before.
>>>>>>> 
>>>>>>> ./make.sh build
>>>>>>> chroot: failed to run command ‘env’: No such file or directory
>>>>>>> Full toolchain compilation
>>>>>>> stage1 [ FAIL ]
>>>>>>> 
>>>>>>>     Jul  8 19:26:39: Building stage1 ====================================== Installing stage1 ...
>>>>>>>     mkdir -pv /tools_x86_64/lib
>>>>>>>     mkdir: cannot create directory '/tools_x86_64': File exists
>>>>>>>     make: *** [stage1:50: /home/ahb/sandbox/ms/ipfire-2.x/log/stage1] Error 1
>>>>>>> 
>>>>>>> ERROR: Building stage1 [ FAIL ]
>>>>>>>     Check /home/ahb/sandbox/ms/ipfire-2.x/log_x86_64/_build.toolchain.log for errors if applicable [ FAIL ]
>>>>>>> 
>>>>>>> so it wasn't as simple as doing a fresh git clone.
>>>>>>> 
>>>>>>> Regards,
>>>>>>> 
>>>>>>> Adolf.
>>>>>>> 
>>>>>>> 
>>>>>>>> So I ran ./make.sh gettoolchain first, as I usually would.
>>>>>>>> 
>>>>>>>> ./make.sh gettoolchain
>>>>>>>> b2sum: cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: No such file or directory
>>>>>>>> cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: FAILED open or read
>>>>>>>> b2sum: WARNING: 1 listed file could not be read
>>>>>>>> 
>>>>>>>> ipfire-2.29-toolchain-20240210-x86_64.tar.zst is present together with its b2 file.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Then ran ./make.sh downloadsrc
>>>>>>>> 
>>>>>>>> Previous version ends with
>>>>>>>> 
>>>>>>>> ***Verifying BLAKE2 checksum
>>>>>>>> all files BLAKE2 checksum match [ DONE]
>>>>>>>> 
>>>>>>>> after zstd has been checked.
>>>>>>>> 
>>>>>>>> New version stops at zstd entry.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> ./make.sh clean gave the message Cleaning Build directory... but was completed very quickly.
>>>>>>>> Log and Build directories have not been cleaned out. The img and iso files are still present.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> ./make.sh build gave message
>>>>>>>> 
>>>>>>>> chroot: failed to run command ‘env’: No such file or directory
>>>>>>>> 
>>>>>>>> and then did a full toolchain compilation which failed with gcc but log is >9000 lines.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> Adolf.
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Thank you for listening to this brain-dump.
>>>>>>>>> 
>>>>>>>>> All the best,
>>>>>>>>> -Michael
>>>>>>>>> 
>>>>>>>>>> On 3 Jul 2024, at 10:58, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hello Adolf,
>>>>>>>>>> 
>>>>>>>>>> This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more.
>>>>>>>>>> 
>>>>>>>>>> I rebooted the machine and it is back up again.
>>>>>>>>>> 
>>>>>>>>>> -Michael
>>>>>>>>>> 
>>>>>>>>>>> On 2 Jul 2024, at 15:42, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Hi Michael and all,
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> I ran the arm builder with the 4.20.2 version of samba to test it out.
>>>>>>>>>>> 
>>>>>>>>>>> The build got to building gdb and then failed.
>>>>>>>>>>> 
>>>>>>>>>>> Interestingly, the nightly build of arm was successful with the same version of gdb.
>>>>>>>>>>> 
>>>>>>>>>>> The build log for gdb is attached. The actual error is at line 618.
>>>>>>>>>>> 
>>>>>>>>>>> Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message.
>>>>>>>>>>> 
>>>>>>>>>>> ------------------------------------------
>>>>>>>>>>> 
>>>>>>>>>>> ssh bonnietwin(a)arm64-01.zrh.ipfire.org
>>>>>>>>>>> PTY allocation request failed on channel 0
>>>>>>>>>>> Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64
>>>>>>>>>>> 
>>>>>>>>>>> The programs included with the Debian GNU/Linux system are free software;
>>>>>>>>>>> the exact distribution terms for each program are described in the
>>>>>>>>>>> individual files in /usr/share/doc/*/copyright.
>>>>>>>>>>> 
>>>>>>>>>>> Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
>>>>>>>>>>> permitted by applicable law.
>>>>>>>>>>> /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied
>>>>>>>>>>> 
>>>>>>>>>>> ------------------------------------------
>>>>>>>>>>> 
>>>>>>>>>>> Regards,
>>>>>>>>>>> 
>>>>>>>>>>> Adolf.
>>>>>>>>>>> 
>>>>>>>>>>> <_build.ipfire.gdb.log>



^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: Problem during building of samba on arm builder
  2024-07-10 14:24                     ` Michael Tremer
@ 2024-07-10 16:53                       ` Adolf Belka
  2024-07-10 18:19                         ` Michael Tremer
  0 siblings, 1 reply; 16+ messages in thread
From: Adolf Belka @ 2024-07-10 16:53 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 23075 bytes --]

Hi Michael,

On 10/07/2024 16:24, Michael Tremer wrote:
> Hello,
> 
>> On 10 Jul 2024, at 14:21, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>
>> Hi Michael,
>>
>> On 10/07/2024 15:05, Adolf Belka wrote:
>>> Hi Michael,
>>>
>>> On 10/07/2024 14:59, Adolf Belka wrote:
>>>> Hi Michael,
>>>>
>>>> On 10/07/2024 12:33, Adolf Belka wrote:
>>>>> Hi Michael,
>>>>>
>>>>> On 10/07/2024 11:57, Michael Tremer wrote:
>>>>>> Hello again,
>>>>>>
>>>>>> I managed to (finally) build the toolchain with the updated system. So hopefully there should not be any more outstanding problems that I know of so far.
>>>>>
>>>>> I just did a git pull on your repo to my clone.
>>>>>
>>>>> Ran ./make.sh gettoolchain and it successfully downloaded the toolchain.
>>>>>
>>>>> Ran ./make.sh downloadsrc and it successfully tested everything.
>>>>>
>>>>> Ran ./make.sh clean and build and log directories were cleared out and removed. As far as I can tell it was successful.
>>>>>
>>>>> Currently running ./make.sh build. So up to this point everything going well. Will let you know how it goes.
>>>>>
>>>> It has got to building popt. In the normal build system this takes around 3 secs. Currently in the new build it is at nearly 2 hours. Even with an empty cache, that seems a long build time for popt, unless I am being too optimistic.
>>>>
>>>> I will let it keep going.
>>>>
>>> I just had a look at the log file and it looks like it completed popt but then is stuck on trying to leave the directory /usr/src/lfs. Here is the output from the log, nothing new is getting written to the log.
>>>
>>> make[3]: Leaving directory '/usr/src/popt-1.19/po'
>>> Making install in tests
>>> make[3]: Entering directory '/usr/src/popt-1.19/tests'
>>> make[4]: Entering directory '/usr/src/popt-1.19/tests'
>>> make[4]: Nothing to be done for 'install-exec-am'.
>>> make[4]: Nothing to be done for 'install-data-am'.
>>> make[4]: Leaving directory '/usr/src/popt-1.19/tests'
>>> make[3]: Leaving directory '/usr/src/popt-1.19/tests'
>>> make[3]: Entering directory '/usr/src/popt-1.19'
>>> make[4]: Entering directory '/usr/src/popt-1.19'
>>> make[4]: Nothing to be done for 'install-exec-am'.
>>>   /bin/mkdir -p '/usr/share/man/man3'
>>>   /usr/bin/install -c -m 644 popt.3 '/usr/share/man/man3'
>>>   /bin/mkdir -p '/usr/lib/pkgconfig'
>>>   /usr/bin/install -c -m 644 popt.pc '/usr/lib/pkgconfig'
>>> make[4]: Leaving directory '/usr/src/popt-1.19'
>>> make[3]: Leaving directory '/usr/src/popt-1.19'
>>> make[2]: Leaving directory '/usr/src/popt-1.19'
>>> make[1]: Leaving directory '/usr/src/popt-1.19'
>>> Updating linker cache...
>>> Install done; saving file list to /usr/src/log/popt-1.19 ...
>>>
>>> make: Leaving directory '/usr/src/lfs'
>>>
>>>
>> I stopped the build with Ctrl-C and then ran ./make.sh build again without doing a clean. It got to popt and went straight to libedit, the next package, and started to build it.
>>
>> Something must have just put the build into a loop of some sort. It was not writing anything to the log file.
> 
> The log file says that popt was done. It might be that you interrupted the build just after it has finished, but the root file was not created, yet. That should however not directly be a problem…
2 hours had already gone by when I checked the log for that message. I then left the system running for a further 20 minutes with no new info in the log file and that is when I interrupted the build.

> 
> If this is however the only problem that you are seeing now than I would be happy :)

I have found another problem when it got to the flash-images section.

The flash-images log stops with the following:-

# Insert the UUID because grub-mkconfig often fails to
# detect that correctly
sed -i /tmp/harddisk/boot/grub/grub.cfg \
	-e "s/root=[A-Za-z0-9\/=-]*/root=UUID=$(blkid -o value -s UUID /dev/mapper/loop0p3)/g"
# Install GRUB
grub-install --force --recheck --no-floppy --target=i386-pc \
	--root-directory=/tmp/harddisk /dev/loop0
Installing for i386-pc platform.
Installation finished. No error reported.
# Install GRUB for EFI
grub-install --target=x86_64-efi --removable --no-nvram \
	--boot-directory=/tmp/harddisk/boot --efi-directory=/tmp/harddisk/boot/efi
Installing for x86_64-efi platform.
Installation finished. No error reported.
# restore orginal defaults
mv -f /tmp/harddisk/etc/default/grub.backup /tmp/harddisk/etc/default/grub
rm -f /tmp/harddisk/etc/grub.d/11_linux_scon
# Set ramdisk mode to automatic
echo RAMDISK_MODE=2 > /tmp/harddisk/etc/sysconfig/ramdisk
# Automatically resize the root partition to its maximum size at first boot
touch /tmp/harddisk/.partresize
# Unmount
umount /tmp/harddisk/proc
umount /tmp/harddisk/sys
umount /tmp/harddisk/dev
umount /tmp/harddisk/boot/efi
umount /tmp/harddisk/boot
umount /tmp/harddisk
# zerofree the ext2 images to get better compression
zerofree /dev/mapper/loop0p1
fsck.ext2 -f -y /dev/mapper/loop0p1
e2fsck 1.47.0 (5-Feb-2023)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/mapper/loop0p1: 631/130048 files (22.2% non-contiguous), 124148/520192 blocks
fsck.ext2 -f -y /dev/mapper/loop0p1
e2fsck 1.47.0 (5-Feb-2023)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/mapper/loop0p1: 631/130048 files (22.2% non-contiguous), 124148/520192 blocks
zerofree /dev/mapper/loop0p3
fsck.ext4 -f -y /dev/mapper/loop0p3
e2fsck 1.47.0 (5-Feb-2023)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/mapper/loop0p3: 25477/118080 files (0.1% non-contiguous), 418098/471661 blocks
fsck.ext4 -f -y /dev/mapper/loop0p3
e2fsck 1.47.0 (5-Feb-2023)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/mapper/loop0p3: 25477/118080 files (0.1% non-contiguous), 418098/471661 blocks
kpartx -d -v /dev/loop0
del devmap : loop0p1
del devmap : loop0p2
del devmap : loop0p3
losetup -d /dev/loop0
# Add padding at the end of the image (to fix alignment issues if the image is
# not copied to a block device)
dd if=/dev/zero bs=1M count=100  >> /tmp/image.img
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.0387438 s, 2.7 GB/s
# Compress Image
xz -T0 -8 < /tmp/image.img > /usr/src/images/ipfire-2.29-core187-x86_64.img.xz
xz: Reduced the number of threads from 16 to 11 to not exceed the memory usage limit of 7860 MiB
rm -rf /tmp/image.img /tmp/harddisk /tmp/cdrom
# Create checksum file
cd /usr/src/images && b2sum "ipfire-2.29-core187-x86_64.img.xz" > "ipfire-2.29-core187-x86_64.img.xz".b2"
/bin/sh: -c: line 1: unexpected EOF while looking for matching `"'
make: *** [flash-images:186: /usr/src/log/flash-image] Error 2
make: Leaving directory '/usr/src/lfs'

In the images directory there is the iso and img files but only a b2 file for the iso.

Prior to the above section of the log there are about 165 entries similar saying:-

ERROR: ld.so: object '/tools_x86_64/lib/libpakfire_preload.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.

Not sure that it has anything to do with the flash-images fail I am having as all those messages say the fail was ignored.

Looking in the flash-images lfs file there looks like

	# Create checksum file
	cd $(IMAGES_DIR) && b2sum "$(notdir $(IMAGE_FILE))" > "$(notdir "$(IMAGE_FILE)").b2"

has "$(IMAGE_FILE)" for the destination while the source only has $(IMAGE_FILE)

So I removed the "" from the destination end and ran the build again.

This time the b2 file was created but now the flash-images failed with

     make: Nothing to be done for 'download'.
     make: Leaving directory '/home/ahb/sandbox/ms/ipfire-2.x/lfs'
     make: Entering directory '/usr/src/lfs'
     rm -rf /install/packages/package /tmp/*
     mkdir -p /install/packages/package
     eval $(cat /usr/src/config/rootfiles/core/187/meta)
     cat: /usr/src/config/rootfiles/core/187/meta: No such file or directory
     #Generate ROOTFILES from filelists
     BUILD_ARCH=x86_64 BUILDTARGET=x86_64-pc-linux-gnu KVER=6.6.32 \
     /usr/src/src/scripts/archive.files \
     /usr/src/config/rootfiles/core/187/filelists \
     /usr/src/config/rootfiles/core/187/files \
     /usr/src/config/rootfiles/core/187/files.x86_64 \
     > /tmp/ROOTFILES.tmp
     #remove excluded files from ROOTFILES
     grep -f /usr/src/config/rootfiles/core/187/exclude -v /tmp/ROOTFILES.tmp > /tmp/ROOTFILES
     make: *** [core-updates:60: core/187] Error 1
     make: Leaving directory '/usr/src/lfs'

The above might have some relation to not doing a clean before the build. So I will leave the change in the flash-images lfs file and do a clean and a fresh new build

Regards,

Adolf.

> 
> -Michael
> 
>>
>> Regards,
>>
>> Adolf.
>>
>>
>>> Regards,
>>>
>>> Adolf.
>>>
>>>
>>>>
>>>> One thing I found. I am running the new build system while I have been running some package updates with the old system with its mount points. The two have each run without any impact on the other.
>>>>
>>>> Regards,
>>>>
>>>> Adolf.
>>>>
>>>>> Regards,
>>>>> Adolf.
>>>>>
>>>>>>
>>>>>> Best,
>>>>>> -Michael
>>>>>>
>>>>>>> On 9 Jul 2024, at 22:29, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>>>>>
>>>>>>> Hello Adolf,
>>>>>>>
>>>>>>> Thank you for testing this.
>>>>>>>
>>>>>>> There have indeed been plenty of problems there… I spent a lot of time on this today and hopefully fixed most of them.
>>>>>>>
>>>>>>> I cannot build the toolchain on my machine and I am not sure why yet, but a build with the packaged toolchain runs through.
>>>>>>>
>>>>>>> I have also spent some time on getting rid of the strip stage because it annoyed me how long it takes and creating the disk images as well as packages should be significantly faster now, too. I hope I didn’t introduce too many new bugs.
>>>>>>>
>>>>>>> Please let me know if you have more success now.
>>>>>>>
>>>>>>> Best,
>>>>>>> -Michael
>>>>>>>
>>>>>>>> On 8 Jul 2024, at 20:34, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>
>>>>>>>> Hi Michael,
>>>>>>>>
>>>>>>>> On 08/07/2024 21:15, Adolf Belka wrote:
>>>>>>>>> Hi Michael,
>>>>>>>>>
>>>>>>>>> On 08/07/2024 18:11, Michael Tremer wrote:
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> I have been spending a lot of time on this problem, because it has been bothering me for a long time. I also saw an opportunity to make more changes to the build system.
>>>>>>>>>>
>>>>>>>>>> Currently this is all a little bit WIP, but I hope that we can merge this into next as soon as the current update has been moved to master.
>>>>>>>>>>
>>>>>>>>>> I am referring to this branch which is currently based on next: https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/unshare
>>>>>>>>>>
>>>>>>>>>> It makes use of the unshare command which creates new namespaces in Linux. That way, we can isolate the build system better from the host system and in case something goes wrong, there is less damage. We can also enforce some more rules…
>>>>>>>>>>
>>>>>>>>>> So, what has changed?
>>>>>>>>>>
>>>>>>>>>> * The make.sh script might re-execute itself into a new mount namespace when it is suitable. This happens for “make.sh build” and “make.sh shell”, but it does not happen for “make.sh downloadsrc” for example.
>>>>>>>>>>
>>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l2129
>>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l2251
>>>>>>>>>>
>>>>>>>>>> * The new mount namespace means that we will no longer see any bind-mounts in the host system and we no longer need to umount anything ourselves which is where we occasionally wiped the entire hard drive of the host system. When the last process exits, the namespace is being cleaned up and everything is being umounted.
>>>>>>>>>>
>>>>>>>>>> * The function that prepares the build environment has been almost entirely rewritten:
>>>>>>>>>>
>>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l426
>>>>>>>>>>
>>>>>>>>>> It used to mount parts of the host system into the build environment which are needed to run anything. Those were /dev, /proc, /sys, etc…
>>>>>>>>>>
>>>>>>>>>> Instead, it now creates a new /dev mount point and creates a minimal amount of device nodes and symlinks. That way, we detach from the host system and no longer allow the build system access to the host’s filesystem and block devices. We also bind-mount the sources in read-only mode now, so that the build system cannot change anything in the source tree. On top of that, cache is read-only, too. ccache and the log directory are the only places that are writable.
>>>>>>>>>>
>>>>>>>>>> We mount a separate /tmp directory.
>>>>>>>>>>
>>>>>>>>>> * When we then build a package, we create more namespaces for each package. These isolate each build process from each other.
>>>>>>>>>>
>>>>>>>>>> Mostly, this is to detach from the host system. A new UTS namespace allows to change the hostname in the build system without affecting the host and so on. We do the same thing with a new time namespace.
>>>>>>>>>>
>>>>>>>>>> We do however create a new PID namespace which means that the build system no longer will see any processes running on the host system. That requires to mount a new instance of /proc with each package. This also has the effect that if the shell that we launched terminates (because the build is done) any background processes will be killed immediately.
>>>>>>>>>>
>>>>>>>>>> Last, we clone the mount namespace that we have created before so that no build command can modify what we set up earlier.
>>>>>>>>>>
>>>>>>>>>> * Since everything is now so decoupled, we gain a couple of new (maybe minor?) features:
>>>>>>>>>>
>>>>>>>>>>     It is now possible to run “make.sh shell” while a build is running. That does not happen a lot, but we can do this now :)
>>>>>>>>>>
>>>>>>>>>>     If the build crashes or the host system is being shut down while a build is running, there is nothing to clean up afterwards.
>>>>>>>>>>
>>>>>>>>>> * I have garnished this all with a lot of code cleanup and I suppose I might have introduced some new bugs here or there :)
>>>>>>>>>>
>>>>>>>>>> * This is probably mostly around a new implementation of the timer that updates the build time. It has been annoying me a lot that it takes a long time to walk through all packages that have been built before to finally get to a package that we want to rebuild. Mostly this was all help up by a call of “sleep 0.1”
>>>>>>>>>>
>>>>>>>>>> Since bash does not really do any concurrency, I had to be creative and replaced the busy-loop with a background process that is launched whenever it is needed and which will “ping” the main make.sh script once a second. That way, we can just run as usual, but regularly get interrupted to update the runtime.
>>>>>>>>>>
>>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l361
>>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l834
>>>>>>>>>>
>>>>>>>>>> We now only fork one extra sub shell and we have to handle the timer events which is a lot cheaper as well as more straight-forward to code.
>>>>>>>>>>
>>>>>>>>>> * As there is no difference between the different stages any more (those stages that we inherited from LFS), I have merged them all into one.
>>>>>>>>>>
>>>>>>>>>> * Last but not least, I have create the option to build for multiple architectures on the same system. Since we can now mount the entire source tree into (many independent) build environments, we might as well… As discussed on the last call, this might not be the best option for ARM, but RISV-C builds at a decent speed even when emulated.
>>>>>>>>>>
>>>>>>>>>> The only thing that I needed to do for this is to suffix the build and log directories which are now called build_${ARCH}, i.e. build_aarch64, build_x86_64, and so on. The packages/ directory is not changed yet, but that will have to happen as well. Most likely I want to merge this with the generated images, but I am not sure what to call this, yet. Happy to hear suggestions. result_x86_64? Just images_x86_64?
>>>>>>>>>>
>>>>>>>>>> ---
>>>>>>>>>>
>>>>>>>>>> I have run a build and this seems to be working just fine on my Debian machine. I am writing to all of you to first of all let you know what I am up to; and secondly to ask to give this a go on your systems. I think it should run just fine, as all the tools that I require should be available everywhere. However, there might be some older kernels that might not support all of this, yet or any other problems I cannot think of yet. Please give me some feedback and send me all the bugs :)
>>>>>>>>> I gave this a go but it didn't work.
>>>>>>>>>
>>>>>>>>> Not sure if I should have run the ./make.sh clean command on the old version before I pulled the unshare branch into my clone of your repo.
>>>>>>>>>
>>>>>>>>> Should I have started with a complete new clone of your repo? I might try that anyway just to see.
>>>>>>>>>
>>>>>>>> I created a completely new clone of you ipfire-2.x repor and then checked out the unshare branch to a branch called unshare in my local repo clone.
>>>>>>>>
>>>>>>>> gettoolchain gave the same issue, except that this time the toolchain directory ended up completely empty.
>>>>>>>>
>>>>>>>> downloadsrc had the same result.
>>>>>>>>
>>>>>>>> clean had nothing to clean up as it was a fresh clone.
>>>>>>>>
>>>>>>>> build then tried to build the toolchain and came up with this error, different from before.
>>>>>>>>
>>>>>>>> ./make.sh build
>>>>>>>> chroot: failed to run command ‘env’: No such file or directory
>>>>>>>> Full toolchain compilation
>>>>>>>> stage1 [ FAIL ]
>>>>>>>>
>>>>>>>>      Jul  8 19:26:39: Building stage1 ====================================== Installing stage1 ...
>>>>>>>>      mkdir -pv /tools_x86_64/lib
>>>>>>>>      mkdir: cannot create directory '/tools_x86_64': File exists
>>>>>>>>      make: *** [stage1:50: /home/ahb/sandbox/ms/ipfire-2.x/log/stage1] Error 1
>>>>>>>>
>>>>>>>> ERROR: Building stage1 [ FAIL ]
>>>>>>>>      Check /home/ahb/sandbox/ms/ipfire-2.x/log_x86_64/_build.toolchain.log for errors if applicable [ FAIL ]
>>>>>>>>
>>>>>>>> so it wasn't as simple as doing a fresh git clone.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Adolf.
>>>>>>>>
>>>>>>>>
>>>>>>>>> So I ran ./make.sh gettoolchain first, as I usually would.
>>>>>>>>>
>>>>>>>>> ./make.sh gettoolchain
>>>>>>>>> b2sum: cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: No such file or directory
>>>>>>>>> cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: FAILED open or read
>>>>>>>>> b2sum: WARNING: 1 listed file could not be read
>>>>>>>>>
>>>>>>>>> ipfire-2.29-toolchain-20240210-x86_64.tar.zst is present together with its b2 file.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Then ran ./make.sh downloadsrc
>>>>>>>>>
>>>>>>>>> Previous version ends with
>>>>>>>>>
>>>>>>>>> ***Verifying BLAKE2 checksum
>>>>>>>>> all files BLAKE2 checksum match [ DONE]
>>>>>>>>>
>>>>>>>>> after zstd has been checked.
>>>>>>>>>
>>>>>>>>> New version stops at zstd entry.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ./make.sh clean gave the message Cleaning Build directory... but was completed very quickly.
>>>>>>>>> Log and Build directories have not been cleaned out. The img and iso files are still present.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ./make.sh build gave message
>>>>>>>>>
>>>>>>>>> chroot: failed to run command ‘env’: No such file or directory
>>>>>>>>>
>>>>>>>>> and then did a full toolchain compilation which failed with gcc but log is >9000 lines.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Adolf.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thank you for listening to this brain-dump.
>>>>>>>>>>
>>>>>>>>>> All the best,
>>>>>>>>>> -Michael
>>>>>>>>>>
>>>>>>>>>>> On 3 Jul 2024, at 10:58, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hello Adolf,
>>>>>>>>>>>
>>>>>>>>>>> This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more.
>>>>>>>>>>>
>>>>>>>>>>> I rebooted the machine and it is back up again.
>>>>>>>>>>>
>>>>>>>>>>> -Michael
>>>>>>>>>>>
>>>>>>>>>>>> On 2 Jul 2024, at 15:42, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Michael and all,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I ran the arm builder with the 4.20.2 version of samba to test it out.
>>>>>>>>>>>>
>>>>>>>>>>>> The build got to building gdb and then failed.
>>>>>>>>>>>>
>>>>>>>>>>>> Interestingly, the nightly build of arm was successful with the same version of gdb.
>>>>>>>>>>>>
>>>>>>>>>>>> The build log for gdb is attached. The actual error is at line 618.
>>>>>>>>>>>>
>>>>>>>>>>>> Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message.
>>>>>>>>>>>>
>>>>>>>>>>>> ------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>> ssh bonnietwin(a)arm64-01.zrh.ipfire.org
>>>>>>>>>>>> PTY allocation request failed on channel 0
>>>>>>>>>>>> Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64
>>>>>>>>>>>>
>>>>>>>>>>>> The programs included with the Debian GNU/Linux system are free software;
>>>>>>>>>>>> the exact distribution terms for each program are described in the
>>>>>>>>>>>> individual files in /usr/share/doc/*/copyright.
>>>>>>>>>>>>
>>>>>>>>>>>> Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
>>>>>>>>>>>> permitted by applicable law.
>>>>>>>>>>>> /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied
>>>>>>>>>>>>
>>>>>>>>>>>> ------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>
>>>>>>>>>>>> <_build.ipfire.gdb.log>
> 
> 

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: Problem during building of samba on arm builder
  2024-07-10 16:53                       ` Adolf Belka
@ 2024-07-10 18:19                         ` Michael Tremer
  2024-07-10 21:57                           ` Adolf Belka
  0 siblings, 1 reply; 16+ messages in thread
From: Michael Tremer @ 2024-07-10 18:19 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 24700 bytes --]

Hello,

> On 10 Jul 2024, at 17:53, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi Michael,
> 
> On 10/07/2024 16:24, Michael Tremer wrote:
>> Hello,
>>> On 10 Jul 2024, at 14:21, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>> 
>>> Hi Michael,
>>> 
>>> On 10/07/2024 15:05, Adolf Belka wrote:
>>>> Hi Michael,
>>>> 
>>>> On 10/07/2024 14:59, Adolf Belka wrote:
>>>>> Hi Michael,
>>>>> 
>>>>> On 10/07/2024 12:33, Adolf Belka wrote:
>>>>>> Hi Michael,
>>>>>> 
>>>>>> On 10/07/2024 11:57, Michael Tremer wrote:
>>>>>>> Hello again,
>>>>>>> 
>>>>>>> I managed to (finally) build the toolchain with the updated system. So hopefully there should not be any more outstanding problems that I know of so far.
>>>>>> 
>>>>>> I just did a git pull on your repo to my clone.
>>>>>> 
>>>>>> Ran ./make.sh gettoolchain and it successfully downloaded the toolchain.
>>>>>> 
>>>>>> Ran ./make.sh downloadsrc and it successfully tested everything.
>>>>>> 
>>>>>> Ran ./make.sh clean and build and log directories were cleared out and removed. As far as I can tell it was successful.
>>>>>> 
>>>>>> Currently running ./make.sh build. So up to this point everything going well. Will let you know how it goes.
>>>>>> 
>>>>> It has got to building popt. In the normal build system this takes around 3 secs. Currently in the new build it is at nearly 2 hours. Even with an empty cache, that seems a long build time for popt, unless I am being too optimistic.
>>>>> 
>>>>> I will let it keep going.
>>>>> 
>>>> I just had a look at the log file and it looks like it completed popt but then is stuck on trying to leave the directory /usr/src/lfs. Here is the output from the log, nothing new is getting written to the log.
>>>> 
>>>> make[3]: Leaving directory '/usr/src/popt-1.19/po'
>>>> Making install in tests
>>>> make[3]: Entering directory '/usr/src/popt-1.19/tests'
>>>> make[4]: Entering directory '/usr/src/popt-1.19/tests'
>>>> make[4]: Nothing to be done for 'install-exec-am'.
>>>> make[4]: Nothing to be done for 'install-data-am'.
>>>> make[4]: Leaving directory '/usr/src/popt-1.19/tests'
>>>> make[3]: Leaving directory '/usr/src/popt-1.19/tests'
>>>> make[3]: Entering directory '/usr/src/popt-1.19'
>>>> make[4]: Entering directory '/usr/src/popt-1.19'
>>>> make[4]: Nothing to be done for 'install-exec-am'.
>>>>  /bin/mkdir -p '/usr/share/man/man3'
>>>>  /usr/bin/install -c -m 644 popt.3 '/usr/share/man/man3'
>>>>  /bin/mkdir -p '/usr/lib/pkgconfig'
>>>>  /usr/bin/install -c -m 644 popt.pc '/usr/lib/pkgconfig'
>>>> make[4]: Leaving directory '/usr/src/popt-1.19'
>>>> make[3]: Leaving directory '/usr/src/popt-1.19'
>>>> make[2]: Leaving directory '/usr/src/popt-1.19'
>>>> make[1]: Leaving directory '/usr/src/popt-1.19'
>>>> Updating linker cache...
>>>> Install done; saving file list to /usr/src/log/popt-1.19 ...
>>>> 
>>>> make: Leaving directory '/usr/src/lfs'
>>>> 
>>>> 
>>> I stopped the build with Ctrl-C and then ran ./make.sh build again without doing a clean. It got to popt and went straight to libedit, the next package, and started to build it.
>>> 
>>> Something must have just put the build into a loop of some sort. It was not writing anything to the log file.
>> The log file says that popt was done. It might be that you interrupted the build just after it has finished, but the root file was not created, yet. That should however not directly be a problem…
> 2 hours had already gone by when I checked the log for that message. I then left the system running for a further 20 minutes with no new info in the log file and that is when I interrupted the build.

Oh, that. Yes, I think I have seen this too, but I am not sure why yet.

It looks like the wait call misses that the process has already been gone here:

  https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=69e1fd7e45b42657f8b4c9352089cc430740b6c4;hb=refs/heads/unshare#l810

> 
>> If this is however the only problem that you are seeing now than I would be happy :)
> 
> I have found another problem when it got to the flash-images section.
> 
> The flash-images log stops with the following:-
> 
> # Insert the UUID because grub-mkconfig often fails to
> # detect that correctly
> sed -i /tmp/harddisk/boot/grub/grub.cfg \
> -e "s/root=[A-Za-z0-9\/=-]*/root=UUID=$(blkid -o value -s UUID /dev/mapper/loop0p3)/g"
> # Install GRUB
> grub-install --force --recheck --no-floppy --target=i386-pc \
> --root-directory=/tmp/harddisk /dev/loop0
> Installing for i386-pc platform.
> Installation finished. No error reported.
> # Install GRUB for EFI
> grub-install --target=x86_64-efi --removable --no-nvram \
> --boot-directory=/tmp/harddisk/boot --efi-directory=/tmp/harddisk/boot/efi
> Installing for x86_64-efi platform.
> Installation finished. No error reported.
> # restore orginal defaults
> mv -f /tmp/harddisk/etc/default/grub.backup /tmp/harddisk/etc/default/grub
> rm -f /tmp/harddisk/etc/grub.d/11_linux_scon
> # Set ramdisk mode to automatic
> echo RAMDISK_MODE=2 > /tmp/harddisk/etc/sysconfig/ramdisk
> # Automatically resize the root partition to its maximum size at first boot
> touch /tmp/harddisk/.partresize
> # Unmount
> umount /tmp/harddisk/proc
> umount /tmp/harddisk/sys
> umount /tmp/harddisk/dev
> umount /tmp/harddisk/boot/efi
> umount /tmp/harddisk/boot
> umount /tmp/harddisk
> # zerofree the ext2 images to get better compression
> zerofree /dev/mapper/loop0p1
> fsck.ext2 -f -y /dev/mapper/loop0p1
> e2fsck 1.47.0 (5-Feb-2023)
> Pass 1: Checking inodes, blocks, and sizes
> Pass 2: Checking directory structure
> Pass 3: Checking directory connectivity
> Pass 4: Checking reference counts
> Pass 5: Checking group summary information
> /dev/mapper/loop0p1: 631/130048 files (22.2% non-contiguous), 124148/520192 blocks
> fsck.ext2 -f -y /dev/mapper/loop0p1
> e2fsck 1.47.0 (5-Feb-2023)
> Pass 1: Checking inodes, blocks, and sizes
> Pass 2: Checking directory structure
> Pass 3: Checking directory connectivity
> Pass 4: Checking reference counts
> Pass 5: Checking group summary information
> /dev/mapper/loop0p1: 631/130048 files (22.2% non-contiguous), 124148/520192 blocks
> zerofree /dev/mapper/loop0p3
> fsck.ext4 -f -y /dev/mapper/loop0p3
> e2fsck 1.47.0 (5-Feb-2023)
> Pass 1: Checking inodes, blocks, and sizes
> Pass 2: Checking directory structure
> Pass 3: Checking directory connectivity
> Pass 4: Checking reference counts
> Pass 5: Checking group summary information
> /dev/mapper/loop0p3: 25477/118080 files (0.1% non-contiguous), 418098/471661 blocks
> fsck.ext4 -f -y /dev/mapper/loop0p3
> e2fsck 1.47.0 (5-Feb-2023)
> Pass 1: Checking inodes, blocks, and sizes
> Pass 2: Checking directory structure
> Pass 3: Checking directory connectivity
> Pass 4: Checking reference counts
> Pass 5: Checking group summary information
> /dev/mapper/loop0p3: 25477/118080 files (0.1% non-contiguous), 418098/471661 blocks
> kpartx -d -v /dev/loop0
> del devmap : loop0p1
> del devmap : loop0p2
> del devmap : loop0p3
> losetup -d /dev/loop0
> # Add padding at the end of the image (to fix alignment issues if the image is
> # not copied to a block device)
> dd if=/dev/zero bs=1M count=100  >> /tmp/image.img
> 100+0 records in
> 100+0 records out
> 104857600 bytes (105 MB, 100 MiB) copied, 0.0387438 s, 2.7 GB/s
> # Compress Image
> xz -T0 -8 < /tmp/image.img > /usr/src/images/ipfire-2.29-core187-x86_64.img.xz
> xz: Reduced the number of threads from 16 to 11 to not exceed the memory usage limit of 7860 MiB
> rm -rf /tmp/image.img /tmp/harddisk /tmp/cdrom
> # Create checksum file
> cd /usr/src/images && b2sum "ipfire-2.29-core187-x86_64.img.xz" > "ipfire-2.29-core187-x86_64.img.xz".b2"
> /bin/sh: -c: line 1: unexpected EOF while looking for matching `"'
> make: *** [flash-images:186: /usr/src/log/flash-image] Error 2
> make: Leaving directory '/usr/src/lfs'

I fixed the extra quotes:

  https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=d24f7382fbe384f81e387f85af06ea19cad908c9

> In the images directory there is the iso and img files but only a b2 file for the iso.
> 
> Prior to the above section of the log there are about 165 entries similar saying:-
> 
> ERROR: ld.so: object '/tools_x86_64/lib/libpakfire_preload.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.

This comes from the commands that we run in a second, inner chroot. I will have to check whether we still need all this stuff to fake the environment. QEMU might do most of the job for us when we are emulating a different architecture, and otherwise we don’t need this any more I believe.

> Not sure that it has anything to do with the flash-images fail I am having as all those messages say the fail was ignored.
> 
> Looking in the flash-images lfs file there looks like
> 
> # Create checksum file
> cd $(IMAGES_DIR) && b2sum "$(notdir $(IMAGE_FILE))" > "$(notdir "$(IMAGE_FILE)").b2"
> 
> has "$(IMAGE_FILE)" for the destination while the source only has $(IMAGE_FILE)
> 
> So I removed the "" from the destination end and ran the build again.
> 
> This time the b2 file was created but now the flash-images failed with
> 
>    make: Nothing to be done for 'download'.
>    make: Leaving directory '/home/ahb/sandbox/ms/ipfire-2.x/lfs'
>    make: Entering directory '/usr/src/lfs'
>    rm -rf /install/packages/package /tmp/*
>    mkdir -p /install/packages/package
>    eval $(cat /usr/src/config/rootfiles/core/187/meta)
>    cat: /usr/src/config/rootfiles/core/187/meta: No such file or directory
>    #Generate ROOTFILES from filelists
>    BUILD_ARCH=x86_64 BUILDTARGET=x86_64-pc-linux-gnu KVER=6.6.32 \
>    /usr/src/src/scripts/archive.files \
>    /usr/src/config/rootfiles/core/187/filelists \
>    /usr/src/config/rootfiles/core/187/files \
>    /usr/src/config/rootfiles/core/187/files.x86_64 \
>    > /tmp/ROOTFILES.tmp
>    #remove excluded files from ROOTFILES
>    grep -f /usr/src/config/rootfiles/core/187/exclude -v /tmp/ROOTFILES.tmp > /tmp/ROOTFILES
>    make: *** [core-updates:60: core/187] Error 1
>    make: Leaving directory '/usr/src/lfs'

I didn’t look at the packaging before, but this has also been dealt with. Please pull again and re-run the build.

> The above might have some relation to not doing a clean before the build. So I will leave the change in the flash-images lfs file and do a clean and a fresh new build

A clean build should not be needed.

-Michael

> 
> Regards,
> 
> Adolf.
> 
>> -Michael
>>> 
>>> Regards,
>>> 
>>> Adolf.
>>> 
>>> 
>>>> Regards,
>>>> 
>>>> Adolf.
>>>> 
>>>> 
>>>>> 
>>>>> One thing I found. I am running the new build system while I have been running some package updates with the old system with its mount points. The two have each run without any impact on the other.
>>>>> 
>>>>> Regards,
>>>>> 
>>>>> Adolf.
>>>>> 
>>>>>> Regards,
>>>>>> Adolf.
>>>>>> 
>>>>>>> 
>>>>>>> Best,
>>>>>>> -Michael
>>>>>>> 
>>>>>>>> On 9 Jul 2024, at 22:29, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>>>>>> 
>>>>>>>> Hello Adolf,
>>>>>>>> 
>>>>>>>> Thank you for testing this.
>>>>>>>> 
>>>>>>>> There have indeed been plenty of problems there… I spent a lot of time on this today and hopefully fixed most of them.
>>>>>>>> 
>>>>>>>> I cannot build the toolchain on my machine and I am not sure why yet, but a build with the packaged toolchain runs through.
>>>>>>>> 
>>>>>>>> I have also spent some time on getting rid of the strip stage because it annoyed me how long it takes and creating the disk images as well as packages should be significantly faster now, too. I hope I didn’t introduce too many new bugs.
>>>>>>>> 
>>>>>>>> Please let me know if you have more success now.
>>>>>>>> 
>>>>>>>> Best,
>>>>>>>> -Michael
>>>>>>>> 
>>>>>>>>> On 8 Jul 2024, at 20:34, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>> 
>>>>>>>>> Hi Michael,
>>>>>>>>> 
>>>>>>>>> On 08/07/2024 21:15, Adolf Belka wrote:
>>>>>>>>>> Hi Michael,
>>>>>>>>>> 
>>>>>>>>>> On 08/07/2024 18:11, Michael Tremer wrote:
>>>>>>>>>>> Hello,
>>>>>>>>>>> 
>>>>>>>>>>> I have been spending a lot of time on this problem, because it has been bothering me for a long time. I also saw an opportunity to make more changes to the build system.
>>>>>>>>>>> 
>>>>>>>>>>> Currently this is all a little bit WIP, but I hope that we can merge this into next as soon as the current update has been moved to master.
>>>>>>>>>>> 
>>>>>>>>>>> I am referring to this branch which is currently based on next: https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/unshare
>>>>>>>>>>> 
>>>>>>>>>>> It makes use of the unshare command which creates new namespaces in Linux. That way, we can isolate the build system better from the host system and in case something goes wrong, there is less damage. We can also enforce some more rules…
>>>>>>>>>>> 
>>>>>>>>>>> So, what has changed?
>>>>>>>>>>> 
>>>>>>>>>>> * The make.sh script might re-execute itself into a new mount namespace when it is suitable. This happens for “make.sh build” and “make.sh shell”, but it does not happen for “make.sh downloadsrc” for example.
>>>>>>>>>>> 
>>>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l2129
>>>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l2251
>>>>>>>>>>> 
>>>>>>>>>>> * The new mount namespace means that we will no longer see any bind-mounts in the host system and we no longer need to umount anything ourselves which is where we occasionally wiped the entire hard drive of the host system. When the last process exits, the namespace is being cleaned up and everything is being umounted.
>>>>>>>>>>> 
>>>>>>>>>>> * The function that prepares the build environment has been almost entirely rewritten:
>>>>>>>>>>> 
>>>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l426
>>>>>>>>>>> 
>>>>>>>>>>> It used to mount parts of the host system into the build environment which are needed to run anything. Those were /dev, /proc, /sys, etc…
>>>>>>>>>>> 
>>>>>>>>>>> Instead, it now creates a new /dev mount point and creates a minimal amount of device nodes and symlinks. That way, we detach from the host system and no longer allow the build system access to the host’s filesystem and block devices. We also bind-mount the sources in read-only mode now, so that the build system cannot change anything in the source tree. On top of that, cache is read-only, too. ccache and the log directory are the only places that are writable.
>>>>>>>>>>> 
>>>>>>>>>>> We mount a separate /tmp directory.
>>>>>>>>>>> 
>>>>>>>>>>> * When we then build a package, we create more namespaces for each package. These isolate each build process from each other.
>>>>>>>>>>> 
>>>>>>>>>>> Mostly, this is to detach from the host system. A new UTS namespace allows to change the hostname in the build system without affecting the host and so on. We do the same thing with a new time namespace.
>>>>>>>>>>> 
>>>>>>>>>>> We do however create a new PID namespace which means that the build system no longer will see any processes running on the host system. That requires to mount a new instance of /proc with each package. This also has the effect that if the shell that we launched terminates (because the build is done) any background processes will be killed immediately.
>>>>>>>>>>> 
>>>>>>>>>>> Last, we clone the mount namespace that we have created before so that no build command can modify what we set up earlier.
>>>>>>>>>>> 
>>>>>>>>>>> * Since everything is now so decoupled, we gain a couple of new (maybe minor?) features:
>>>>>>>>>>> 
>>>>>>>>>>>    It is now possible to run “make.sh shell” while a build is running. That does not happen a lot, but we can do this now :)
>>>>>>>>>>> 
>>>>>>>>>>>    If the build crashes or the host system is being shut down while a build is running, there is nothing to clean up afterwards.
>>>>>>>>>>> 
>>>>>>>>>>> * I have garnished this all with a lot of code cleanup and I suppose I might have introduced some new bugs here or there :)
>>>>>>>>>>> 
>>>>>>>>>>> * This is probably mostly around a new implementation of the timer that updates the build time. It has been annoying me a lot that it takes a long time to walk through all packages that have been built before to finally get to a package that we want to rebuild. Mostly this was all help up by a call of “sleep 0.1”
>>>>>>>>>>> 
>>>>>>>>>>> Since bash does not really do any concurrency, I had to be creative and replaced the busy-loop with a background process that is launched whenever it is needed and which will “ping” the main make.sh script once a second. That way, we can just run as usual, but regularly get interrupted to update the runtime.
>>>>>>>>>>> 
>>>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l361
>>>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l834
>>>>>>>>>>> 
>>>>>>>>>>> We now only fork one extra sub shell and we have to handle the timer events which is a lot cheaper as well as more straight-forward to code.
>>>>>>>>>>> 
>>>>>>>>>>> * As there is no difference between the different stages any more (those stages that we inherited from LFS), I have merged them all into one.
>>>>>>>>>>> 
>>>>>>>>>>> * Last but not least, I have create the option to build for multiple architectures on the same system. Since we can now mount the entire source tree into (many independent) build environments, we might as well… As discussed on the last call, this might not be the best option for ARM, but RISV-C builds at a decent speed even when emulated.
>>>>>>>>>>> 
>>>>>>>>>>> The only thing that I needed to do for this is to suffix the build and log directories which are now called build_${ARCH}, i.e. build_aarch64, build_x86_64, and so on. The packages/ directory is not changed yet, but that will have to happen as well. Most likely I want to merge this with the generated images, but I am not sure what to call this, yet. Happy to hear suggestions. result_x86_64? Just images_x86_64?
>>>>>>>>>>> 
>>>>>>>>>>> ---
>>>>>>>>>>> 
>>>>>>>>>>> I have run a build and this seems to be working just fine on my Debian machine. I am writing to all of you to first of all let you know what I am up to; and secondly to ask to give this a go on your systems. I think it should run just fine, as all the tools that I require should be available everywhere. However, there might be some older kernels that might not support all of this, yet or any other problems I cannot think of yet. Please give me some feedback and send me all the bugs :)
>>>>>>>>>> I gave this a go but it didn't work.
>>>>>>>>>> 
>>>>>>>>>> Not sure if I should have run the ./make.sh clean command on the old version before I pulled the unshare branch into my clone of your repo.
>>>>>>>>>> 
>>>>>>>>>> Should I have started with a complete new clone of your repo? I might try that anyway just to see.
>>>>>>>>>> 
>>>>>>>>> I created a completely new clone of you ipfire-2.x repor and then checked out the unshare branch to a branch called unshare in my local repo clone.
>>>>>>>>> 
>>>>>>>>> gettoolchain gave the same issue, except that this time the toolchain directory ended up completely empty.
>>>>>>>>> 
>>>>>>>>> downloadsrc had the same result.
>>>>>>>>> 
>>>>>>>>> clean had nothing to clean up as it was a fresh clone.
>>>>>>>>> 
>>>>>>>>> build then tried to build the toolchain and came up with this error, different from before.
>>>>>>>>> 
>>>>>>>>> ./make.sh build
>>>>>>>>> chroot: failed to run command ‘env’: No such file or directory
>>>>>>>>> Full toolchain compilation
>>>>>>>>> stage1 [ FAIL ]
>>>>>>>>> 
>>>>>>>>>     Jul  8 19:26:39: Building stage1 ====================================== Installing stage1 ...
>>>>>>>>>     mkdir -pv /tools_x86_64/lib
>>>>>>>>>     mkdir: cannot create directory '/tools_x86_64': File exists
>>>>>>>>>     make: *** [stage1:50: /home/ahb/sandbox/ms/ipfire-2.x/log/stage1] Error 1
>>>>>>>>> 
>>>>>>>>> ERROR: Building stage1 [ FAIL ]
>>>>>>>>>     Check /home/ahb/sandbox/ms/ipfire-2.x/log_x86_64/_build.toolchain.log for errors if applicable [ FAIL ]
>>>>>>>>> 
>>>>>>>>> so it wasn't as simple as doing a fresh git clone.
>>>>>>>>> 
>>>>>>>>> Regards,
>>>>>>>>> 
>>>>>>>>> Adolf.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> So I ran ./make.sh gettoolchain first, as I usually would.
>>>>>>>>>> 
>>>>>>>>>> ./make.sh gettoolchain
>>>>>>>>>> b2sum: cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: No such file or directory
>>>>>>>>>> cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: FAILED open or read
>>>>>>>>>> b2sum: WARNING: 1 listed file could not be read
>>>>>>>>>> 
>>>>>>>>>> ipfire-2.29-toolchain-20240210-x86_64.tar.zst is present together with its b2 file.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Then ran ./make.sh downloadsrc
>>>>>>>>>> 
>>>>>>>>>> Previous version ends with
>>>>>>>>>> 
>>>>>>>>>> ***Verifying BLAKE2 checksum
>>>>>>>>>> all files BLAKE2 checksum match [ DONE]
>>>>>>>>>> 
>>>>>>>>>> after zstd has been checked.
>>>>>>>>>> 
>>>>>>>>>> New version stops at zstd entry.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> ./make.sh clean gave the message Cleaning Build directory... but was completed very quickly.
>>>>>>>>>> Log and Build directories have not been cleaned out. The img and iso files are still present.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> ./make.sh build gave message
>>>>>>>>>> 
>>>>>>>>>> chroot: failed to run command ‘env’: No such file or directory
>>>>>>>>>> 
>>>>>>>>>> and then did a full toolchain compilation which failed with gcc but log is >9000 lines.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Regards,
>>>>>>>>>> Adolf.
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Thank you for listening to this brain-dump.
>>>>>>>>>>> 
>>>>>>>>>>> All the best,
>>>>>>>>>>> -Michael
>>>>>>>>>>> 
>>>>>>>>>>>> On 3 Jul 2024, at 10:58, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hello Adolf,
>>>>>>>>>>>> 
>>>>>>>>>>>> This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more.
>>>>>>>>>>>> 
>>>>>>>>>>>> I rebooted the machine and it is back up again.
>>>>>>>>>>>> 
>>>>>>>>>>>> -Michael
>>>>>>>>>>>> 
>>>>>>>>>>>>> On 2 Jul 2024, at 15:42, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi Michael and all,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I ran the arm builder with the 4.20.2 version of samba to test it out.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The build got to building gdb and then failed.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Interestingly, the nightly build of arm was successful with the same version of gdb.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The build log for gdb is attached. The actual error is at line 618.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> ------------------------------------------
>>>>>>>>>>>>> 
>>>>>>>>>>>>> ssh bonnietwin(a)arm64-01.zrh.ipfire.org
>>>>>>>>>>>>> PTY allocation request failed on channel 0
>>>>>>>>>>>>> Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The programs included with the Debian GNU/Linux system are free software;
>>>>>>>>>>>>> the exact distribution terms for each program are described in the
>>>>>>>>>>>>> individual files in /usr/share/doc/*/copyright.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
>>>>>>>>>>>>> permitted by applicable law.
>>>>>>>>>>>>> /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied
>>>>>>>>>>>>> 
>>>>>>>>>>>>> ------------------------------------------
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> <_build.ipfire.gdb.log>



^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: Problem during building of samba on arm builder
  2024-07-10 18:19                         ` Michael Tremer
@ 2024-07-10 21:57                           ` Adolf Belka
  0 siblings, 0 replies; 16+ messages in thread
From: Adolf Belka @ 2024-07-10 21:57 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 25283 bytes --]

Hi Michael,

On 10/07/2024 20:19, Michael Tremer wrote:
> Hello,
> 
>> On 10 Jul 2024, at 17:53, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>
>> Hi Michael,
>>
>> On 10/07/2024 16:24, Michael Tremer wrote:
>>> Hello,
>>>> On 10 Jul 2024, at 14:21, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>
>>>> Hi Michael,
>>>>
>>>> On 10/07/2024 15:05, Adolf Belka wrote:
>>>>> Hi Michael,
>>>>>
>>>>> On 10/07/2024 14:59, Adolf Belka wrote:
>>>>>> Hi Michael,
>>>>>>
>>>>>> On 10/07/2024 12:33, Adolf Belka wrote:
>>>>>>> Hi Michael,
>>>>>>>
>>>>>>> On 10/07/2024 11:57, Michael Tremer wrote:
>>>>>>>> Hello again,
>>>>>>>>
>>>>>>>> I managed to (finally) build the toolchain with the updated system. So hopefully there should not be any more outstanding problems that I know of so far.
>>>>>>>
>>>>>>> I just did a git pull on your repo to my clone.
>>>>>>>
>>>>>>> Ran ./make.sh gettoolchain and it successfully downloaded the toolchain.
>>>>>>>
>>>>>>> Ran ./make.sh downloadsrc and it successfully tested everything.
>>>>>>>
>>>>>>> Ran ./make.sh clean and build and log directories were cleared out and removed. As far as I can tell it was successful.
>>>>>>>
>>>>>>> Currently running ./make.sh build. So up to this point everything going well. Will let you know how it goes.
>>>>>>>
>>>>>> It has got to building popt. In the normal build system this takes around 3 secs. Currently in the new build it is at nearly 2 hours. Even with an empty cache, that seems a long build time for popt, unless I am being too optimistic.
>>>>>>
>>>>>> I will let it keep going.
>>>>>>
>>>>> I just had a look at the log file and it looks like it completed popt but then is stuck on trying to leave the directory /usr/src/lfs. Here is the output from the log, nothing new is getting written to the log.
>>>>>
>>>>> make[3]: Leaving directory '/usr/src/popt-1.19/po'
>>>>> Making install in tests
>>>>> make[3]: Entering directory '/usr/src/popt-1.19/tests'
>>>>> make[4]: Entering directory '/usr/src/popt-1.19/tests'
>>>>> make[4]: Nothing to be done for 'install-exec-am'.
>>>>> make[4]: Nothing to be done for 'install-data-am'.
>>>>> make[4]: Leaving directory '/usr/src/popt-1.19/tests'
>>>>> make[3]: Leaving directory '/usr/src/popt-1.19/tests'
>>>>> make[3]: Entering directory '/usr/src/popt-1.19'
>>>>> make[4]: Entering directory '/usr/src/popt-1.19'
>>>>> make[4]: Nothing to be done for 'install-exec-am'.
>>>>>   /bin/mkdir -p '/usr/share/man/man3'
>>>>>   /usr/bin/install -c -m 644 popt.3 '/usr/share/man/man3'
>>>>>   /bin/mkdir -p '/usr/lib/pkgconfig'
>>>>>   /usr/bin/install -c -m 644 popt.pc '/usr/lib/pkgconfig'
>>>>> make[4]: Leaving directory '/usr/src/popt-1.19'
>>>>> make[3]: Leaving directory '/usr/src/popt-1.19'
>>>>> make[2]: Leaving directory '/usr/src/popt-1.19'
>>>>> make[1]: Leaving directory '/usr/src/popt-1.19'
>>>>> Updating linker cache...
>>>>> Install done; saving file list to /usr/src/log/popt-1.19 ...
>>>>>
>>>>> make: Leaving directory '/usr/src/lfs'
>>>>>
>>>>>
>>>> I stopped the build with Ctrl-C and then ran ./make.sh build again without doing a clean. It got to popt and went straight to libedit, the next package, and started to build it.
>>>>
>>>> Something must have just put the build into a loop of some sort. It was not writing anything to the log file.
>>> The log file says that popt was done. It might be that you interrupted the build just after it has finished, but the root file was not created, yet. That should however not directly be a problem…
>> 2 hours had already gone by when I checked the log for that message. I then left the system running for a further 20 minutes with no new info in the log file and that is when I interrupted the build.
> 
> Oh, that. Yes, I think I have seen this too, but I am not sure why yet.
> 
> It looks like the wait call misses that the process has already been gone here:
> 
>    https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=69e1fd7e45b42657f8b4c9352089cc430740b6c4;hb=refs/heads/unshare#l810
> 
>>
>>> If this is however the only problem that you are seeing now than I would be happy :)
>>
>> I have found another problem when it got to the flash-images section.
>>
>> The flash-images log stops with the following:-
>>
>> # Insert the UUID because grub-mkconfig often fails to
>> # detect that correctly
>> sed -i /tmp/harddisk/boot/grub/grub.cfg \
>> -e "s/root=[A-Za-z0-9\/=-]*/root=UUID=$(blkid -o value -s UUID /dev/mapper/loop0p3)/g"
>> # Install GRUB
>> grub-install --force --recheck --no-floppy --target=i386-pc \
>> --root-directory=/tmp/harddisk /dev/loop0
>> Installing for i386-pc platform.
>> Installation finished. No error reported.
>> # Install GRUB for EFI
>> grub-install --target=x86_64-efi --removable --no-nvram \
>> --boot-directory=/tmp/harddisk/boot --efi-directory=/tmp/harddisk/boot/efi
>> Installing for x86_64-efi platform.
>> Installation finished. No error reported.
>> # restore orginal defaults
>> mv -f /tmp/harddisk/etc/default/grub.backup /tmp/harddisk/etc/default/grub
>> rm -f /tmp/harddisk/etc/grub.d/11_linux_scon
>> # Set ramdisk mode to automatic
>> echo RAMDISK_MODE=2 > /tmp/harddisk/etc/sysconfig/ramdisk
>> # Automatically resize the root partition to its maximum size at first boot
>> touch /tmp/harddisk/.partresize
>> # Unmount
>> umount /tmp/harddisk/proc
>> umount /tmp/harddisk/sys
>> umount /tmp/harddisk/dev
>> umount /tmp/harddisk/boot/efi
>> umount /tmp/harddisk/boot
>> umount /tmp/harddisk
>> # zerofree the ext2 images to get better compression
>> zerofree /dev/mapper/loop0p1
>> fsck.ext2 -f -y /dev/mapper/loop0p1
>> e2fsck 1.47.0 (5-Feb-2023)
>> Pass 1: Checking inodes, blocks, and sizes
>> Pass 2: Checking directory structure
>> Pass 3: Checking directory connectivity
>> Pass 4: Checking reference counts
>> Pass 5: Checking group summary information
>> /dev/mapper/loop0p1: 631/130048 files (22.2% non-contiguous), 124148/520192 blocks
>> fsck.ext2 -f -y /dev/mapper/loop0p1
>> e2fsck 1.47.0 (5-Feb-2023)
>> Pass 1: Checking inodes, blocks, and sizes
>> Pass 2: Checking directory structure
>> Pass 3: Checking directory connectivity
>> Pass 4: Checking reference counts
>> Pass 5: Checking group summary information
>> /dev/mapper/loop0p1: 631/130048 files (22.2% non-contiguous), 124148/520192 blocks
>> zerofree /dev/mapper/loop0p3
>> fsck.ext4 -f -y /dev/mapper/loop0p3
>> e2fsck 1.47.0 (5-Feb-2023)
>> Pass 1: Checking inodes, blocks, and sizes
>> Pass 2: Checking directory structure
>> Pass 3: Checking directory connectivity
>> Pass 4: Checking reference counts
>> Pass 5: Checking group summary information
>> /dev/mapper/loop0p3: 25477/118080 files (0.1% non-contiguous), 418098/471661 blocks
>> fsck.ext4 -f -y /dev/mapper/loop0p3
>> e2fsck 1.47.0 (5-Feb-2023)
>> Pass 1: Checking inodes, blocks, and sizes
>> Pass 2: Checking directory structure
>> Pass 3: Checking directory connectivity
>> Pass 4: Checking reference counts
>> Pass 5: Checking group summary information
>> /dev/mapper/loop0p3: 25477/118080 files (0.1% non-contiguous), 418098/471661 blocks
>> kpartx -d -v /dev/loop0
>> del devmap : loop0p1
>> del devmap : loop0p2
>> del devmap : loop0p3
>> losetup -d /dev/loop0
>> # Add padding at the end of the image (to fix alignment issues if the image is
>> # not copied to a block device)
>> dd if=/dev/zero bs=1M count=100  >> /tmp/image.img
>> 100+0 records in
>> 100+0 records out
>> 104857600 bytes (105 MB, 100 MiB) copied, 0.0387438 s, 2.7 GB/s
>> # Compress Image
>> xz -T0 -8 < /tmp/image.img > /usr/src/images/ipfire-2.29-core187-x86_64.img.xz
>> xz: Reduced the number of threads from 16 to 11 to not exceed the memory usage limit of 7860 MiB
>> rm -rf /tmp/image.img /tmp/harddisk /tmp/cdrom
>> # Create checksum file
>> cd /usr/src/images && b2sum "ipfire-2.29-core187-x86_64.img.xz" > "ipfire-2.29-core187-x86_64.img.xz".b2"
>> /bin/sh: -c: line 1: unexpected EOF while looking for matching `"'
>> make: *** [flash-images:186: /usr/src/log/flash-image] Error 2
>> make: Leaving directory '/usr/src/lfs'
> 
> I fixed the extra quotes:
> 
>    https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=d24f7382fbe384f81e387f85af06ea19cad908c9
> 
>> In the images directory there is the iso and img files but only a b2 file for the iso.
>>
>> Prior to the above section of the log there are about 165 entries similar saying:-
>>
>> ERROR: ld.so: object '/tools_x86_64/lib/libpakfire_preload.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
> 
> This comes from the commands that we run in a second, inner chroot. I will have to check whether we still need all this stuff to fake the environment. QEMU might do most of the job for us when we are emulating a different architecture, and otherwise we don’t need this any more I believe.
> 
>> Not sure that it has anything to do with the flash-images fail I am having as all those messages say the fail was ignored.
>>
>> Looking in the flash-images lfs file there looks like
>>
>> # Create checksum file
>> cd $(IMAGES_DIR) && b2sum "$(notdir $(IMAGE_FILE))" > "$(notdir "$(IMAGE_FILE)").b2"
>>
>> has "$(IMAGE_FILE)" for the destination while the source only has $(IMAGE_FILE)
>>
>> So I removed the "" from the destination end and ran the build again.
>>
>> This time the b2 file was created but now the flash-images failed with
>>
>>     make: Nothing to be done for 'download'.
>>     make: Leaving directory '/home/ahb/sandbox/ms/ipfire-2.x/lfs'
>>     make: Entering directory '/usr/src/lfs'
>>     rm -rf /install/packages/package /tmp/*
>>     mkdir -p /install/packages/package
>>     eval $(cat /usr/src/config/rootfiles/core/187/meta)
>>     cat: /usr/src/config/rootfiles/core/187/meta: No such file or directory
>>     #Generate ROOTFILES from filelists
>>     BUILD_ARCH=x86_64 BUILDTARGET=x86_64-pc-linux-gnu KVER=6.6.32 \
>>     /usr/src/src/scripts/archive.files \
>>     /usr/src/config/rootfiles/core/187/filelists \
>>     /usr/src/config/rootfiles/core/187/files \
>>     /usr/src/config/rootfiles/core/187/files.x86_64 \
>>     > /tmp/ROOTFILES.tmp
>>     #remove excluded files from ROOTFILES
>>     grep -f /usr/src/config/rootfiles/core/187/exclude -v /tmp/ROOTFILES.tmp > /tmp/ROOTFILES
>>     make: *** [core-updates:60: core/187] Error 1
>>     make: Leaving directory '/usr/src/lfs'
> 
> I didn’t look at the packaging before, but this has also been dealt with. Please pull again and re-run the build.
> 
>> The above might have some relation to not doing a clean before the build. So I will leave the change in the flash-images lfs file and do a clean and a fresh new build

I did a git pull and then did a full clean and build and this time the whole process was run through without any fails.

Regards,

Adolf.
> 
> A clean build should not be needed.
> 
> -Michael
> 
>>
>> Regards,
>>
>> Adolf.
>>
>>> -Michael
>>>>
>>>> Regards,
>>>>
>>>> Adolf.
>>>>
>>>>
>>>>> Regards,
>>>>>
>>>>> Adolf.
>>>>>
>>>>>
>>>>>>
>>>>>> One thing I found. I am running the new build system while I have been running some package updates with the old system with its mount points. The two have each run without any impact on the other.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Adolf.
>>>>>>
>>>>>>> Regards,
>>>>>>> Adolf.
>>>>>>>
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> -Michael
>>>>>>>>
>>>>>>>>> On 9 Jul 2024, at 22:29, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>>>>>>>
>>>>>>>>> Hello Adolf,
>>>>>>>>>
>>>>>>>>> Thank you for testing this.
>>>>>>>>>
>>>>>>>>> There have indeed been plenty of problems there… I spent a lot of time on this today and hopefully fixed most of them.
>>>>>>>>>
>>>>>>>>> I cannot build the toolchain on my machine and I am not sure why yet, but a build with the packaged toolchain runs through.
>>>>>>>>>
>>>>>>>>> I have also spent some time on getting rid of the strip stage because it annoyed me how long it takes and creating the disk images as well as packages should be significantly faster now, too. I hope I didn’t introduce too many new bugs.
>>>>>>>>>
>>>>>>>>> Please let me know if you have more success now.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> -Michael
>>>>>>>>>
>>>>>>>>>> On 8 Jul 2024, at 20:34, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Michael,
>>>>>>>>>>
>>>>>>>>>> On 08/07/2024 21:15, Adolf Belka wrote:
>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>
>>>>>>>>>>> On 08/07/2024 18:11, Michael Tremer wrote:
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>
>>>>>>>>>>>> I have been spending a lot of time on this problem, because it has been bothering me for a long time. I also saw an opportunity to make more changes to the build system.
>>>>>>>>>>>>
>>>>>>>>>>>> Currently this is all a little bit WIP, but I hope that we can merge this into next as soon as the current update has been moved to master.
>>>>>>>>>>>>
>>>>>>>>>>>> I am referring to this branch which is currently based on next: https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/unshare
>>>>>>>>>>>>
>>>>>>>>>>>> It makes use of the unshare command which creates new namespaces in Linux. That way, we can isolate the build system better from the host system and in case something goes wrong, there is less damage. We can also enforce some more rules…
>>>>>>>>>>>>
>>>>>>>>>>>> So, what has changed?
>>>>>>>>>>>>
>>>>>>>>>>>> * The make.sh script might re-execute itself into a new mount namespace when it is suitable. This happens for “make.sh build” and “make.sh shell”, but it does not happen for “make.sh downloadsrc” for example.
>>>>>>>>>>>>
>>>>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l2129
>>>>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l2251
>>>>>>>>>>>>
>>>>>>>>>>>> * The new mount namespace means that we will no longer see any bind-mounts in the host system and we no longer need to umount anything ourselves which is where we occasionally wiped the entire hard drive of the host system. When the last process exits, the namespace is being cleaned up and everything is being umounted.
>>>>>>>>>>>>
>>>>>>>>>>>> * The function that prepares the build environment has been almost entirely rewritten:
>>>>>>>>>>>>
>>>>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l426
>>>>>>>>>>>>
>>>>>>>>>>>> It used to mount parts of the host system into the build environment which are needed to run anything. Those were /dev, /proc, /sys, etc…
>>>>>>>>>>>>
>>>>>>>>>>>> Instead, it now creates a new /dev mount point and creates a minimal amount of device nodes and symlinks. That way, we detach from the host system and no longer allow the build system access to the host’s filesystem and block devices. We also bind-mount the sources in read-only mode now, so that the build system cannot change anything in the source tree. On top of that, cache is read-only, too. ccache and the log directory are the only places that are writable.
>>>>>>>>>>>>
>>>>>>>>>>>> We mount a separate /tmp directory.
>>>>>>>>>>>>
>>>>>>>>>>>> * When we then build a package, we create more namespaces for each package. These isolate each build process from each other.
>>>>>>>>>>>>
>>>>>>>>>>>> Mostly, this is to detach from the host system. A new UTS namespace allows to change the hostname in the build system without affecting the host and so on. We do the same thing with a new time namespace.
>>>>>>>>>>>>
>>>>>>>>>>>> We do however create a new PID namespace which means that the build system no longer will see any processes running on the host system. That requires to mount a new instance of /proc with each package. This also has the effect that if the shell that we launched terminates (because the build is done) any background processes will be killed immediately.
>>>>>>>>>>>>
>>>>>>>>>>>> Last, we clone the mount namespace that we have created before so that no build command can modify what we set up earlier.
>>>>>>>>>>>>
>>>>>>>>>>>> * Since everything is now so decoupled, we gain a couple of new (maybe minor?) features:
>>>>>>>>>>>>
>>>>>>>>>>>>     It is now possible to run “make.sh shell” while a build is running. That does not happen a lot, but we can do this now :)
>>>>>>>>>>>>
>>>>>>>>>>>>     If the build crashes or the host system is being shut down while a build is running, there is nothing to clean up afterwards.
>>>>>>>>>>>>
>>>>>>>>>>>> * I have garnished this all with a lot of code cleanup and I suppose I might have introduced some new bugs here or there :)
>>>>>>>>>>>>
>>>>>>>>>>>> * This is probably mostly around a new implementation of the timer that updates the build time. It has been annoying me a lot that it takes a long time to walk through all packages that have been built before to finally get to a package that we want to rebuild. Mostly this was all help up by a call of “sleep 0.1”
>>>>>>>>>>>>
>>>>>>>>>>>> Since bash does not really do any concurrency, I had to be creative and replaced the busy-loop with a background process that is launched whenever it is needed and which will “ping” the main make.sh script once a second. That way, we can just run as usual, but regularly get interrupted to update the runtime.
>>>>>>>>>>>>
>>>>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l361
>>>>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=blob;f=make.sh;h=b952627782a0d5ef4ac75f17315b689fcb3b4fe0;hb=refs/heads/unshare#l834
>>>>>>>>>>>>
>>>>>>>>>>>> We now only fork one extra sub shell and we have to handle the timer events which is a lot cheaper as well as more straight-forward to code.
>>>>>>>>>>>>
>>>>>>>>>>>> * As there is no difference between the different stages any more (those stages that we inherited from LFS), I have merged them all into one.
>>>>>>>>>>>>
>>>>>>>>>>>> * Last but not least, I have create the option to build for multiple architectures on the same system. Since we can now mount the entire source tree into (many independent) build environments, we might as well… As discussed on the last call, this might not be the best option for ARM, but RISV-C builds at a decent speed even when emulated.
>>>>>>>>>>>>
>>>>>>>>>>>> The only thing that I needed to do for this is to suffix the build and log directories which are now called build_${ARCH}, i.e. build_aarch64, build_x86_64, and so on. The packages/ directory is not changed yet, but that will have to happen as well. Most likely I want to merge this with the generated images, but I am not sure what to call this, yet. Happy to hear suggestions. result_x86_64? Just images_x86_64?
>>>>>>>>>>>>
>>>>>>>>>>>> ---
>>>>>>>>>>>>
>>>>>>>>>>>> I have run a build and this seems to be working just fine on my Debian machine. I am writing to all of you to first of all let you know what I am up to; and secondly to ask to give this a go on your systems. I think it should run just fine, as all the tools that I require should be available everywhere. However, there might be some older kernels that might not support all of this, yet or any other problems I cannot think of yet. Please give me some feedback and send me all the bugs :)
>>>>>>>>>>> I gave this a go but it didn't work.
>>>>>>>>>>>
>>>>>>>>>>> Not sure if I should have run the ./make.sh clean command on the old version before I pulled the unshare branch into my clone of your repo.
>>>>>>>>>>>
>>>>>>>>>>> Should I have started with a complete new clone of your repo? I might try that anyway just to see.
>>>>>>>>>>>
>>>>>>>>>> I created a completely new clone of you ipfire-2.x repor and then checked out the unshare branch to a branch called unshare in my local repo clone.
>>>>>>>>>>
>>>>>>>>>> gettoolchain gave the same issue, except that this time the toolchain directory ended up completely empty.
>>>>>>>>>>
>>>>>>>>>> downloadsrc had the same result.
>>>>>>>>>>
>>>>>>>>>> clean had nothing to clean up as it was a fresh clone.
>>>>>>>>>>
>>>>>>>>>> build then tried to build the toolchain and came up with this error, different from before.
>>>>>>>>>>
>>>>>>>>>> ./make.sh build
>>>>>>>>>> chroot: failed to run command ‘env’: No such file or directory
>>>>>>>>>> Full toolchain compilation
>>>>>>>>>> stage1 [ FAIL ]
>>>>>>>>>>
>>>>>>>>>>      Jul  8 19:26:39: Building stage1 ====================================== Installing stage1 ...
>>>>>>>>>>      mkdir -pv /tools_x86_64/lib
>>>>>>>>>>      mkdir: cannot create directory '/tools_x86_64': File exists
>>>>>>>>>>      make: *** [stage1:50: /home/ahb/sandbox/ms/ipfire-2.x/log/stage1] Error 1
>>>>>>>>>>
>>>>>>>>>> ERROR: Building stage1 [ FAIL ]
>>>>>>>>>>      Check /home/ahb/sandbox/ms/ipfire-2.x/log_x86_64/_build.toolchain.log for errors if applicable [ FAIL ]
>>>>>>>>>>
>>>>>>>>>> so it wasn't as simple as doing a fresh git clone.
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>> Adolf.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> So I ran ./make.sh gettoolchain first, as I usually would.
>>>>>>>>>>>
>>>>>>>>>>> ./make.sh gettoolchain
>>>>>>>>>>> b2sum: cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: No such file or directory
>>>>>>>>>>> cache/toolchains/ipfire-2.29-toolchain-20240521-x86_64.tar.zst: FAILED open or read
>>>>>>>>>>> b2sum: WARNING: 1 listed file could not be read
>>>>>>>>>>>
>>>>>>>>>>> ipfire-2.29-toolchain-20240210-x86_64.tar.zst is present together with its b2 file.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Then ran ./make.sh downloadsrc
>>>>>>>>>>>
>>>>>>>>>>> Previous version ends with
>>>>>>>>>>>
>>>>>>>>>>> ***Verifying BLAKE2 checksum
>>>>>>>>>>> all files BLAKE2 checksum match [ DONE]
>>>>>>>>>>>
>>>>>>>>>>> after zstd has been checked.
>>>>>>>>>>>
>>>>>>>>>>> New version stops at zstd entry.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ./make.sh clean gave the message Cleaning Build directory... but was completed very quickly.
>>>>>>>>>>> Log and Build directories have not been cleaned out. The img and iso files are still present.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ./make.sh build gave message
>>>>>>>>>>>
>>>>>>>>>>> chroot: failed to run command ‘env’: No such file or directory
>>>>>>>>>>>
>>>>>>>>>>> and then did a full toolchain compilation which failed with gcc but log is >9000 lines.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Adolf.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you for listening to this brain-dump.
>>>>>>>>>>>>
>>>>>>>>>>>> All the best,
>>>>>>>>>>>> -Michael
>>>>>>>>>>>>
>>>>>>>>>>>>> On 3 Jul 2024, at 10:58, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hello Adolf,
>>>>>>>>>>>>>
>>>>>>>>>>>>> This happens occasionally that the buildsystem umounts /dev and then nothing will really work any more.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I rebooted the machine and it is back up again.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 2 Jul 2024, at 15:42, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Michael and all,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I ran the arm builder with the 4.20.2 version of samba to test it out.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The build got to building gdb and then failed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Interestingly, the nightly build of arm was successful with the same version of gdb.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The build log for gdb is attached. The actual error is at line 618.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Another thing I found is that I just tried to go back into the arm builder. I successfully got into people.ipfire.org but then trying to scp into the arm builder failed with the following message.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ------------------------------------------
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ssh bonnietwin(a)arm64-01.zrh.ipfire.org
>>>>>>>>>>>>>> PTY allocation request failed on channel 0
>>>>>>>>>>>>>> Linux arm64-01.zrh.ipfire.org 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The programs included with the Debian GNU/Linux system are free software;
>>>>>>>>>>>>>> the exact distribution terms for each program are described in the
>>>>>>>>>>>>>> individual files in /usr/share/doc/*/copyright.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
>>>>>>>>>>>>>> permitted by applicable law.
>>>>>>>>>>>>>> /etc/profile.d/Z99-cloud-locale-test.sh: line 14: /dev/null: Permission denied
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ------------------------------------------
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> <_build.ipfire.gdb.log>
> 
> 

^ permalink raw reply	[flat|nested] 16+ messages in thread

end of thread, other threads:[~2024-07-10 21:57 UTC | newest]

Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-07-02 14:42 Problem during building of samba on arm builder Adolf Belka
2024-07-03  9:58 ` Michael Tremer
2024-07-03 20:44   ` Adolf Belka
2024-07-08 16:11   ` Michael Tremer
2024-07-08 19:15     ` Adolf Belka
2024-07-08 19:34       ` Adolf Belka
2024-07-09 21:29         ` Michael Tremer
2024-07-10  9:57           ` Michael Tremer
2024-07-10 10:33             ` Adolf Belka
2024-07-10 12:59               ` Adolf Belka
2024-07-10 13:05                 ` Adolf Belka
2024-07-10 13:21                   ` Adolf Belka
2024-07-10 14:24                     ` Michael Tremer
2024-07-10 16:53                       ` Adolf Belka
2024-07-10 18:19                         ` Michael Tremer
2024-07-10 21:57                           ` Adolf Belka

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox