From mboxrd@z Thu Jan 1 00:00:00 1970 From: Matthias Fischer To: development@lists.ipfire.org Subject: Re: [PATCH 0/1] Parallelized build for several packages Date: Fri, 08 Mar 2019 16:58:14 +0100 Message-ID: <5c166d77-6766-f3ed-70b5-154ae8e4a13b@ipfire.org> In-Reply-To: <8998D83E-82FA-48E0-896E-446DE2BEDA56@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2428134487292854535==" List-Id: --===============2428134487292854535== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi, On 08.03.2019 11:17, Michael Tremer wrote: > Hi, >=20 > Are you guys still there? Yep. ;-) Your branch 'faster-build' refuses to build here - I get 'slang' errors: ***SNIP*** cd /usr/src/slang-2.3.0 && make -j9 make[1]: Entering directory '/usr/src/slang-2.3.0' cd src; make elf make[2]: Entering directory '/usr/src/slang-2.3.0/src' /usr/src/slang-2.3.0/autoconf/mkinsdir.sh /usr/src/slang-2.3.0/src/elfobjs cp sysconf.h config.h cp terminfo/default.inc terminfo.inc cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall -fexceptions -f= PIC -march=3Di586 -mindirect-branch=3Dthunk -mfunction-return=3Dthunk -mtune= =3Dgeneric -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=3D2 -Wp,-D_GLIBCXX_ASSE= RTIONS -fstack-protector-strong -fPIC -Dunix -DSLANG /usr/src/slang-2.3.0/= src/sldisply.c cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall -fexceptions -f= PIC -march=3Di586 -mindirect-branch=3Dthunk -mfunction-return=3Dthunk -mtune= =3Dgeneric -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=3D2 -Wp,-D_GLIBCXX_ASSE= RTIONS -fstack-protector-strong -fPIC -Dunix -DSLANG /usr/src/slang-2.3.0/= src/slutty.c cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall -fexceptions -f= PIC -march=3Di586 -mindirect-branch=3Dthunk -mfunction-return=3Dthunk -mtune= =3Dgeneric -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=3D2 -Wp,-D_GLIBCXX_ASSE= RTIONS -fstack-protector-strong -fPIC -Dunix -DSLANG /usr/src/slang-2.3.0/= src/slang.c /bin/sh: line 0: cd: /usr/src/slang-2.3.0/src/elfobjs: No such file or direct= ory mkdir -p -- /usr/src/slang-2.3.0/src/elfobjs make[2]: *** [Makefile:472: /usr/src/slang-2.3.0/src/elfobjs/sldisply.o] Erro= r 1 make[2]: *** Waiting for unfinished jobs.... /usr/src/slang-2.3.0/src/slang.c: In function 'inner_interp': /usr/src/slang-2.3.0/src/slang.c:5733:9: warning: 'test' may be used uninitia= lized in this function [-Wmaybe-uninitialized] if ((0 =3D=3D pop_int (&test)) ~~~~~~~~~~~~~~~~~~~~~~ && (test =3D=3D 0)) ^~~~~~~~~~~~~~ /usr/src/slang-2.3.0/src/slang.c: In function 'lang_do_loops': /usr/src/slang-2.3.0/src/slang.c:3725:10: warning: 'blks[0]' may be used unin= itialized in this function [-Wmaybe-uninitialized] block =3D blks[0]; ~~~~~~^~~~~~~~~ /usr/src/slang-2.3.0/src/slang.c:3924:7: warning: 'first' may be used uniniti= alized in this function [-Wmaybe-uninitialized] i +=3D ctrl; ~~^~~~~~~ /usr/src/slang-2.3.0/src/slang.c:3900:13: warning: 'last' may be used uniniti= alized in this function [-Wmaybe-uninitialized] if (i > last) break; ^ make[2]: Leaving directory '/usr/src/slang-2.3.0/src' make[1]: *** [Makefile:55: elf] Error 2 make[1]: Leaving directory '/usr/src/slang-2.3.0' make: *** [slang:74: /usr/src/log/slang-2.3.0] Error 2 ***SNAP*** Don't know why. I tried two builds, same error. Best, Matthias > I think that we are saving about 30 min to an hour on the nightly builds no= w=E2=80=A6 Did anybody else run some benchmarks? >=20 > -Michael >=20 >> On 6 Mar 2019, at 16:36, Michael Tremer wrot= e: >>=20 >> Hi, >>=20 >> Sorry for not being able to get around to this earlier, but I have investe= d some time on this: >>=20 >> First of all, I merged Wolfgang=E2=80=99s patch. He is right, the build is= too slow. It sometimes looks a bit like it is not taking enough advantage of= fast systems. On the other hand with have small systems, which are (in their= own right) quite fast. Those are the ARM systems. They usually have 1G of RA= M. 2GB when you are lucky. >>=20 >> So I adjusted make.sh a little bit and removed composing the MAKETUNING va= riable from it. >>=20 >> Now, make.sh only figures out the value that usually comes after the -j. I= t is set in DEFAULT_PARALLELISM. >>=20 >> In the individual LFS files, we can now set a new variable called MAX_PARA= LLELISM which is optional. In boost, I now calculate this from the amount of = memory that is available. That will make sure that we have a cap on this that= will be high on systems that have the memory and low on those that don=E2=80= =99t. Before, we hardcoded -j2 which of course runs very slow on large system= s. >>=20 >> The code is here: >>=20 >> https://git.ipfire.org/?p=3Dpeople/ms/ipfire-2.x.git;a=3Dshortlog;h=3Dref= s/heads/faster-build=20 >>=20 >> Please review. I am now running this on a couple of builders and will see = what comes out. Please do the same and send me feedback. >>=20 >> Best, >> -Michael >>=20 >>> On 21 Feb 2019, at 17:28, Wolfgang Apolinarski wrote: >>>=20 >>> Hi Arne, >>>=20 >>> of course I did delete the ccache between my multiple builds to verify th= e commit. To play it safe, I hard reset using git, checked that all folders (= build, ccache, etc.) are gone and restarted the build after gettoolchain and = downloadsrc. >>>=20 >>> Sorry, I did not know that there are still a lot of machines with low ram= that we are using to build ipfire. My hope was that these machines have disa= ppeared over the last years. >>> Additionally, I do not have the ability to test all ipfire arches, as men= tioned, the patch needs testing by others. >>>=20 >>> I also thought of adding a manual parameter for developers that somehow m= anually sets the parameters for parallel build, but since the MAKETUNING para= meter already exists and is used with other packages this would just somehow = mess everything up. >>>=20 >>> For my builds, I can just manually apply the patch to speed up the full b= uild. My thought was that sharing the patch would make sense for others. >>>=20 >>>> -----Urspr=C3=BCngliche Nachricht----- >>>> Von: Arne Fitzenreiter >>>> Gesendet: Donnerstag, 21. Februar 2019 09:44 >>>> An: Wolfgang Apolinarski >>>> Cc: development(a)lists.ipfire.org >>>> Betreff: Re: [PATCH 0/1] Parallelized build for several packages >>>>=20 >>>> On the most of this packages (eg. boot and cmake) i have disabled >>>> parallel build because it fails on machines with low ram (less than 2GB) >>>> or on arm. >>>> So such changes need tested on an 1GB machine, on all architectures and >>>> build two times after deleting ccache. Because the build need much lower >>>> resources if it has the file already prebuilt in cache. >>>=20 >>=20 >=20 >=20 --===============2428134487292854535==--