From mboxrd@z Thu Jan 1 00:00:00 1970 From: Matthias Fischer To: development@lists.ipfire.org Subject: Re: [PATCH 0/1] Parallelized build for several packages Date: Fri, 08 Mar 2019 19:16:18 +0100 Message-ID: In-Reply-To: <5c166d77-6766-f3ed-70b5-154ae8e4a13b@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============7834025155556667454==" List-Id: --===============7834025155556667454== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi, third try, now building current 'next', identical error: 'slang 2.3.0' doesn't like "cd $(DIR_APP) && make $(MAKETUNING)" here. After reverting https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dblob;f=3Dlfs/slang;h=3D217e74c= 77317d4c829913f934458779fd278bf29;hb=3D23164efba5f57b3d8ccb07a166b613f2f951e1= b6, build continues... Best, Matthias On 08.03.2019 16:58, Matthias Fischer wrote: > Hi, >=20 > On 08.03.2019 11:17, Michael Tremer wrote: >> Hi, >>=20 >> Are you guys still there? >=20 > Yep. ;-) >=20 > Your branch 'faster-build' refuses to build here - I get 'slang' errors: >=20 > ***SNIP*** > cd /usr/src/slang-2.3.0 && make -j9 > make[1]: Entering directory '/usr/src/slang-2.3.0' > cd src; make elf > make[2]: Entering directory '/usr/src/slang-2.3.0/src' > /usr/src/slang-2.3.0/autoconf/mkinsdir.sh /usr/src/slang-2.3.0/src/elfobjs > cp sysconf.h config.h > cp terminfo/default.inc terminfo.inc > cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall -fexceptions = -fPIC -march=3Di586 -mindirect-branch=3Dthunk -mfunction-return=3Dthunk -mtun= e=3Dgeneric -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=3D2 -Wp,-D_GLIBCXX_ASS= ERTIONS -fstack-protector-strong -fPIC -Dunix -DSLANG /usr/src/slang-2.3.0= /src/sldisply.c > cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall -fexceptions = -fPIC -march=3Di586 -mindirect-branch=3Dthunk -mfunction-return=3Dthunk -mtun= e=3Dgeneric -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=3D2 -Wp,-D_GLIBCXX_ASS= ERTIONS -fstack-protector-strong -fPIC -Dunix -DSLANG /usr/src/slang-2.3.0= /src/slutty.c > cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall -fexceptions = -fPIC -march=3Di586 -mindirect-branch=3Dthunk -mfunction-return=3Dthunk -mtun= e=3Dgeneric -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=3D2 -Wp,-D_GLIBCXX_ASS= ERTIONS -fstack-protector-strong -fPIC -Dunix -DSLANG /usr/src/slang-2.3.0= /src/slang.c > /bin/sh: line 0: cd: /usr/src/slang-2.3.0/src/elfobjs: No such file or dire= ctory > mkdir -p -- /usr/src/slang-2.3.0/src/elfobjs > make[2]: *** [Makefile:472: /usr/src/slang-2.3.0/src/elfobjs/sldisply.o] Er= ror 1 > make[2]: *** Waiting for unfinished jobs.... > /usr/src/slang-2.3.0/src/slang.c: In function 'inner_interp': > /usr/src/slang-2.3.0/src/slang.c:5733:9: warning: 'test' may be used uninit= ialized in this function [-Wmaybe-uninitialized] > if ((0 =3D=3D pop_int (&test)) > ~~~~~~~~~~~~~~~~~~~~~~ > && (test =3D=3D 0)) > ^~~~~~~~~~~~~~ > /usr/src/slang-2.3.0/src/slang.c: In function 'lang_do_loops': > /usr/src/slang-2.3.0/src/slang.c:3725:10: warning: 'blks[0]' may be used un= initialized in this function [-Wmaybe-uninitialized] > block =3D blks[0]; > ~~~~~~^~~~~~~~~ > /usr/src/slang-2.3.0/src/slang.c:3924:7: warning: 'first' may be used unini= tialized in this function [-Wmaybe-uninitialized] > i +=3D ctrl; > ~~^~~~~~~ > /usr/src/slang-2.3.0/src/slang.c:3900:13: warning: 'last' may be used unini= tialized in this function [-Wmaybe-uninitialized] > if (i > last) break; > ^ > make[2]: Leaving directory '/usr/src/slang-2.3.0/src' > make[1]: *** [Makefile:55: elf] Error 2 > make[1]: Leaving directory '/usr/src/slang-2.3.0' > make: *** [slang:74: /usr/src/log/slang-2.3.0] Error 2 > ***SNAP*** >=20 > Don't know why. I tried two builds, same error. >=20 > Best, > Matthias >=20 >> I think that we are saving about 30 min to an hour on the nightly builds n= ow=E2=80=A6 Did anybody else run some benchmarks? >>=20 >> -Michael >>=20 >>> On 6 Mar 2019, at 16:36, Michael Tremer wro= te: >>>=20 >>> Hi, >>>=20 >>> Sorry for not being able to get around to this earlier, but I have invest= ed some time on this: >>>=20 >>> First of all, I merged Wolfgang=E2=80=99s patch. He is right, the build i= s too slow. It sometimes looks a bit like it is not taking enough advantage o= f fast systems. On the other hand with have small systems, which are (in thei= r own right) quite fast. Those are the ARM systems. They usually have 1G of R= AM. 2GB when you are lucky. >>>=20 >>> So I adjusted make.sh a little bit and removed composing the MAKETUNING v= ariable from it. >>>=20 >>> Now, make.sh only figures out the value that usually comes after the -j. = It is set in DEFAULT_PARALLELISM. >>>=20 >>> In the individual LFS files, we can now set a new variable called MAX_PAR= ALLELISM which is optional. In boost, I now calculate this from the amount of= memory that is available. That will make sure that we have a cap on this tha= t will be high on systems that have the memory and low on those that don=E2= =80=99t. Before, we hardcoded -j2 which of course runs very slow on large sys= tems. >>>=20 >>> The code is here: >>>=20 >>> https://git.ipfire.org/?p=3Dpeople/ms/ipfire-2.x.git;a=3Dshortlog;h=3Dre= fs/heads/faster-build=20 >>>=20 >>> Please review. I am now running this on a couple of builders and will see= what comes out. Please do the same and send me feedback. >>>=20 >>> Best, >>> -Michael >>>=20 >>>> On 21 Feb 2019, at 17:28, Wolfgang Apolinarski wrote: >>>>=20 >>>> Hi Arne, >>>>=20 >>>> of course I did delete the ccache between my multiple builds to verify t= he commit. To play it safe, I hard reset using git, checked that all folders = (build, ccache, etc.) are gone and restarted the build after gettoolchain and= downloadsrc. >>>>=20 >>>> Sorry, I did not know that there are still a lot of machines with low ra= m that we are using to build ipfire. My hope was that these machines have dis= appeared over the last years. >>>> Additionally, I do not have the ability to test all ipfire arches, as me= ntioned, the patch needs testing by others. >>>>=20 >>>> I also thought of adding a manual parameter for developers that somehow = manually sets the parameters for parallel build, but since the MAKETUNING par= ameter already exists and is used with other packages this would just somehow= mess everything up. >>>>=20 >>>> For my builds, I can just manually apply the patch to speed up the full = build. My thought was that sharing the patch would make sense for others. >>>>=20 >>>>> -----Urspr=C3=BCngliche Nachricht----- >>>>> Von: Arne Fitzenreiter >>>>> Gesendet: Donnerstag, 21. Februar 2019 09:44 >>>>> An: Wolfgang Apolinarski >>>>> Cc: development(a)lists.ipfire.org >>>>> Betreff: Re: [PATCH 0/1] Parallelized build for several packages >>>>>=20 >>>>> On the most of this packages (eg. boot and cmake) i have disabled >>>>> parallel build because it fails on machines with low ram (less than 2GB) >>>>> or on arm. >>>>> So such changes need tested on an 1GB machine, on all architectures and >>>>> build two times after deleting ccache. Because the build need much lower >>>>> resources if it has the file already prebuilt in cache. >>>>=20 >>>=20 >>=20 >>=20 >=20 >=20 --===============7834025155556667454==--