From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: [PATCH 0/1] Parallelized build for several packages Date: Sun, 10 Mar 2019 18:24:16 +0000 Message-ID: <68DDCFCA-8600-4EEF-81AD-F8315154FA99@ipfire.org> In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============3741187045801659118==" List-Id: --===============3741187045801659118== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi, I merged the patch. And I am surprised it is only the one=E2=80=A6 -Michael > On 9 Mar 2019, at 20:36, Matthias Fischer w= rote: >=20 > On 09.03.2019 21:24, Wolfgang Apolinarski wrote: >> Hi, >>=20 >> sorry, that I did not respond in time. I also had issues with "slang" duri= ng >> one of my latest test builds, especially when using something like "-j3" (= it >> always worked fine when using higher values). >> The lfs manual also stated that the build of "slang" cannot be parallelize= d. >> Sorry, I apparently overlooked that (I have now checked all other packages >> and did not find incompabilities). >=20 > No problem, current 'next' - with commented $(MAKETUNING) for 'slang' - bui= lt without problems. >=20 > For the records: > Last clean "Build finished in 7:02:45". Hardware: i7/2600, 8 GB RAM, 500GB = SATA HDD, Ubuntu 18.04.2 LTS / 32bit (yes, I know... ;-) ). >=20 > Best, > Matthias >=20 >> Today, I started again some more builds (with parallel build disabled for >> slang), the test continues tomorrow. >>=20 >> Best regards, >> Wolfgang >> PS.: The builds always use the branch that Michael created. >>=20 >> -----Urspr=C3=BCngliche Nachricht----- >> Von: Matthias Fischer =20 >> Gesendet: Freitag, 8. M=C3=A4rz 2019 19:16 >> An: Michael Tremer ; Wolfgang Apolinarski >> >> Cc: development(a)lists.ipfire.org >> Betreff: Re: [PATCH 0/1] Parallelized build for several packages >>=20 >> Hi, >>=20 >> third try, now building current 'next', identical error: >>=20 >> 'slang 2.3.0' doesn't like "cd $(DIR_APP) && make $(MAKETUNING)" here. >>=20 >> After reverting >> https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dblob;f=3Dlfs/slang;h=3D217e= 74c77317d4 >> c829913f934458779fd278bf29;hb=3D23164efba5f57b3d8ccb07a166b613f2f951e1b6, >> build continues... >>=20 >> Best, >> Matthias >>=20 >> On 08.03.2019 16:58, Matthias Fischer wrote: >>> Hi, >>>=20 >>> On 08.03.2019 11:17, Michael Tremer wrote: >>>> Hi, >>>>=20 >>>> Are you guys still there? >>>=20 >>> Yep. ;-) >>>=20 >>> Your branch 'faster-build' refuses to build here - I get 'slang' errors: >>>=20 >>> ***SNIP*** >>> cd /usr/src/slang-2.3.0 && make -j9 >>> make[1]: Entering directory '/usr/src/slang-2.3.0' >>> cd src; make elf >>> make[2]: Entering directory '/usr/src/slang-2.3.0/src' >>> /usr/src/slang-2.3.0/autoconf/mkinsdir.sh=20 >>> /usr/src/slang-2.3.0/src/elfobjs cp sysconf.h config.h cp=20 >>> terminfo/default.inc terminfo.inc cd /usr/src/slang-2.3.0/src/elfobjs=20 >>> && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=3Di586=20 >>> -mindirect-branch=3Dthunk -mfunction-return=3Dthunk -mtune=3Dgeneric=20 >>> -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=3D2 -Wp,-D_GLIBCXX_ASSERTIONS = >>> -fstack-protector-strong -fPIC -Dunix -DSLANG =20 >>> /usr/src/slang-2.3.0/src/sldisply.c >>> cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall=20 >>> -fexceptions -fPIC -march=3Di586 -mindirect-branch=3Dthunk=20 >>> -mfunction-return=3Dthunk -mtune=3Dgeneric -fomit-frame-pointer=20 >>> -Wp,-D_FORTIFY_SOURCE=3D2 -Wp,-D_GLIBCXX_ASSERTIONS=20 >>> -fstack-protector-strong -fPIC -Dunix -DSLANG =20 >>> /usr/src/slang-2.3.0/src/slutty.c cd /usr/src/slang-2.3.0/src/elfobjs=20 >>> && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=3Di586=20 >>> -mindirect-branch=3Dthunk -mfunction-return=3Dthunk -mtune=3Dgeneric=20 >>> -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=3D2 -Wp,-D_GLIBCXX_ASSERTIONS = >>> -fstack-protector-strong -fPIC -Dunix -DSLANG =20 >>> /usr/src/slang-2.3.0/src/slang.c >>> /bin/sh: line 0: cd: /usr/src/slang-2.3.0/src/elfobjs: No such file or=20 >>> directory mkdir -p -- /usr/src/slang-2.3.0/src/elfobjs >>> make[2]: *** [Makefile:472:=20 >>> /usr/src/slang-2.3.0/src/elfobjs/sldisply.o] Error 1 >>> make[2]: *** Waiting for unfinished jobs.... >>> /usr/src/slang-2.3.0/src/slang.c: In function 'inner_interp': >>> /usr/src/slang-2.3.0/src/slang.c:5733:9: warning: 'test' may be used >> uninitialized in this function [-Wmaybe-uninitialized] >>> if ((0 =3D=3D pop_int (&test)) >>> ~~~~~~~~~~~~~~~~~~~~~~ >>> && (test =3D=3D 0)) >>> ^~~~~~~~~~~~~~ >>> /usr/src/slang-2.3.0/src/slang.c: In function 'lang_do_loops': >>> /usr/src/slang-2.3.0/src/slang.c:3725:10: warning: 'blks[0]' may be used >> uninitialized in this function [-Wmaybe-uninitialized] >>> block =3D blks[0]; >>> ~~~~~~^~~~~~~~~ >>> /usr/src/slang-2.3.0/src/slang.c:3924:7: warning: 'first' may be used >> uninitialized in this function [-Wmaybe-uninitialized] >>> i +=3D ctrl; >>> ~~^~~~~~~ >>> /usr/src/slang-2.3.0/src/slang.c:3900:13: warning: 'last' may be used >> uninitialized in this function [-Wmaybe-uninitialized] >>> if (i > last) break; >>> ^ >>> make[2]: Leaving directory '/usr/src/slang-2.3.0/src' >>> make[1]: *** [Makefile:55: elf] Error 2 >>> make[1]: Leaving directory '/usr/src/slang-2.3.0' >>> make: *** [slang:74: /usr/src/log/slang-2.3.0] Error 2 >>> ***SNAP*** >>>=20 >>> Don't know why. I tried two builds, same error. >>>=20 >>> Best, >>> Matthias >>>=20 >>>> I think that we are saving about 30 min to an hour on the nightly builds >> now=E2=80=A6 Did anybody else run some benchmarks? >>>>=20 >>>> -Michael >>>>=20 >>>>> On 6 Mar 2019, at 16:36, Michael Tremer >> wrote: >>>>>=20 >>>>> Hi, >>>>>=20 >>>>> Sorry for not being able to get around to this earlier, but I have >> invested some time on this: >>>>>=20 >>>>> First of all, I merged Wolfgang=E2=80=99s patch. He is right, the build= is too >> slow. It sometimes looks a bit like it is not taking enough advantage of >> fast systems. On the other hand with have small systems, which are (in the= ir >> own right) quite fast. Those are the ARM systems. They usually have 1G of >> RAM. 2GB when you are lucky. >>>>>=20 >>>>> So I adjusted make.sh a little bit and removed composing the MAKETUNING >> variable from it. >>>>>=20 >>>>> Now, make.sh only figures out the value that usually comes after the -j. >> It is set in DEFAULT_PARALLELISM. >>>>>=20 >>>>> In the individual LFS files, we can now set a new variable called >> MAX_PARALLELISM which is optional. In boost, I now calculate this from the >> amount of memory that is available. That will make sure that we have a cap >> on this that will be high on systems that have the memory and low on those >> that don=E2=80=99t. Before, we hardcoded -j2 which of course runs very slo= w on large >> systems. >>>>>=20 >>>>> The code is here: >>>>>=20 >>>>>=20 >>>>> https://git.ipfire.org/?p=3Dpeople/ms/ipfire-2.x.git;a=3Dshortlog;h=3Dr= efs >>>>> /heads/faster-build >>>>>=20 >>>>> Please review. I am now running this on a couple of builders and will >> see what comes out. Please do the same and send me feedback. >>>>>=20 >>>>> Best, >>>>> -Michael >>>>>=20 >>>>>> On 21 Feb 2019, at 17:28, Wolfgang Apolinarski >> wrote: >>>>>>=20 >>>>>> Hi Arne, >>>>>>=20 >>>>>> of course I did delete the ccache between my multiple builds to verify >> the commit. To play it safe, I hard reset using git, checked that all >> folders (build, ccache, etc.) are gone and restarted the build after >> gettoolchain and downloadsrc. >>>>>>=20 >>>>>> Sorry, I did not know that there are still a lot of machines with low >> ram that we are using to build ipfire. My hope was that these machines have >> disappeared over the last years. >>>>>> Additionally, I do not have the ability to test all ipfire arches, as >> mentioned, the patch needs testing by others. >>>>>>=20 >>>>>> I also thought of adding a manual parameter for developers that somehow >> manually sets the parameters for parallel build, but since the MAKETUNING >> parameter already exists and is used with other packages this would just >> somehow mess everything up. >>>>>>=20 >>>>>> For my builds, I can just manually apply the patch to speed up the full >> build. My thought was that sharing the patch would make sense for others. >>>>>>=20 >>>>>>> -----Urspr=C3=BCngliche Nachricht----- >>>>>>> Von: Arne Fitzenreiter >>>>>>> Gesendet: Donnerstag, 21. Februar 2019 09:44 >>>>>>> An: Wolfgang Apolinarski >>>>>>> Cc: development(a)lists.ipfire.org >>>>>>> Betreff: Re: [PATCH 0/1] Parallelized build for several packages >>>>>>>=20 >>>>>>> On the most of this packages (eg. boot and cmake) i have disabled=20 >>>>>>> parallel build because it fails on machines with low ram (less=20 >>>>>>> than 2GB) or on arm. >>>>>>> So such changes need tested on an 1GB machine, on all=20 >>>>>>> architectures and build two times after deleting ccache. Because=20 >>>>>>> the build need much lower resources if it has the file already >> prebuilt in cache. >=20 --===============3741187045801659118==--