From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: [PATCH 0/1] Parallelized build for several packages Date: Mon, 11 Mar 2019 09:37:50 +0000 Message-ID: In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============7276130214755606295==" List-Id: --===============7276130214755606295== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Yes, probably. It uses a lot of memory when compiling because it is C++. I will adjust the number of concurrent processes just like we did for boost. Best, -Michael > On 10 Mar 2019, at 19:17, ummeegge wrote: >=20 > Hi all, >=20 > On So, 2019-03-10 at 18:24 +0000, Michael Tremer wrote: >> Hi, >>=20 >> I merged the patch. And I am surprised it is only the one=E2=80=A6 > dnsdist do have problems to build here in next currently: >=20 > checking for boost/foreach.hpp... yes > checking for the toolset name used by Boost for g++... configure: WARNING: = could not figure out which toolset name to use for g++ > .... >=20 > g++: internal compiler error: Killed (program cc1plus) > Please submit a full bug report, > with preprocessed source if appropriate. > See for instructions. > make[3]: *** [Makefile:743: dnsdist-lua2.o] Error 4 > make[3]: *** Waiting for unfinished jobs.... >=20 >=20 > May this is possibly related to this topic ? >=20 > Best, >=20 > Erik >=20 >>=20 >> -Michael >>=20 >>> On 9 Mar 2019, at 20:36, Matthias Fischer < >>> matthias.fischer(a)ipfire.org> wrote: >>>=20 >>> On 09.03.2019 21:24, Wolfgang Apolinarski wrote: >>>> Hi, >>>>=20 >>>> sorry, that I did not respond in time. I also had issues with >>>> "slang" during >>>> one of my latest test builds, especially when using something >>>> like "-j3" (it >>>> always worked fine when using higher values). >>>> The lfs manual also stated that the build of "slang" cannot be >>>> parallelized. >>>> Sorry, I apparently overlooked that (I have now checked all other >>>> packages >>>> and did not find incompabilities). >>>=20 >>> No problem, current 'next' - with commented $(MAKETUNING) for >>> 'slang' - built without problems. >>>=20 >>> For the records: >>> Last clean "Build finished in 7:02:45". Hardware: i7/2600, 8 GB >>> RAM, 500GB SATA HDD, Ubuntu 18.04.2 LTS / 32bit (yes, I know... ;-)=20 >>> ). >>>=20 >>> Best, >>> Matthias >>>=20 >>>> Today, I started again some more builds (with parallel build >>>> disabled for >>>> slang), the test continues tomorrow. >>>>=20 >>>> Best regards, >>>> Wolfgang >>>> PS.: The builds always use the branch that Michael created. >>>>=20 >>>> -----Urspr=C3=BCngliche Nachricht----- >>>> Von: Matthias Fischer =20 >>>> Gesendet: Freitag, 8. M=C3=A4rz 2019 19:16 >>>> An: Michael Tremer ; Wolfgang >>>> Apolinarski >>>> >>>> Cc: development(a)lists.ipfire.org >>>> Betreff: Re: [PATCH 0/1] Parallelized build for several packages >>>>=20 >>>> Hi, >>>>=20 >>>> third try, now building current 'next', identical error: >>>>=20 >>>> 'slang 2.3.0' doesn't like "cd $(DIR_APP) && make $(MAKETUNING)" >>>> here. >>>>=20 >>>> After reverting >>>>=20 > https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dblob;f=3Dlfs/slang;h=3D217e7= 4c77317d4 >>>> c829913f934458779fd278bf29;hb=3D23164efba5f57b3d8ccb07a166b613f2f95 >>>> 1e1b6, >>>> build continues... >>>>=20 >>>> Best, >>>> Matthias >>>>=20 >>>> On 08.03.2019 16:58, Matthias Fischer wrote: >>>>> Hi, >>>>>=20 >>>>> On 08.03.2019 11:17, Michael Tremer wrote: >>>>>> Hi, >>>>>>=20 >>>>>> Are you guys still there? >>>>>=20 >>>>> Yep. ;-) >>>>>=20 >>>>> Your branch 'faster-build' refuses to build here - I get >>>>> 'slang' errors: >>>>>=20 >>>>> ***SNIP*** >>>>> cd /usr/src/slang-2.3.0 && make -j9 >>>>> make[1]: Entering directory '/usr/src/slang-2.3.0' >>>>> cd src; make elf >>>>> make[2]: Entering directory '/usr/src/slang-2.3.0/src' >>>>> /usr/src/slang-2.3.0/autoconf/mkinsdir.sh=20 >>>>> /usr/src/slang-2.3.0/src/elfobjs cp sysconf.h config.h cp=20 >>>>> terminfo/default.inc terminfo.inc cd /usr/src/slang- >>>>> 2.3.0/src/elfobjs=20 >>>>> && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=3Di586=20 >>>>> -mindirect-branch=3Dthunk -mfunction-return=3Dthunk -mtune=3Dgeneric=20 >>>>> -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=3D2 -Wp,- >>>>> D_GLIBCXX_ASSERTIONS=20 >>>>> -fstack-protector-strong -fPIC -Dunix -DSLANG =20 >>>>> /usr/src/slang-2.3.0/src/sldisply.c >>>>> cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall=20 >>>>> -fexceptions -fPIC -march=3Di586 -mindirect-branch=3Dthunk=20 >>>>> -mfunction-return=3Dthunk -mtune=3Dgeneric -fomit-frame-pointer=20 >>>>> -Wp,-D_FORTIFY_SOURCE=3D2 -Wp,-D_GLIBCXX_ASSERTIONS=20 >>>>> -fstack-protector-strong -fPIC -Dunix -DSLANG =20 >>>>> /usr/src/slang-2.3.0/src/slutty.c cd /usr/src/slang- >>>>> 2.3.0/src/elfobjs=20 >>>>> && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=3Di586=20 >>>>> -mindirect-branch=3Dthunk -mfunction-return=3Dthunk -mtune=3Dgeneric=20 >>>>> -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=3D2 -Wp,- >>>>> D_GLIBCXX_ASSERTIONS=20 >>>>> -fstack-protector-strong -fPIC -Dunix -DSLANG =20 >>>>> /usr/src/slang-2.3.0/src/slang.c >>>>> /bin/sh: line 0: cd: /usr/src/slang-2.3.0/src/elfobjs: No such >>>>> file or=20 >>>>> directory mkdir -p -- /usr/src/slang-2.3.0/src/elfobjs >>>>> make[2]: *** [Makefile:472:=20 >>>>> /usr/src/slang-2.3.0/src/elfobjs/sldisply.o] Error 1 >>>>> make[2]: *** Waiting for unfinished jobs.... >>>>> /usr/src/slang-2.3.0/src/slang.c: In function 'inner_interp': >>>>> /usr/src/slang-2.3.0/src/slang.c:5733:9: warning: 'test' may be >>>>> used >>>>=20 >>>> uninitialized in this function [-Wmaybe-uninitialized] >>>>> if ((0 =3D=3D pop_int (&test)) >>>>> ~~~~~~~~~~~~~~~~~~~~~~ >>>>> && (test =3D=3D 0)) >>>>> ^~~~~~~~~~~~~~ >>>>> /usr/src/slang-2.3.0/src/slang.c: In function 'lang_do_loops': >>>>> /usr/src/slang-2.3.0/src/slang.c:3725:10: warning: 'blks[0]' >>>>> may be used >>>>=20 >>>> uninitialized in this function [-Wmaybe-uninitialized] >>>>> block =3D blks[0]; >>>>> ~~~~~~^~~~~~~~~ >>>>> /usr/src/slang-2.3.0/src/slang.c:3924:7: warning: 'first' may >>>>> be used >>>>=20 >>>> uninitialized in this function [-Wmaybe-uninitialized] >>>>> i +=3D ctrl; >>>>> ~~^~~~~~~ >>>>> /usr/src/slang-2.3.0/src/slang.c:3900:13: warning: 'last' may >>>>> be used >>>>=20 >>>> uninitialized in this function [-Wmaybe-uninitialized] >>>>> if (i > last) break; >>>>> ^ >>>>> make[2]: Leaving directory '/usr/src/slang-2.3.0/src' >>>>> make[1]: *** [Makefile:55: elf] Error 2 >>>>> make[1]: Leaving directory '/usr/src/slang-2.3.0' >>>>> make: *** [slang:74: /usr/src/log/slang-2.3.0] Error 2 >>>>> ***SNAP*** >>>>>=20 >>>>> Don't know why. I tried two builds, same error. >>>>>=20 >>>>> Best, >>>>> Matthias >>>>>=20 >>>>>> I think that we are saving about 30 min to an hour on the >>>>>> nightly builds >>>>=20 >>>> now=E2=80=A6 Did anybody else run some benchmarks? >>>>>>=20 >>>>>> -Michael >>>>>>=20 >>>>>>> On 6 Mar 2019, at 16:36, Michael Tremer < >>>>>>> michael.tremer(a)ipfire.org> >>>>=20 >>>> wrote: >>>>>>>=20 >>>>>>> Hi, >>>>>>>=20 >>>>>>> Sorry for not being able to get around to this earlier, but >>>>>>> I have >>>>=20 >>>> invested some time on this: >>>>>>>=20 >>>>>>> First of all, I merged Wolfgang=E2=80=99s patch. He is right, the >>>>>>> build is too >>>>=20 >>>> slow. It sometimes looks a bit like it is not taking enough >>>> advantage of >>>> fast systems. On the other hand with have small systems, which >>>> are (in their >>>> own right) quite fast. Those are the ARM systems. They usually >>>> have 1G of >>>> RAM. 2GB when you are lucky. >>>>>>>=20 >>>>>>> So I adjusted make.sh a little bit and removed composing >>>>>>> the MAKETUNING >>>>=20 >>>> variable from it. >>>>>>>=20 >>>>>>> Now, make.sh only figures out the value that usually comes >>>>>>> after the -j. >>>>=20 >>>> It is set in DEFAULT_PARALLELISM. >>>>>>>=20 >>>>>>> In the individual LFS files, we can now set a new variable >>>>>>> called >>>>=20 >>>> MAX_PARALLELISM which is optional. In boost, I now calculate this >>>> from the >>>> amount of memory that is available. That will make sure that we >>>> have a cap >>>> on this that will be high on systems that have the memory and low >>>> on those >>>> that don=E2=80=99t. Before, we hardcoded -j2 which of course runs very >>>> slow on large >>>> systems. >>>>>>>=20 >>>>>>> The code is here: >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 > https://git.ipfire.org/?p=3Dpeople/ms/ipfire-2.x.git;a=3Dshortlog;h=3Drefs >>>>>>> /heads/faster-build >>>>>>>=20 >>>>>>> Please review. I am now running this on a couple of >>>>>>> builders and will >>>>=20 >>>> see what comes out. Please do the same and send me feedback. >>>>>>>=20 >>>>>>> Best, >>>>>>> -Michael >>>>>>>=20 >>>>>>>> On 21 Feb 2019, at 17:28, Wolfgang Apolinarski >>>>=20 >>>> wrote: >>>>>>>>=20 >>>>>>>> Hi Arne, >>>>>>>>=20 >>>>>>>> of course I did delete the ccache between my multiple >>>>>>>> builds to verify >>>>=20 >>>> the commit. To play it safe, I hard reset using git, checked that >>>> all >>>> folders (build, ccache, etc.) are gone and restarted the build >>>> after >>>> gettoolchain and downloadsrc. >>>>>>>>=20 >>>>>>>> Sorry, I did not know that there are still a lot of >>>>>>>> machines with low >>>>=20 >>>> ram that we are using to build ipfire. My hope was that these >>>> machines have >>>> disappeared over the last years. >>>>>>>> Additionally, I do not have the ability to test all >>>>>>>> ipfire arches, as >>>>=20 >>>> mentioned, the patch needs testing by others. >>>>>>>>=20 >>>>>>>> I also thought of adding a manual parameter for >>>>>>>> developers that somehow >>>>=20 >>>> manually sets the parameters for parallel build, but since the >>>> MAKETUNING >>>> parameter already exists and is used with other packages this >>>> would just >>>> somehow mess everything up. >>>>>>>>=20 >>>>>>>> For my builds, I can just manually apply the patch to >>>>>>>> speed up the full >>>>=20 >>>> build. My thought was that sharing the patch would make sense for >>>> others. >>>>>>>>=20 >>>>>>>>> -----Urspr=C3=BCngliche Nachricht----- >>>>>>>>> Von: Arne Fitzenreiter >>>>>>>>> Gesendet: Donnerstag, 21. Februar 2019 09:44 >>>>>>>>> An: Wolfgang Apolinarski < >>>>>>>>> wolfgang.apolinarski(a)ipfire.org> >>>>>>>>> Cc: development(a)lists.ipfire.org >>>>>>>>> Betreff: Re: [PATCH 0/1] Parallelized build for several >>>>>>>>> packages >>>>>>>>>=20 >>>>>>>>> On the most of this packages (eg. boot and cmake) i >>>>>>>>> have disabled=20 >>>>>>>>> parallel build because it fails on machines with low >>>>>>>>> ram (less=20 >>>>>>>>> than 2GB) or on arm. >>>>>>>>> So such changes need tested on an 1GB machine, on all=20 >>>>>>>>> architectures and build two times after deleting >>>>>>>>> ccache. Because=20 >>>>>>>>> the build need much lower resources if it has the file >>>>>>>>> already >>>>=20 >>>> prebuilt in cache. >>=20 >>=20 >=20 --===============7276130214755606295==--