From mboxrd@z Thu Jan 1 00:00:00 1970 From: Wolfgang Apolinarski To: development@lists.ipfire.org Subject: Re: [PATCH 0/1] Parallelized build for several packages Date: Sat, 09 Mar 2019 21:24:27 +0100 Message-ID: <002e01d4d6b6$18bf9e30$4a3eda90$@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0222617268710639223==" List-Id: --===============0222617268710639223== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit Hi, sorry, that I did not respond in time. I also had issues with "slang" during one of my latest test builds, especially when using something like "-j3" (it always worked fine when using higher values). The lfs manual also stated that the build of "slang" cannot be parallelized. Sorry, I apparently overlooked that (I have now checked all other packages and did not find incompabilities). Today, I started again some more builds (with parallel build disabled for slang), the test continues tomorrow. Best regards, Wolfgang PS.: The builds always use the branch that Michael created. -----Ursprüngliche Nachricht----- Von: Matthias Fischer Gesendet: Freitag, 8. März 2019 19:16 An: Michael Tremer ; Wolfgang Apolinarski Cc: development(a)lists.ipfire.org Betreff: Re: [PATCH 0/1] Parallelized build for several packages Hi, third try, now building current 'next', identical error: 'slang 2.3.0' doesn't like "cd $(DIR_APP) && make $(MAKETUNING)" here. After reverting https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=lfs/slang;h=217e74c77317d4 c829913f934458779fd278bf29;hb=23164efba5f57b3d8ccb07a166b613f2f951e1b6, build continues... Best, Matthias On 08.03.2019 16:58, Matthias Fischer wrote: > Hi, > > On 08.03.2019 11:17, Michael Tremer wrote: >> Hi, >> >> Are you guys still there? > > Yep. ;-) > > Your branch 'faster-build' refuses to build here - I get 'slang' errors: > > ***SNIP*** > cd /usr/src/slang-2.3.0 && make -j9 > make[1]: Entering directory '/usr/src/slang-2.3.0' > cd src; make elf > make[2]: Entering directory '/usr/src/slang-2.3.0/src' > /usr/src/slang-2.3.0/autoconf/mkinsdir.sh > /usr/src/slang-2.3.0/src/elfobjs cp sysconf.h config.h cp > terminfo/default.inc terminfo.inc cd /usr/src/slang-2.3.0/src/elfobjs > && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586 > -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic > -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS > -fstack-protector-strong -fPIC -Dunix -DSLANG > /usr/src/slang-2.3.0/src/sldisply.c > cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall > -fexceptions -fPIC -march=i586 -mindirect-branch=thunk > -mfunction-return=thunk -mtune=generic -fomit-frame-pointer > -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS > -fstack-protector-strong -fPIC -Dunix -DSLANG > /usr/src/slang-2.3.0/src/slutty.c cd /usr/src/slang-2.3.0/src/elfobjs > && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586 > -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic > -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS > -fstack-protector-strong -fPIC -Dunix -DSLANG > /usr/src/slang-2.3.0/src/slang.c > /bin/sh: line 0: cd: /usr/src/slang-2.3.0/src/elfobjs: No such file or > directory mkdir -p -- /usr/src/slang-2.3.0/src/elfobjs > make[2]: *** [Makefile:472: > /usr/src/slang-2.3.0/src/elfobjs/sldisply.o] Error 1 > make[2]: *** Waiting for unfinished jobs.... > /usr/src/slang-2.3.0/src/slang.c: In function 'inner_interp': > /usr/src/slang-2.3.0/src/slang.c:5733:9: warning: 'test' may be used uninitialized in this function [-Wmaybe-uninitialized] > if ((0 == pop_int (&test)) > ~~~~~~~~~~~~~~~~~~~~~~ > && (test == 0)) > ^~~~~~~~~~~~~~ > /usr/src/slang-2.3.0/src/slang.c: In function 'lang_do_loops': > /usr/src/slang-2.3.0/src/slang.c:3725:10: warning: 'blks[0]' may be used uninitialized in this function [-Wmaybe-uninitialized] > block = blks[0]; > ~~~~~~^~~~~~~~~ > /usr/src/slang-2.3.0/src/slang.c:3924:7: warning: 'first' may be used uninitialized in this function [-Wmaybe-uninitialized] > i += ctrl; > ~~^~~~~~~ > /usr/src/slang-2.3.0/src/slang.c:3900:13: warning: 'last' may be used uninitialized in this function [-Wmaybe-uninitialized] > if (i > last) break; > ^ > make[2]: Leaving directory '/usr/src/slang-2.3.0/src' > make[1]: *** [Makefile:55: elf] Error 2 > make[1]: Leaving directory '/usr/src/slang-2.3.0' > make: *** [slang:74: /usr/src/log/slang-2.3.0] Error 2 > ***SNAP*** > > Don't know why. I tried two builds, same error. > > Best, > Matthias > >> I think that we are saving about 30 min to an hour on the nightly builds now… Did anybody else run some benchmarks? >> >> -Michael >> >>> On 6 Mar 2019, at 16:36, Michael Tremer wrote: >>> >>> Hi, >>> >>> Sorry for not being able to get around to this earlier, but I have invested some time on this: >>> >>> First of all, I merged Wolfgang’s patch. He is right, the build is too slow. It sometimes looks a bit like it is not taking enough advantage of fast systems. On the other hand with have small systems, which are (in their own right) quite fast. Those are the ARM systems. They usually have 1G of RAM. 2GB when you are lucky. >>> >>> So I adjusted make.sh a little bit and removed composing the MAKETUNING variable from it. >>> >>> Now, make.sh only figures out the value that usually comes after the -j. It is set in DEFAULT_PARALLELISM. >>> >>> In the individual LFS files, we can now set a new variable called MAX_PARALLELISM which is optional. In boost, I now calculate this from the amount of memory that is available. That will make sure that we have a cap on this that will be high on systems that have the memory and low on those that don’t. Before, we hardcoded -j2 which of course runs very slow on large systems. >>> >>> The code is here: >>> >>> >>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs >>> /heads/faster-build >>> >>> Please review. I am now running this on a couple of builders and will see what comes out. Please do the same and send me feedback. >>> >>> Best, >>> -Michael >>> >>>> On 21 Feb 2019, at 17:28, Wolfgang Apolinarski wrote: >>>> >>>> Hi Arne, >>>> >>>> of course I did delete the ccache between my multiple builds to verify the commit. To play it safe, I hard reset using git, checked that all folders (build, ccache, etc.) are gone and restarted the build after gettoolchain and downloadsrc. >>>> >>>> Sorry, I did not know that there are still a lot of machines with low ram that we are using to build ipfire. My hope was that these machines have disappeared over the last years. >>>> Additionally, I do not have the ability to test all ipfire arches, as mentioned, the patch needs testing by others. >>>> >>>> I also thought of adding a manual parameter for developers that somehow manually sets the parameters for parallel build, but since the MAKETUNING parameter already exists and is used with other packages this would just somehow mess everything up. >>>> >>>> For my builds, I can just manually apply the patch to speed up the full build. My thought was that sharing the patch would make sense for others. >>>> >>>>> -----Ursprüngliche Nachricht----- >>>>> Von: Arne Fitzenreiter >>>>> Gesendet: Donnerstag, 21. Februar 2019 09:44 >>>>> An: Wolfgang Apolinarski >>>>> Cc: development(a)lists.ipfire.org >>>>> Betreff: Re: [PATCH 0/1] Parallelized build for several packages >>>>> >>>>> On the most of this packages (eg. boot and cmake) i have disabled >>>>> parallel build because it fails on machines with low ram (less >>>>> than 2GB) or on arm. >>>>> So such changes need tested on an 1GB machine, on all >>>>> architectures and build two times after deleting ccache. Because >>>>> the build need much lower resources if it has the file already prebuilt in cache. >>>> >>> >> >> > > --===============0222617268710639223==--