On 09.03.2019 21:24, Wolfgang Apolinarski wrote: > Hi, > > sorry, that I did not respond in time. I also had issues with "slang" during > one of my latest test builds, especially when using something like "-j3" (it > always worked fine when using higher values). > The lfs manual also stated that the build of "slang" cannot be parallelized. > Sorry, I apparently overlooked that (I have now checked all other packages > and did not find incompabilities). No problem, current 'next' - with commented $(MAKETUNING) for 'slang' - built without problems. For the records: Last clean "Build finished in 7:02:45". Hardware: i7/2600, 8 GB RAM, 500GB SATA HDD, Ubuntu 18.04.2 LTS / 32bit (yes, I know... ;-) ). Best, Matthias > Today, I started again some more builds (with parallel build disabled for > slang), the test continues tomorrow. > > Best regards, > Wolfgang > PS.: The builds always use the branch that Michael created. > > -----Ursprüngliche Nachricht----- > Von: Matthias Fischer > Gesendet: Freitag, 8. März 2019 19:16 > An: Michael Tremer ; Wolfgang Apolinarski > > Cc: development(a)lists.ipfire.org > Betreff: Re: [PATCH 0/1] Parallelized build for several packages > > Hi, > > third try, now building current 'next', identical error: > > 'slang 2.3.0' doesn't like "cd $(DIR_APP) && make $(MAKETUNING)" here. > > After reverting > https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=lfs/slang;h=217e74c77317d4 > c829913f934458779fd278bf29;hb=23164efba5f57b3d8ccb07a166b613f2f951e1b6, > build continues... > > Best, > Matthias > > On 08.03.2019 16:58, Matthias Fischer wrote: >> Hi, >> >> On 08.03.2019 11:17, Michael Tremer wrote: >>> Hi, >>> >>> Are you guys still there? >> >> Yep. ;-) >> >> Your branch 'faster-build' refuses to build here - I get 'slang' errors: >> >> ***SNIP*** >> cd /usr/src/slang-2.3.0 && make -j9 >> make[1]: Entering directory '/usr/src/slang-2.3.0' >> cd src; make elf >> make[2]: Entering directory '/usr/src/slang-2.3.0/src' >> /usr/src/slang-2.3.0/autoconf/mkinsdir.sh >> /usr/src/slang-2.3.0/src/elfobjs cp sysconf.h config.h cp >> terminfo/default.inc terminfo.inc cd /usr/src/slang-2.3.0/src/elfobjs >> && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586 >> -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic >> -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS >> -fstack-protector-strong -fPIC -Dunix -DSLANG >> /usr/src/slang-2.3.0/src/sldisply.c >> cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall >> -fexceptions -fPIC -march=i586 -mindirect-branch=thunk >> -mfunction-return=thunk -mtune=generic -fomit-frame-pointer >> -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS >> -fstack-protector-strong -fPIC -Dunix -DSLANG >> /usr/src/slang-2.3.0/src/slutty.c cd /usr/src/slang-2.3.0/src/elfobjs >> && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586 >> -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic >> -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS >> -fstack-protector-strong -fPIC -Dunix -DSLANG >> /usr/src/slang-2.3.0/src/slang.c >> /bin/sh: line 0: cd: /usr/src/slang-2.3.0/src/elfobjs: No such file or >> directory mkdir -p -- /usr/src/slang-2.3.0/src/elfobjs >> make[2]: *** [Makefile:472: >> /usr/src/slang-2.3.0/src/elfobjs/sldisply.o] Error 1 >> make[2]: *** Waiting for unfinished jobs.... >> /usr/src/slang-2.3.0/src/slang.c: In function 'inner_interp': >> /usr/src/slang-2.3.0/src/slang.c:5733:9: warning: 'test' may be used > uninitialized in this function [-Wmaybe-uninitialized] >> if ((0 == pop_int (&test)) >> ~~~~~~~~~~~~~~~~~~~~~~ >> && (test == 0)) >> ^~~~~~~~~~~~~~ >> /usr/src/slang-2.3.0/src/slang.c: In function 'lang_do_loops': >> /usr/src/slang-2.3.0/src/slang.c:3725:10: warning: 'blks[0]' may be used > uninitialized in this function [-Wmaybe-uninitialized] >> block = blks[0]; >> ~~~~~~^~~~~~~~~ >> /usr/src/slang-2.3.0/src/slang.c:3924:7: warning: 'first' may be used > uninitialized in this function [-Wmaybe-uninitialized] >> i += ctrl; >> ~~^~~~~~~ >> /usr/src/slang-2.3.0/src/slang.c:3900:13: warning: 'last' may be used > uninitialized in this function [-Wmaybe-uninitialized] >> if (i > last) break; >> ^ >> make[2]: Leaving directory '/usr/src/slang-2.3.0/src' >> make[1]: *** [Makefile:55: elf] Error 2 >> make[1]: Leaving directory '/usr/src/slang-2.3.0' >> make: *** [slang:74: /usr/src/log/slang-2.3.0] Error 2 >> ***SNAP*** >> >> Don't know why. I tried two builds, same error. >> >> Best, >> Matthias >> >>> I think that we are saving about 30 min to an hour on the nightly builds > now… Did anybody else run some benchmarks? >>> >>> -Michael >>> >>>> On 6 Mar 2019, at 16:36, Michael Tremer > wrote: >>>> >>>> Hi, >>>> >>>> Sorry for not being able to get around to this earlier, but I have > invested some time on this: >>>> >>>> First of all, I merged Wolfgang’s patch. He is right, the build is too > slow. It sometimes looks a bit like it is not taking enough advantage of > fast systems. On the other hand with have small systems, which are (in their > own right) quite fast. Those are the ARM systems. They usually have 1G of > RAM. 2GB when you are lucky. >>>> >>>> So I adjusted make.sh a little bit and removed composing the MAKETUNING > variable from it. >>>> >>>> Now, make.sh only figures out the value that usually comes after the -j. > It is set in DEFAULT_PARALLELISM. >>>> >>>> In the individual LFS files, we can now set a new variable called > MAX_PARALLELISM which is optional. In boost, I now calculate this from the > amount of memory that is available. That will make sure that we have a cap > on this that will be high on systems that have the memory and low on those > that don’t. Before, we hardcoded -j2 which of course runs very slow on large > systems. >>>> >>>> The code is here: >>>> >>>> >>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs >>>> /heads/faster-build >>>> >>>> Please review. I am now running this on a couple of builders and will > see what comes out. Please do the same and send me feedback. >>>> >>>> Best, >>>> -Michael >>>> >>>>> On 21 Feb 2019, at 17:28, Wolfgang Apolinarski > wrote: >>>>> >>>>> Hi Arne, >>>>> >>>>> of course I did delete the ccache between my multiple builds to verify > the commit. To play it safe, I hard reset using git, checked that all > folders (build, ccache, etc.) are gone and restarted the build after > gettoolchain and downloadsrc. >>>>> >>>>> Sorry, I did not know that there are still a lot of machines with low > ram that we are using to build ipfire. My hope was that these machines have > disappeared over the last years. >>>>> Additionally, I do not have the ability to test all ipfire arches, as > mentioned, the patch needs testing by others. >>>>> >>>>> I also thought of adding a manual parameter for developers that somehow > manually sets the parameters for parallel build, but since the MAKETUNING > parameter already exists and is used with other packages this would just > somehow mess everything up. >>>>> >>>>> For my builds, I can just manually apply the patch to speed up the full > build. My thought was that sharing the patch would make sense for others. >>>>> >>>>>> -----Ursprüngliche Nachricht----- >>>>>> Von: Arne Fitzenreiter >>>>>> Gesendet: Donnerstag, 21. Februar 2019 09:44 >>>>>> An: Wolfgang Apolinarski >>>>>> Cc: development(a)lists.ipfire.org >>>>>> Betreff: Re: [PATCH 0/1] Parallelized build for several packages >>>>>> >>>>>> On the most of this packages (eg. boot and cmake) i have disabled >>>>>> parallel build because it fails on machines with low ram (less >>>>>> than 2GB) or on arm. >>>>>> So such changes need tested on an 1GB machine, on all >>>>>> architectures and build two times after deleting ccache. Because >>>>>> the build need much lower resources if it has the file already > prebuilt in cache.