Hi! > From: Michael Tremer > Yes, probably. > > It uses a lot of memory when compiling because it is C++. > > I will adjust the number of concurrent processes just like we did for boost. That is probably a good idea. I tested a build with 12 cores and 1 GB of RAM (CentOS) and it actually compiled fine (x86_64), but it is a good idea to play it safe here (dnsdist). I also performed other tests at the weekend and all my builds have been successful, so far, I hope it stays that way... Best regards, Wolfgang > > Best, > -Michael > > > On 10 Mar 2019, at 19:17, ummeegge wrote: > > > > Hi all, > > > > On So, 2019-03-10 at 18:24 +0000, Michael Tremer wrote: > >> Hi, > >> > >> I merged the patch. And I am surprised it is only the one… > > dnsdist do have problems to build here in next currently: > > > > checking for boost/foreach.hpp... yes > > checking for the toolset name used by Boost for g++... configure: > > WARNING: could not figure out which toolset name to use for g++ .... > > > > g++: internal compiler error: Killed (program cc1plus) > > Please submit a full bug report, > > with preprocessed source if appropriate. > > See for instructions. > > make[3]: *** [Makefile:743: dnsdist-lua2.o] Error 4 > > make[3]: *** Waiting for unfinished jobs.... > > > > > > May this is possibly related to this topic ? > > > > Best, > > > > Erik > > > >> > >> -Michael > >> > >>> On 9 Mar 2019, at 20:36, Matthias Fischer < > >>> matthias.fischer(a)ipfire.org> wrote: > >>> > >>> On 09.03.2019 21:24, Wolfgang Apolinarski wrote: > >>>> Hi, > >>>> > >>>> sorry, that I did not respond in time. I also had issues with > >>>> "slang" during one of my latest test builds, especially when using > >>>> something like "-j3" (it always worked fine when using higher > >>>> values). > >>>> The lfs manual also stated that the build of "slang" cannot be > >>>> parallelized. > >>>> Sorry, I apparently overlooked that (I have now checked all other > >>>> packages and did not find incompabilities). > >>> > >>> No problem, current 'next' - with commented $(MAKETUNING) for > >>> 'slang' - built without problems. > >>> > >>> For the records: > >>> Last clean "Build finished in 7:02:45". Hardware: i7/2600, 8 GB RAM, > >>> 500GB SATA HDD, Ubuntu 18.04.2 LTS / 32bit (yes, I know... ;-) ). > >>> > >>> Best, > >>> Matthias > >>> > >>>> Today, I started again some more builds (with parallel build > >>>> disabled for slang), the test continues tomorrow. > >>>> > >>>> Best regards, > >>>> Wolfgang > >>>> PS.: The builds always use the branch that Michael created. > >>>> > >>>> -----Ursprüngliche Nachricht----- > >>>> Von: Matthias Fischer > >>>> Gesendet: Freitag, 8. März 2019 19:16 > >>>> An: Michael Tremer ; Wolfgang > >>>> Apolinarski > >>>> Cc: development(a)lists.ipfire.org > >>>> Betreff: Re: [PATCH 0/1] Parallelized build for several packages > >>>> > >>>> Hi, > >>>> > >>>> third try, now building current 'next', identical error: > >>>> > >>>> 'slang 2.3.0' doesn't like "cd $(DIR_APP) && make $(MAKETUNING)" > >>>> here. > >>>> > >>>> After reverting > >>>> > > https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=lfs/slang;h=217e74c7 > > 7317d4 > >>>> > c829913f934458779fd278bf29;hb=23164efba5f57b3d8ccb07a166b613f2f95 > >>>> 1e1b6, > >>>> build continues... > >>>> > >>>> Best, > >>>> Matthias > >>>> > >>>> On 08.03.2019 16:58, Matthias Fischer wrote: > >>>>> Hi, > >>>>> > >>>>> On 08.03.2019 11:17, Michael Tremer wrote: > >>>>>> Hi, > >>>>>> > >>>>>> Are you guys still there? > >>>>> > >>>>> Yep. ;-) > >>>>> > >>>>> Your branch 'faster-build' refuses to build here - I get 'slang' > >>>>> errors: > >>>>> > >>>>> ***SNIP*** > >>>>> cd /usr/src/slang-2.3.0 && make -j9 > >>>>> make[1]: Entering directory '/usr/src/slang-2.3.0' > >>>>> cd src; make elf > >>>>> make[2]: Entering directory '/usr/src/slang-2.3.0/src' > >>>>> /usr/src/slang-2.3.0/autoconf/mkinsdir.sh > >>>>> /usr/src/slang-2.3.0/src/elfobjs cp sysconf.h config.h cp > >>>>> terminfo/default.inc terminfo.inc cd /usr/src/slang- > >>>>> 2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall -fexceptions -fPIC > >>>>> -march=i586 -mindirect-branch=thunk -mfunction-return=thunk > >>>>> -mtune=generic -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 - > Wp,- > >>>>> D_GLIBCXX_ASSERTIONS -fstack-protector-strong -fPIC -Dunix > >>>>> -DSLANG /usr/src/slang-2.3.0/src/sldisply.c > >>>>> cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall > >>>>> -fexceptions -fPIC -march=i586 -mindirect-branch=thunk > >>>>> -mfunction-return=thunk -mtune=generic -fomit-frame-pointer > >>>>> -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS > >>>>> -fstack-protector-strong -fPIC -Dunix -DSLANG > >>>>> /usr/src/slang-2.3.0/src/slutty.c cd /usr/src/slang- > >>>>> 2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall -fexceptions -fPIC > >>>>> -march=i586 -mindirect-branch=thunk -mfunction-return=thunk > >>>>> -mtune=generic -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 - > Wp,- > >>>>> D_GLIBCXX_ASSERTIONS -fstack-protector-strong -fPIC -Dunix > >>>>> -DSLANG /usr/src/slang-2.3.0/src/slang.c > >>>>> /bin/sh: line 0: cd: /usr/src/slang-2.3.0/src/elfobjs: No such > >>>>> file or directory mkdir -p -- /usr/src/slang-2.3.0/src/elfobjs > >>>>> make[2]: *** [Makefile:472: > >>>>> /usr/src/slang-2.3.0/src/elfobjs/sldisply.o] Error 1 > >>>>> make[2]: *** Waiting for unfinished jobs.... > >>>>> /usr/src/slang-2.3.0/src/slang.c: In function 'inner_interp': > >>>>> /usr/src/slang-2.3.0/src/slang.c:5733:9: warning: 'test' may be > >>>>> used > >>>> > >>>> uninitialized in this function [-Wmaybe-uninitialized] > >>>>> if ((0 == pop_int (&test)) > >>>>> ~~~~~~~~~~~~~~~~~~~~~~ > >>>>> && (test == 0)) > >>>>> ^~~~~~~~~~~~~~ > >>>>> /usr/src/slang-2.3.0/src/slang.c: In function 'lang_do_loops': > >>>>> /usr/src/slang-2.3.0/src/slang.c:3725:10: warning: 'blks[0]' > >>>>> may be used > >>>> > >>>> uninitialized in this function [-Wmaybe-uninitialized] > >>>>> block = blks[0]; > >>>>> ~~~~~~^~~~~~~~~ > >>>>> /usr/src/slang-2.3.0/src/slang.c:3924:7: warning: 'first' may be > >>>>> used > >>>> > >>>> uninitialized in this function [-Wmaybe-uninitialized] > >>>>> i += ctrl; > >>>>> ~~^~~~~~~ > >>>>> /usr/src/slang-2.3.0/src/slang.c:3900:13: warning: 'last' may be > >>>>> used > >>>> > >>>> uninitialized in this function [-Wmaybe-uninitialized] > >>>>> if (i > last) break; > >>>>> ^ > >>>>> make[2]: Leaving directory '/usr/src/slang-2.3.0/src' > >>>>> make[1]: *** [Makefile:55: elf] Error 2 > >>>>> make[1]: Leaving directory '/usr/src/slang-2.3.0' > >>>>> make: *** [slang:74: /usr/src/log/slang-2.3.0] Error 2 > >>>>> ***SNAP*** > >>>>> > >>>>> Don't know why. I tried two builds, same error. > >>>>> > >>>>> Best, > >>>>> Matthias > >>>>> > >>>>>> I think that we are saving about 30 min to an hour on the nightly > >>>>>> builds > >>>> > >>>> now… Did anybody else run some benchmarks? > >>>>>> > >>>>>> -Michael > >>>>>> > >>>>>>> On 6 Mar 2019, at 16:36, Michael Tremer < > >>>>>>> michael.tremer(a)ipfire.org> > >>>> > >>>> wrote: > >>>>>>> > >>>>>>> Hi, > >>>>>>> > >>>>>>> Sorry for not being able to get around to this earlier, but I > >>>>>>> have > >>>> > >>>> invested some time on this: > >>>>>>> > >>>>>>> First of all, I merged Wolfgang’s patch. He is right, the build > >>>>>>> is too > >>>> > >>>> slow. It sometimes looks a bit like it is not taking enough > >>>> advantage of fast systems. On the other hand with have small > >>>> systems, which are (in their own right) quite fast. Those are the > >>>> ARM systems. They usually have 1G of RAM. 2GB when you are lucky. > >>>>>>> > >>>>>>> So I adjusted make.sh a little bit and removed composing the > >>>>>>> MAKETUNING > >>>> > >>>> variable from it. > >>>>>>> > >>>>>>> Now, make.sh only figures out the value that usually comes after > >>>>>>> the -j. > >>>> > >>>> It is set in DEFAULT_PARALLELISM. > >>>>>>> > >>>>>>> In the individual LFS files, we can now set a new variable > >>>>>>> called > >>>> > >>>> MAX_PARALLELISM which is optional. In boost, I now calculate this > >>>> from the amount of memory that is available. That will make sure > >>>> that we have a cap on this that will be high on systems that have > >>>> the memory and low on those that don’t. Before, we hardcoded -j2 > >>>> which of course runs very slow on large systems. > >>>>>>> > >>>>>>> The code is here: > >>>>>>> > >>>>>>> > >>>>>>> > > https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs > >>>>>>> /heads/faster-build > >>>>>>> > >>>>>>> Please review. I am now running this on a couple of builders and > >>>>>>> will > >>>> > >>>> see what comes out. Please do the same and send me feedback. > >>>>>>> > >>>>>>> Best, > >>>>>>> -Michael > >>>>>>> > >>>>>>>> On 21 Feb 2019, at 17:28, Wolfgang Apolinarski > >>>> > >>>> wrote: > >>>>>>>> > >>>>>>>> Hi Arne, > >>>>>>>> > >>>>>>>> of course I did delete the ccache between my multiple builds to > >>>>>>>> verify > >>>> > >>>> the commit. To play it safe, I hard reset using git, checked that > >>>> all folders (build, ccache, etc.) are gone and restarted the build > >>>> after gettoolchain and downloadsrc. > >>>>>>>> > >>>>>>>> Sorry, I did not know that there are still a lot of machines > >>>>>>>> with low > >>>> > >>>> ram that we are using to build ipfire. My hope was that these > >>>> machines have disappeared over the last years. > >>>>>>>> Additionally, I do not have the ability to test all ipfire > >>>>>>>> arches, as > >>>> > >>>> mentioned, the patch needs testing by others. > >>>>>>>> > >>>>>>>> I also thought of adding a manual parameter for developers that > >>>>>>>> somehow > >>>> > >>>> manually sets the parameters for parallel build, but since the > >>>> MAKETUNING parameter already exists and is used with other > packages > >>>> this would just somehow mess everything up. > >>>>>>>> > >>>>>>>> For my builds, I can just manually apply the patch to speed up > >>>>>>>> the full > >>>> > >>>> build. My thought was that sharing the patch would make sense for > >>>> others. > >>>>>>>> > >>>>>>>>> -----Ursprüngliche Nachricht----- > >>>>>>>>> Von: Arne Fitzenreiter > >>>>>>>>> Gesendet: Donnerstag, 21. Februar 2019 09:44 > >>>>>>>>> An: Wolfgang Apolinarski < > >>>>>>>>> wolfgang.apolinarski(a)ipfire.org> > >>>>>>>>> Cc: development(a)lists.ipfire.org > >>>>>>>>> Betreff: Re: [PATCH 0/1] Parallelized build for several > >>>>>>>>> packages > >>>>>>>>> > >>>>>>>>> On the most of this packages (eg. boot and cmake) i have > >>>>>>>>> disabled parallel build because it fails on machines with low > >>>>>>>>> ram (less than 2GB) or on arm. > >>>>>>>>> So such changes need tested on an 1GB machine, on all > >>>>>>>>> architectures and build two times after deleting ccache. > >>>>>>>>> Because the build need much lower resources if it has the file > >>>>>>>>> already > >>>> > >>>> prebuilt in cache. > >> > >> > >