RE: [PATCH 0/1] Parallelized build for several packages

[Wolfgang Apolinarski <wolfgang.apolinarski@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 11454 bytes --]

Hi!

> From: Michael Tremer

> Yes, probably.
> 
> It uses a lot of memory when compiling because it is C++.
> 
> I will adjust the number of concurrent processes just like we did for boost.

That is probably a good idea.
I tested a build with 12 cores and 1 GB of RAM (CentOS) and it actually compiled fine (x86_64), but it is a good idea to play it safe here (dnsdist). I also performed other tests at the weekend and all my builds have been successful, so far, I hope it stays that way...

Best regards,
Wolfgang

> 
> Best,
> -Michael
> 
> > On 10 Mar 2019, at 19:17, ummeegge <ummeegge(a)ipfire.org> wrote:
> >
> > Hi all,
> >
> > On So, 2019-03-10 at 18:24 +0000, Michael Tremer wrote:
> >> Hi,
> >>
> >> I merged the patch. And I am surprised it is only the one…
> > dnsdist do have problems to build here in next currently:
> >
> > checking for boost/foreach.hpp... yes
> > checking for the toolset name used by Boost for g++... configure:
> > WARNING: could not figure out which toolset name to use for g++ ....
> >
> > g++: internal compiler error: Killed (program cc1plus)
> > Please submit a full bug report,
> > with preprocessed source if appropriate.
> > See <http://bugtracker.ipfire.org> for instructions.
> > make[3]: *** [Makefile:743: dnsdist-lua2.o] Error 4
> > make[3]: *** Waiting for unfinished jobs....
> >
> >
> > May this is possibly related to this topic ?
> >
> > Best,
> >
> > Erik
> >
> >>
> >> -Michael
> >>
> >>> On 9 Mar 2019, at 20:36, Matthias Fischer <
> >>> matthias.fischer(a)ipfire.org> wrote:
> >>>
> >>> On 09.03.2019 21:24, Wolfgang Apolinarski wrote:
> >>>> Hi,
> >>>>
> >>>> sorry, that I did not respond in time. I also had issues with
> >>>> "slang" during one of my latest test builds, especially when using
> >>>> something like "-j3" (it always worked fine when using higher
> >>>> values).
> >>>> The lfs manual also stated that the build of "slang" cannot be
> >>>> parallelized.
> >>>> Sorry, I apparently overlooked that (I have now checked all other
> >>>> packages and did not find incompabilities).
> >>>
> >>> No problem, current 'next' - with commented $(MAKETUNING) for
> >>> 'slang' - built without problems.
> >>>
> >>> For the records:
> >>> Last clean "Build finished in 7:02:45". Hardware: i7/2600, 8 GB RAM,
> >>> 500GB SATA HDD, Ubuntu 18.04.2 LTS / 32bit (yes, I know... ;-) ).
> >>>
> >>> Best,
> >>> Matthias
> >>>
> >>>> Today, I started again some more builds (with parallel build
> >>>> disabled for slang), the test continues tomorrow.
> >>>>
> >>>> Best regards,
> >>>> Wolfgang
> >>>> PS.: The builds always use the branch that Michael created.
> >>>>
> >>>> -----Ursprüngliche Nachricht-----
> >>>> Von: Matthias Fischer <matthias.fischer(a)ipfire.org>
> >>>> Gesendet: Freitag, 8. März 2019 19:16
> >>>> An: Michael Tremer <michael.tremer(a)ipfire.org>; Wolfgang
> >>>> Apolinarski <wolfgang.apolinarski(a)ipfire.org>
> >>>> Cc: development(a)lists.ipfire.org
> >>>> Betreff: Re: [PATCH 0/1] Parallelized build for several packages
> >>>>
> >>>> Hi,
> >>>>
> >>>> third try, now building current 'next', identical error:
> >>>>
> >>>> 'slang 2.3.0' doesn't like "cd $(DIR_APP) && make $(MAKETUNING)"
> >>>> here.
> >>>>
> >>>> After reverting
> >>>>
> > https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=lfs/slang;h=217e74c7
> > 7317d4
> >>>>
> c829913f934458779fd278bf29;hb=23164efba5f57b3d8ccb07a166b613f2f95
> >>>> 1e1b6,
> >>>> build continues...
> >>>>
> >>>> Best,
> >>>> Matthias
> >>>>
> >>>> On 08.03.2019 16:58, Matthias Fischer wrote:
> >>>>> Hi,
> >>>>>
> >>>>> On 08.03.2019 11:17, Michael Tremer wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> Are you guys still there?
> >>>>>
> >>>>> Yep. ;-)
> >>>>>
> >>>>> Your branch 'faster-build' refuses to build here - I get 'slang'
> >>>>> errors:
> >>>>>
> >>>>> ***SNIP***
> >>>>> cd /usr/src/slang-2.3.0 && make -j9
> >>>>> make[1]: Entering directory '/usr/src/slang-2.3.0'
> >>>>> cd src; make elf
> >>>>> make[2]: Entering directory '/usr/src/slang-2.3.0/src'
> >>>>> /usr/src/slang-2.3.0/autoconf/mkinsdir.sh
> >>>>> /usr/src/slang-2.3.0/src/elfobjs cp sysconf.h config.h cp
> >>>>> terminfo/default.inc terminfo.inc cd /usr/src/slang-
> >>>>> 2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall -fexceptions -fPIC
> >>>>> -march=i586 -mindirect-branch=thunk -mfunction-return=thunk
> >>>>> -mtune=generic -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -
> Wp,-
> >>>>> D_GLIBCXX_ASSERTIONS -fstack-protector-strong  -fPIC  -Dunix
> >>>>> -DSLANG /usr/src/slang-2.3.0/src/sldisply.c
> >>>>> cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall
> >>>>> -fexceptions -fPIC -march=i586 -mindirect-branch=thunk
> >>>>> -mfunction-return=thunk -mtune=generic -fomit-frame-pointer
> >>>>> -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
> >>>>> -fstack-protector-strong  -fPIC  -Dunix -DSLANG
> >>>>> /usr/src/slang-2.3.0/src/slutty.c cd /usr/src/slang-
> >>>>> 2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall -fexceptions -fPIC
> >>>>> -march=i586 -mindirect-branch=thunk -mfunction-return=thunk
> >>>>> -mtune=generic -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -
> Wp,-
> >>>>> D_GLIBCXX_ASSERTIONS -fstack-protector-strong  -fPIC  -Dunix
> >>>>> -DSLANG /usr/src/slang-2.3.0/src/slang.c
> >>>>> /bin/sh: line 0: cd: /usr/src/slang-2.3.0/src/elfobjs: No such
> >>>>> file or directory mkdir -p -- /usr/src/slang-2.3.0/src/elfobjs
> >>>>> make[2]: *** [Makefile:472:
> >>>>> /usr/src/slang-2.3.0/src/elfobjs/sldisply.o] Error 1
> >>>>> make[2]: *** Waiting for unfinished jobs....
> >>>>> /usr/src/slang-2.3.0/src/slang.c: In function 'inner_interp':
> >>>>> /usr/src/slang-2.3.0/src/slang.c:5733:9: warning: 'test' may be
> >>>>> used
> >>>>
> >>>> uninitialized in this function [-Wmaybe-uninitialized]
> >>>>>    if ((0 == pop_int (&test))
> >>>>>        ~~~~~~~~~~~~~~~~~~~~~~
> >>>>>        && (test == 0))
> >>>>>        ^~~~~~~~~~~~~~
> >>>>> /usr/src/slang-2.3.0/src/slang.c: In function 'lang_do_loops':
> >>>>> /usr/src/slang-2.3.0/src/slang.c:3725:10: warning: 'blks[0]'
> >>>>> may be used
> >>>>
> >>>> uninitialized in this function [-Wmaybe-uninitialized]
> >>>>>   block = blks[0];
> >>>>>   ~~~~~~^~~~~~~~~
> >>>>> /usr/src/slang-2.3.0/src/slang.c:3924:7: warning: 'first' may be
> >>>>> used
> >>>>
> >>>> uninitialized in this function [-Wmaybe-uninitialized]
> >>>>>    i += ctrl;
> >>>>>    ~~^~~~~~~
> >>>>> /usr/src/slang-2.3.0/src/slang.c:3900:13: warning: 'last' may be
> >>>>> used
> >>>>
> >>>> uninitialized in this function [-Wmaybe-uninitialized]
> >>>>>         if (i > last) break;
> >>>>>            ^
> >>>>> make[2]: Leaving directory '/usr/src/slang-2.3.0/src'
> >>>>> make[1]: *** [Makefile:55: elf] Error 2
> >>>>> make[1]: Leaving directory '/usr/src/slang-2.3.0'
> >>>>> make: *** [slang:74: /usr/src/log/slang-2.3.0] Error 2
> >>>>> ***SNAP***
> >>>>>
> >>>>> Don't know why. I tried two builds, same error.
> >>>>>
> >>>>> Best,
> >>>>> Matthias
> >>>>>
> >>>>>> I think that we are saving about 30 min to an hour on the nightly
> >>>>>> builds
> >>>>
> >>>> now… Did anybody else run some benchmarks?
> >>>>>>
> >>>>>> -Michael
> >>>>>>
> >>>>>>> On 6 Mar 2019, at 16:36, Michael Tremer <
> >>>>>>> michael.tremer(a)ipfire.org>
> >>>>
> >>>> wrote:
> >>>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> Sorry for not being able to get around to this earlier, but I
> >>>>>>> have
> >>>>
> >>>> invested some time on this:
> >>>>>>>
> >>>>>>> First of all, I merged Wolfgang’s patch. He is right, the build
> >>>>>>> is too
> >>>>
> >>>> slow. It sometimes looks a bit like it is not taking enough
> >>>> advantage of fast systems. On the other hand with have small
> >>>> systems, which are (in their own right) quite fast. Those are the
> >>>> ARM systems. They usually have 1G of RAM. 2GB when you are lucky.
> >>>>>>>
> >>>>>>> So I adjusted make.sh a little bit and removed composing the
> >>>>>>> MAKETUNING
> >>>>
> >>>> variable from it.
> >>>>>>>
> >>>>>>> Now, make.sh only figures out the value that usually comes after
> >>>>>>> the -j.
> >>>>
> >>>> It is set in DEFAULT_PARALLELISM.
> >>>>>>>
> >>>>>>> In the individual LFS files, we can now set a new variable
> >>>>>>> called
> >>>>
> >>>> MAX_PARALLELISM which is optional. In boost, I now calculate this
> >>>> from the amount of memory that is available. That will make sure
> >>>> that we have a cap on this that will be high on systems that have
> >>>> the memory and low on those that don’t. Before, we hardcoded -j2
> >>>> which of course runs very slow on large systems.
> >>>>>>>
> >>>>>>> The code is here:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> > https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs
> >>>>>>> /heads/faster-build
> >>>>>>>
> >>>>>>> Please review. I am now running this on a couple of builders and
> >>>>>>> will
> >>>>
> >>>> see what comes out. Please do the same and send me feedback.
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> -Michael
> >>>>>>>
> >>>>>>>> On 21 Feb 2019, at 17:28, Wolfgang Apolinarski
> >>>>
> >>>> <wolfgang.apolinarski(a)ipfire.org> wrote:
> >>>>>>>>
> >>>>>>>> Hi Arne,
> >>>>>>>>
> >>>>>>>> of course I did delete the ccache between my multiple builds to
> >>>>>>>> verify
> >>>>
> >>>> the commit. To play it safe, I hard reset using git, checked that
> >>>> all folders (build, ccache, etc.) are gone and restarted the build
> >>>> after gettoolchain and downloadsrc.
> >>>>>>>>
> >>>>>>>> Sorry, I did not know that there are still a lot of machines
> >>>>>>>> with low
> >>>>
> >>>> ram that we are using to build ipfire. My hope was that these
> >>>> machines have disappeared over the last years.
> >>>>>>>> Additionally, I do not have the ability to test all ipfire
> >>>>>>>> arches, as
> >>>>
> >>>> mentioned, the patch needs testing by others.
> >>>>>>>>
> >>>>>>>> I also thought of adding a manual parameter for developers that
> >>>>>>>> somehow
> >>>>
> >>>> manually sets the parameters for parallel build, but since the
> >>>> MAKETUNING parameter already exists and is used with other
> packages
> >>>> this would just somehow mess everything up.
> >>>>>>>>
> >>>>>>>> For my builds, I can just manually apply the patch to speed up
> >>>>>>>> the full
> >>>>
> >>>> build. My thought was that sharing the patch would make sense for
> >>>> others.
> >>>>>>>>
> >>>>>>>>> -----Ursprüngliche Nachricht-----
> >>>>>>>>> Von: Arne Fitzenreiter <Arne.Fitzenreiter(a)ipfire.org>
> >>>>>>>>> Gesendet: Donnerstag, 21. Februar 2019 09:44
> >>>>>>>>> An: Wolfgang Apolinarski <
> >>>>>>>>> wolfgang.apolinarski(a)ipfire.org>
> >>>>>>>>> Cc: development(a)lists.ipfire.org
> >>>>>>>>> Betreff: Re: [PATCH 0/1] Parallelized build for several
> >>>>>>>>> packages
> >>>>>>>>>
> >>>>>>>>> On the most of this packages (eg. boot and cmake) i have
> >>>>>>>>> disabled parallel build because it fails on machines with low
> >>>>>>>>> ram (less than 2GB) or on arm.
> >>>>>>>>> So such changes need tested on an 1GB machine, on all
> >>>>>>>>> architectures and build two times after deleting ccache.
> >>>>>>>>> Because the build need much lower resources if it has the file
> >>>>>>>>> already
> >>>>
> >>>> prebuilt in cache.
> >>
> >>
> >