Re: [PATCH 0/1] Parallelized build for several packages

[ummeegge <ummeegge@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 12850 bytes --]

Hi,

On Mo, 2019-03-11 at 09:37 +0000, Michael Tremer wrote:
> Yes, probably.
> 
> It uses a lot of memory when compiling because it is C++.
> 
> I will adjust the number of concurrent processes just like we did for
> boost.
This fixed it. Build went through.

Best,

Erik

> 
> Best,
> -Michael
> 
> > On 10 Mar 2019, at 19:17, ummeegge <ummeegge(a)ipfire.org> wrote:
> > 
> > Hi all,
> > 
> > On So, 2019-03-10 at 18:24 +0000, Michael Tremer wrote:
> > > Hi,
> > > 
> > > I merged the patch. And I am surprised it is only the one…
> > 
> > dnsdist do have problems to build here in next currently:
> > 
> > checking for boost/foreach.hpp... yes
> > checking for the toolset name used by Boost for g++... configure:
> > WARNING: could not figure out which toolset name to use for g++
> > ....
> > 
> > g++: internal compiler error: Killed (program cc1plus)
> > Please submit a full bug report,
> > with preprocessed source if appropriate.
> > See <http://bugtracker.ipfire.org> for instructions.
> > make[3]: *** [Makefile:743: dnsdist-lua2.o] Error 4
> > make[3]: *** Waiting for unfinished jobs....
> > 
> > 
> > May this is possibly related to this topic ?
> > 
> > Best,
> > 
> > Erik
> > 
> > > 
> > > -Michael
> > > 
> > > > On 9 Mar 2019, at 20:36, Matthias Fischer <
> > > > matthias.fischer(a)ipfire.org> wrote:
> > > > 
> > > > On 09.03.2019 21:24, Wolfgang Apolinarski wrote:
> > > > > Hi,
> > > > > 
> > > > > sorry, that I did not respond in time. I also had issues with
> > > > > "slang" during
> > > > > one of my latest test builds, especially when using something
> > > > > like "-j3" (it
> > > > > always worked fine when using higher values).
> > > > > The lfs manual also stated that the build of "slang" cannot
> > > > > be
> > > > > parallelized.
> > > > > Sorry, I apparently overlooked that (I have now checked all
> > > > > other
> > > > > packages
> > > > > and did not find incompabilities).
> > > > 
> > > > No problem, current 'next' - with commented $(MAKETUNING) for
> > > > 'slang' - built without problems.
> > > > 
> > > > For the records:
> > > > Last clean "Build finished in 7:02:45". Hardware: i7/2600, 8 GB
> > > > RAM, 500GB SATA HDD, Ubuntu 18.04.2 LTS / 32bit (yes, I know...
> > > > ;-) 
> > > > ).
> > > > 
> > > > Best,
> > > > Matthias
> > > > 
> > > > > Today, I started again some more builds (with parallel build
> > > > > disabled for
> > > > > slang), the test continues tomorrow.
> > > > > 
> > > > > Best regards,
> > > > > Wolfgang
> > > > > PS.: The builds always use the branch that Michael created.
> > > > > 
> > > > > -----Ursprüngliche Nachricht-----
> > > > > Von: Matthias Fischer <matthias.fischer(a)ipfire.org> 
> > > > > Gesendet: Freitag, 8. März 2019 19:16
> > > > > An: Michael Tremer <michael.tremer(a)ipfire.org>; Wolfgang
> > > > > Apolinarski
> > > > > <wolfgang.apolinarski(a)ipfire.org>
> > > > > Cc: development(a)lists.ipfire.org
> > > > > Betreff: Re: [PATCH 0/1] Parallelized build for several
> > > > > packages
> > > > > 
> > > > > Hi,
> > > > > 
> > > > > third try, now building current 'next', identical error:
> > > > > 
> > > > > 'slang 2.3.0' doesn't like "cd $(DIR_APP) && make
> > > > > $(MAKETUNING)"
> > > > > here.
> > > > > 
> > > > > After reverting
> > > > > 
> > 
> > 
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=lfs/slang;h=217e74c77317d4
> > > > > c829913f934458779fd278bf29;hb=23164efba5f57b3d8ccb07a166b613f
> > > > > 2f95
> > > > > 1e1b6,
> > > > > build continues...
> > > > > 
> > > > > Best,
> > > > > Matthias
> > > > > 
> > > > > On 08.03.2019 16:58, Matthias Fischer wrote:
> > > > > > Hi,
> > > > > > 
> > > > > > On 08.03.2019 11:17, Michael Tremer wrote:
> > > > > > > Hi,
> > > > > > > 
> > > > > > > Are you guys still there?
> > > > > > 
> > > > > > Yep. ;-)
> > > > > > 
> > > > > > Your branch 'faster-build' refuses to build here - I get
> > > > > > 'slang' errors:
> > > > > > 
> > > > > > ***SNIP***
> > > > > > cd /usr/src/slang-2.3.0 && make -j9
> > > > > > make[1]: Entering directory '/usr/src/slang-2.3.0'
> > > > > > cd src; make elf
> > > > > > make[2]: Entering directory '/usr/src/slang-2.3.0/src'
> > > > > > /usr/src/slang-2.3.0/autoconf/mkinsdir.sh 
> > > > > > /usr/src/slang-2.3.0/src/elfobjs cp sysconf.h config.h cp 
> > > > > > terminfo/default.inc terminfo.inc cd /usr/src/slang-
> > > > > > 2.3.0/src/elfobjs 
> > > > > > && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586 
> > > > > > -mindirect-branch=thunk -mfunction-return=thunk
> > > > > > -mtune=generic 
> > > > > > -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-
> > > > > > D_GLIBCXX_ASSERTIONS 
> > > > > > -fstack-protector-strong  -fPIC  -Dunix -DSLANG  
> > > > > > /usr/src/slang-2.3.0/src/sldisply.c
> > > > > > cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe
> > > > > > -Wall 
> > > > > > -fexceptions -fPIC -march=i586 -mindirect-branch=thunk 
> > > > > > -mfunction-return=thunk -mtune=generic -fomit-frame-
> > > > > > pointer 
> > > > > > -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS 
> > > > > > -fstack-protector-strong  -fPIC  -Dunix -DSLANG  
> > > > > > /usr/src/slang-2.3.0/src/slutty.c cd /usr/src/slang-
> > > > > > 2.3.0/src/elfobjs 
> > > > > > && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586 
> > > > > > -mindirect-branch=thunk -mfunction-return=thunk
> > > > > > -mtune=generic 
> > > > > > -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-
> > > > > > D_GLIBCXX_ASSERTIONS 
> > > > > > -fstack-protector-strong  -fPIC  -Dunix -DSLANG  
> > > > > > /usr/src/slang-2.3.0/src/slang.c
> > > > > > /bin/sh: line 0: cd: /usr/src/slang-2.3.0/src/elfobjs: No
> > > > > > such
> > > > > > file or 
> > > > > > directory mkdir -p -- /usr/src/slang-2.3.0/src/elfobjs
> > > > > > make[2]: *** [Makefile:472: 
> > > > > > /usr/src/slang-2.3.0/src/elfobjs/sldisply.o] Error 1
> > > > > > make[2]: *** Waiting for unfinished jobs....
> > > > > > /usr/src/slang-2.3.0/src/slang.c: In function
> > > > > > 'inner_interp':
> > > > > > /usr/src/slang-2.3.0/src/slang.c:5733:9: warning: 'test'
> > > > > > may be
> > > > > > used
> > > > > 
> > > > > uninitialized in this function [-Wmaybe-uninitialized]
> > > > > >    if ((0 == pop_int (&test))
> > > > > >        ~~~~~~~~~~~~~~~~~~~~~~
> > > > > >        && (test == 0))
> > > > > >        ^~~~~~~~~~~~~~
> > > > > > /usr/src/slang-2.3.0/src/slang.c: In function
> > > > > > 'lang_do_loops':
> > > > > > /usr/src/slang-2.3.0/src/slang.c:3725:10: warning:
> > > > > > 'blks[0]'
> > > > > > may be used
> > > > > 
> > > > > uninitialized in this function [-Wmaybe-uninitialized]
> > > > > >   block = blks[0];
> > > > > >   ~~~~~~^~~~~~~~~
> > > > > > /usr/src/slang-2.3.0/src/slang.c:3924:7: warning: 'first'
> > > > > > may
> > > > > > be used
> > > > > 
> > > > > uninitialized in this function [-Wmaybe-uninitialized]
> > > > > >    i += ctrl;
> > > > > >    ~~^~~~~~~
> > > > > > /usr/src/slang-2.3.0/src/slang.c:3900:13: warning: 'last'
> > > > > > may
> > > > > > be used
> > > > > 
> > > > > uninitialized in this function [-Wmaybe-uninitialized]
> > > > > >         if (i > last) break;
> > > > > >            ^
> > > > > > make[2]: Leaving directory '/usr/src/slang-2.3.0/src'
> > > > > > make[1]: *** [Makefile:55: elf] Error 2
> > > > > > make[1]: Leaving directory '/usr/src/slang-2.3.0'
> > > > > > make: *** [slang:74: /usr/src/log/slang-2.3.0] Error 2
> > > > > > ***SNAP***
> > > > > > 
> > > > > > Don't know why. I tried two builds, same error.
> > > > > > 
> > > > > > Best,
> > > > > > Matthias
> > > > > > 
> > > > > > > I think that we are saving about 30 min to an hour on the
> > > > > > > nightly builds
> > > > > 
> > > > > now… Did anybody else run some benchmarks?
> > > > > > > 
> > > > > > > -Michael
> > > > > > > 
> > > > > > > > On 6 Mar 2019, at 16:36, Michael Tremer <
> > > > > > > > michael.tremer(a)ipfire.org>
> > > > > 
> > > > > wrote:
> > > > > > > > 
> > > > > > > > Hi,
> > > > > > > > 
> > > > > > > > Sorry for not being able to get around to this earlier,
> > > > > > > > but
> > > > > > > > I have
> > > > > 
> > > > > invested some time on this:
> > > > > > > > 
> > > > > > > > First of all, I merged Wolfgang’s patch. He is right,
> > > > > > > > the
> > > > > > > > build is too
> > > > > 
> > > > > slow. It sometimes looks a bit like it is not taking enough
> > > > > advantage of
> > > > > fast systems. On the other hand with have small systems,
> > > > > which
> > > > > are (in their
> > > > > own right) quite fast. Those are the ARM systems. They
> > > > > usually
> > > > > have 1G of
> > > > > RAM. 2GB when you are lucky.
> > > > > > > > 
> > > > > > > > So I adjusted make.sh a little bit and removed
> > > > > > > > composing
> > > > > > > > the MAKETUNING
> > > > > 
> > > > > variable from it.
> > > > > > > > 
> > > > > > > > Now, make.sh only figures out the value that usually
> > > > > > > > comes
> > > > > > > > after the -j.
> > > > > 
> > > > > It is set in DEFAULT_PARALLELISM.
> > > > > > > > 
> > > > > > > > In the individual LFS files, we can now set a new
> > > > > > > > variable
> > > > > > > > called
> > > > > 
> > > > > MAX_PARALLELISM which is optional. In boost, I now calculate
> > > > > this
> > > > > from the
> > > > > amount of memory that is available. That will make sure that
> > > > > we
> > > > > have a cap
> > > > > on this that will be high on systems that have the memory and
> > > > > low
> > > > > on those
> > > > > that don’t. Before, we hardcoded -j2 which of course runs
> > > > > very
> > > > > slow on large
> > > > > systems.
> > > > > > > > 
> > > > > > > > The code is here:
> > > > > > > > 
> > > > > > > > 
> > > > > > > > 
> > 
> > 
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs
> > > > > > > > /heads/faster-build
> > > > > > > > 
> > > > > > > > Please review. I am now running this on a couple of
> > > > > > > > builders and will
> > > > > 
> > > > > see what comes out. Please do the same and send me feedback.
> > > > > > > > 
> > > > > > > > Best,
> > > > > > > > -Michael
> > > > > > > > 
> > > > > > > > > On 21 Feb 2019, at 17:28, Wolfgang Apolinarski
> > > > > 
> > > > > <wolfgang.apolinarski(a)ipfire.org> wrote:
> > > > > > > > > 
> > > > > > > > > Hi Arne,
> > > > > > > > > 
> > > > > > > > > of course I did delete the ccache between my multiple
> > > > > > > > > builds to verify
> > > > > 
> > > > > the commit. To play it safe, I hard reset using git, checked
> > > > > that
> > > > > all
> > > > > folders (build, ccache, etc.) are gone and restarted the
> > > > > build
> > > > > after
> > > > > gettoolchain and downloadsrc.
> > > > > > > > > 
> > > > > > > > > Sorry, I did not know that there are still a lot of
> > > > > > > > > machines with low
> > > > > 
> > > > > ram that we are using to build ipfire. My hope was that these
> > > > > machines have
> > > > > disappeared over the last years.
> > > > > > > > > Additionally, I do not have the ability to test all
> > > > > > > > > ipfire arches, as
> > > > > 
> > > > > mentioned, the patch needs testing by others.
> > > > > > > > > 
> > > > > > > > > I also thought of adding a manual parameter for
> > > > > > > > > developers that somehow
> > > > > 
> > > > > manually sets the parameters for parallel build, but since
> > > > > the
> > > > > MAKETUNING
> > > > > parameter already exists and is used with other packages this
> > > > > would just
> > > > > somehow mess everything up.
> > > > > > > > > 
> > > > > > > > > For my builds, I can just manually apply the patch to
> > > > > > > > > speed up the full
> > > > > 
> > > > > build. My thought was that sharing the patch would make sense
> > > > > for
> > > > > others.
> > > > > > > > > 
> > > > > > > > > > -----Ursprüngliche Nachricht-----
> > > > > > > > > > Von: Arne Fitzenreiter <
> > > > > > > > > > Arne.Fitzenreiter(a)ipfire.org>
> > > > > > > > > > Gesendet: Donnerstag, 21. Februar 2019 09:44
> > > > > > > > > > An: Wolfgang Apolinarski <
> > > > > > > > > > wolfgang.apolinarski(a)ipfire.org>
> > > > > > > > > > Cc: development(a)lists.ipfire.org
> > > > > > > > > > Betreff: Re: [PATCH 0/1] Parallelized build for
> > > > > > > > > > several
> > > > > > > > > > packages
> > > > > > > > > > 
> > > > > > > > > > On the most of this packages (eg. boot and cmake) i
> > > > > > > > > > have disabled 
> > > > > > > > > > parallel build because it fails on machines with
> > > > > > > > > > low
> > > > > > > > > > ram (less 
> > > > > > > > > > than 2GB) or on arm.
> > > > > > > > > > So such changes need tested on an 1GB machine, on
> > > > > > > > > > all 
> > > > > > > > > > architectures and build two times after deleting
> > > > > > > > > > ccache. Because 
> > > > > > > > > > the build need much lower resources if it has the
> > > > > > > > > > file
> > > > > > > > > > already
> > > > > 
> > > > > prebuilt in cache.
> > > 
> > > 
> 
>