* RE: [PATCH 0/1] Parallelized build for several packages
@ 2019-03-11 17:50 Wolfgang Apolinarski
0 siblings, 0 replies; 14+ messages in thread
From: Wolfgang Apolinarski @ 2019-03-11 17:50 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 11454 bytes --]
Hi!
> From: Michael Tremer
> Yes, probably.
>
> It uses a lot of memory when compiling because it is C++.
>
> I will adjust the number of concurrent processes just like we did for boost.
That is probably a good idea.
I tested a build with 12 cores and 1 GB of RAM (CentOS) and it actually compiled fine (x86_64), but it is a good idea to play it safe here (dnsdist). I also performed other tests at the weekend and all my builds have been successful, so far, I hope it stays that way...
Best regards,
Wolfgang
>
> Best,
> -Michael
>
> > On 10 Mar 2019, at 19:17, ummeegge <ummeegge(a)ipfire.org> wrote:
> >
> > Hi all,
> >
> > On So, 2019-03-10 at 18:24 +0000, Michael Tremer wrote:
> >> Hi,
> >>
> >> I merged the patch. And I am surprised it is only the one…
> > dnsdist do have problems to build here in next currently:
> >
> > checking for boost/foreach.hpp... yes
> > checking for the toolset name used by Boost for g++... configure:
> > WARNING: could not figure out which toolset name to use for g++ ....
> >
> > g++: internal compiler error: Killed (program cc1plus)
> > Please submit a full bug report,
> > with preprocessed source if appropriate.
> > See <http://bugtracker.ipfire.org> for instructions.
> > make[3]: *** [Makefile:743: dnsdist-lua2.o] Error 4
> > make[3]: *** Waiting for unfinished jobs....
> >
> >
> > May this is possibly related to this topic ?
> >
> > Best,
> >
> > Erik
> >
> >>
> >> -Michael
> >>
> >>> On 9 Mar 2019, at 20:36, Matthias Fischer <
> >>> matthias.fischer(a)ipfire.org> wrote:
> >>>
> >>> On 09.03.2019 21:24, Wolfgang Apolinarski wrote:
> >>>> Hi,
> >>>>
> >>>> sorry, that I did not respond in time. I also had issues with
> >>>> "slang" during one of my latest test builds, especially when using
> >>>> something like "-j3" (it always worked fine when using higher
> >>>> values).
> >>>> The lfs manual also stated that the build of "slang" cannot be
> >>>> parallelized.
> >>>> Sorry, I apparently overlooked that (I have now checked all other
> >>>> packages and did not find incompabilities).
> >>>
> >>> No problem, current 'next' - with commented $(MAKETUNING) for
> >>> 'slang' - built without problems.
> >>>
> >>> For the records:
> >>> Last clean "Build finished in 7:02:45". Hardware: i7/2600, 8 GB RAM,
> >>> 500GB SATA HDD, Ubuntu 18.04.2 LTS / 32bit (yes, I know... ;-) ).
> >>>
> >>> Best,
> >>> Matthias
> >>>
> >>>> Today, I started again some more builds (with parallel build
> >>>> disabled for slang), the test continues tomorrow.
> >>>>
> >>>> Best regards,
> >>>> Wolfgang
> >>>> PS.: The builds always use the branch that Michael created.
> >>>>
> >>>> -----Ursprüngliche Nachricht-----
> >>>> Von: Matthias Fischer <matthias.fischer(a)ipfire.org>
> >>>> Gesendet: Freitag, 8. März 2019 19:16
> >>>> An: Michael Tremer <michael.tremer(a)ipfire.org>; Wolfgang
> >>>> Apolinarski <wolfgang.apolinarski(a)ipfire.org>
> >>>> Cc: development(a)lists.ipfire.org
> >>>> Betreff: Re: [PATCH 0/1] Parallelized build for several packages
> >>>>
> >>>> Hi,
> >>>>
> >>>> third try, now building current 'next', identical error:
> >>>>
> >>>> 'slang 2.3.0' doesn't like "cd $(DIR_APP) && make $(MAKETUNING)"
> >>>> here.
> >>>>
> >>>> After reverting
> >>>>
> > https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=lfs/slang;h=217e74c7
> > 7317d4
> >>>>
> c829913f934458779fd278bf29;hb=23164efba5f57b3d8ccb07a166b613f2f95
> >>>> 1e1b6,
> >>>> build continues...
> >>>>
> >>>> Best,
> >>>> Matthias
> >>>>
> >>>> On 08.03.2019 16:58, Matthias Fischer wrote:
> >>>>> Hi,
> >>>>>
> >>>>> On 08.03.2019 11:17, Michael Tremer wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> Are you guys still there?
> >>>>>
> >>>>> Yep. ;-)
> >>>>>
> >>>>> Your branch 'faster-build' refuses to build here - I get 'slang'
> >>>>> errors:
> >>>>>
> >>>>> ***SNIP***
> >>>>> cd /usr/src/slang-2.3.0 && make -j9
> >>>>> make[1]: Entering directory '/usr/src/slang-2.3.0'
> >>>>> cd src; make elf
> >>>>> make[2]: Entering directory '/usr/src/slang-2.3.0/src'
> >>>>> /usr/src/slang-2.3.0/autoconf/mkinsdir.sh
> >>>>> /usr/src/slang-2.3.0/src/elfobjs cp sysconf.h config.h cp
> >>>>> terminfo/default.inc terminfo.inc cd /usr/src/slang-
> >>>>> 2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall -fexceptions -fPIC
> >>>>> -march=i586 -mindirect-branch=thunk -mfunction-return=thunk
> >>>>> -mtune=generic -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -
> Wp,-
> >>>>> D_GLIBCXX_ASSERTIONS -fstack-protector-strong -fPIC -Dunix
> >>>>> -DSLANG /usr/src/slang-2.3.0/src/sldisply.c
> >>>>> cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall
> >>>>> -fexceptions -fPIC -march=i586 -mindirect-branch=thunk
> >>>>> -mfunction-return=thunk -mtune=generic -fomit-frame-pointer
> >>>>> -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
> >>>>> -fstack-protector-strong -fPIC -Dunix -DSLANG
> >>>>> /usr/src/slang-2.3.0/src/slutty.c cd /usr/src/slang-
> >>>>> 2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall -fexceptions -fPIC
> >>>>> -march=i586 -mindirect-branch=thunk -mfunction-return=thunk
> >>>>> -mtune=generic -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -
> Wp,-
> >>>>> D_GLIBCXX_ASSERTIONS -fstack-protector-strong -fPIC -Dunix
> >>>>> -DSLANG /usr/src/slang-2.3.0/src/slang.c
> >>>>> /bin/sh: line 0: cd: /usr/src/slang-2.3.0/src/elfobjs: No such
> >>>>> file or directory mkdir -p -- /usr/src/slang-2.3.0/src/elfobjs
> >>>>> make[2]: *** [Makefile:472:
> >>>>> /usr/src/slang-2.3.0/src/elfobjs/sldisply.o] Error 1
> >>>>> make[2]: *** Waiting for unfinished jobs....
> >>>>> /usr/src/slang-2.3.0/src/slang.c: In function 'inner_interp':
> >>>>> /usr/src/slang-2.3.0/src/slang.c:5733:9: warning: 'test' may be
> >>>>> used
> >>>>
> >>>> uninitialized in this function [-Wmaybe-uninitialized]
> >>>>> if ((0 == pop_int (&test))
> >>>>> ~~~~~~~~~~~~~~~~~~~~~~
> >>>>> && (test == 0))
> >>>>> ^~~~~~~~~~~~~~
> >>>>> /usr/src/slang-2.3.0/src/slang.c: In function 'lang_do_loops':
> >>>>> /usr/src/slang-2.3.0/src/slang.c:3725:10: warning: 'blks[0]'
> >>>>> may be used
> >>>>
> >>>> uninitialized in this function [-Wmaybe-uninitialized]
> >>>>> block = blks[0];
> >>>>> ~~~~~~^~~~~~~~~
> >>>>> /usr/src/slang-2.3.0/src/slang.c:3924:7: warning: 'first' may be
> >>>>> used
> >>>>
> >>>> uninitialized in this function [-Wmaybe-uninitialized]
> >>>>> i += ctrl;
> >>>>> ~~^~~~~~~
> >>>>> /usr/src/slang-2.3.0/src/slang.c:3900:13: warning: 'last' may be
> >>>>> used
> >>>>
> >>>> uninitialized in this function [-Wmaybe-uninitialized]
> >>>>> if (i > last) break;
> >>>>> ^
> >>>>> make[2]: Leaving directory '/usr/src/slang-2.3.0/src'
> >>>>> make[1]: *** [Makefile:55: elf] Error 2
> >>>>> make[1]: Leaving directory '/usr/src/slang-2.3.0'
> >>>>> make: *** [slang:74: /usr/src/log/slang-2.3.0] Error 2
> >>>>> ***SNAP***
> >>>>>
> >>>>> Don't know why. I tried two builds, same error.
> >>>>>
> >>>>> Best,
> >>>>> Matthias
> >>>>>
> >>>>>> I think that we are saving about 30 min to an hour on the nightly
> >>>>>> builds
> >>>>
> >>>> now… Did anybody else run some benchmarks?
> >>>>>>
> >>>>>> -Michael
> >>>>>>
> >>>>>>> On 6 Mar 2019, at 16:36, Michael Tremer <
> >>>>>>> michael.tremer(a)ipfire.org>
> >>>>
> >>>> wrote:
> >>>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> Sorry for not being able to get around to this earlier, but I
> >>>>>>> have
> >>>>
> >>>> invested some time on this:
> >>>>>>>
> >>>>>>> First of all, I merged Wolfgang’s patch. He is right, the build
> >>>>>>> is too
> >>>>
> >>>> slow. It sometimes looks a bit like it is not taking enough
> >>>> advantage of fast systems. On the other hand with have small
> >>>> systems, which are (in their own right) quite fast. Those are the
> >>>> ARM systems. They usually have 1G of RAM. 2GB when you are lucky.
> >>>>>>>
> >>>>>>> So I adjusted make.sh a little bit and removed composing the
> >>>>>>> MAKETUNING
> >>>>
> >>>> variable from it.
> >>>>>>>
> >>>>>>> Now, make.sh only figures out the value that usually comes after
> >>>>>>> the -j.
> >>>>
> >>>> It is set in DEFAULT_PARALLELISM.
> >>>>>>>
> >>>>>>> In the individual LFS files, we can now set a new variable
> >>>>>>> called
> >>>>
> >>>> MAX_PARALLELISM which is optional. In boost, I now calculate this
> >>>> from the amount of memory that is available. That will make sure
> >>>> that we have a cap on this that will be high on systems that have
> >>>> the memory and low on those that don’t. Before, we hardcoded -j2
> >>>> which of course runs very slow on large systems.
> >>>>>>>
> >>>>>>> The code is here:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> > https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs
> >>>>>>> /heads/faster-build
> >>>>>>>
> >>>>>>> Please review. I am now running this on a couple of builders and
> >>>>>>> will
> >>>>
> >>>> see what comes out. Please do the same and send me feedback.
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> -Michael
> >>>>>>>
> >>>>>>>> On 21 Feb 2019, at 17:28, Wolfgang Apolinarski
> >>>>
> >>>> <wolfgang.apolinarski(a)ipfire.org> wrote:
> >>>>>>>>
> >>>>>>>> Hi Arne,
> >>>>>>>>
> >>>>>>>> of course I did delete the ccache between my multiple builds to
> >>>>>>>> verify
> >>>>
> >>>> the commit. To play it safe, I hard reset using git, checked that
> >>>> all folders (build, ccache, etc.) are gone and restarted the build
> >>>> after gettoolchain and downloadsrc.
> >>>>>>>>
> >>>>>>>> Sorry, I did not know that there are still a lot of machines
> >>>>>>>> with low
> >>>>
> >>>> ram that we are using to build ipfire. My hope was that these
> >>>> machines have disappeared over the last years.
> >>>>>>>> Additionally, I do not have the ability to test all ipfire
> >>>>>>>> arches, as
> >>>>
> >>>> mentioned, the patch needs testing by others.
> >>>>>>>>
> >>>>>>>> I also thought of adding a manual parameter for developers that
> >>>>>>>> somehow
> >>>>
> >>>> manually sets the parameters for parallel build, but since the
> >>>> MAKETUNING parameter already exists and is used with other
> packages
> >>>> this would just somehow mess everything up.
> >>>>>>>>
> >>>>>>>> For my builds, I can just manually apply the patch to speed up
> >>>>>>>> the full
> >>>>
> >>>> build. My thought was that sharing the patch would make sense for
> >>>> others.
> >>>>>>>>
> >>>>>>>>> -----Ursprüngliche Nachricht-----
> >>>>>>>>> Von: Arne Fitzenreiter <Arne.Fitzenreiter(a)ipfire.org>
> >>>>>>>>> Gesendet: Donnerstag, 21. Februar 2019 09:44
> >>>>>>>>> An: Wolfgang Apolinarski <
> >>>>>>>>> wolfgang.apolinarski(a)ipfire.org>
> >>>>>>>>> Cc: development(a)lists.ipfire.org
> >>>>>>>>> Betreff: Re: [PATCH 0/1] Parallelized build for several
> >>>>>>>>> packages
> >>>>>>>>>
> >>>>>>>>> On the most of this packages (eg. boot and cmake) i have
> >>>>>>>>> disabled parallel build because it fails on machines with low
> >>>>>>>>> ram (less than 2GB) or on arm.
> >>>>>>>>> So such changes need tested on an 1GB machine, on all
> >>>>>>>>> architectures and build two times after deleting ccache.
> >>>>>>>>> Because the build need much lower resources if it has the file
> >>>>>>>>> already
> >>>>
> >>>> prebuilt in cache.
> >>
> >>
> >
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 0/1] Parallelized build for several packages
2019-03-11 9:37 ` Michael Tremer
@ 2019-03-11 17:07 ` ummeegge
0 siblings, 0 replies; 14+ messages in thread
From: ummeegge @ 2019-03-11 17:07 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 12850 bytes --]
Hi,
On Mo, 2019-03-11 at 09:37 +0000, Michael Tremer wrote:
> Yes, probably.
>
> It uses a lot of memory when compiling because it is C++.
>
> I will adjust the number of concurrent processes just like we did for
> boost.
This fixed it. Build went through.
Best,
Erik
>
> Best,
> -Michael
>
> > On 10 Mar 2019, at 19:17, ummeegge <ummeegge(a)ipfire.org> wrote:
> >
> > Hi all,
> >
> > On So, 2019-03-10 at 18:24 +0000, Michael Tremer wrote:
> > > Hi,
> > >
> > > I merged the patch. And I am surprised it is only the one…
> >
> > dnsdist do have problems to build here in next currently:
> >
> > checking for boost/foreach.hpp... yes
> > checking for the toolset name used by Boost for g++... configure:
> > WARNING: could not figure out which toolset name to use for g++
> > ....
> >
> > g++: internal compiler error: Killed (program cc1plus)
> > Please submit a full bug report,
> > with preprocessed source if appropriate.
> > See <http://bugtracker.ipfire.org> for instructions.
> > make[3]: *** [Makefile:743: dnsdist-lua2.o] Error 4
> > make[3]: *** Waiting for unfinished jobs....
> >
> >
> > May this is possibly related to this topic ?
> >
> > Best,
> >
> > Erik
> >
> > >
> > > -Michael
> > >
> > > > On 9 Mar 2019, at 20:36, Matthias Fischer <
> > > > matthias.fischer(a)ipfire.org> wrote:
> > > >
> > > > On 09.03.2019 21:24, Wolfgang Apolinarski wrote:
> > > > > Hi,
> > > > >
> > > > > sorry, that I did not respond in time. I also had issues with
> > > > > "slang" during
> > > > > one of my latest test builds, especially when using something
> > > > > like "-j3" (it
> > > > > always worked fine when using higher values).
> > > > > The lfs manual also stated that the build of "slang" cannot
> > > > > be
> > > > > parallelized.
> > > > > Sorry, I apparently overlooked that (I have now checked all
> > > > > other
> > > > > packages
> > > > > and did not find incompabilities).
> > > >
> > > > No problem, current 'next' - with commented $(MAKETUNING) for
> > > > 'slang' - built without problems.
> > > >
> > > > For the records:
> > > > Last clean "Build finished in 7:02:45". Hardware: i7/2600, 8 GB
> > > > RAM, 500GB SATA HDD, Ubuntu 18.04.2 LTS / 32bit (yes, I know...
> > > > ;-)
> > > > ).
> > > >
> > > > Best,
> > > > Matthias
> > > >
> > > > > Today, I started again some more builds (with parallel build
> > > > > disabled for
> > > > > slang), the test continues tomorrow.
> > > > >
> > > > > Best regards,
> > > > > Wolfgang
> > > > > PS.: The builds always use the branch that Michael created.
> > > > >
> > > > > -----Ursprüngliche Nachricht-----
> > > > > Von: Matthias Fischer <matthias.fischer(a)ipfire.org>
> > > > > Gesendet: Freitag, 8. März 2019 19:16
> > > > > An: Michael Tremer <michael.tremer(a)ipfire.org>; Wolfgang
> > > > > Apolinarski
> > > > > <wolfgang.apolinarski(a)ipfire.org>
> > > > > Cc: development(a)lists.ipfire.org
> > > > > Betreff: Re: [PATCH 0/1] Parallelized build for several
> > > > > packages
> > > > >
> > > > > Hi,
> > > > >
> > > > > third try, now building current 'next', identical error:
> > > > >
> > > > > 'slang 2.3.0' doesn't like "cd $(DIR_APP) && make
> > > > > $(MAKETUNING)"
> > > > > here.
> > > > >
> > > > > After reverting
> > > > >
> >
> >
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=lfs/slang;h=217e74c77317d4
> > > > > c829913f934458779fd278bf29;hb=23164efba5f57b3d8ccb07a166b613f
> > > > > 2f95
> > > > > 1e1b6,
> > > > > build continues...
> > > > >
> > > > > Best,
> > > > > Matthias
> > > > >
> > > > > On 08.03.2019 16:58, Matthias Fischer wrote:
> > > > > > Hi,
> > > > > >
> > > > > > On 08.03.2019 11:17, Michael Tremer wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > Are you guys still there?
> > > > > >
> > > > > > Yep. ;-)
> > > > > >
> > > > > > Your branch 'faster-build' refuses to build here - I get
> > > > > > 'slang' errors:
> > > > > >
> > > > > > ***SNIP***
> > > > > > cd /usr/src/slang-2.3.0 && make -j9
> > > > > > make[1]: Entering directory '/usr/src/slang-2.3.0'
> > > > > > cd src; make elf
> > > > > > make[2]: Entering directory '/usr/src/slang-2.3.0/src'
> > > > > > /usr/src/slang-2.3.0/autoconf/mkinsdir.sh
> > > > > > /usr/src/slang-2.3.0/src/elfobjs cp sysconf.h config.h cp
> > > > > > terminfo/default.inc terminfo.inc cd /usr/src/slang-
> > > > > > 2.3.0/src/elfobjs
> > > > > > && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586
> > > > > > -mindirect-branch=thunk -mfunction-return=thunk
> > > > > > -mtune=generic
> > > > > > -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-
> > > > > > D_GLIBCXX_ASSERTIONS
> > > > > > -fstack-protector-strong -fPIC -Dunix -DSLANG
> > > > > > /usr/src/slang-2.3.0/src/sldisply.c
> > > > > > cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe
> > > > > > -Wall
> > > > > > -fexceptions -fPIC -march=i586 -mindirect-branch=thunk
> > > > > > -mfunction-return=thunk -mtune=generic -fomit-frame-
> > > > > > pointer
> > > > > > -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
> > > > > > -fstack-protector-strong -fPIC -Dunix -DSLANG
> > > > > > /usr/src/slang-2.3.0/src/slutty.c cd /usr/src/slang-
> > > > > > 2.3.0/src/elfobjs
> > > > > > && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586
> > > > > > -mindirect-branch=thunk -mfunction-return=thunk
> > > > > > -mtune=generic
> > > > > > -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-
> > > > > > D_GLIBCXX_ASSERTIONS
> > > > > > -fstack-protector-strong -fPIC -Dunix -DSLANG
> > > > > > /usr/src/slang-2.3.0/src/slang.c
> > > > > > /bin/sh: line 0: cd: /usr/src/slang-2.3.0/src/elfobjs: No
> > > > > > such
> > > > > > file or
> > > > > > directory mkdir -p -- /usr/src/slang-2.3.0/src/elfobjs
> > > > > > make[2]: *** [Makefile:472:
> > > > > > /usr/src/slang-2.3.0/src/elfobjs/sldisply.o] Error 1
> > > > > > make[2]: *** Waiting for unfinished jobs....
> > > > > > /usr/src/slang-2.3.0/src/slang.c: In function
> > > > > > 'inner_interp':
> > > > > > /usr/src/slang-2.3.0/src/slang.c:5733:9: warning: 'test'
> > > > > > may be
> > > > > > used
> > > > >
> > > > > uninitialized in this function [-Wmaybe-uninitialized]
> > > > > > if ((0 == pop_int (&test))
> > > > > > ~~~~~~~~~~~~~~~~~~~~~~
> > > > > > && (test == 0))
> > > > > > ^~~~~~~~~~~~~~
> > > > > > /usr/src/slang-2.3.0/src/slang.c: In function
> > > > > > 'lang_do_loops':
> > > > > > /usr/src/slang-2.3.0/src/slang.c:3725:10: warning:
> > > > > > 'blks[0]'
> > > > > > may be used
> > > > >
> > > > > uninitialized in this function [-Wmaybe-uninitialized]
> > > > > > block = blks[0];
> > > > > > ~~~~~~^~~~~~~~~
> > > > > > /usr/src/slang-2.3.0/src/slang.c:3924:7: warning: 'first'
> > > > > > may
> > > > > > be used
> > > > >
> > > > > uninitialized in this function [-Wmaybe-uninitialized]
> > > > > > i += ctrl;
> > > > > > ~~^~~~~~~
> > > > > > /usr/src/slang-2.3.0/src/slang.c:3900:13: warning: 'last'
> > > > > > may
> > > > > > be used
> > > > >
> > > > > uninitialized in this function [-Wmaybe-uninitialized]
> > > > > > if (i > last) break;
> > > > > > ^
> > > > > > make[2]: Leaving directory '/usr/src/slang-2.3.0/src'
> > > > > > make[1]: *** [Makefile:55: elf] Error 2
> > > > > > make[1]: Leaving directory '/usr/src/slang-2.3.0'
> > > > > > make: *** [slang:74: /usr/src/log/slang-2.3.0] Error 2
> > > > > > ***SNAP***
> > > > > >
> > > > > > Don't know why. I tried two builds, same error.
> > > > > >
> > > > > > Best,
> > > > > > Matthias
> > > > > >
> > > > > > > I think that we are saving about 30 min to an hour on the
> > > > > > > nightly builds
> > > > >
> > > > > now… Did anybody else run some benchmarks?
> > > > > > >
> > > > > > > -Michael
> > > > > > >
> > > > > > > > On 6 Mar 2019, at 16:36, Michael Tremer <
> > > > > > > > michael.tremer(a)ipfire.org>
> > > > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > Sorry for not being able to get around to this earlier,
> > > > > > > > but
> > > > > > > > I have
> > > > >
> > > > > invested some time on this:
> > > > > > > >
> > > > > > > > First of all, I merged Wolfgang’s patch. He is right,
> > > > > > > > the
> > > > > > > > build is too
> > > > >
> > > > > slow. It sometimes looks a bit like it is not taking enough
> > > > > advantage of
> > > > > fast systems. On the other hand with have small systems,
> > > > > which
> > > > > are (in their
> > > > > own right) quite fast. Those are the ARM systems. They
> > > > > usually
> > > > > have 1G of
> > > > > RAM. 2GB when you are lucky.
> > > > > > > >
> > > > > > > > So I adjusted make.sh a little bit and removed
> > > > > > > > composing
> > > > > > > > the MAKETUNING
> > > > >
> > > > > variable from it.
> > > > > > > >
> > > > > > > > Now, make.sh only figures out the value that usually
> > > > > > > > comes
> > > > > > > > after the -j.
> > > > >
> > > > > It is set in DEFAULT_PARALLELISM.
> > > > > > > >
> > > > > > > > In the individual LFS files, we can now set a new
> > > > > > > > variable
> > > > > > > > called
> > > > >
> > > > > MAX_PARALLELISM which is optional. In boost, I now calculate
> > > > > this
> > > > > from the
> > > > > amount of memory that is available. That will make sure that
> > > > > we
> > > > > have a cap
> > > > > on this that will be high on systems that have the memory and
> > > > > low
> > > > > on those
> > > > > that don’t. Before, we hardcoded -j2 which of course runs
> > > > > very
> > > > > slow on large
> > > > > systems.
> > > > > > > >
> > > > > > > > The code is here:
> > > > > > > >
> > > > > > > >
> > > > > > > >
> >
> >
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs
> > > > > > > > /heads/faster-build
> > > > > > > >
> > > > > > > > Please review. I am now running this on a couple of
> > > > > > > > builders and will
> > > > >
> > > > > see what comes out. Please do the same and send me feedback.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > -Michael
> > > > > > > >
> > > > > > > > > On 21 Feb 2019, at 17:28, Wolfgang Apolinarski
> > > > >
> > > > > <wolfgang.apolinarski(a)ipfire.org> wrote:
> > > > > > > > >
> > > > > > > > > Hi Arne,
> > > > > > > > >
> > > > > > > > > of course I did delete the ccache between my multiple
> > > > > > > > > builds to verify
> > > > >
> > > > > the commit. To play it safe, I hard reset using git, checked
> > > > > that
> > > > > all
> > > > > folders (build, ccache, etc.) are gone and restarted the
> > > > > build
> > > > > after
> > > > > gettoolchain and downloadsrc.
> > > > > > > > >
> > > > > > > > > Sorry, I did not know that there are still a lot of
> > > > > > > > > machines with low
> > > > >
> > > > > ram that we are using to build ipfire. My hope was that these
> > > > > machines have
> > > > > disappeared over the last years.
> > > > > > > > > Additionally, I do not have the ability to test all
> > > > > > > > > ipfire arches, as
> > > > >
> > > > > mentioned, the patch needs testing by others.
> > > > > > > > >
> > > > > > > > > I also thought of adding a manual parameter for
> > > > > > > > > developers that somehow
> > > > >
> > > > > manually sets the parameters for parallel build, but since
> > > > > the
> > > > > MAKETUNING
> > > > > parameter already exists and is used with other packages this
> > > > > would just
> > > > > somehow mess everything up.
> > > > > > > > >
> > > > > > > > > For my builds, I can just manually apply the patch to
> > > > > > > > > speed up the full
> > > > >
> > > > > build. My thought was that sharing the patch would make sense
> > > > > for
> > > > > others.
> > > > > > > > >
> > > > > > > > > > -----Ursprüngliche Nachricht-----
> > > > > > > > > > Von: Arne Fitzenreiter <
> > > > > > > > > > Arne.Fitzenreiter(a)ipfire.org>
> > > > > > > > > > Gesendet: Donnerstag, 21. Februar 2019 09:44
> > > > > > > > > > An: Wolfgang Apolinarski <
> > > > > > > > > > wolfgang.apolinarski(a)ipfire.org>
> > > > > > > > > > Cc: development(a)lists.ipfire.org
> > > > > > > > > > Betreff: Re: [PATCH 0/1] Parallelized build for
> > > > > > > > > > several
> > > > > > > > > > packages
> > > > > > > > > >
> > > > > > > > > > On the most of this packages (eg. boot and cmake) i
> > > > > > > > > > have disabled
> > > > > > > > > > parallel build because it fails on machines with
> > > > > > > > > > low
> > > > > > > > > > ram (less
> > > > > > > > > > than 2GB) or on arm.
> > > > > > > > > > So such changes need tested on an 1GB machine, on
> > > > > > > > > > all
> > > > > > > > > > architectures and build two times after deleting
> > > > > > > > > > ccache. Because
> > > > > > > > > > the build need much lower resources if it has the
> > > > > > > > > > file
> > > > > > > > > > already
> > > > >
> > > > > prebuilt in cache.
> > >
> > >
>
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 0/1] Parallelized build for several packages
2019-03-10 19:17 ` ummeegge
@ 2019-03-11 9:37 ` Michael Tremer
2019-03-11 17:07 ` ummeegge
0 siblings, 1 reply; 14+ messages in thread
From: Michael Tremer @ 2019-03-11 9:37 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 10750 bytes --]
Yes, probably.
It uses a lot of memory when compiling because it is C++.
I will adjust the number of concurrent processes just like we did for boost.
Best,
-Michael
> On 10 Mar 2019, at 19:17, ummeegge <ummeegge(a)ipfire.org> wrote:
>
> Hi all,
>
> On So, 2019-03-10 at 18:24 +0000, Michael Tremer wrote:
>> Hi,
>>
>> I merged the patch. And I am surprised it is only the one…
> dnsdist do have problems to build here in next currently:
>
> checking for boost/foreach.hpp... yes
> checking for the toolset name used by Boost for g++... configure: WARNING: could not figure out which toolset name to use for g++
> ....
>
> g++: internal compiler error: Killed (program cc1plus)
> Please submit a full bug report,
> with preprocessed source if appropriate.
> See <http://bugtracker.ipfire.org> for instructions.
> make[3]: *** [Makefile:743: dnsdist-lua2.o] Error 4
> make[3]: *** Waiting for unfinished jobs....
>
>
> May this is possibly related to this topic ?
>
> Best,
>
> Erik
>
>>
>> -Michael
>>
>>> On 9 Mar 2019, at 20:36, Matthias Fischer <
>>> matthias.fischer(a)ipfire.org> wrote:
>>>
>>> On 09.03.2019 21:24, Wolfgang Apolinarski wrote:
>>>> Hi,
>>>>
>>>> sorry, that I did not respond in time. I also had issues with
>>>> "slang" during
>>>> one of my latest test builds, especially when using something
>>>> like "-j3" (it
>>>> always worked fine when using higher values).
>>>> The lfs manual also stated that the build of "slang" cannot be
>>>> parallelized.
>>>> Sorry, I apparently overlooked that (I have now checked all other
>>>> packages
>>>> and did not find incompabilities).
>>>
>>> No problem, current 'next' - with commented $(MAKETUNING) for
>>> 'slang' - built without problems.
>>>
>>> For the records:
>>> Last clean "Build finished in 7:02:45". Hardware: i7/2600, 8 GB
>>> RAM, 500GB SATA HDD, Ubuntu 18.04.2 LTS / 32bit (yes, I know... ;-)
>>> ).
>>>
>>> Best,
>>> Matthias
>>>
>>>> Today, I started again some more builds (with parallel build
>>>> disabled for
>>>> slang), the test continues tomorrow.
>>>>
>>>> Best regards,
>>>> Wolfgang
>>>> PS.: The builds always use the branch that Michael created.
>>>>
>>>> -----Ursprüngliche Nachricht-----
>>>> Von: Matthias Fischer <matthias.fischer(a)ipfire.org>
>>>> Gesendet: Freitag, 8. März 2019 19:16
>>>> An: Michael Tremer <michael.tremer(a)ipfire.org>; Wolfgang
>>>> Apolinarski
>>>> <wolfgang.apolinarski(a)ipfire.org>
>>>> Cc: development(a)lists.ipfire.org
>>>> Betreff: Re: [PATCH 0/1] Parallelized build for several packages
>>>>
>>>> Hi,
>>>>
>>>> third try, now building current 'next', identical error:
>>>>
>>>> 'slang 2.3.0' doesn't like "cd $(DIR_APP) && make $(MAKETUNING)"
>>>> here.
>>>>
>>>> After reverting
>>>>
> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=lfs/slang;h=217e74c77317d4
>>>> c829913f934458779fd278bf29;hb=23164efba5f57b3d8ccb07a166b613f2f95
>>>> 1e1b6,
>>>> build continues...
>>>>
>>>> Best,
>>>> Matthias
>>>>
>>>> On 08.03.2019 16:58, Matthias Fischer wrote:
>>>>> Hi,
>>>>>
>>>>> On 08.03.2019 11:17, Michael Tremer wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Are you guys still there?
>>>>>
>>>>> Yep. ;-)
>>>>>
>>>>> Your branch 'faster-build' refuses to build here - I get
>>>>> 'slang' errors:
>>>>>
>>>>> ***SNIP***
>>>>> cd /usr/src/slang-2.3.0 && make -j9
>>>>> make[1]: Entering directory '/usr/src/slang-2.3.0'
>>>>> cd src; make elf
>>>>> make[2]: Entering directory '/usr/src/slang-2.3.0/src'
>>>>> /usr/src/slang-2.3.0/autoconf/mkinsdir.sh
>>>>> /usr/src/slang-2.3.0/src/elfobjs cp sysconf.h config.h cp
>>>>> terminfo/default.inc terminfo.inc cd /usr/src/slang-
>>>>> 2.3.0/src/elfobjs
>>>>> && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586
>>>>> -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic
>>>>> -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-
>>>>> D_GLIBCXX_ASSERTIONS
>>>>> -fstack-protector-strong -fPIC -Dunix -DSLANG
>>>>> /usr/src/slang-2.3.0/src/sldisply.c
>>>>> cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall
>>>>> -fexceptions -fPIC -march=i586 -mindirect-branch=thunk
>>>>> -mfunction-return=thunk -mtune=generic -fomit-frame-pointer
>>>>> -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
>>>>> -fstack-protector-strong -fPIC -Dunix -DSLANG
>>>>> /usr/src/slang-2.3.0/src/slutty.c cd /usr/src/slang-
>>>>> 2.3.0/src/elfobjs
>>>>> && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586
>>>>> -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic
>>>>> -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-
>>>>> D_GLIBCXX_ASSERTIONS
>>>>> -fstack-protector-strong -fPIC -Dunix -DSLANG
>>>>> /usr/src/slang-2.3.0/src/slang.c
>>>>> /bin/sh: line 0: cd: /usr/src/slang-2.3.0/src/elfobjs: No such
>>>>> file or
>>>>> directory mkdir -p -- /usr/src/slang-2.3.0/src/elfobjs
>>>>> make[2]: *** [Makefile:472:
>>>>> /usr/src/slang-2.3.0/src/elfobjs/sldisply.o] Error 1
>>>>> make[2]: *** Waiting for unfinished jobs....
>>>>> /usr/src/slang-2.3.0/src/slang.c: In function 'inner_interp':
>>>>> /usr/src/slang-2.3.0/src/slang.c:5733:9: warning: 'test' may be
>>>>> used
>>>>
>>>> uninitialized in this function [-Wmaybe-uninitialized]
>>>>> if ((0 == pop_int (&test))
>>>>> ~~~~~~~~~~~~~~~~~~~~~~
>>>>> && (test == 0))
>>>>> ^~~~~~~~~~~~~~
>>>>> /usr/src/slang-2.3.0/src/slang.c: In function 'lang_do_loops':
>>>>> /usr/src/slang-2.3.0/src/slang.c:3725:10: warning: 'blks[0]'
>>>>> may be used
>>>>
>>>> uninitialized in this function [-Wmaybe-uninitialized]
>>>>> block = blks[0];
>>>>> ~~~~~~^~~~~~~~~
>>>>> /usr/src/slang-2.3.0/src/slang.c:3924:7: warning: 'first' may
>>>>> be used
>>>>
>>>> uninitialized in this function [-Wmaybe-uninitialized]
>>>>> i += ctrl;
>>>>> ~~^~~~~~~
>>>>> /usr/src/slang-2.3.0/src/slang.c:3900:13: warning: 'last' may
>>>>> be used
>>>>
>>>> uninitialized in this function [-Wmaybe-uninitialized]
>>>>> if (i > last) break;
>>>>> ^
>>>>> make[2]: Leaving directory '/usr/src/slang-2.3.0/src'
>>>>> make[1]: *** [Makefile:55: elf] Error 2
>>>>> make[1]: Leaving directory '/usr/src/slang-2.3.0'
>>>>> make: *** [slang:74: /usr/src/log/slang-2.3.0] Error 2
>>>>> ***SNAP***
>>>>>
>>>>> Don't know why. I tried two builds, same error.
>>>>>
>>>>> Best,
>>>>> Matthias
>>>>>
>>>>>> I think that we are saving about 30 min to an hour on the
>>>>>> nightly builds
>>>>
>>>> now… Did anybody else run some benchmarks?
>>>>>>
>>>>>> -Michael
>>>>>>
>>>>>>> On 6 Mar 2019, at 16:36, Michael Tremer <
>>>>>>> michael.tremer(a)ipfire.org>
>>>>
>>>> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Sorry for not being able to get around to this earlier, but
>>>>>>> I have
>>>>
>>>> invested some time on this:
>>>>>>>
>>>>>>> First of all, I merged Wolfgang’s patch. He is right, the
>>>>>>> build is too
>>>>
>>>> slow. It sometimes looks a bit like it is not taking enough
>>>> advantage of
>>>> fast systems. On the other hand with have small systems, which
>>>> are (in their
>>>> own right) quite fast. Those are the ARM systems. They usually
>>>> have 1G of
>>>> RAM. 2GB when you are lucky.
>>>>>>>
>>>>>>> So I adjusted make.sh a little bit and removed composing
>>>>>>> the MAKETUNING
>>>>
>>>> variable from it.
>>>>>>>
>>>>>>> Now, make.sh only figures out the value that usually comes
>>>>>>> after the -j.
>>>>
>>>> It is set in DEFAULT_PARALLELISM.
>>>>>>>
>>>>>>> In the individual LFS files, we can now set a new variable
>>>>>>> called
>>>>
>>>> MAX_PARALLELISM which is optional. In boost, I now calculate this
>>>> from the
>>>> amount of memory that is available. That will make sure that we
>>>> have a cap
>>>> on this that will be high on systems that have the memory and low
>>>> on those
>>>> that don’t. Before, we hardcoded -j2 which of course runs very
>>>> slow on large
>>>> systems.
>>>>>>>
>>>>>>> The code is here:
>>>>>>>
>>>>>>>
>>>>>>>
> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs
>>>>>>> /heads/faster-build
>>>>>>>
>>>>>>> Please review. I am now running this on a couple of
>>>>>>> builders and will
>>>>
>>>> see what comes out. Please do the same and send me feedback.
>>>>>>>
>>>>>>> Best,
>>>>>>> -Michael
>>>>>>>
>>>>>>>> On 21 Feb 2019, at 17:28, Wolfgang Apolinarski
>>>>
>>>> <wolfgang.apolinarski(a)ipfire.org> wrote:
>>>>>>>>
>>>>>>>> Hi Arne,
>>>>>>>>
>>>>>>>> of course I did delete the ccache between my multiple
>>>>>>>> builds to verify
>>>>
>>>> the commit. To play it safe, I hard reset using git, checked that
>>>> all
>>>> folders (build, ccache, etc.) are gone and restarted the build
>>>> after
>>>> gettoolchain and downloadsrc.
>>>>>>>>
>>>>>>>> Sorry, I did not know that there are still a lot of
>>>>>>>> machines with low
>>>>
>>>> ram that we are using to build ipfire. My hope was that these
>>>> machines have
>>>> disappeared over the last years.
>>>>>>>> Additionally, I do not have the ability to test all
>>>>>>>> ipfire arches, as
>>>>
>>>> mentioned, the patch needs testing by others.
>>>>>>>>
>>>>>>>> I also thought of adding a manual parameter for
>>>>>>>> developers that somehow
>>>>
>>>> manually sets the parameters for parallel build, but since the
>>>> MAKETUNING
>>>> parameter already exists and is used with other packages this
>>>> would just
>>>> somehow mess everything up.
>>>>>>>>
>>>>>>>> For my builds, I can just manually apply the patch to
>>>>>>>> speed up the full
>>>>
>>>> build. My thought was that sharing the patch would make sense for
>>>> others.
>>>>>>>>
>>>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>>>> Von: Arne Fitzenreiter <Arne.Fitzenreiter(a)ipfire.org>
>>>>>>>>> Gesendet: Donnerstag, 21. Februar 2019 09:44
>>>>>>>>> An: Wolfgang Apolinarski <
>>>>>>>>> wolfgang.apolinarski(a)ipfire.org>
>>>>>>>>> Cc: development(a)lists.ipfire.org
>>>>>>>>> Betreff: Re: [PATCH 0/1] Parallelized build for several
>>>>>>>>> packages
>>>>>>>>>
>>>>>>>>> On the most of this packages (eg. boot and cmake) i
>>>>>>>>> have disabled
>>>>>>>>> parallel build because it fails on machines with low
>>>>>>>>> ram (less
>>>>>>>>> than 2GB) or on arm.
>>>>>>>>> So such changes need tested on an 1GB machine, on all
>>>>>>>>> architectures and build two times after deleting
>>>>>>>>> ccache. Because
>>>>>>>>> the build need much lower resources if it has the file
>>>>>>>>> already
>>>>
>>>> prebuilt in cache.
>>
>>
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 0/1] Parallelized build for several packages
2019-03-10 18:24 ` Michael Tremer
@ 2019-03-10 19:17 ` ummeegge
2019-03-11 9:37 ` Michael Tremer
0 siblings, 1 reply; 14+ messages in thread
From: ummeegge @ 2019-03-10 19:17 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 11060 bytes --]
Hi all,
On So, 2019-03-10 at 18:24 +0000, Michael Tremer wrote:
> Hi,
>
> I merged the patch. And I am surprised it is only the one…
dnsdist do have problems to build here in next currently:
checking for boost/foreach.hpp... yes
checking for the toolset name used by Boost for g++... configure: WARNING: could not figure out which toolset name to use for g++
....
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://bugtracker.ipfire.org> for instructions.
make[3]: *** [Makefile:743: dnsdist-lua2.o] Error 4
make[3]: *** Waiting for unfinished jobs....
May this is possibly related to this topic ?
Best,
Erik
>
> -Michael
>
> > On 9 Mar 2019, at 20:36, Matthias Fischer <
> > matthias.fischer(a)ipfire.org> wrote:
> >
> > On 09.03.2019 21:24, Wolfgang Apolinarski wrote:
> > > Hi,
> > >
> > > sorry, that I did not respond in time. I also had issues with
> > > "slang" during
> > > one of my latest test builds, especially when using something
> > > like "-j3" (it
> > > always worked fine when using higher values).
> > > The lfs manual also stated that the build of "slang" cannot be
> > > parallelized.
> > > Sorry, I apparently overlooked that (I have now checked all other
> > > packages
> > > and did not find incompabilities).
> >
> > No problem, current 'next' - with commented $(MAKETUNING) for
> > 'slang' - built without problems.
> >
> > For the records:
> > Last clean "Build finished in 7:02:45". Hardware: i7/2600, 8 GB
> > RAM, 500GB SATA HDD, Ubuntu 18.04.2 LTS / 32bit (yes, I know... ;-)
> > ).
> >
> > Best,
> > Matthias
> >
> > > Today, I started again some more builds (with parallel build
> > > disabled for
> > > slang), the test continues tomorrow.
> > >
> > > Best regards,
> > > Wolfgang
> > > PS.: The builds always use the branch that Michael created.
> > >
> > > -----Ursprüngliche Nachricht-----
> > > Von: Matthias Fischer <matthias.fischer(a)ipfire.org>
> > > Gesendet: Freitag, 8. März 2019 19:16
> > > An: Michael Tremer <michael.tremer(a)ipfire.org>; Wolfgang
> > > Apolinarski
> > > <wolfgang.apolinarski(a)ipfire.org>
> > > Cc: development(a)lists.ipfire.org
> > > Betreff: Re: [PATCH 0/1] Parallelized build for several packages
> > >
> > > Hi,
> > >
> > > third try, now building current 'next', identical error:
> > >
> > > 'slang 2.3.0' doesn't like "cd $(DIR_APP) && make $(MAKETUNING)"
> > > here.
> > >
> > > After reverting
> > >
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=lfs/slang;h=217e74c77317d4
> > > c829913f934458779fd278bf29;hb=23164efba5f57b3d8ccb07a166b613f2f95
> > > 1e1b6,
> > > build continues...
> > >
> > > Best,
> > > Matthias
> > >
> > > On 08.03.2019 16:58, Matthias Fischer wrote:
> > > > Hi,
> > > >
> > > > On 08.03.2019 11:17, Michael Tremer wrote:
> > > > > Hi,
> > > > >
> > > > > Are you guys still there?
> > > >
> > > > Yep. ;-)
> > > >
> > > > Your branch 'faster-build' refuses to build here - I get
> > > > 'slang' errors:
> > > >
> > > > ***SNIP***
> > > > cd /usr/src/slang-2.3.0 && make -j9
> > > > make[1]: Entering directory '/usr/src/slang-2.3.0'
> > > > cd src; make elf
> > > > make[2]: Entering directory '/usr/src/slang-2.3.0/src'
> > > > /usr/src/slang-2.3.0/autoconf/mkinsdir.sh
> > > > /usr/src/slang-2.3.0/src/elfobjs cp sysconf.h config.h cp
> > > > terminfo/default.inc terminfo.inc cd /usr/src/slang-
> > > > 2.3.0/src/elfobjs
> > > > && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586
> > > > -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic
> > > > -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-
> > > > D_GLIBCXX_ASSERTIONS
> > > > -fstack-protector-strong -fPIC -Dunix -DSLANG
> > > > /usr/src/slang-2.3.0/src/sldisply.c
> > > > cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall
> > > > -fexceptions -fPIC -march=i586 -mindirect-branch=thunk
> > > > -mfunction-return=thunk -mtune=generic -fomit-frame-pointer
> > > > -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
> > > > -fstack-protector-strong -fPIC -Dunix -DSLANG
> > > > /usr/src/slang-2.3.0/src/slutty.c cd /usr/src/slang-
> > > > 2.3.0/src/elfobjs
> > > > && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586
> > > > -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic
> > > > -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-
> > > > D_GLIBCXX_ASSERTIONS
> > > > -fstack-protector-strong -fPIC -Dunix -DSLANG
> > > > /usr/src/slang-2.3.0/src/slang.c
> > > > /bin/sh: line 0: cd: /usr/src/slang-2.3.0/src/elfobjs: No such
> > > > file or
> > > > directory mkdir -p -- /usr/src/slang-2.3.0/src/elfobjs
> > > > make[2]: *** [Makefile:472:
> > > > /usr/src/slang-2.3.0/src/elfobjs/sldisply.o] Error 1
> > > > make[2]: *** Waiting for unfinished jobs....
> > > > /usr/src/slang-2.3.0/src/slang.c: In function 'inner_interp':
> > > > /usr/src/slang-2.3.0/src/slang.c:5733:9: warning: 'test' may be
> > > > used
> > >
> > > uninitialized in this function [-Wmaybe-uninitialized]
> > > > if ((0 == pop_int (&test))
> > > > ~~~~~~~~~~~~~~~~~~~~~~
> > > > && (test == 0))
> > > > ^~~~~~~~~~~~~~
> > > > /usr/src/slang-2.3.0/src/slang.c: In function 'lang_do_loops':
> > > > /usr/src/slang-2.3.0/src/slang.c:3725:10: warning: 'blks[0]'
> > > > may be used
> > >
> > > uninitialized in this function [-Wmaybe-uninitialized]
> > > > block = blks[0];
> > > > ~~~~~~^~~~~~~~~
> > > > /usr/src/slang-2.3.0/src/slang.c:3924:7: warning: 'first' may
> > > > be used
> > >
> > > uninitialized in this function [-Wmaybe-uninitialized]
> > > > i += ctrl;
> > > > ~~^~~~~~~
> > > > /usr/src/slang-2.3.0/src/slang.c:3900:13: warning: 'last' may
> > > > be used
> > >
> > > uninitialized in this function [-Wmaybe-uninitialized]
> > > > if (i > last) break;
> > > > ^
> > > > make[2]: Leaving directory '/usr/src/slang-2.3.0/src'
> > > > make[1]: *** [Makefile:55: elf] Error 2
> > > > make[1]: Leaving directory '/usr/src/slang-2.3.0'
> > > > make: *** [slang:74: /usr/src/log/slang-2.3.0] Error 2
> > > > ***SNAP***
> > > >
> > > > Don't know why. I tried two builds, same error.
> > > >
> > > > Best,
> > > > Matthias
> > > >
> > > > > I think that we are saving about 30 min to an hour on the
> > > > > nightly builds
> > >
> > > now… Did anybody else run some benchmarks?
> > > > >
> > > > > -Michael
> > > > >
> > > > > > On 6 Mar 2019, at 16:36, Michael Tremer <
> > > > > > michael.tremer(a)ipfire.org>
> > >
> > > wrote:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Sorry for not being able to get around to this earlier, but
> > > > > > I have
> > >
> > > invested some time on this:
> > > > > >
> > > > > > First of all, I merged Wolfgang’s patch. He is right, the
> > > > > > build is too
> > >
> > > slow. It sometimes looks a bit like it is not taking enough
> > > advantage of
> > > fast systems. On the other hand with have small systems, which
> > > are (in their
> > > own right) quite fast. Those are the ARM systems. They usually
> > > have 1G of
> > > RAM. 2GB when you are lucky.
> > > > > >
> > > > > > So I adjusted make.sh a little bit and removed composing
> > > > > > the MAKETUNING
> > >
> > > variable from it.
> > > > > >
> > > > > > Now, make.sh only figures out the value that usually comes
> > > > > > after the -j.
> > >
> > > It is set in DEFAULT_PARALLELISM.
> > > > > >
> > > > > > In the individual LFS files, we can now set a new variable
> > > > > > called
> > >
> > > MAX_PARALLELISM which is optional. In boost, I now calculate this
> > > from the
> > > amount of memory that is available. That will make sure that we
> > > have a cap
> > > on this that will be high on systems that have the memory and low
> > > on those
> > > that don’t. Before, we hardcoded -j2 which of course runs very
> > > slow on large
> > > systems.
> > > > > >
> > > > > > The code is here:
> > > > > >
> > > > > >
> > > > > >
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs
> > > > > > /heads/faster-build
> > > > > >
> > > > > > Please review. I am now running this on a couple of
> > > > > > builders and will
> > >
> > > see what comes out. Please do the same and send me feedback.
> > > > > >
> > > > > > Best,
> > > > > > -Michael
> > > > > >
> > > > > > > On 21 Feb 2019, at 17:28, Wolfgang Apolinarski
> > >
> > > <wolfgang.apolinarski(a)ipfire.org> wrote:
> > > > > > >
> > > > > > > Hi Arne,
> > > > > > >
> > > > > > > of course I did delete the ccache between my multiple
> > > > > > > builds to verify
> > >
> > > the commit. To play it safe, I hard reset using git, checked that
> > > all
> > > folders (build, ccache, etc.) are gone and restarted the build
> > > after
> > > gettoolchain and downloadsrc.
> > > > > > >
> > > > > > > Sorry, I did not know that there are still a lot of
> > > > > > > machines with low
> > >
> > > ram that we are using to build ipfire. My hope was that these
> > > machines have
> > > disappeared over the last years.
> > > > > > > Additionally, I do not have the ability to test all
> > > > > > > ipfire arches, as
> > >
> > > mentioned, the patch needs testing by others.
> > > > > > >
> > > > > > > I also thought of adding a manual parameter for
> > > > > > > developers that somehow
> > >
> > > manually sets the parameters for parallel build, but since the
> > > MAKETUNING
> > > parameter already exists and is used with other packages this
> > > would just
> > > somehow mess everything up.
> > > > > > >
> > > > > > > For my builds, I can just manually apply the patch to
> > > > > > > speed up the full
> > >
> > > build. My thought was that sharing the patch would make sense for
> > > others.
> > > > > > >
> > > > > > > > -----Ursprüngliche Nachricht-----
> > > > > > > > Von: Arne Fitzenreiter <Arne.Fitzenreiter(a)ipfire.org>
> > > > > > > > Gesendet: Donnerstag, 21. Februar 2019 09:44
> > > > > > > > An: Wolfgang Apolinarski <
> > > > > > > > wolfgang.apolinarski(a)ipfire.org>
> > > > > > > > Cc: development(a)lists.ipfire.org
> > > > > > > > Betreff: Re: [PATCH 0/1] Parallelized build for several
> > > > > > > > packages
> > > > > > > >
> > > > > > > > On the most of this packages (eg. boot and cmake) i
> > > > > > > > have disabled
> > > > > > > > parallel build because it fails on machines with low
> > > > > > > > ram (less
> > > > > > > > than 2GB) or on arm.
> > > > > > > > So such changes need tested on an 1GB machine, on all
> > > > > > > > architectures and build two times after deleting
> > > > > > > > ccache. Because
> > > > > > > > the build need much lower resources if it has the file
> > > > > > > > already
> > >
> > > prebuilt in cache.
>
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 0/1] Parallelized build for several packages
2019-03-09 20:36 ` Matthias Fischer
@ 2019-03-10 18:24 ` Michael Tremer
2019-03-10 19:17 ` ummeegge
0 siblings, 1 reply; 14+ messages in thread
From: Michael Tremer @ 2019-03-10 18:24 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 8815 bytes --]
Hi,
I merged the patch. And I am surprised it is only the one…
-Michael
> On 9 Mar 2019, at 20:36, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
>
> On 09.03.2019 21:24, Wolfgang Apolinarski wrote:
>> Hi,
>>
>> sorry, that I did not respond in time. I also had issues with "slang" during
>> one of my latest test builds, especially when using something like "-j3" (it
>> always worked fine when using higher values).
>> The lfs manual also stated that the build of "slang" cannot be parallelized.
>> Sorry, I apparently overlooked that (I have now checked all other packages
>> and did not find incompabilities).
>
> No problem, current 'next' - with commented $(MAKETUNING) for 'slang' - built without problems.
>
> For the records:
> Last clean "Build finished in 7:02:45". Hardware: i7/2600, 8 GB RAM, 500GB SATA HDD, Ubuntu 18.04.2 LTS / 32bit (yes, I know... ;-) ).
>
> Best,
> Matthias
>
>> Today, I started again some more builds (with parallel build disabled for
>> slang), the test continues tomorrow.
>>
>> Best regards,
>> Wolfgang
>> PS.: The builds always use the branch that Michael created.
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Matthias Fischer <matthias.fischer(a)ipfire.org>
>> Gesendet: Freitag, 8. März 2019 19:16
>> An: Michael Tremer <michael.tremer(a)ipfire.org>; Wolfgang Apolinarski
>> <wolfgang.apolinarski(a)ipfire.org>
>> Cc: development(a)lists.ipfire.org
>> Betreff: Re: [PATCH 0/1] Parallelized build for several packages
>>
>> Hi,
>>
>> third try, now building current 'next', identical error:
>>
>> 'slang 2.3.0' doesn't like "cd $(DIR_APP) && make $(MAKETUNING)" here.
>>
>> After reverting
>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=lfs/slang;h=217e74c77317d4
>> c829913f934458779fd278bf29;hb=23164efba5f57b3d8ccb07a166b613f2f951e1b6,
>> build continues...
>>
>> Best,
>> Matthias
>>
>> On 08.03.2019 16:58, Matthias Fischer wrote:
>>> Hi,
>>>
>>> On 08.03.2019 11:17, Michael Tremer wrote:
>>>> Hi,
>>>>
>>>> Are you guys still there?
>>>
>>> Yep. ;-)
>>>
>>> Your branch 'faster-build' refuses to build here - I get 'slang' errors:
>>>
>>> ***SNIP***
>>> cd /usr/src/slang-2.3.0 && make -j9
>>> make[1]: Entering directory '/usr/src/slang-2.3.0'
>>> cd src; make elf
>>> make[2]: Entering directory '/usr/src/slang-2.3.0/src'
>>> /usr/src/slang-2.3.0/autoconf/mkinsdir.sh
>>> /usr/src/slang-2.3.0/src/elfobjs cp sysconf.h config.h cp
>>> terminfo/default.inc terminfo.inc cd /usr/src/slang-2.3.0/src/elfobjs
>>> && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586
>>> -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic
>>> -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
>>> -fstack-protector-strong -fPIC -Dunix -DSLANG
>>> /usr/src/slang-2.3.0/src/sldisply.c
>>> cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall
>>> -fexceptions -fPIC -march=i586 -mindirect-branch=thunk
>>> -mfunction-return=thunk -mtune=generic -fomit-frame-pointer
>>> -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
>>> -fstack-protector-strong -fPIC -Dunix -DSLANG
>>> /usr/src/slang-2.3.0/src/slutty.c cd /usr/src/slang-2.3.0/src/elfobjs
>>> && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586
>>> -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic
>>> -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
>>> -fstack-protector-strong -fPIC -Dunix -DSLANG
>>> /usr/src/slang-2.3.0/src/slang.c
>>> /bin/sh: line 0: cd: /usr/src/slang-2.3.0/src/elfobjs: No such file or
>>> directory mkdir -p -- /usr/src/slang-2.3.0/src/elfobjs
>>> make[2]: *** [Makefile:472:
>>> /usr/src/slang-2.3.0/src/elfobjs/sldisply.o] Error 1
>>> make[2]: *** Waiting for unfinished jobs....
>>> /usr/src/slang-2.3.0/src/slang.c: In function 'inner_interp':
>>> /usr/src/slang-2.3.0/src/slang.c:5733:9: warning: 'test' may be used
>> uninitialized in this function [-Wmaybe-uninitialized]
>>> if ((0 == pop_int (&test))
>>> ~~~~~~~~~~~~~~~~~~~~~~
>>> && (test == 0))
>>> ^~~~~~~~~~~~~~
>>> /usr/src/slang-2.3.0/src/slang.c: In function 'lang_do_loops':
>>> /usr/src/slang-2.3.0/src/slang.c:3725:10: warning: 'blks[0]' may be used
>> uninitialized in this function [-Wmaybe-uninitialized]
>>> block = blks[0];
>>> ~~~~~~^~~~~~~~~
>>> /usr/src/slang-2.3.0/src/slang.c:3924:7: warning: 'first' may be used
>> uninitialized in this function [-Wmaybe-uninitialized]
>>> i += ctrl;
>>> ~~^~~~~~~
>>> /usr/src/slang-2.3.0/src/slang.c:3900:13: warning: 'last' may be used
>> uninitialized in this function [-Wmaybe-uninitialized]
>>> if (i > last) break;
>>> ^
>>> make[2]: Leaving directory '/usr/src/slang-2.3.0/src'
>>> make[1]: *** [Makefile:55: elf] Error 2
>>> make[1]: Leaving directory '/usr/src/slang-2.3.0'
>>> make: *** [slang:74: /usr/src/log/slang-2.3.0] Error 2
>>> ***SNAP***
>>>
>>> Don't know why. I tried two builds, same error.
>>>
>>> Best,
>>> Matthias
>>>
>>>> I think that we are saving about 30 min to an hour on the nightly builds
>> now… Did anybody else run some benchmarks?
>>>>
>>>> -Michael
>>>>
>>>>> On 6 Mar 2019, at 16:36, Michael Tremer <michael.tremer(a)ipfire.org>
>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> Sorry for not being able to get around to this earlier, but I have
>> invested some time on this:
>>>>>
>>>>> First of all, I merged Wolfgang’s patch. He is right, the build is too
>> slow. It sometimes looks a bit like it is not taking enough advantage of
>> fast systems. On the other hand with have small systems, which are (in their
>> own right) quite fast. Those are the ARM systems. They usually have 1G of
>> RAM. 2GB when you are lucky.
>>>>>
>>>>> So I adjusted make.sh a little bit and removed composing the MAKETUNING
>> variable from it.
>>>>>
>>>>> Now, make.sh only figures out the value that usually comes after the -j.
>> It is set in DEFAULT_PARALLELISM.
>>>>>
>>>>> In the individual LFS files, we can now set a new variable called
>> MAX_PARALLELISM which is optional. In boost, I now calculate this from the
>> amount of memory that is available. That will make sure that we have a cap
>> on this that will be high on systems that have the memory and low on those
>> that don’t. Before, we hardcoded -j2 which of course runs very slow on large
>> systems.
>>>>>
>>>>> The code is here:
>>>>>
>>>>>
>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs
>>>>> /heads/faster-build
>>>>>
>>>>> Please review. I am now running this on a couple of builders and will
>> see what comes out. Please do the same and send me feedback.
>>>>>
>>>>> Best,
>>>>> -Michael
>>>>>
>>>>>> On 21 Feb 2019, at 17:28, Wolfgang Apolinarski
>> <wolfgang.apolinarski(a)ipfire.org> wrote:
>>>>>>
>>>>>> Hi Arne,
>>>>>>
>>>>>> of course I did delete the ccache between my multiple builds to verify
>> the commit. To play it safe, I hard reset using git, checked that all
>> folders (build, ccache, etc.) are gone and restarted the build after
>> gettoolchain and downloadsrc.
>>>>>>
>>>>>> Sorry, I did not know that there are still a lot of machines with low
>> ram that we are using to build ipfire. My hope was that these machines have
>> disappeared over the last years.
>>>>>> Additionally, I do not have the ability to test all ipfire arches, as
>> mentioned, the patch needs testing by others.
>>>>>>
>>>>>> I also thought of adding a manual parameter for developers that somehow
>> manually sets the parameters for parallel build, but since the MAKETUNING
>> parameter already exists and is used with other packages this would just
>> somehow mess everything up.
>>>>>>
>>>>>> For my builds, I can just manually apply the patch to speed up the full
>> build. My thought was that sharing the patch would make sense for others.
>>>>>>
>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>> Von: Arne Fitzenreiter <Arne.Fitzenreiter(a)ipfire.org>
>>>>>>> Gesendet: Donnerstag, 21. Februar 2019 09:44
>>>>>>> An: Wolfgang Apolinarski <wolfgang.apolinarski(a)ipfire.org>
>>>>>>> Cc: development(a)lists.ipfire.org
>>>>>>> Betreff: Re: [PATCH 0/1] Parallelized build for several packages
>>>>>>>
>>>>>>> On the most of this packages (eg. boot and cmake) i have disabled
>>>>>>> parallel build because it fails on machines with low ram (less
>>>>>>> than 2GB) or on arm.
>>>>>>> So such changes need tested on an 1GB machine, on all
>>>>>>> architectures and build two times after deleting ccache. Because
>>>>>>> the build need much lower resources if it has the file already
>> prebuilt in cache.
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 0/1] Parallelized build for several packages
2019-03-09 20:24 Wolfgang Apolinarski
@ 2019-03-09 20:36 ` Matthias Fischer
2019-03-10 18:24 ` Michael Tremer
0 siblings, 1 reply; 14+ messages in thread
From: Matthias Fischer @ 2019-03-09 20:36 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 8440 bytes --]
On 09.03.2019 21:24, Wolfgang Apolinarski wrote:
> Hi,
>
> sorry, that I did not respond in time. I also had issues with "slang" during
> one of my latest test builds, especially when using something like "-j3" (it
> always worked fine when using higher values).
> The lfs manual also stated that the build of "slang" cannot be parallelized.
> Sorry, I apparently overlooked that (I have now checked all other packages
> and did not find incompabilities).
No problem, current 'next' - with commented $(MAKETUNING) for 'slang' - built without problems.
For the records:
Last clean "Build finished in 7:02:45". Hardware: i7/2600, 8 GB RAM, 500GB SATA HDD, Ubuntu 18.04.2 LTS / 32bit (yes, I know... ;-) ).
Best,
Matthias
> Today, I started again some more builds (with parallel build disabled for
> slang), the test continues tomorrow.
>
> Best regards,
> Wolfgang
> PS.: The builds always use the branch that Michael created.
>
> -----Ursprüngliche Nachricht-----
> Von: Matthias Fischer <matthias.fischer(a)ipfire.org>
> Gesendet: Freitag, 8. März 2019 19:16
> An: Michael Tremer <michael.tremer(a)ipfire.org>; Wolfgang Apolinarski
> <wolfgang.apolinarski(a)ipfire.org>
> Cc: development(a)lists.ipfire.org
> Betreff: Re: [PATCH 0/1] Parallelized build for several packages
>
> Hi,
>
> third try, now building current 'next', identical error:
>
> 'slang 2.3.0' doesn't like "cd $(DIR_APP) && make $(MAKETUNING)" here.
>
> After reverting
> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=lfs/slang;h=217e74c77317d4
> c829913f934458779fd278bf29;hb=23164efba5f57b3d8ccb07a166b613f2f951e1b6,
> build continues...
>
> Best,
> Matthias
>
> On 08.03.2019 16:58, Matthias Fischer wrote:
>> Hi,
>>
>> On 08.03.2019 11:17, Michael Tremer wrote:
>>> Hi,
>>>
>>> Are you guys still there?
>>
>> Yep. ;-)
>>
>> Your branch 'faster-build' refuses to build here - I get 'slang' errors:
>>
>> ***SNIP***
>> cd /usr/src/slang-2.3.0 && make -j9
>> make[1]: Entering directory '/usr/src/slang-2.3.0'
>> cd src; make elf
>> make[2]: Entering directory '/usr/src/slang-2.3.0/src'
>> /usr/src/slang-2.3.0/autoconf/mkinsdir.sh
>> /usr/src/slang-2.3.0/src/elfobjs cp sysconf.h config.h cp
>> terminfo/default.inc terminfo.inc cd /usr/src/slang-2.3.0/src/elfobjs
>> && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586
>> -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic
>> -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
>> -fstack-protector-strong -fPIC -Dunix -DSLANG
>> /usr/src/slang-2.3.0/src/sldisply.c
>> cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall
>> -fexceptions -fPIC -march=i586 -mindirect-branch=thunk
>> -mfunction-return=thunk -mtune=generic -fomit-frame-pointer
>> -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
>> -fstack-protector-strong -fPIC -Dunix -DSLANG
>> /usr/src/slang-2.3.0/src/slutty.c cd /usr/src/slang-2.3.0/src/elfobjs
>> && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586
>> -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic
>> -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
>> -fstack-protector-strong -fPIC -Dunix -DSLANG
>> /usr/src/slang-2.3.0/src/slang.c
>> /bin/sh: line 0: cd: /usr/src/slang-2.3.0/src/elfobjs: No such file or
>> directory mkdir -p -- /usr/src/slang-2.3.0/src/elfobjs
>> make[2]: *** [Makefile:472:
>> /usr/src/slang-2.3.0/src/elfobjs/sldisply.o] Error 1
>> make[2]: *** Waiting for unfinished jobs....
>> /usr/src/slang-2.3.0/src/slang.c: In function 'inner_interp':
>> /usr/src/slang-2.3.0/src/slang.c:5733:9: warning: 'test' may be used
> uninitialized in this function [-Wmaybe-uninitialized]
>> if ((0 == pop_int (&test))
>> ~~~~~~~~~~~~~~~~~~~~~~
>> && (test == 0))
>> ^~~~~~~~~~~~~~
>> /usr/src/slang-2.3.0/src/slang.c: In function 'lang_do_loops':
>> /usr/src/slang-2.3.0/src/slang.c:3725:10: warning: 'blks[0]' may be used
> uninitialized in this function [-Wmaybe-uninitialized]
>> block = blks[0];
>> ~~~~~~^~~~~~~~~
>> /usr/src/slang-2.3.0/src/slang.c:3924:7: warning: 'first' may be used
> uninitialized in this function [-Wmaybe-uninitialized]
>> i += ctrl;
>> ~~^~~~~~~
>> /usr/src/slang-2.3.0/src/slang.c:3900:13: warning: 'last' may be used
> uninitialized in this function [-Wmaybe-uninitialized]
>> if (i > last) break;
>> ^
>> make[2]: Leaving directory '/usr/src/slang-2.3.0/src'
>> make[1]: *** [Makefile:55: elf] Error 2
>> make[1]: Leaving directory '/usr/src/slang-2.3.0'
>> make: *** [slang:74: /usr/src/log/slang-2.3.0] Error 2
>> ***SNAP***
>>
>> Don't know why. I tried two builds, same error.
>>
>> Best,
>> Matthias
>>
>>> I think that we are saving about 30 min to an hour on the nightly builds
> now… Did anybody else run some benchmarks?
>>>
>>> -Michael
>>>
>>>> On 6 Mar 2019, at 16:36, Michael Tremer <michael.tremer(a)ipfire.org>
> wrote:
>>>>
>>>> Hi,
>>>>
>>>> Sorry for not being able to get around to this earlier, but I have
> invested some time on this:
>>>>
>>>> First of all, I merged Wolfgang’s patch. He is right, the build is too
> slow. It sometimes looks a bit like it is not taking enough advantage of
> fast systems. On the other hand with have small systems, which are (in their
> own right) quite fast. Those are the ARM systems. They usually have 1G of
> RAM. 2GB when you are lucky.
>>>>
>>>> So I adjusted make.sh a little bit and removed composing the MAKETUNING
> variable from it.
>>>>
>>>> Now, make.sh only figures out the value that usually comes after the -j.
> It is set in DEFAULT_PARALLELISM.
>>>>
>>>> In the individual LFS files, we can now set a new variable called
> MAX_PARALLELISM which is optional. In boost, I now calculate this from the
> amount of memory that is available. That will make sure that we have a cap
> on this that will be high on systems that have the memory and low on those
> that don’t. Before, we hardcoded -j2 which of course runs very slow on large
> systems.
>>>>
>>>> The code is here:
>>>>
>>>>
>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs
>>>> /heads/faster-build
>>>>
>>>> Please review. I am now running this on a couple of builders and will
> see what comes out. Please do the same and send me feedback.
>>>>
>>>> Best,
>>>> -Michael
>>>>
>>>>> On 21 Feb 2019, at 17:28, Wolfgang Apolinarski
> <wolfgang.apolinarski(a)ipfire.org> wrote:
>>>>>
>>>>> Hi Arne,
>>>>>
>>>>> of course I did delete the ccache between my multiple builds to verify
> the commit. To play it safe, I hard reset using git, checked that all
> folders (build, ccache, etc.) are gone and restarted the build after
> gettoolchain and downloadsrc.
>>>>>
>>>>> Sorry, I did not know that there are still a lot of machines with low
> ram that we are using to build ipfire. My hope was that these machines have
> disappeared over the last years.
>>>>> Additionally, I do not have the ability to test all ipfire arches, as
> mentioned, the patch needs testing by others.
>>>>>
>>>>> I also thought of adding a manual parameter for developers that somehow
> manually sets the parameters for parallel build, but since the MAKETUNING
> parameter already exists and is used with other packages this would just
> somehow mess everything up.
>>>>>
>>>>> For my builds, I can just manually apply the patch to speed up the full
> build. My thought was that sharing the patch would make sense for others.
>>>>>
>>>>>> -----Ursprüngliche Nachricht-----
>>>>>> Von: Arne Fitzenreiter <Arne.Fitzenreiter(a)ipfire.org>
>>>>>> Gesendet: Donnerstag, 21. Februar 2019 09:44
>>>>>> An: Wolfgang Apolinarski <wolfgang.apolinarski(a)ipfire.org>
>>>>>> Cc: development(a)lists.ipfire.org
>>>>>> Betreff: Re: [PATCH 0/1] Parallelized build for several packages
>>>>>>
>>>>>> On the most of this packages (eg. boot and cmake) i have disabled
>>>>>> parallel build because it fails on machines with low ram (less
>>>>>> than 2GB) or on arm.
>>>>>> So such changes need tested on an 1GB machine, on all
>>>>>> architectures and build two times after deleting ccache. Because
>>>>>> the build need much lower resources if it has the file already
> prebuilt in cache.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 0/1] Parallelized build for several packages
@ 2019-03-09 20:24 Wolfgang Apolinarski
2019-03-09 20:36 ` Matthias Fischer
0 siblings, 1 reply; 14+ messages in thread
From: Wolfgang Apolinarski @ 2019-03-09 20:24 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 7686 bytes --]
Hi,
sorry, that I did not respond in time. I also had issues with "slang" during
one of my latest test builds, especially when using something like "-j3" (it
always worked fine when using higher values).
The lfs manual also stated that the build of "slang" cannot be parallelized.
Sorry, I apparently overlooked that (I have now checked all other packages
and did not find incompabilities).
Today, I started again some more builds (with parallel build disabled for
slang), the test continues tomorrow.
Best regards,
Wolfgang
PS.: The builds always use the branch that Michael created.
-----Ursprüngliche Nachricht-----
Von: Matthias Fischer <matthias.fischer(a)ipfire.org>
Gesendet: Freitag, 8. März 2019 19:16
An: Michael Tremer <michael.tremer(a)ipfire.org>; Wolfgang Apolinarski
<wolfgang.apolinarski(a)ipfire.org>
Cc: development(a)lists.ipfire.org
Betreff: Re: [PATCH 0/1] Parallelized build for several packages
Hi,
third try, now building current 'next', identical error:
'slang 2.3.0' doesn't like "cd $(DIR_APP) && make $(MAKETUNING)" here.
After reverting
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=lfs/slang;h=217e74c77317d4
c829913f934458779fd278bf29;hb=23164efba5f57b3d8ccb07a166b613f2f951e1b6,
build continues...
Best,
Matthias
On 08.03.2019 16:58, Matthias Fischer wrote:
> Hi,
>
> On 08.03.2019 11:17, Michael Tremer wrote:
>> Hi,
>>
>> Are you guys still there?
>
> Yep. ;-)
>
> Your branch 'faster-build' refuses to build here - I get 'slang' errors:
>
> ***SNIP***
> cd /usr/src/slang-2.3.0 && make -j9
> make[1]: Entering directory '/usr/src/slang-2.3.0'
> cd src; make elf
> make[2]: Entering directory '/usr/src/slang-2.3.0/src'
> /usr/src/slang-2.3.0/autoconf/mkinsdir.sh
> /usr/src/slang-2.3.0/src/elfobjs cp sysconf.h config.h cp
> terminfo/default.inc terminfo.inc cd /usr/src/slang-2.3.0/src/elfobjs
> && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586
> -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic
> -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
> -fstack-protector-strong -fPIC -Dunix -DSLANG
> /usr/src/slang-2.3.0/src/sldisply.c
> cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall
> -fexceptions -fPIC -march=i586 -mindirect-branch=thunk
> -mfunction-return=thunk -mtune=generic -fomit-frame-pointer
> -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
> -fstack-protector-strong -fPIC -Dunix -DSLANG
> /usr/src/slang-2.3.0/src/slutty.c cd /usr/src/slang-2.3.0/src/elfobjs
> && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586
> -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic
> -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
> -fstack-protector-strong -fPIC -Dunix -DSLANG
> /usr/src/slang-2.3.0/src/slang.c
> /bin/sh: line 0: cd: /usr/src/slang-2.3.0/src/elfobjs: No such file or
> directory mkdir -p -- /usr/src/slang-2.3.0/src/elfobjs
> make[2]: *** [Makefile:472:
> /usr/src/slang-2.3.0/src/elfobjs/sldisply.o] Error 1
> make[2]: *** Waiting for unfinished jobs....
> /usr/src/slang-2.3.0/src/slang.c: In function 'inner_interp':
> /usr/src/slang-2.3.0/src/slang.c:5733:9: warning: 'test' may be used
uninitialized in this function [-Wmaybe-uninitialized]
> if ((0 == pop_int (&test))
> ~~~~~~~~~~~~~~~~~~~~~~
> && (test == 0))
> ^~~~~~~~~~~~~~
> /usr/src/slang-2.3.0/src/slang.c: In function 'lang_do_loops':
> /usr/src/slang-2.3.0/src/slang.c:3725:10: warning: 'blks[0]' may be used
uninitialized in this function [-Wmaybe-uninitialized]
> block = blks[0];
> ~~~~~~^~~~~~~~~
> /usr/src/slang-2.3.0/src/slang.c:3924:7: warning: 'first' may be used
uninitialized in this function [-Wmaybe-uninitialized]
> i += ctrl;
> ~~^~~~~~~
> /usr/src/slang-2.3.0/src/slang.c:3900:13: warning: 'last' may be used
uninitialized in this function [-Wmaybe-uninitialized]
> if (i > last) break;
> ^
> make[2]: Leaving directory '/usr/src/slang-2.3.0/src'
> make[1]: *** [Makefile:55: elf] Error 2
> make[1]: Leaving directory '/usr/src/slang-2.3.0'
> make: *** [slang:74: /usr/src/log/slang-2.3.0] Error 2
> ***SNAP***
>
> Don't know why. I tried two builds, same error.
>
> Best,
> Matthias
>
>> I think that we are saving about 30 min to an hour on the nightly builds
now… Did anybody else run some benchmarks?
>>
>> -Michael
>>
>>> On 6 Mar 2019, at 16:36, Michael Tremer <michael.tremer(a)ipfire.org>
wrote:
>>>
>>> Hi,
>>>
>>> Sorry for not being able to get around to this earlier, but I have
invested some time on this:
>>>
>>> First of all, I merged Wolfgang’s patch. He is right, the build is too
slow. It sometimes looks a bit like it is not taking enough advantage of
fast systems. On the other hand with have small systems, which are (in their
own right) quite fast. Those are the ARM systems. They usually have 1G of
RAM. 2GB when you are lucky.
>>>
>>> So I adjusted make.sh a little bit and removed composing the MAKETUNING
variable from it.
>>>
>>> Now, make.sh only figures out the value that usually comes after the -j.
It is set in DEFAULT_PARALLELISM.
>>>
>>> In the individual LFS files, we can now set a new variable called
MAX_PARALLELISM which is optional. In boost, I now calculate this from the
amount of memory that is available. That will make sure that we have a cap
on this that will be high on systems that have the memory and low on those
that don’t. Before, we hardcoded -j2 which of course runs very slow on large
systems.
>>>
>>> The code is here:
>>>
>>>
>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs
>>> /heads/faster-build
>>>
>>> Please review. I am now running this on a couple of builders and will
see what comes out. Please do the same and send me feedback.
>>>
>>> Best,
>>> -Michael
>>>
>>>> On 21 Feb 2019, at 17:28, Wolfgang Apolinarski
<wolfgang.apolinarski(a)ipfire.org> wrote:
>>>>
>>>> Hi Arne,
>>>>
>>>> of course I did delete the ccache between my multiple builds to verify
the commit. To play it safe, I hard reset using git, checked that all
folders (build, ccache, etc.) are gone and restarted the build after
gettoolchain and downloadsrc.
>>>>
>>>> Sorry, I did not know that there are still a lot of machines with low
ram that we are using to build ipfire. My hope was that these machines have
disappeared over the last years.
>>>> Additionally, I do not have the ability to test all ipfire arches, as
mentioned, the patch needs testing by others.
>>>>
>>>> I also thought of adding a manual parameter for developers that somehow
manually sets the parameters for parallel build, but since the MAKETUNING
parameter already exists and is used with other packages this would just
somehow mess everything up.
>>>>
>>>> For my builds, I can just manually apply the patch to speed up the full
build. My thought was that sharing the patch would make sense for others.
>>>>
>>>>> -----Ursprüngliche Nachricht-----
>>>>> Von: Arne Fitzenreiter <Arne.Fitzenreiter(a)ipfire.org>
>>>>> Gesendet: Donnerstag, 21. Februar 2019 09:44
>>>>> An: Wolfgang Apolinarski <wolfgang.apolinarski(a)ipfire.org>
>>>>> Cc: development(a)lists.ipfire.org
>>>>> Betreff: Re: [PATCH 0/1] Parallelized build for several packages
>>>>>
>>>>> On the most of this packages (eg. boot and cmake) i have disabled
>>>>> parallel build because it fails on machines with low ram (less
>>>>> than 2GB) or on arm.
>>>>> So such changes need tested on an 1GB machine, on all
>>>>> architectures and build two times after deleting ccache. Because
>>>>> the build need much lower resources if it has the file already
prebuilt in cache.
>>>>
>>>
>>
>>
>
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 0/1] Parallelized build for several packages
2019-03-08 15:58 ` Matthias Fischer
@ 2019-03-08 18:16 ` Matthias Fischer
0 siblings, 0 replies; 14+ messages in thread
From: Matthias Fischer @ 2019-03-08 18:16 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 6828 bytes --]
Hi,
third try, now building current 'next', identical error:
'slang 2.3.0' doesn't like "cd $(DIR_APP) && make $(MAKETUNING)" here.
After reverting
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=lfs/slang;h=217e74c77317d4c829913f934458779fd278bf29;hb=23164efba5f57b3d8ccb07a166b613f2f951e1b6,
build continues...
Best,
Matthias
On 08.03.2019 16:58, Matthias Fischer wrote:
> Hi,
>
> On 08.03.2019 11:17, Michael Tremer wrote:
>> Hi,
>>
>> Are you guys still there?
>
> Yep. ;-)
>
> Your branch 'faster-build' refuses to build here - I get 'slang' errors:
>
> ***SNIP***
> cd /usr/src/slang-2.3.0 && make -j9
> make[1]: Entering directory '/usr/src/slang-2.3.0'
> cd src; make elf
> make[2]: Entering directory '/usr/src/slang-2.3.0/src'
> /usr/src/slang-2.3.0/autoconf/mkinsdir.sh /usr/src/slang-2.3.0/src/elfobjs
> cp sysconf.h config.h
> cp terminfo/default.inc terminfo.inc
> cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586 -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -fPIC -Dunix -DSLANG /usr/src/slang-2.3.0/src/sldisply.c
> cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586 -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -fPIC -Dunix -DSLANG /usr/src/slang-2.3.0/src/slutty.c
> cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586 -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -fPIC -Dunix -DSLANG /usr/src/slang-2.3.0/src/slang.c
> /bin/sh: line 0: cd: /usr/src/slang-2.3.0/src/elfobjs: No such file or directory
> mkdir -p -- /usr/src/slang-2.3.0/src/elfobjs
> make[2]: *** [Makefile:472: /usr/src/slang-2.3.0/src/elfobjs/sldisply.o] Error 1
> make[2]: *** Waiting for unfinished jobs....
> /usr/src/slang-2.3.0/src/slang.c: In function 'inner_interp':
> /usr/src/slang-2.3.0/src/slang.c:5733:9: warning: 'test' may be used uninitialized in this function [-Wmaybe-uninitialized]
> if ((0 == pop_int (&test))
> ~~~~~~~~~~~~~~~~~~~~~~
> && (test == 0))
> ^~~~~~~~~~~~~~
> /usr/src/slang-2.3.0/src/slang.c: In function 'lang_do_loops':
> /usr/src/slang-2.3.0/src/slang.c:3725:10: warning: 'blks[0]' may be used uninitialized in this function [-Wmaybe-uninitialized]
> block = blks[0];
> ~~~~~~^~~~~~~~~
> /usr/src/slang-2.3.0/src/slang.c:3924:7: warning: 'first' may be used uninitialized in this function [-Wmaybe-uninitialized]
> i += ctrl;
> ~~^~~~~~~
> /usr/src/slang-2.3.0/src/slang.c:3900:13: warning: 'last' may be used uninitialized in this function [-Wmaybe-uninitialized]
> if (i > last) break;
> ^
> make[2]: Leaving directory '/usr/src/slang-2.3.0/src'
> make[1]: *** [Makefile:55: elf] Error 2
> make[1]: Leaving directory '/usr/src/slang-2.3.0'
> make: *** [slang:74: /usr/src/log/slang-2.3.0] Error 2
> ***SNAP***
>
> Don't know why. I tried two builds, same error.
>
> Best,
> Matthias
>
>> I think that we are saving about 30 min to an hour on the nightly builds now… Did anybody else run some benchmarks?
>>
>> -Michael
>>
>>> On 6 Mar 2019, at 16:36, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>
>>> Hi,
>>>
>>> Sorry for not being able to get around to this earlier, but I have invested some time on this:
>>>
>>> First of all, I merged Wolfgang’s patch. He is right, the build is too slow. It sometimes looks a bit like it is not taking enough advantage of fast systems. On the other hand with have small systems, which are (in their own right) quite fast. Those are the ARM systems. They usually have 1G of RAM. 2GB when you are lucky.
>>>
>>> So I adjusted make.sh a little bit and removed composing the MAKETUNING variable from it.
>>>
>>> Now, make.sh only figures out the value that usually comes after the -j. It is set in DEFAULT_PARALLELISM.
>>>
>>> In the individual LFS files, we can now set a new variable called MAX_PARALLELISM which is optional. In boost, I now calculate this from the amount of memory that is available. That will make sure that we have a cap on this that will be high on systems that have the memory and low on those that don’t. Before, we hardcoded -j2 which of course runs very slow on large systems.
>>>
>>> The code is here:
>>>
>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/faster-build
>>>
>>> Please review. I am now running this on a couple of builders and will see what comes out. Please do the same and send me feedback.
>>>
>>> Best,
>>> -Michael
>>>
>>>> On 21 Feb 2019, at 17:28, Wolfgang Apolinarski <wolfgang.apolinarski(a)ipfire.org> wrote:
>>>>
>>>> Hi Arne,
>>>>
>>>> of course I did delete the ccache between my multiple builds to verify the commit. To play it safe, I hard reset using git, checked that all folders (build, ccache, etc.) are gone and restarted the build after gettoolchain and downloadsrc.
>>>>
>>>> Sorry, I did not know that there are still a lot of machines with low ram that we are using to build ipfire. My hope was that these machines have disappeared over the last years.
>>>> Additionally, I do not have the ability to test all ipfire arches, as mentioned, the patch needs testing by others.
>>>>
>>>> I also thought of adding a manual parameter for developers that somehow manually sets the parameters for parallel build, but since the MAKETUNING parameter already exists and is used with other packages this would just somehow mess everything up.
>>>>
>>>> For my builds, I can just manually apply the patch to speed up the full build. My thought was that sharing the patch would make sense for others.
>>>>
>>>>> -----Ursprüngliche Nachricht-----
>>>>> Von: Arne Fitzenreiter <Arne.Fitzenreiter(a)ipfire.org>
>>>>> Gesendet: Donnerstag, 21. Februar 2019 09:44
>>>>> An: Wolfgang Apolinarski <wolfgang.apolinarski(a)ipfire.org>
>>>>> Cc: development(a)lists.ipfire.org
>>>>> Betreff: Re: [PATCH 0/1] Parallelized build for several packages
>>>>>
>>>>> On the most of this packages (eg. boot and cmake) i have disabled
>>>>> parallel build because it fails on machines with low ram (less than 2GB)
>>>>> or on arm.
>>>>> So such changes need tested on an 1GB machine, on all architectures and
>>>>> build two times after deleting ccache. Because the build need much lower
>>>>> resources if it has the file already prebuilt in cache.
>>>>
>>>
>>
>>
>
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 0/1] Parallelized build for several packages
2019-03-08 10:17 ` Michael Tremer
@ 2019-03-08 15:58 ` Matthias Fischer
2019-03-08 18:16 ` Matthias Fischer
0 siblings, 1 reply; 14+ messages in thread
From: Matthias Fischer @ 2019-03-08 15:58 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 6266 bytes --]
Hi,
On 08.03.2019 11:17, Michael Tremer wrote:
> Hi,
>
> Are you guys still there?
Yep. ;-)
Your branch 'faster-build' refuses to build here - I get 'slang' errors:
***SNIP***
cd /usr/src/slang-2.3.0 && make -j9
make[1]: Entering directory '/usr/src/slang-2.3.0'
cd src; make elf
make[2]: Entering directory '/usr/src/slang-2.3.0/src'
/usr/src/slang-2.3.0/autoconf/mkinsdir.sh /usr/src/slang-2.3.0/src/elfobjs
cp sysconf.h config.h
cp terminfo/default.inc terminfo.inc
cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586 -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -fPIC -Dunix -DSLANG /usr/src/slang-2.3.0/src/sldisply.c
cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586 -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -fPIC -Dunix -DSLANG /usr/src/slang-2.3.0/src/slutty.c
cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586 -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -fPIC -Dunix -DSLANG /usr/src/slang-2.3.0/src/slang.c
/bin/sh: line 0: cd: /usr/src/slang-2.3.0/src/elfobjs: No such file or directory
mkdir -p -- /usr/src/slang-2.3.0/src/elfobjs
make[2]: *** [Makefile:472: /usr/src/slang-2.3.0/src/elfobjs/sldisply.o] Error 1
make[2]: *** Waiting for unfinished jobs....
/usr/src/slang-2.3.0/src/slang.c: In function 'inner_interp':
/usr/src/slang-2.3.0/src/slang.c:5733:9: warning: 'test' may be used uninitialized in this function [-Wmaybe-uninitialized]
if ((0 == pop_int (&test))
~~~~~~~~~~~~~~~~~~~~~~
&& (test == 0))
^~~~~~~~~~~~~~
/usr/src/slang-2.3.0/src/slang.c: In function 'lang_do_loops':
/usr/src/slang-2.3.0/src/slang.c:3725:10: warning: 'blks[0]' may be used uninitialized in this function [-Wmaybe-uninitialized]
block = blks[0];
~~~~~~^~~~~~~~~
/usr/src/slang-2.3.0/src/slang.c:3924:7: warning: 'first' may be used uninitialized in this function [-Wmaybe-uninitialized]
i += ctrl;
~~^~~~~~~
/usr/src/slang-2.3.0/src/slang.c:3900:13: warning: 'last' may be used uninitialized in this function [-Wmaybe-uninitialized]
if (i > last) break;
^
make[2]: Leaving directory '/usr/src/slang-2.3.0/src'
make[1]: *** [Makefile:55: elf] Error 2
make[1]: Leaving directory '/usr/src/slang-2.3.0'
make: *** [slang:74: /usr/src/log/slang-2.3.0] Error 2
***SNAP***
Don't know why. I tried two builds, same error.
Best,
Matthias
> I think that we are saving about 30 min to an hour on the nightly builds now… Did anybody else run some benchmarks?
>
> -Michael
>
>> On 6 Mar 2019, at 16:36, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>
>> Hi,
>>
>> Sorry for not being able to get around to this earlier, but I have invested some time on this:
>>
>> First of all, I merged Wolfgang’s patch. He is right, the build is too slow. It sometimes looks a bit like it is not taking enough advantage of fast systems. On the other hand with have small systems, which are (in their own right) quite fast. Those are the ARM systems. They usually have 1G of RAM. 2GB when you are lucky.
>>
>> So I adjusted make.sh a little bit and removed composing the MAKETUNING variable from it.
>>
>> Now, make.sh only figures out the value that usually comes after the -j. It is set in DEFAULT_PARALLELISM.
>>
>> In the individual LFS files, we can now set a new variable called MAX_PARALLELISM which is optional. In boost, I now calculate this from the amount of memory that is available. That will make sure that we have a cap on this that will be high on systems that have the memory and low on those that don’t. Before, we hardcoded -j2 which of course runs very slow on large systems.
>>
>> The code is here:
>>
>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/faster-build
>>
>> Please review. I am now running this on a couple of builders and will see what comes out. Please do the same and send me feedback.
>>
>> Best,
>> -Michael
>>
>>> On 21 Feb 2019, at 17:28, Wolfgang Apolinarski <wolfgang.apolinarski(a)ipfire.org> wrote:
>>>
>>> Hi Arne,
>>>
>>> of course I did delete the ccache between my multiple builds to verify the commit. To play it safe, I hard reset using git, checked that all folders (build, ccache, etc.) are gone and restarted the build after gettoolchain and downloadsrc.
>>>
>>> Sorry, I did not know that there are still a lot of machines with low ram that we are using to build ipfire. My hope was that these machines have disappeared over the last years.
>>> Additionally, I do not have the ability to test all ipfire arches, as mentioned, the patch needs testing by others.
>>>
>>> I also thought of adding a manual parameter for developers that somehow manually sets the parameters for parallel build, but since the MAKETUNING parameter already exists and is used with other packages this would just somehow mess everything up.
>>>
>>> For my builds, I can just manually apply the patch to speed up the full build. My thought was that sharing the patch would make sense for others.
>>>
>>>> -----Ursprüngliche Nachricht-----
>>>> Von: Arne Fitzenreiter <Arne.Fitzenreiter(a)ipfire.org>
>>>> Gesendet: Donnerstag, 21. Februar 2019 09:44
>>>> An: Wolfgang Apolinarski <wolfgang.apolinarski(a)ipfire.org>
>>>> Cc: development(a)lists.ipfire.org
>>>> Betreff: Re: [PATCH 0/1] Parallelized build for several packages
>>>>
>>>> On the most of this packages (eg. boot and cmake) i have disabled
>>>> parallel build because it fails on machines with low ram (less than 2GB)
>>>> or on arm.
>>>> So such changes need tested on an 1GB machine, on all architectures and
>>>> build two times after deleting ccache. Because the build need much lower
>>>> resources if it has the file already prebuilt in cache.
>>>
>>
>
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 0/1] Parallelized build for several packages
2019-03-06 16:36 ` Michael Tremer
@ 2019-03-08 10:17 ` Michael Tremer
2019-03-08 15:58 ` Matthias Fischer
0 siblings, 1 reply; 14+ messages in thread
From: Michael Tremer @ 2019-03-08 10:17 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 3391 bytes --]
Hi,
Are you guys still there?
I think that we are saving about 30 min to an hour on the nightly builds now… Did anybody else run some benchmarks?
-Michael
> On 6 Mar 2019, at 16:36, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>
> Hi,
>
> Sorry for not being able to get around to this earlier, but I have invested some time on this:
>
> First of all, I merged Wolfgang’s patch. He is right, the build is too slow. It sometimes looks a bit like it is not taking enough advantage of fast systems. On the other hand with have small systems, which are (in their own right) quite fast. Those are the ARM systems. They usually have 1G of RAM. 2GB when you are lucky.
>
> So I adjusted make.sh a little bit and removed composing the MAKETUNING variable from it.
>
> Now, make.sh only figures out the value that usually comes after the -j. It is set in DEFAULT_PARALLELISM.
>
> In the individual LFS files, we can now set a new variable called MAX_PARALLELISM which is optional. In boost, I now calculate this from the amount of memory that is available. That will make sure that we have a cap on this that will be high on systems that have the memory and low on those that don’t. Before, we hardcoded -j2 which of course runs very slow on large systems.
>
> The code is here:
>
> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/faster-build
>
> Please review. I am now running this on a couple of builders and will see what comes out. Please do the same and send me feedback.
>
> Best,
> -Michael
>
>> On 21 Feb 2019, at 17:28, Wolfgang Apolinarski <wolfgang.apolinarski(a)ipfire.org> wrote:
>>
>> Hi Arne,
>>
>> of course I did delete the ccache between my multiple builds to verify the commit. To play it safe, I hard reset using git, checked that all folders (build, ccache, etc.) are gone and restarted the build after gettoolchain and downloadsrc.
>>
>> Sorry, I did not know that there are still a lot of machines with low ram that we are using to build ipfire. My hope was that these machines have disappeared over the last years.
>> Additionally, I do not have the ability to test all ipfire arches, as mentioned, the patch needs testing by others.
>>
>> I also thought of adding a manual parameter for developers that somehow manually sets the parameters for parallel build, but since the MAKETUNING parameter already exists and is used with other packages this would just somehow mess everything up.
>>
>> For my builds, I can just manually apply the patch to speed up the full build. My thought was that sharing the patch would make sense for others.
>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: Arne Fitzenreiter <Arne.Fitzenreiter(a)ipfire.org>
>>> Gesendet: Donnerstag, 21. Februar 2019 09:44
>>> An: Wolfgang Apolinarski <wolfgang.apolinarski(a)ipfire.org>
>>> Cc: development(a)lists.ipfire.org
>>> Betreff: Re: [PATCH 0/1] Parallelized build for several packages
>>>
>>> On the most of this packages (eg. boot and cmake) i have disabled
>>> parallel build because it fails on machines with low ram (less than 2GB)
>>> or on arm.
>>> So such changes need tested on an 1GB machine, on all architectures and
>>> build two times after deleting ccache. Because the build need much lower
>>> resources if it has the file already prebuilt in cache.
>>
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 0/1] Parallelized build for several packages
2019-02-21 17:28 Wolfgang Apolinarski
@ 2019-03-06 16:36 ` Michael Tremer
2019-03-08 10:17 ` Michael Tremer
0 siblings, 1 reply; 14+ messages in thread
From: Michael Tremer @ 2019-03-06 16:36 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 3066 bytes --]
Hi,
Sorry for not being able to get around to this earlier, but I have invested some time on this:
First of all, I merged Wolfgang’s patch. He is right, the build is too slow. It sometimes looks a bit like it is not taking enough advantage of fast systems. On the other hand with have small systems, which are (in their own right) quite fast. Those are the ARM systems. They usually have 1G of RAM. 2GB when you are lucky.
So I adjusted make.sh a little bit and removed composing the MAKETUNING variable from it.
Now, make.sh only figures out the value that usually comes after the -j. It is set in DEFAULT_PARALLELISM.
In the individual LFS files, we can now set a new variable called MAX_PARALLELISM which is optional. In boost, I now calculate this from the amount of memory that is available. That will make sure that we have a cap on this that will be high on systems that have the memory and low on those that don’t. Before, we hardcoded -j2 which of course runs very slow on large systems.
The code is here:
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/faster-build
Please review. I am now running this on a couple of builders and will see what comes out. Please do the same and send me feedback.
Best,
-Michael
> On 21 Feb 2019, at 17:28, Wolfgang Apolinarski <wolfgang.apolinarski(a)ipfire.org> wrote:
>
> Hi Arne,
>
> of course I did delete the ccache between my multiple builds to verify the commit. To play it safe, I hard reset using git, checked that all folders (build, ccache, etc.) are gone and restarted the build after gettoolchain and downloadsrc.
>
> Sorry, I did not know that there are still a lot of machines with low ram that we are using to build ipfire. My hope was that these machines have disappeared over the last years.
> Additionally, I do not have the ability to test all ipfire arches, as mentioned, the patch needs testing by others.
>
> I also thought of adding a manual parameter for developers that somehow manually sets the parameters for parallel build, but since the MAKETUNING parameter already exists and is used with other packages this would just somehow mess everything up.
>
> For my builds, I can just manually apply the patch to speed up the full build. My thought was that sharing the patch would make sense for others.
>
>> -----Ursprüngliche Nachricht-----
>> Von: Arne Fitzenreiter <Arne.Fitzenreiter(a)ipfire.org>
>> Gesendet: Donnerstag, 21. Februar 2019 09:44
>> An: Wolfgang Apolinarski <wolfgang.apolinarski(a)ipfire.org>
>> Cc: development(a)lists.ipfire.org
>> Betreff: Re: [PATCH 0/1] Parallelized build for several packages
>>
>> On the most of this packages (eg. boot and cmake) i have disabled
>> parallel build because it fails on machines with low ram (less than 2GB)
>> or on arm.
>> So such changes need tested on an 1GB machine, on all architectures and
>> build two times after deleting ccache. Because the build need much lower
>> resources if it has the file already prebuilt in cache.
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 0/1] Parallelized build for several packages
@ 2019-02-21 17:28 Wolfgang Apolinarski
2019-03-06 16:36 ` Michael Tremer
0 siblings, 1 reply; 14+ messages in thread
From: Wolfgang Apolinarski @ 2019-02-21 17:28 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 1643 bytes --]
Hi Arne,
of course I did delete the ccache between my multiple builds to verify the commit. To play it safe, I hard reset using git, checked that all folders (build, ccache, etc.) are gone and restarted the build after gettoolchain and downloadsrc.
Sorry, I did not know that there are still a lot of machines with low ram that we are using to build ipfire. My hope was that these machines have disappeared over the last years.
Additionally, I do not have the ability to test all ipfire arches, as mentioned, the patch needs testing by others.
I also thought of adding a manual parameter for developers that somehow manually sets the parameters for parallel build, but since the MAKETUNING parameter already exists and is used with other packages this would just somehow mess everything up.
For my builds, I can just manually apply the patch to speed up the full build. My thought was that sharing the patch would make sense for others.
> -----Ursprüngliche Nachricht-----
> Von: Arne Fitzenreiter <Arne.Fitzenreiter(a)ipfire.org>
> Gesendet: Donnerstag, 21. Februar 2019 09:44
> An: Wolfgang Apolinarski <wolfgang.apolinarski(a)ipfire.org>
> Cc: development(a)lists.ipfire.org
> Betreff: Re: [PATCH 0/1] Parallelized build for several packages
>
> On the most of this packages (eg. boot and cmake) i have disabled
> parallel build because it fails on machines with low ram (less than 2GB)
> or on arm.
> So such changes need tested on an 1GB machine, on all architectures and
> build two times after deleting ccache. Because the build need much lower
> resources if it has the file already prebuilt in cache.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 0/1] Parallelized build for several packages
2019-02-20 19:18 Wolfgang Apolinarski
@ 2019-02-21 8:44 ` Arne Fitzenreiter
0 siblings, 0 replies; 14+ messages in thread
From: Arne Fitzenreiter @ 2019-02-21 8:44 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 355 bytes --]
On the most of this packages (eg. boot and cmake) i have disabled
parallel build because it fails on machines with low ram (less than 2GB)
or on arm.
So such changes need tested on an 1GB machine, on all architectures and
build two times after deleting ccache. Because the build need much lower
resources if it has the file already prebuilt in cache.
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH 0/1] Parallelized build for several packages
@ 2019-02-20 19:18 Wolfgang Apolinarski
2019-02-21 8:44 ` Arne Fitzenreiter
0 siblings, 1 reply; 14+ messages in thread
From: Wolfgang Apolinarski @ 2019-02-20 19:18 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 2096 bytes --]
With this patch, I tried to optimize the build (because it really takes
a long time if I compile from scratch for testing).
Nowadays, most packages are already optimizing for parallel build. I was
testing the build on an 8 core system (i.e., MAKETUNING is -j9).
I also found out something else, the collectd package build crashes when
libstatgrab is already built:
https://github.com/collectd/collectd/issues/445
Someone else already noticed this on the list, without discovering the
cause. An upgrade of collectd would be required to solve this problem.
Since I am not sure for what collectd is actually used (and a major
upgrade would be required), I did not perform this.
Speed-up:
Package | Before | After
boost | 1:20:41 | 13:39
cyrus-imapd | 3:50 | 0:48
dnsdist | 5:23 | 1:18
gcc | 46:15 | 11:29
gettext | 9:55 | 4:56
gutenprint | 3:14 | 1:22
ipfire-netboot | 9:33 | 2:07
krb5 | 5:28 | 1:27
lcd4linux | 1:04 | 0:27
netsnmpd | 8:00 | 3:33
nut | 1:59 | 0:21
openssl | 7:57 | 2:37
rrdtool | 1:31 | 0:32
samba | 5:03 | 0:55
slang | 1:04 | 0:19
snort | 5:11 | 0:56
xfsprogs | 5:41 | 1:46
I also marked packages that (in their current version) crash when being
built in parallel. I performed multiple builds (and tested the ISO
image), so I hope that this patch is reliable for everyone.
Nevertheless, please test the patch on different build environments
before applying it.
Wolfgang Apolinarski (1):
Parallelized build for several packages
lfs/boost | 2 +-
lfs/collectd | 2 +-
lfs/cyrus-imapd | 2 +-
lfs/dhcp | 2 +-
lfs/dnsdist | 2 +-
lfs/gcc | 4 ++--
lfs/gettext | 8 ++++----
lfs/groff | 2 +-
lfs/gutenprint | 2 +-
lfs/ipfire-netboot | 4 ++--
lfs/krb5 | 2 +-
lfs/lcd4linux | 2 +-
lfs/netpbm | 2 +-
lfs/netsnmpd | 2 +-
lfs/nut | 2 +-
lfs/openssl | 2 +-
lfs/rrdtool | 2 +-
lfs/samba | 4 ++--
lfs/slang | 2 +-
lfs/snort | 2 +-
lfs/xfsprogs | 2 +-
21 files changed, 27 insertions(+), 27 deletions(-)
--
2.19.2
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2019-03-11 17:50 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-03-11 17:50 [PATCH 0/1] Parallelized build for several packages Wolfgang Apolinarski
-- strict thread matches above, loose matches on Subject: below --
2019-03-09 20:24 Wolfgang Apolinarski
2019-03-09 20:36 ` Matthias Fischer
2019-03-10 18:24 ` Michael Tremer
2019-03-10 19:17 ` ummeegge
2019-03-11 9:37 ` Michael Tremer
2019-03-11 17:07 ` ummeegge
2019-02-21 17:28 Wolfgang Apolinarski
2019-03-06 16:36 ` Michael Tremer
2019-03-08 10:17 ` Michael Tremer
2019-03-08 15:58 ` Matthias Fischer
2019-03-08 18:16 ` Matthias Fischer
2019-02-20 19:18 Wolfgang Apolinarski
2019-02-21 8:44 ` Arne Fitzenreiter
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox