Hi!
From: Michael Tremer
Yes, probably.
It uses a lot of memory when compiling because it is C++.
I will adjust the number of concurrent processes just like we did for boost.
That is probably a good idea. I tested a build with 12 cores and 1 GB of RAM (CentOS) and it actually compiled fine (x86_64), but it is a good idea to play it safe here (dnsdist). I also performed other tests at the weekend and all my builds have been successful, so far, I hope it stays that way...
Best regards, Wolfgang
Best, -Michael
On 10 Mar 2019, at 19:17, ummeegge ummeegge@ipfire.org wrote:
Hi all,
On So, 2019-03-10 at 18:24 +0000, Michael Tremer wrote:
Hi,
I merged the patch. And I am surprised it is only the one…
dnsdist do have problems to build here in next currently:
checking for boost/foreach.hpp... yes checking for the toolset name used by Boost for g++... configure: WARNING: could not figure out which toolset name to use for g++ ....
g++: internal compiler error: Killed (program cc1plus) Please submit a full bug report, with preprocessed source if appropriate. See http://bugtracker.ipfire.org for instructions. make[3]: *** [Makefile:743: dnsdist-lua2.o] Error 4 make[3]: *** Waiting for unfinished jobs....
May this is possibly related to this topic ?
Best,
Erik
-Michael
On 9 Mar 2019, at 20:36, Matthias Fischer < matthias.fischer@ipfire.org> wrote:
On 09.03.2019 21:24, Wolfgang Apolinarski wrote:
Hi,
sorry, that I did not respond in time. I also had issues with "slang" during one of my latest test builds, especially when using something like "-j3" (it always worked fine when using higher values). The lfs manual also stated that the build of "slang" cannot be parallelized. Sorry, I apparently overlooked that (I have now checked all other packages and did not find incompabilities).
No problem, current 'next' - with commented $(MAKETUNING) for 'slang' - built without problems.
For the records: Last clean "Build finished in 7:02:45". Hardware: i7/2600, 8 GB RAM, 500GB SATA HDD, Ubuntu 18.04.2 LTS / 32bit (yes, I know... ;-) ).
Best, Matthias
Today, I started again some more builds (with parallel build disabled for slang), the test continues tomorrow.
Best regards, Wolfgang PS.: The builds always use the branch that Michael created.
-----Ursprüngliche Nachricht----- Von: Matthias Fischer matthias.fischer@ipfire.org Gesendet: Freitag, 8. März 2019 19:16 An: Michael Tremer michael.tremer@ipfire.org; Wolfgang Apolinarski wolfgang.apolinarski@ipfire.org Cc: development@lists.ipfire.org Betreff: Re: [PATCH 0/1] Parallelized build for several packages
Hi,
third try, now building current 'next', identical error:
'slang 2.3.0' doesn't like "cd $(DIR_APP) && make $(MAKETUNING)" here.
After reverting
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=lfs/slang;h=217e74c7 7317d4
c829913f934458779fd278bf29;hb=23164efba5f57b3d8ccb07a166b613f2f95
1e1b6, build continues...
Best, Matthias
On 08.03.2019 16:58, Matthias Fischer wrote:
Hi,
On 08.03.2019 11:17, Michael Tremer wrote: > Hi, > > Are you guys still there?
Yep. ;-)
Your branch 'faster-build' refuses to build here - I get 'slang' errors:
***SNIP*** cd /usr/src/slang-2.3.0 && make -j9 make[1]: Entering directory '/usr/src/slang-2.3.0' cd src; make elf make[2]: Entering directory '/usr/src/slang-2.3.0/src' /usr/src/slang-2.3.0/autoconf/mkinsdir.sh /usr/src/slang-2.3.0/src/elfobjs cp sysconf.h config.h cp terminfo/default.inc terminfo.inc cd /usr/src/slang- 2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586 -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -
Wp,-
D_GLIBCXX_ASSERTIONS -fstack-protector-strong -fPIC -Dunix -DSLANG /usr/src/slang-2.3.0/src/sldisply.c cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586 -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -fPIC -Dunix -DSLANG /usr/src/slang-2.3.0/src/slutty.c cd /usr/src/slang- 2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586 -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -
Wp,-
D_GLIBCXX_ASSERTIONS -fstack-protector-strong -fPIC -Dunix -DSLANG /usr/src/slang-2.3.0/src/slang.c /bin/sh: line 0: cd: /usr/src/slang-2.3.0/src/elfobjs: No such file or directory mkdir -p -- /usr/src/slang-2.3.0/src/elfobjs make[2]: *** [Makefile:472: /usr/src/slang-2.3.0/src/elfobjs/sldisply.o] Error 1 make[2]: *** Waiting for unfinished jobs.... /usr/src/slang-2.3.0/src/slang.c: In function 'inner_interp': /usr/src/slang-2.3.0/src/slang.c:5733:9: warning: 'test' may be used
uninitialized in this function [-Wmaybe-uninitialized]
if ((0 == pop_int (&test)) ~~~~~~~~~~~~~~~~~~~~~~ && (test == 0)) ^~~~~~~~~~~~~~ /usr/src/slang-2.3.0/src/slang.c: In function 'lang_do_loops': /usr/src/slang-2.3.0/src/slang.c:3725:10: warning: 'blks[0]' may be used
uninitialized in this function [-Wmaybe-uninitialized]
block = blks[0];
/usr/src/slang-2.3.0/src/slang.c:3924:7: warning: 'first' may be useduninitialized in this function [-Wmaybe-uninitialized]
i += ctrl; ~~^~~~~~~ /usr/src/slang-2.3.0/src/slang.c:3900:13: warning: 'last' may be used
uninitialized in this function [-Wmaybe-uninitialized]
if (i > last) break; ^make[2]: Leaving directory '/usr/src/slang-2.3.0/src' make[1]: *** [Makefile:55: elf] Error 2 make[1]: Leaving directory '/usr/src/slang-2.3.0' make: *** [slang:74: /usr/src/log/slang-2.3.0] Error 2 ***SNAP***
Don't know why. I tried two builds, same error.
Best, Matthias
> I think that we are saving about 30 min to an hour on the nightly > builds
now… Did anybody else run some benchmarks?
> > -Michael > >> On 6 Mar 2019, at 16:36, Michael Tremer < >> michael.tremer@ipfire.org>
wrote:
>> >> Hi, >> >> Sorry for not being able to get around to this earlier, but I >> have
invested some time on this:
>> >> First of all, I merged Wolfgang’s patch. He is right, the build >> is too
slow. It sometimes looks a bit like it is not taking enough advantage of fast systems. On the other hand with have small systems, which are (in their own right) quite fast. Those are the ARM systems. They usually have 1G of RAM. 2GB when you are lucky.
>> >> So I adjusted make.sh a little bit and removed composing the >> MAKETUNING
variable from it.
>> >> Now, make.sh only figures out the value that usually comes after >> the -j.
It is set in DEFAULT_PARALLELISM.
>> >> In the individual LFS files, we can now set a new variable >> called
MAX_PARALLELISM which is optional. In boost, I now calculate this from the amount of memory that is available. That will make sure that we have a cap on this that will be high on systems that have the memory and low on those that don’t. Before, we hardcoded -j2 which of course runs very slow on large systems.
>> >> The code is here: >> >> >>
https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs
>> /heads/faster-build >> >> Please review. I am now running this on a couple of builders and >> will
see what comes out. Please do the same and send me feedback.
>> >> Best, >> -Michael >> >>> On 21 Feb 2019, at 17:28, Wolfgang Apolinarski
wolfgang.apolinarski@ipfire.org wrote:
>>> >>> Hi Arne, >>> >>> of course I did delete the ccache between my multiple builds to >>> verify
the commit. To play it safe, I hard reset using git, checked that all folders (build, ccache, etc.) are gone and restarted the build after gettoolchain and downloadsrc.
>>> >>> Sorry, I did not know that there are still a lot of machines >>> with low
ram that we are using to build ipfire. My hope was that these machines have disappeared over the last years.
>>> Additionally, I do not have the ability to test all ipfire >>> arches, as
mentioned, the patch needs testing by others.
>>> >>> I also thought of adding a manual parameter for developers that >>> somehow
manually sets the parameters for parallel build, but since the MAKETUNING parameter already exists and is used with other
packages
this would just somehow mess everything up.
>>> >>> For my builds, I can just manually apply the patch to speed up >>> the full
build. My thought was that sharing the patch would make sense for others.
>>> >>>> -----Ursprüngliche Nachricht----- >>>> Von: Arne Fitzenreiter Arne.Fitzenreiter@ipfire.org >>>> Gesendet: Donnerstag, 21. Februar 2019 09:44 >>>> An: Wolfgang Apolinarski < >>>> wolfgang.apolinarski@ipfire.org> >>>> Cc: development@lists.ipfire.org >>>> Betreff: Re: [PATCH 0/1] Parallelized build for several >>>> packages >>>> >>>> On the most of this packages (eg. boot and cmake) i have >>>> disabled parallel build because it fails on machines with low >>>> ram (less than 2GB) or on arm. >>>> So such changes need tested on an 1GB machine, on all >>>> architectures and build two times after deleting ccache. >>>> Because the build need much lower resources if it has the file >>>> already
prebuilt in cache.