Re: [PATCH 0/1] Parallelized build for several packages

[Michael Tremer <michael.tremer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 10750 bytes --]

Yes, probably.

It uses a lot of memory when compiling because it is C++.

I will adjust the number of concurrent processes just like we did for boost.

Best,
-Michael

> On 10 Mar 2019, at 19:17, ummeegge <ummeegge(a)ipfire.org> wrote:
> 
> Hi all,
> 
> On So, 2019-03-10 at 18:24 +0000, Michael Tremer wrote:
>> Hi,
>> 
>> I merged the patch. And I am surprised it is only the one…
> dnsdist do have problems to build here in next currently:
> 
> checking for boost/foreach.hpp... yes
> checking for the toolset name used by Boost for g++... configure: WARNING: could not figure out which toolset name to use for g++
> ....
> 
> g++: internal compiler error: Killed (program cc1plus)
> Please submit a full bug report,
> with preprocessed source if appropriate.
> See <http://bugtracker.ipfire.org> for instructions.
> make[3]: *** [Makefile:743: dnsdist-lua2.o] Error 4
> make[3]: *** Waiting for unfinished jobs....
> 
> 
> May this is possibly related to this topic ?
> 
> Best,
> 
> Erik
> 
>> 
>> -Michael
>> 
>>> On 9 Mar 2019, at 20:36, Matthias Fischer <
>>> matthias.fischer(a)ipfire.org> wrote:
>>> 
>>> On 09.03.2019 21:24, Wolfgang Apolinarski wrote:
>>>> Hi,
>>>> 
>>>> sorry, that I did not respond in time. I also had issues with
>>>> "slang" during
>>>> one of my latest test builds, especially when using something
>>>> like "-j3" (it
>>>> always worked fine when using higher values).
>>>> The lfs manual also stated that the build of "slang" cannot be
>>>> parallelized.
>>>> Sorry, I apparently overlooked that (I have now checked all other
>>>> packages
>>>> and did not find incompabilities).
>>> 
>>> No problem, current 'next' - with commented $(MAKETUNING) for
>>> 'slang' - built without problems.
>>> 
>>> For the records:
>>> Last clean "Build finished in 7:02:45". Hardware: i7/2600, 8 GB
>>> RAM, 500GB SATA HDD, Ubuntu 18.04.2 LTS / 32bit (yes, I know... ;-) 
>>> ).
>>> 
>>> Best,
>>> Matthias
>>> 
>>>> Today, I started again some more builds (with parallel build
>>>> disabled for
>>>> slang), the test continues tomorrow.
>>>> 
>>>> Best regards,
>>>> Wolfgang
>>>> PS.: The builds always use the branch that Michael created.
>>>> 
>>>> -----Ursprüngliche Nachricht-----
>>>> Von: Matthias Fischer <matthias.fischer(a)ipfire.org> 
>>>> Gesendet: Freitag, 8. März 2019 19:16
>>>> An: Michael Tremer <michael.tremer(a)ipfire.org>; Wolfgang
>>>> Apolinarski
>>>> <wolfgang.apolinarski(a)ipfire.org>
>>>> Cc: development(a)lists.ipfire.org
>>>> Betreff: Re: [PATCH 0/1] Parallelized build for several packages
>>>> 
>>>> Hi,
>>>> 
>>>> third try, now building current 'next', identical error:
>>>> 
>>>> 'slang 2.3.0' doesn't like "cd $(DIR_APP) && make $(MAKETUNING)"
>>>> here.
>>>> 
>>>> After reverting
>>>> 
> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=lfs/slang;h=217e74c77317d4
>>>> c829913f934458779fd278bf29;hb=23164efba5f57b3d8ccb07a166b613f2f95
>>>> 1e1b6,
>>>> build continues...
>>>> 
>>>> Best,
>>>> Matthias
>>>> 
>>>> On 08.03.2019 16:58, Matthias Fischer wrote:
>>>>> Hi,
>>>>> 
>>>>> On 08.03.2019 11:17, Michael Tremer wrote:
>>>>>> Hi,
>>>>>> 
>>>>>> Are you guys still there?
>>>>> 
>>>>> Yep. ;-)
>>>>> 
>>>>> Your branch 'faster-build' refuses to build here - I get
>>>>> 'slang' errors:
>>>>> 
>>>>> ***SNIP***
>>>>> cd /usr/src/slang-2.3.0 && make -j9
>>>>> make[1]: Entering directory '/usr/src/slang-2.3.0'
>>>>> cd src; make elf
>>>>> make[2]: Entering directory '/usr/src/slang-2.3.0/src'
>>>>> /usr/src/slang-2.3.0/autoconf/mkinsdir.sh 
>>>>> /usr/src/slang-2.3.0/src/elfobjs cp sysconf.h config.h cp 
>>>>> terminfo/default.inc terminfo.inc cd /usr/src/slang-
>>>>> 2.3.0/src/elfobjs 
>>>>> && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586 
>>>>> -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic 
>>>>> -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-
>>>>> D_GLIBCXX_ASSERTIONS 
>>>>> -fstack-protector-strong  -fPIC  -Dunix -DSLANG  
>>>>> /usr/src/slang-2.3.0/src/sldisply.c
>>>>> cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall 
>>>>> -fexceptions -fPIC -march=i586 -mindirect-branch=thunk 
>>>>> -mfunction-return=thunk -mtune=generic -fomit-frame-pointer 
>>>>> -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS 
>>>>> -fstack-protector-strong  -fPIC  -Dunix -DSLANG  
>>>>> /usr/src/slang-2.3.0/src/slutty.c cd /usr/src/slang-
>>>>> 2.3.0/src/elfobjs 
>>>>> && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586 
>>>>> -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic 
>>>>> -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-
>>>>> D_GLIBCXX_ASSERTIONS 
>>>>> -fstack-protector-strong  -fPIC  -Dunix -DSLANG  
>>>>> /usr/src/slang-2.3.0/src/slang.c
>>>>> /bin/sh: line 0: cd: /usr/src/slang-2.3.0/src/elfobjs: No such
>>>>> file or 
>>>>> directory mkdir -p -- /usr/src/slang-2.3.0/src/elfobjs
>>>>> make[2]: *** [Makefile:472: 
>>>>> /usr/src/slang-2.3.0/src/elfobjs/sldisply.o] Error 1
>>>>> make[2]: *** Waiting for unfinished jobs....
>>>>> /usr/src/slang-2.3.0/src/slang.c: In function 'inner_interp':
>>>>> /usr/src/slang-2.3.0/src/slang.c:5733:9: warning: 'test' may be
>>>>> used
>>>> 
>>>> uninitialized in this function [-Wmaybe-uninitialized]
>>>>>    if ((0 == pop_int (&test))
>>>>>        ~~~~~~~~~~~~~~~~~~~~~~
>>>>>        && (test == 0))
>>>>>        ^~~~~~~~~~~~~~
>>>>> /usr/src/slang-2.3.0/src/slang.c: In function 'lang_do_loops':
>>>>> /usr/src/slang-2.3.0/src/slang.c:3725:10: warning: 'blks[0]'
>>>>> may be used
>>>> 
>>>> uninitialized in this function [-Wmaybe-uninitialized]
>>>>>   block = blks[0];
>>>>>   ~~~~~~^~~~~~~~~
>>>>> /usr/src/slang-2.3.0/src/slang.c:3924:7: warning: 'first' may
>>>>> be used
>>>> 
>>>> uninitialized in this function [-Wmaybe-uninitialized]
>>>>>    i += ctrl;
>>>>>    ~~^~~~~~~
>>>>> /usr/src/slang-2.3.0/src/slang.c:3900:13: warning: 'last' may
>>>>> be used
>>>> 
>>>> uninitialized in this function [-Wmaybe-uninitialized]
>>>>>         if (i > last) break;
>>>>>            ^
>>>>> make[2]: Leaving directory '/usr/src/slang-2.3.0/src'
>>>>> make[1]: *** [Makefile:55: elf] Error 2
>>>>> make[1]: Leaving directory '/usr/src/slang-2.3.0'
>>>>> make: *** [slang:74: /usr/src/log/slang-2.3.0] Error 2
>>>>> ***SNAP***
>>>>> 
>>>>> Don't know why. I tried two builds, same error.
>>>>> 
>>>>> Best,
>>>>> Matthias
>>>>> 
>>>>>> I think that we are saving about 30 min to an hour on the
>>>>>> nightly builds
>>>> 
>>>> now… Did anybody else run some benchmarks?
>>>>>> 
>>>>>> -Michael
>>>>>> 
>>>>>>> On 6 Mar 2019, at 16:36, Michael Tremer <
>>>>>>> michael.tremer(a)ipfire.org>
>>>> 
>>>> wrote:
>>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> Sorry for not being able to get around to this earlier, but
>>>>>>> I have
>>>> 
>>>> invested some time on this:
>>>>>>> 
>>>>>>> First of all, I merged Wolfgang’s patch. He is right, the
>>>>>>> build is too
>>>> 
>>>> slow. It sometimes looks a bit like it is not taking enough
>>>> advantage of
>>>> fast systems. On the other hand with have small systems, which
>>>> are (in their
>>>> own right) quite fast. Those are the ARM systems. They usually
>>>> have 1G of
>>>> RAM. 2GB when you are lucky.
>>>>>>> 
>>>>>>> So I adjusted make.sh a little bit and removed composing
>>>>>>> the MAKETUNING
>>>> 
>>>> variable from it.
>>>>>>> 
>>>>>>> Now, make.sh only figures out the value that usually comes
>>>>>>> after the -j.
>>>> 
>>>> It is set in DEFAULT_PARALLELISM.
>>>>>>> 
>>>>>>> In the individual LFS files, we can now set a new variable
>>>>>>> called
>>>> 
>>>> MAX_PARALLELISM which is optional. In boost, I now calculate this
>>>> from the
>>>> amount of memory that is available. That will make sure that we
>>>> have a cap
>>>> on this that will be high on systems that have the memory and low
>>>> on those
>>>> that don’t. Before, we hardcoded -j2 which of course runs very
>>>> slow on large
>>>> systems.
>>>>>>> 
>>>>>>> The code is here:
>>>>>>> 
>>>>>>> 
>>>>>>> 
> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs
>>>>>>> /heads/faster-build
>>>>>>> 
>>>>>>> Please review. I am now running this on a couple of
>>>>>>> builders and will
>>>> 
>>>> see what comes out. Please do the same and send me feedback.
>>>>>>> 
>>>>>>> Best,
>>>>>>> -Michael
>>>>>>> 
>>>>>>>> On 21 Feb 2019, at 17:28, Wolfgang Apolinarski
>>>> 
>>>> <wolfgang.apolinarski(a)ipfire.org> wrote:
>>>>>>>> 
>>>>>>>> Hi Arne,
>>>>>>>> 
>>>>>>>> of course I did delete the ccache between my multiple
>>>>>>>> builds to verify
>>>> 
>>>> the commit. To play it safe, I hard reset using git, checked that
>>>> all
>>>> folders (build, ccache, etc.) are gone and restarted the build
>>>> after
>>>> gettoolchain and downloadsrc.
>>>>>>>> 
>>>>>>>> Sorry, I did not know that there are still a lot of
>>>>>>>> machines with low
>>>> 
>>>> ram that we are using to build ipfire. My hope was that these
>>>> machines have
>>>> disappeared over the last years.
>>>>>>>> Additionally, I do not have the ability to test all
>>>>>>>> ipfire arches, as
>>>> 
>>>> mentioned, the patch needs testing by others.
>>>>>>>> 
>>>>>>>> I also thought of adding a manual parameter for
>>>>>>>> developers that somehow
>>>> 
>>>> manually sets the parameters for parallel build, but since the
>>>> MAKETUNING
>>>> parameter already exists and is used with other packages this
>>>> would just
>>>> somehow mess everything up.
>>>>>>>> 
>>>>>>>> For my builds, I can just manually apply the patch to
>>>>>>>> speed up the full
>>>> 
>>>> build. My thought was that sharing the patch would make sense for
>>>> others.
>>>>>>>> 
>>>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>>>> Von: Arne Fitzenreiter <Arne.Fitzenreiter(a)ipfire.org>
>>>>>>>>> Gesendet: Donnerstag, 21. Februar 2019 09:44
>>>>>>>>> An: Wolfgang Apolinarski <
>>>>>>>>> wolfgang.apolinarski(a)ipfire.org>
>>>>>>>>> Cc: development(a)lists.ipfire.org
>>>>>>>>> Betreff: Re: [PATCH 0/1] Parallelized build for several
>>>>>>>>> packages
>>>>>>>>> 
>>>>>>>>> On the most of this packages (eg. boot and cmake) i
>>>>>>>>> have disabled 
>>>>>>>>> parallel build because it fails on machines with low
>>>>>>>>> ram (less 
>>>>>>>>> than 2GB) or on arm.
>>>>>>>>> So such changes need tested on an 1GB machine, on all 
>>>>>>>>> architectures and build two times after deleting
>>>>>>>>> ccache. Because 
>>>>>>>>> the build need much lower resources if it has the file
>>>>>>>>> already
>>>> 
>>>> prebuilt in cache.
>> 
>> 
>