Re: [PATCH 0/1] Parallelized build for several packages

[Michael Tremer <michael.tremer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 8815 bytes --]

Hi,

I merged the patch. And I am surprised it is only the one…

-Michael

> On 9 Mar 2019, at 20:36, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
> 
> On 09.03.2019 21:24, Wolfgang Apolinarski wrote:
>> Hi,
>> 
>> sorry, that I did not respond in time. I also had issues with "slang" during
>> one of my latest test builds, especially when using something like "-j3" (it
>> always worked fine when using higher values).
>> The lfs manual also stated that the build of "slang" cannot be parallelized.
>> Sorry, I apparently overlooked that (I have now checked all other packages
>> and did not find incompabilities).
> 
> No problem, current 'next' - with commented $(MAKETUNING) for 'slang' - built without problems.
> 
> For the records:
> Last clean "Build finished in 7:02:45". Hardware: i7/2600, 8 GB RAM, 500GB SATA HDD, Ubuntu 18.04.2 LTS / 32bit (yes, I know... ;-) ).
> 
> Best,
> Matthias
> 
>> Today, I started again some more builds (with parallel build disabled for
>> slang), the test continues tomorrow.
>> 
>> Best regards,
>> Wolfgang
>> PS.: The builds always use the branch that Michael created.
>> 
>> -----Ursprüngliche Nachricht-----
>> Von: Matthias Fischer <matthias.fischer(a)ipfire.org> 
>> Gesendet: Freitag, 8. März 2019 19:16
>> An: Michael Tremer <michael.tremer(a)ipfire.org>; Wolfgang Apolinarski
>> <wolfgang.apolinarski(a)ipfire.org>
>> Cc: development(a)lists.ipfire.org
>> Betreff: Re: [PATCH 0/1] Parallelized build for several packages
>> 
>> Hi,
>> 
>> third try, now building current 'next', identical error:
>> 
>> 'slang 2.3.0' doesn't like "cd $(DIR_APP) && make $(MAKETUNING)" here.
>> 
>> After reverting
>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=lfs/slang;h=217e74c77317d4
>> c829913f934458779fd278bf29;hb=23164efba5f57b3d8ccb07a166b613f2f951e1b6,
>> build continues...
>> 
>> Best,
>> Matthias
>> 
>> On 08.03.2019 16:58, Matthias Fischer wrote:
>>> Hi,
>>> 
>>> On 08.03.2019 11:17, Michael Tremer wrote:
>>>> Hi,
>>>> 
>>>> Are you guys still there?
>>> 
>>> Yep. ;-)
>>> 
>>> Your branch 'faster-build' refuses to build here - I get 'slang' errors:
>>> 
>>> ***SNIP***
>>> cd /usr/src/slang-2.3.0 && make -j9
>>> make[1]: Entering directory '/usr/src/slang-2.3.0'
>>> cd src; make elf
>>> make[2]: Entering directory '/usr/src/slang-2.3.0/src'
>>> /usr/src/slang-2.3.0/autoconf/mkinsdir.sh 
>>> /usr/src/slang-2.3.0/src/elfobjs cp sysconf.h config.h cp 
>>> terminfo/default.inc terminfo.inc cd /usr/src/slang-2.3.0/src/elfobjs 
>>> && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586 
>>> -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic 
>>> -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS 
>>> -fstack-protector-strong  -fPIC  -Dunix -DSLANG  
>>> /usr/src/slang-2.3.0/src/sldisply.c
>>> cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall 
>>> -fexceptions -fPIC -march=i586 -mindirect-branch=thunk 
>>> -mfunction-return=thunk -mtune=generic -fomit-frame-pointer 
>>> -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS 
>>> -fstack-protector-strong  -fPIC  -Dunix -DSLANG  
>>> /usr/src/slang-2.3.0/src/slutty.c cd /usr/src/slang-2.3.0/src/elfobjs 
>>> && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586 
>>> -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic 
>>> -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS 
>>> -fstack-protector-strong  -fPIC  -Dunix -DSLANG  
>>> /usr/src/slang-2.3.0/src/slang.c
>>> /bin/sh: line 0: cd: /usr/src/slang-2.3.0/src/elfobjs: No such file or 
>>> directory mkdir -p -- /usr/src/slang-2.3.0/src/elfobjs
>>> make[2]: *** [Makefile:472: 
>>> /usr/src/slang-2.3.0/src/elfobjs/sldisply.o] Error 1
>>> make[2]: *** Waiting for unfinished jobs....
>>> /usr/src/slang-2.3.0/src/slang.c: In function 'inner_interp':
>>> /usr/src/slang-2.3.0/src/slang.c:5733:9: warning: 'test' may be used
>> uninitialized in this function [-Wmaybe-uninitialized]
>>>     if ((0 == pop_int (&test))
>>>         ~~~~~~~~~~~~~~~~~~~~~~
>>>         && (test == 0))
>>>         ^~~~~~~~~~~~~~
>>> /usr/src/slang-2.3.0/src/slang.c: In function 'lang_do_loops':
>>> /usr/src/slang-2.3.0/src/slang.c:3725:10: warning: 'blks[0]' may be used
>> uninitialized in this function [-Wmaybe-uninitialized]
>>>    block = blks[0];
>>>    ~~~~~~^~~~~~~~~
>>> /usr/src/slang-2.3.0/src/slang.c:3924:7: warning: 'first' may be used
>> uninitialized in this function [-Wmaybe-uninitialized]
>>>     i += ctrl;
>>>     ~~^~~~~~~
>>> /usr/src/slang-2.3.0/src/slang.c:3900:13: warning: 'last' may be used
>> uninitialized in this function [-Wmaybe-uninitialized]
>>>          if (i > last) break;
>>>             ^
>>> make[2]: Leaving directory '/usr/src/slang-2.3.0/src'
>>> make[1]: *** [Makefile:55: elf] Error 2
>>> make[1]: Leaving directory '/usr/src/slang-2.3.0'
>>> make: *** [slang:74: /usr/src/log/slang-2.3.0] Error 2
>>> ***SNAP***
>>> 
>>> Don't know why. I tried two builds, same error.
>>> 
>>> Best,
>>> Matthias
>>> 
>>>> I think that we are saving about 30 min to an hour on the nightly builds
>> now… Did anybody else run some benchmarks?
>>>> 
>>>> -Michael
>>>> 
>>>>> On 6 Mar 2019, at 16:36, Michael Tremer <michael.tremer(a)ipfire.org>
>> wrote:
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> Sorry for not being able to get around to this earlier, but I have
>> invested some time on this:
>>>>> 
>>>>> First of all, I merged Wolfgang’s patch. He is right, the build is too
>> slow. It sometimes looks a bit like it is not taking enough advantage of
>> fast systems. On the other hand with have small systems, which are (in their
>> own right) quite fast. Those are the ARM systems. They usually have 1G of
>> RAM. 2GB when you are lucky.
>>>>> 
>>>>> So I adjusted make.sh a little bit and removed composing the MAKETUNING
>> variable from it.
>>>>> 
>>>>> Now, make.sh only figures out the value that usually comes after the -j.
>> It is set in DEFAULT_PARALLELISM.
>>>>> 
>>>>> In the individual LFS files, we can now set a new variable called
>> MAX_PARALLELISM which is optional. In boost, I now calculate this from the
>> amount of memory that is available. That will make sure that we have a cap
>> on this that will be high on systems that have the memory and low on those
>> that don’t. Before, we hardcoded -j2 which of course runs very slow on large
>> systems.
>>>>> 
>>>>> The code is here:
>>>>> 
>>>>> 
>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs
>>>>> /heads/faster-build
>>>>> 
>>>>> Please review. I am now running this on a couple of builders and will
>> see what comes out. Please do the same and send me feedback.
>>>>> 
>>>>> Best,
>>>>> -Michael
>>>>> 
>>>>>> On 21 Feb 2019, at 17:28, Wolfgang Apolinarski
>> <wolfgang.apolinarski(a)ipfire.org> wrote:
>>>>>> 
>>>>>> Hi Arne,
>>>>>> 
>>>>>> of course I did delete the ccache between my multiple builds to verify
>> the commit. To play it safe, I hard reset using git, checked that all
>> folders (build, ccache, etc.) are gone and restarted the build after
>> gettoolchain and downloadsrc.
>>>>>> 
>>>>>> Sorry, I did not know that there are still a lot of machines with low
>> ram that we are using to build ipfire. My hope was that these machines have
>> disappeared over the last years.
>>>>>> Additionally, I do not have the ability to test all ipfire arches, as
>> mentioned, the patch needs testing by others.
>>>>>> 
>>>>>> I also thought of adding a manual parameter for developers that somehow
>> manually sets the parameters for parallel build, but since the MAKETUNING
>> parameter already exists and is used with other packages this would just
>> somehow mess everything up.
>>>>>> 
>>>>>> For my builds, I can just manually apply the patch to speed up the full
>> build. My thought was that sharing the patch would make sense for others.
>>>>>> 
>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>> Von: Arne Fitzenreiter <Arne.Fitzenreiter(a)ipfire.org>
>>>>>>> Gesendet: Donnerstag, 21. Februar 2019 09:44
>>>>>>> An: Wolfgang Apolinarski <wolfgang.apolinarski(a)ipfire.org>
>>>>>>> Cc: development(a)lists.ipfire.org
>>>>>>> Betreff: Re: [PATCH 0/1] Parallelized build for several packages
>>>>>>> 
>>>>>>> On the most of this packages (eg. boot and cmake) i have disabled 
>>>>>>> parallel build because it fails on machines with low ram (less 
>>>>>>> than 2GB) or on arm.
>>>>>>> So such changes need tested on an 1GB machine, on all 
>>>>>>> architectures and build two times after deleting ccache. Because 
>>>>>>> the build need much lower resources if it has the file already
>> prebuilt in cache.
>