Re: [PATCH 0/1] Parallelized build for several packages

[Wolfgang Apolinarski <wolfgang.apolinarski@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 7686 bytes --]

Hi,

sorry, that I did not respond in time. I also had issues with "slang" during
one of my latest test builds, especially when using something like "-j3" (it
always worked fine when using higher values).
The lfs manual also stated that the build of "slang" cannot be parallelized.
Sorry, I apparently overlooked that (I have now checked all other packages
and did not find incompabilities).

Today, I started again some more builds (with parallel build disabled for
slang), the test continues tomorrow.

Best regards,
Wolfgang
PS.: The builds always use the branch that Michael created.

-----Ursprüngliche Nachricht-----
Von: Matthias Fischer <matthias.fischer(a)ipfire.org> 
Gesendet: Freitag, 8. März 2019 19:16
An: Michael Tremer <michael.tremer(a)ipfire.org>; Wolfgang Apolinarski
<wolfgang.apolinarski(a)ipfire.org>
Cc: development(a)lists.ipfire.org
Betreff: Re: [PATCH 0/1] Parallelized build for several packages

Hi,

third try, now building current 'next', identical error:

'slang 2.3.0' doesn't like "cd $(DIR_APP) && make $(MAKETUNING)" here.

After reverting
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=lfs/slang;h=217e74c77317d4
c829913f934458779fd278bf29;hb=23164efba5f57b3d8ccb07a166b613f2f951e1b6,
build continues...

Best,
Matthias

On 08.03.2019 16:58, Matthias Fischer wrote:
> Hi,
> 
> On 08.03.2019 11:17, Michael Tremer wrote:
>> Hi,
>> 
>> Are you guys still there?
> 
> Yep. ;-)
> 
> Your branch 'faster-build' refuses to build here - I get 'slang' errors:
> 
> ***SNIP***
> cd /usr/src/slang-2.3.0 && make -j9
> make[1]: Entering directory '/usr/src/slang-2.3.0'
> cd src; make elf
> make[2]: Entering directory '/usr/src/slang-2.3.0/src'
> /usr/src/slang-2.3.0/autoconf/mkinsdir.sh 
> /usr/src/slang-2.3.0/src/elfobjs cp sysconf.h config.h cp 
> terminfo/default.inc terminfo.inc cd /usr/src/slang-2.3.0/src/elfobjs 
> && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586 
> -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic 
> -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS 
> -fstack-protector-strong  -fPIC  -Dunix -DSLANG  
> /usr/src/slang-2.3.0/src/sldisply.c
> cd /usr/src/slang-2.3.0/src/elfobjs && gcc -c -O2 -pipe -Wall 
> -fexceptions -fPIC -march=i586 -mindirect-branch=thunk 
> -mfunction-return=thunk -mtune=generic -fomit-frame-pointer 
> -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS 
> -fstack-protector-strong  -fPIC  -Dunix -DSLANG  
> /usr/src/slang-2.3.0/src/slutty.c cd /usr/src/slang-2.3.0/src/elfobjs 
> && gcc -c -O2 -pipe -Wall -fexceptions -fPIC -march=i586 
> -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic 
> -fomit-frame-pointer -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS 
> -fstack-protector-strong  -fPIC  -Dunix -DSLANG  
> /usr/src/slang-2.3.0/src/slang.c
> /bin/sh: line 0: cd: /usr/src/slang-2.3.0/src/elfobjs: No such file or 
> directory mkdir -p -- /usr/src/slang-2.3.0/src/elfobjs
> make[2]: *** [Makefile:472: 
> /usr/src/slang-2.3.0/src/elfobjs/sldisply.o] Error 1
> make[2]: *** Waiting for unfinished jobs....
> /usr/src/slang-2.3.0/src/slang.c: In function 'inner_interp':
> /usr/src/slang-2.3.0/src/slang.c:5733:9: warning: 'test' may be used
uninitialized in this function [-Wmaybe-uninitialized]
>      if ((0 == pop_int (&test))
>          ~~~~~~~~~~~~~~~~~~~~~~
>          && (test == 0))
>          ^~~~~~~~~~~~~~
> /usr/src/slang-2.3.0/src/slang.c: In function 'lang_do_loops':
> /usr/src/slang-2.3.0/src/slang.c:3725:10: warning: 'blks[0]' may be used
uninitialized in this function [-Wmaybe-uninitialized]
>     block = blks[0];
>     ~~~~~~^~~~~~~~~
> /usr/src/slang-2.3.0/src/slang.c:3924:7: warning: 'first' may be used
uninitialized in this function [-Wmaybe-uninitialized]
>      i += ctrl;
>      ~~^~~~~~~
> /usr/src/slang-2.3.0/src/slang.c:3900:13: warning: 'last' may be used
uninitialized in this function [-Wmaybe-uninitialized]
>           if (i > last) break;
>              ^
> make[2]: Leaving directory '/usr/src/slang-2.3.0/src'
> make[1]: *** [Makefile:55: elf] Error 2
> make[1]: Leaving directory '/usr/src/slang-2.3.0'
> make: *** [slang:74: /usr/src/log/slang-2.3.0] Error 2
> ***SNAP***
> 
> Don't know why. I tried two builds, same error.
> 
> Best,
> Matthias
> 
>> I think that we are saving about 30 min to an hour on the nightly builds
now… Did anybody else run some benchmarks?
>> 
>> -Michael
>> 
>>> On 6 Mar 2019, at 16:36, Michael Tremer <michael.tremer(a)ipfire.org>
wrote:
>>> 
>>> Hi,
>>> 
>>> Sorry for not being able to get around to this earlier, but I have
invested some time on this:
>>> 
>>> First of all, I merged Wolfgang’s patch. He is right, the build is too
slow. It sometimes looks a bit like it is not taking enough advantage of
fast systems. On the other hand with have small systems, which are (in their
own right) quite fast. Those are the ARM systems. They usually have 1G of
RAM. 2GB when you are lucky.
>>> 
>>> So I adjusted make.sh a little bit and removed composing the MAKETUNING
variable from it.
>>> 
>>> Now, make.sh only figures out the value that usually comes after the -j.
It is set in DEFAULT_PARALLELISM.
>>> 
>>> In the individual LFS files, we can now set a new variable called
MAX_PARALLELISM which is optional. In boost, I now calculate this from the
amount of memory that is available. That will make sure that we have a cap
on this that will be high on systems that have the memory and low on those
that don’t. Before, we hardcoded -j2 which of course runs very slow on large
systems.
>>> 
>>> The code is here:
>>> 
>>>  
>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs
>>> /heads/faster-build
>>> 
>>> Please review. I am now running this on a couple of builders and will
see what comes out. Please do the same and send me feedback.
>>> 
>>> Best,
>>> -Michael
>>> 
>>>> On 21 Feb 2019, at 17:28, Wolfgang Apolinarski
<wolfgang.apolinarski(a)ipfire.org> wrote:
>>>> 
>>>> Hi Arne,
>>>> 
>>>> of course I did delete the ccache between my multiple builds to verify
the commit. To play it safe, I hard reset using git, checked that all
folders (build, ccache, etc.) are gone and restarted the build after
gettoolchain and downloadsrc.
>>>> 
>>>> Sorry, I did not know that there are still a lot of machines with low
ram that we are using to build ipfire. My hope was that these machines have
disappeared over the last years.
>>>> Additionally, I do not have the ability to test all ipfire arches, as
mentioned, the patch needs testing by others.
>>>> 
>>>> I also thought of adding a manual parameter for developers that somehow
manually sets the parameters for parallel build, but since the MAKETUNING
parameter already exists and is used with other packages this would just
somehow mess everything up.
>>>> 
>>>> For my builds, I can just manually apply the patch to speed up the full
build. My thought was that sharing the patch would make sense for others.
>>>> 
>>>>> -----Ursprüngliche Nachricht-----
>>>>> Von: Arne Fitzenreiter <Arne.Fitzenreiter(a)ipfire.org>
>>>>> Gesendet: Donnerstag, 21. Februar 2019 09:44
>>>>> An: Wolfgang Apolinarski <wolfgang.apolinarski(a)ipfire.org>
>>>>> Cc: development(a)lists.ipfire.org
>>>>> Betreff: Re: [PATCH 0/1] Parallelized build for several packages
>>>>> 
>>>>> On the most of this packages (eg. boot and cmake) i have disabled 
>>>>> parallel build because it fails on machines with low ram (less 
>>>>> than 2GB) or on arm.
>>>>> So such changes need tested on an 1GB machine, on all 
>>>>> architectures and build two times after deleting ccache. Because 
>>>>> the build need much lower resources if it has the file already
prebuilt in cache.
>>>> 
>>> 
>> 
>> 
> 
>