From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: [PATCH 0/1] Parallelized build for several packages Date: Fri, 08 Mar 2019 10:17:13 +0000 Message-ID: <8998D83E-82FA-48E0-896E-446DE2BEDA56@ipfire.org> In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============7515896188585072471==" List-Id: --===============7515896188585072471== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi, Are you guys still there? I think that we are saving about 30 min to an hour on the nightly builds now= =E2=80=A6 Did anybody else run some benchmarks? -Michael > On 6 Mar 2019, at 16:36, Michael Tremer wrote: >=20 > Hi, >=20 > Sorry for not being able to get around to this earlier, but I have invested= some time on this: >=20 > First of all, I merged Wolfgang=E2=80=99s patch. He is right, the build is = too slow. It sometimes looks a bit like it is not taking enough advantage of = fast systems. On the other hand with have small systems, which are (in their = own right) quite fast. Those are the ARM systems. They usually have 1G of RAM= . 2GB when you are lucky. >=20 > So I adjusted make.sh a little bit and removed composing the MAKETUNING var= iable from it. >=20 > Now, make.sh only figures out the value that usually comes after the -j. It= is set in DEFAULT_PARALLELISM. >=20 > In the individual LFS files, we can now set a new variable called MAX_PARAL= LELISM which is optional. In boost, I now calculate this from the amount of m= emory that is available. That will make sure that we have a cap on this that = will be high on systems that have the memory and low on those that don=E2=80= =99t. Before, we hardcoded -j2 which of course runs very slow on large system= s. >=20 > The code is here: >=20 > https://git.ipfire.org/?p=3Dpeople/ms/ipfire-2.x.git;a=3Dshortlog;h=3Drefs= /heads/faster-build=20 >=20 > Please review. I am now running this on a couple of builders and will see w= hat comes out. Please do the same and send me feedback. >=20 > Best, > -Michael >=20 >> On 21 Feb 2019, at 17:28, Wolfgang Apolinarski wrote: >>=20 >> Hi Arne, >>=20 >> of course I did delete the ccache between my multiple builds to verify the= commit. To play it safe, I hard reset using git, checked that all folders (b= uild, ccache, etc.) are gone and restarted the build after gettoolchain and d= ownloadsrc. >>=20 >> Sorry, I did not know that there are still a lot of machines with low ram = that we are using to build ipfire. My hope was that these machines have disap= peared over the last years. >> Additionally, I do not have the ability to test all ipfire arches, as ment= ioned, the patch needs testing by others. >>=20 >> I also thought of adding a manual parameter for developers that somehow ma= nually sets the parameters for parallel build, but since the MAKETUNING param= eter already exists and is used with other packages this would just somehow m= ess everything up. >>=20 >> For my builds, I can just manually apply the patch to speed up the full bu= ild. My thought was that sharing the patch would make sense for others. >>=20 >>> -----Urspr=C3=BCngliche Nachricht----- >>> Von: Arne Fitzenreiter >>> Gesendet: Donnerstag, 21. Februar 2019 09:44 >>> An: Wolfgang Apolinarski >>> Cc: development(a)lists.ipfire.org >>> Betreff: Re: [PATCH 0/1] Parallelized build for several packages >>>=20 >>> On the most of this packages (eg. boot and cmake) i have disabled >>> parallel build because it fails on machines with low ram (less than 2GB) >>> or on arm. >>> So such changes need tested on an 1GB machine, on all architectures and >>> build two times after deleting ccache. Because the build need much lower >>> resources if it has the file already prebuilt in cache. >>=20 >=20 --===============7515896188585072471==--