From mboxrd@z Thu Jan 1 00:00:00 1970 From: Matthias Fischer To: development@lists.ipfire.org Subject: Re: Planning out Core Update 117 Date: Tue, 21 Nov 2017 18:35:23 +0100 Message-ID: In-Reply-To: <1511275374.4838.600.camel@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0645880907861695564==" List-Id: --===============0645880907861695564== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi, On 21.11.2017 15:42, Michael Tremer wrote: > Hello, >=20 > any progress on this? Not much, because I've been struggling with a flu for the past two weeks, and its not over yet, but: In the meantime I ran several builds with different parameters. IMHO we could use: 'export XZ_OPT =3D --threads=3D0 -8 --memory=3D700MiB' in 'lfs/cdrom' and cd /install/packages/package/tmp/ && XZ_OPT=3D"-T0 -8 --memory=3D700MiB" tar = -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz * in 'lfs/Config'. Using '9' or '--extreme'-parameter brought no further improvements, on the other hand, build time and RAM consumption increased strongly. With '-8', minimal RAM consumption was about 680 MiB, using '700 Mib' was always safe. So this is the minimum I would recommend. Results: (Buildtime: 5 hours 20 minutes 59 seconds) 255866764 ipfire-2.19.1gb-ext4.i586-full-core117.img.gz 252262006 ipfire-2.19.1gb-ext4-scon.i586-full-core117.img.gz 175112192 ipfire-2.19.i586-full-core117.iso =20 > -Michael >=20 > On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote: >> Hi, >>=20 >> On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer wrote: >> > Hi, >> >=20 >> > On 07.11.2017 16:05, Michael Tremer wrote: >> > > Hi, >> > >=20 >> > > maybe this function helps: >> > >=20 >> > > https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dcommitdiff;h=3D5190eea2= 4f9822a63d >> > > c5d06d214b48f973b14f29 >> >=20 >> > Thanks. I'll take a look... This would help, indeed, but how to implement(!?), and... (see below) >> > > On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote: >> > > > On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer wrote: >> > > > ... >> > > > >=20 >> > > > > I forgot: Tuning the XZ-parameters... ;-) >> > > >=20 >> > > > Good that you are bringing this up. So I think what we should do her= e is >> > > > bringing back XZ compression with -8 so that we do not waste too much >> > > > space >> > > > when >> > > > extracting the image again. >> >=20 >> > Right now I'm testing how much RAM I need. >> >=20 >> > Current 'next'-build is running with: >> >=20 >> > 'lfs/cdrom': >> > ... >> > export XZ_OPT =3D --threads=3D0 -8 --memory=3D500MiB >> > ... >> >=20 >> > 'lfs/Config': >> > ... >> > cd /install/packages/package/tmp/ && XZ_OPT=3D"-T0 -8" ... >> > ... >> >=20 >> > I'm waiting for the results. Takes a while. >>=20 >> You can just run it like this and it will tell you: >>=20 >> [ms(a)hughes ~]$ xz -vv8 -T0 >> xz: Filter chain: -- >> lzma2=3Ddict=3D32MiB,lc=3D3,lp=3D0,pb=3D2,mode=3Dnormal,nice=3D64,mf=3Dbt4= ,depth=3D0 >> xz: Using up to 4 threads. >> xz: 2,629 MiB of memory is required. The limiter is disabled. >> xz: Decompression will need 33 MiB of memory. >> xz: Compressed data cannot be written to a terminal >> xz: Try `xz --help' for more information. >>=20 >> > > > To not run into memory limits we should detect how much memory the b= uild >> > > > host >> > > > has and set the memory limit to maybe 80% of that. xz will then >> > > > automatically >> > > > scale down to a number of parallel processes that it can fit into >> > > > memory. This is something I couldn't get to work. First, I couldn't specify not less than '700 MiB'. If I specify '600 MiB', building stops and tells me it needs at least about 680 MiB. If I use '70%', building stops. And I never saw that 'xz' "automatically scaled down". It just stopped everytime. =20 >> > I'm a bit puzzled about this: I tried using '--memory=3D70%' as mentioned >> > in the xz man pages, but is seems to have no effect. It looks as if the >> > percent parameter isn't recognized and ignored. It always ends up in >> > "cannot allocate memory". Any hints on this? Do both 'lfs/cdrom' AND >> > 'lfs/Config' need the '--memory'-parameter? This was the next problem: 'percent'-parameter does not work as expected. >> Yes, but it would be good to have one place where the parameters are >> being generated and then we just use them from a variable. Probably beyond my capabilities for the moment. Any thoughts? >> > > > ... >> > > > We will at least need about half a GB which should be a sensible >> > > > requirement >> > > > for a build system any ways. I'm afraid '500 MiB' are not enough... >> > Agreed. Because of this I'm testing with '500Mib'. >> >=20 >> > > > Will you work on this bringing it into the build system?! >> >=20 >> > "I'll do my very best!" :-)) >>=20 >> Very well Miss Sophie... >> ... Best=E2=84=A2, Matthias --===============0645880907861695564==--