Re: Planning out Core Update 117

[Matthias Fischer <matthias.fischer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 4483 bytes --]

Hi,

On 21.11.2017 15:42, Michael Tremer wrote:
> Hello,
> 
> any progress on this?

Not much, because I've been struggling with a flu for the past two
weeks, and its not over yet, but:

In the meantime I ran several builds with different parameters.

IMHO we could use:

'export XZ_OPT = --threads=0 -8 --memory=700MiB'

in 'lfs/cdrom'

and

cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8 --memory=700MiB" tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz *

in 'lfs/Config'.

Using '9' or '--extreme'-parameter brought no further improvements,
on the other hand, build time and RAM consumption increased
strongly. With '-8', minimal RAM consumption was about 680 MiB, using
'700 Mib' was always safe. So this is the minimum I would recommend.

Results:

(Buildtime: 5 hours 20 minutes 59 seconds)

255866764  ipfire-2.19.1gb-ext4.i586-full-core117.img.gz
252262006  ipfire-2.19.1gb-ext4-scon.i586-full-core117.img.gz
175112192  ipfire-2.19.i586-full-core117.iso
 
> -Michael
> 
> On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote:
>> Hi,
>> 
>> On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer wrote:
>> > Hi,
>> > 
>> > On 07.11.2017 16:05, Michael Tremer wrote:
>> > > Hi,
>> > > 
>> > > maybe this function helps:
>> > > 
>> > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=5190eea24f9822a63d
>> > > c5d06d214b48f973b14f29
>> > 
>> > Thanks. I'll take a look...

This would help, indeed, but how to implement(!?), and... (see below)

>> > > On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote:
>> > > > On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer wrote:
>> > > > ...
>> > > > > 
>> > > > > I forgot: Tuning the XZ-parameters... ;-)
>> > > > 
>> > > > Good that you are bringing this up. So I think what we should do here is
>> > > > bringing back XZ compression with -8 so that we do not waste too much
>> > > > space
>> > > > when
>> > > > extracting the image again.
>> > 
>> > Right now I'm testing how much RAM I need.
>> > 
>> > Current 'next'-build is running with:
>> > 
>> > 'lfs/cdrom':
>> > ...
>> > export XZ_OPT = --threads=0 -8 --memory=500MiB
>> > ...
>> > 
>> > 'lfs/Config':
>> > ...
>> > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8" ...
>> > ...
>> > 
>> > I'm waiting for the results. Takes a while.
>> 
>> You can just run it like this and it will tell you:
>> 
>> [ms(a)hughes ~]$ xz -vv8 -T0
>> xz: Filter chain: --
>> lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
>> xz: Using up to 4 threads.
>> xz: 2,629 MiB of memory is required. The limiter is disabled.
>> xz: Decompression will need 33 MiB of memory.
>> xz: Compressed data cannot be written to a terminal
>> xz: Try `xz --help' for more information.
>> 
>> > > > To not run into memory limits we should detect how much memory the build
>> > > > host
>> > > > has and set the memory limit to maybe 80% of that. xz will then
>> > > > automatically
>> > > > scale down to a number of parallel processes that it can fit into
>> > > > memory.

This is something I couldn't get to work. First, I couldn't specify not less
than '700 MiB'. If I specify '600 MiB', building stops and tells me it needs
at least about 680 MiB. If I use '70%', building stops. And I never saw that
'xz' "automatically scaled down". It just stopped everytime.
 
>> > I'm a bit puzzled about this: I tried using '--memory=70%' as mentioned
>> > in the xz man pages, but is seems to have no effect. It looks as if the
>> > percent parameter isn't recognized and ignored. It always ends up in
>> > "cannot allocate memory". Any hints on this? Do both 'lfs/cdrom' AND
>> > 'lfs/Config' need the '--memory'-parameter?

This was the next problem: 'percent'-parameter does not work as expected.

>> Yes, but it would be good to have one place where the parameters are
>> being generated and then we just use them from a variable.

Probably beyond my capabilities for the moment. Any thoughts?

>> > > > ...
>> > > > We will at least need about half a GB which should be a sensible
>> > > > requirement
>> > > > for a build system any ways.

I'm afraid '500 MiB' are not enough...

>> > Agreed. Because of this I'm testing with '500Mib'.
>> > 
>> > > > Will you work on this bringing it into the build system?!
>> > 
>> > "I'll do my very best!" :-))
>> 
>> Very well Miss Sophie...
>> ...

Best™,
Matthias