Compress images with XZ (was: Planning out Core Update 117)

[Michael Tremer <michael.tremer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 6625 bytes --]

Hello Matthias,

any update on this? Are you still on this?

Best,
-Michael

On Wed, 2017-11-22 at 16:37 +0000, Michael Tremer wrote:
> Hi,
> 
> On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer wrote:
> > Hi,
> > 
> > On 21.11.2017 15:42, Michael Tremer wrote:
> > > Hello,
> > > 
> > > any progress on this?
> > 
> > Not much, because I've been struggling with a flu for the past two
> > weeks, and its not over yet, but:
> > 
> > In the meantime I ran several builds with different parameters.
> > 
> > IMHO we could use:
> > 
> > 'export XZ_OPT = --threads=0 -8 --memory=700MiB'
> > 
> > in 'lfs/cdrom'
> > 
> > and
> > 
> > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8 --memory=700MiB" tar -c
> > -p
> > --numeric-owner -J -f /install/packages/package/files.tar.xz *
> > 
> > in 'lfs/Config'.
> > 
> > Using '9' or '--extreme'-parameter brought no further improvements,
> > on the other hand, build time and RAM consumption increased
> > strongly. With '-8', minimal RAM consumption was about 680 MiB, using
> > '700 Mib' was always safe. So this is the minimum I would recommend.
> 
> Yes, I observed the same thing. -9 compresses a little bit better and extreme
> doesn't change a bit.
> 
> -8 seems to be optimal since it won't require too much memory to decompress
> either. So let's go with that.
> 
> > Results:
> > 
> > (Buildtime: 5 hours 20 minutes 59 seconds)
> > 
> > 255866764  ipfire-2.19.1gb-ext4.i586-full-core117.img.gz
> > 252262006  ipfire-2.19.1gb-ext4-scon.i586-full-core117.img.gz
> > 175112192  ipfire-2.19.i586-full-core117.iso
> >  
> > > -Michael
> > > 
> > > On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote:
> > > > Hi,
> > > > 
> > > > On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer wrote:
> > > > > Hi,
> > > > > 
> > > > > On 07.11.2017 16:05, Michael Tremer wrote:
> > > > > > Hi,
> > > > > > 
> > > > > > maybe this function helps:
> > > > > > 
> > > > > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=5190eea24f98
> > > > > > 22
> > > > > > a63d
> > > > > > c5d06d214b48f973b14f29
> > > > > 
> > > > > Thanks. I'll take a look...
> > 
> > This would help, indeed, but how to implement(!?), and... (see below)
> 
> Just run it somewhere in the script, figure out what is about 80% of that and
> if
> that is smaller than 800 MB, we should just fail.
> 
> It would be good to have a function that determines the xz parameters and we
> use
> that where ever we call xz. We don't want any duplicated code.
> 
> > > > > > On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote:
> > > > > > > On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer wrote:
> > > > > > > ...
> > > > > > > > 
> > > > > > > > I forgot: Tuning the XZ-parameters... ;-)
> > > > > > > 
> > > > > > > Good that you are bringing this up. So I think what we should do
> > > > > > > here is
> > > > > > > bringing back XZ compression with -8 so that we do not waste too
> > > > > > > much
> > > > > > > space
> > > > > > > when
> > > > > > > extracting the image again.
> > > > > 
> > > > > Right now I'm testing how much RAM I need.
> > > > > 
> > > > > Current 'next'-build is running with:
> > > > > 
> > > > > 'lfs/cdrom':
> > > > > ...
> > > > > export XZ_OPT = --threads=0 -8 --memory=500MiB
> > > > > ...
> > > > > 
> > > > > 'lfs/Config':
> > > > > ...
> > > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8" ...
> > > > > ...
> > > > > 
> > > > > I'm waiting for the results. Takes a while.
> > > > 
> > > > You can just run it like this and it will tell you:
> > > > 
> > > > [ms(a)hughes ~]$ xz -vv8 -T0
> > > > xz: Filter chain: --
> > > > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
> > > > xz: Using up to 4 threads.
> > > > xz: 2,629 MiB of memory is required. The limiter is disabled.
> > > > xz: Decompression will need 33 MiB of memory.
> > > > xz: Compressed data cannot be written to a terminal
> > > > xz: Try `xz --help' for more information.
> > > > 
> > > > > > > To not run into memory limits we should detect how much memory the
> > > > > > > build
> > > > > > > host
> > > > > > > has and set the memory limit to maybe 80% of that. xz will then
> > > > > > > automatically
> > > > > > > scale down to a number of parallel processes that it can fit into
> > > > > > > memory.
> > 
> > This is something I couldn't get to work. First, I couldn't specify not less
> > than '700 MiB'. If I specify '600 MiB', building stops and tells me it needs
> > at least about 680 MiB. If I use '70%', building stops. And I never saw that
> > 'xz' "automatically scaled down". It just stopped everytime.
> 
> Yes, ~700M is the minimum.
> 
> It will just reduce the number of threads again to make sure everything fits
> into memory.
> 
> >  
> > > > > I'm a bit puzzled about this: I tried using '--memory=70%' as
> > > > > mentioned
> > > > > in the xz man pages, but is seems to have no effect. It looks as if
> > > > > the
> > > > > percent parameter isn't recognized and ignored. It always ends up in
> > > > > "cannot allocate memory". Any hints on this? Do both 'lfs/cdrom' AND
> > > > > 'lfs/Config' need the '--memory'-parameter?
> > 
> > This was the next problem: 'percent'-parameter does not work as expected.
> 
> No, don't use it. Just determine yourself what it is.
> 
> I think we could even try to set 100% of the memory, because as far as I can
> see
> this is worst case and xz won't use everything that we allow.
> 
> If that does't work, let's scale down to 90%, or 80%, etc...
> 
> > > > Yes, but it would be good to have one place where the parameters are
> > > > being generated and then we just use them from a variable.
> > 
> > Probably beyond my capabilities for the moment. Any thoughts?
> > 
> > > > > > > ...
> > > > > > > We will at least need about half a GB which should be a sensible
> > > > > > > requirement
> > > > > > > for a build system any ways.
> > 
> > I'm afraid '500 MiB' are not enough...
> 
> No, so we will now make 1G the minimum for a build.
> 
> I tried to build the OS on an ARM board with only 512MB of RAM the other day
> and
> even with a lot of swap it doesn't work. So we are past this any way.
> 
> > 
> > > > > Agreed. Because of this I'm testing with '500Mib'.
> > > > > 
> > > > > > > Will you work on this bringing it into the build system?!
> > > > > 
> > > > > "I'll do my very best!" :-))
> > > > 
> > > > Very well Miss Sophie...
> > > > ...
> > 
> > Best™,
> > Matthias
> 
> -Michael

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]