Re: Compress images with XZ

[Matthias Fischer <matthias.fischer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 10512 bytes --]

Hi,

On 30.01.2018 21:03, Michael Tremer wrote:
> Hi,
> 
> ...
>> 
>> I'll consider how I would do this and send it in as soon as I get a grip
>> on it.
> 
> Well, I added some functions already somewhere. I suppose you just need to do
> the maths to set that to X% of the host system which is just a simple
> multiplication of the output of ${HOST_MEM} in make.sh.

Ok, I take my heart in my hand. Here are my first results - but remember,
I'm not familiar with this. I built a short test script which looks as if it
is working as wanted, but...
 
> And that needs to be passed to the lfs scripts that are supposed to make the
> images.

...this is a problem I'm struggling with right now:
How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?

I think this can be done using 'export XZ_MEM' or something else but I'm not sure.

This is the script I'm testing with:

***SNIP***
#!/bin/bash

system_memory() {
	local key val unit

	while read -r key val unit; do
		case "${key}" in
			MemTotal:*)
				# Convert to MB
				echo "$(( ${val} / 1024 ))"
				break
				;;
		esac
	done < /proc/meminfo
}

HOST_MEM=$(system_memory)

# Checking host memory
if [ $HOST_MEM -ge 1024 ]; then
	echo Host-Memory: $HOST_MEM'MiB'
	echo "Host memory is OK (must be at least 1024MiB), starting build."
else
	echo Host-Memory: $HOST_MEM'MiB'
	echo "Host memory too low (less than 1024MiB), building stopped."
exit 1
fi

# Host memory is ok, calculating needed XZ memory to about 70% of host memory
((XZ_MEM=(($HOST_MEM)/10)*7))
	XZ_MEM=$XZ_MEM'Mib'
	echo XZ-Memory: $XZ_MEM
	echo "XZ memory size is OK (must be at least 700MiB), starting build."
***SNAP***

The 'echoes' are mostly for debugging.

Calculation looks ok, 'MiB' must added without blanks (XZ man page: "In most
places where an integer argument is expected, an optional suffix is supported
to easily indicate large integers. There must be no space between the integer
and the suffix.")

> Please send questions where ever you need help. There is no reason to do this
> alone.

Please see above:

1. Is this code what you where thinking of? I would leave the other parameters
as they are (--threads=0 -8).

2. Where should I add such code in the first place? I'd like to start with
'make.sh', so building stops as soon as possible if there is not enough memory. Where's
the best place? Right after function 'system_memory() {...}'?

3. How to I pass contents of 'XZ_MEM' to 'cdrom' and 'Config'?
I consider something like:
...
export XZ_OPT = --threads=0 -8 --memory=$XZ_MEM
...

Best,
Matthias
 
> Best,
> -Michael
> 
>> 
>> Best,
>> Matthias
>>  > Best,
>> > -Michael
>> > 
>> > On Wed, 2017-11-22 at 16:37 +0000, Michael Tremer wrote:
>> > > Hi,
>> > > 
>> > > On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer wrote:
>> > > > Hi,
>> > > > 
>> > > > On 21.11.2017 15:42, Michael Tremer wrote:
>> > > > > Hello,
>> > > > > 
>> > > > > any progress on this?
>> > > > 
>> > > > Not much, because I've been struggling with a flu for the past two
>> > > > weeks, and its not over yet, but:
>> > > > 
>> > > > In the meantime I ran several builds with different parameters.
>> > > > 
>> > > > IMHO we could use:
>> > > > 
>> > > > 'export XZ_OPT = --threads=0 -8 --memory=700MiB'
>> > > > 
>> > > > in 'lfs/cdrom'
>> > > > 
>> > > > and
>> > > > 
>> > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8 --memory=700MiB" tar
>> > > > -c
>> > > > -p
>> > > > --numeric-owner -J -f /install/packages/package/files.tar.xz *
>> > > > 
>> > > > in 'lfs/Config'.
>> > > > 
>> > > > Using '9' or '--extreme'-parameter brought no further improvements,
>> > > > on the other hand, build time and RAM consumption increased
>> > > > strongly. With '-8', minimal RAM consumption was about 680 MiB, using
>> > > > '700 Mib' was always safe. So this is the minimum I would recommend.
>> > > 
>> > > Yes, I observed the same thing. -9 compresses a little bit better and
>> > > extreme
>> > > doesn't change a bit.
>> > > 
>> > > -8 seems to be optimal since it won't require too much memory to
>> > > decompress
>> > > either. So let's go with that.
>> > > 
>> > > > Results:
>> > > > 
>> > > > (Buildtime: 5 hours 20 minutes 59 seconds)
>> > > > 
>> > > > 255866764  ipfire-2.19.1gb-ext4.i586-full-core117.img.gz
>> > > > 252262006  ipfire-2.19.1gb-ext4-scon.i586-full-core117.img.gz
>> > > > 175112192  ipfire-2.19.i586-full-core117.iso
>> > > >  
>> > > > > -Michael
>> > > > > 
>> > > > > On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote:
>> > > > > > Hi,
>> > > > > > 
>> > > > > > On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer wrote:
>> > > > > > > Hi,
>> > > > > > > 
>> > > > > > > On 07.11.2017 16:05, Michael Tremer wrote:
>> > > > > > > > Hi,
>> > > > > > > > 
>> > > > > > > > maybe this function helps:
>> > > > > > > > 
>> > > > > > > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=5190eea2
>> > > > > > > > 4f98
>> > > > > > > > 22
>> > > > > > > > a63d
>> > > > > > > > c5d06d214b48f973b14f29
>> > > > > > > 
>> > > > > > > Thanks. I'll take a look...
>> > > > 
>> > > > This would help, indeed, but how to implement(!?), and... (see below)
>> > > 
>> > > Just run it somewhere in the script, figure out what is about 80% of that
>> > > and
>> > > if
>> > > that is smaller than 800 MB, we should just fail.
>> > > 
>> > > It would be good to have a function that determines the xz parameters and
>> > > we
>> > > use
>> > > that where ever we call xz. We don't want any duplicated code.
>> > > 
>> > > > > > > > On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote:
>> > > > > > > > > On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer wrote:
>> > > > > > > > > ...
>> > > > > > > > > > 
>> > > > > > > > > > I forgot: Tuning the XZ-parameters... ;-)
>> > > > > > > > > 
>> > > > > > > > > Good that you are bringing this up. So I think what we should
>> > > > > > > > > do
>> > > > > > > > > here is
>> > > > > > > > > bringing back XZ compression with -8 so that we do not waste
>> > > > > > > > > too
>> > > > > > > > > much
>> > > > > > > > > space
>> > > > > > > > > when
>> > > > > > > > > extracting the image again.
>> > > > > > > 
>> > > > > > > Right now I'm testing how much RAM I need.
>> > > > > > > 
>> > > > > > > Current 'next'-build is running with:
>> > > > > > > 
>> > > > > > > 'lfs/cdrom':
>> > > > > > > ...
>> > > > > > > export XZ_OPT = --threads=0 -8 --memory=500MiB
>> > > > > > > ...
>> > > > > > > 
>> > > > > > > 'lfs/Config':
>> > > > > > > ...
>> > > > > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8" ...
>> > > > > > > ...
>> > > > > > > 
>> > > > > > > I'm waiting for the results. Takes a while.
>> > > > > > 
>> > > > > > You can just run it like this and it will tell you:
>> > > > > > 
>> > > > > > [ms(a)hughes ~]$ xz -vv8 -T0
>> > > > > > xz: Filter chain: --
>> > > > > > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
>> > > > > > xz: Using up to 4 threads.
>> > > > > > xz: 2,629 MiB of memory is required. The limiter is disabled.
>> > > > > > xz: Decompression will need 33 MiB of memory.
>> > > > > > xz: Compressed data cannot be written to a terminal
>> > > > > > xz: Try `xz --help' for more information.
>> > > > > > 
>> > > > > > > > > To not run into memory limits we should detect how much memory
>> > > > > > > > > the
>> > > > > > > > > build
>> > > > > > > > > host
>> > > > > > > > > has and set the memory limit to maybe 80% of that. xz will
>> > > > > > > > > then
>> > > > > > > > > automatically
>> > > > > > > > > scale down to a number of parallel processes that it can fit
>> > > > > > > > > into
>> > > > > > > > > memory.
>> > > > 
>> > > > This is something I couldn't get to work. First, I couldn't specify not
>> > > > less
>> > > > than '700 MiB'. If I specify '600 MiB', building stops and tells me it
>> > > > needs
>> > > > at least about 680 MiB. If I use '70%', building stops. And I never saw
>> > > > that
>> > > > 'xz' "automatically scaled down". It just stopped everytime.
>> > > 
>> > > Yes, ~700M is the minimum.
>> > > 
>> > > It will just reduce the number of threads again to make sure everything
>> > > fits
>> > > into memory.
>> > > 
>> > > >  
>> > > > > > > I'm a bit puzzled about this: I tried using '--memory=70%' as
>> > > > > > > mentioned
>> > > > > > > in the xz man pages, but is seems to have no effect. It looks as
>> > > > > > > if
>> > > > > > > the
>> > > > > > > percent parameter isn't recognized and ignored. It always ends up
>> > > > > > > in
>> > > > > > > "cannot allocate memory". Any hints on this? Do both 'lfs/cdrom'
>> > > > > > > AND
>> > > > > > > 'lfs/Config' need the '--memory'-parameter?
>> > > > 
>> > > > This was the next problem: 'percent'-parameter does not work as
>> > > > expected.
>> > > 
>> > > No, don't use it. Just determine yourself what it is.
>> > > 
>> > > I think we could even try to set 100% of the memory, because as far as I
>> > > can
>> > > see
>> > > this is worst case and xz won't use everything that we allow.
>> > > 
>> > > If that does't work, let's scale down to 90%, or 80%, etc...
>> > > 
>> > > > > > Yes, but it would be good to have one place where the parameters are
>> > > > > > being generated and then we just use them from a variable.
>> > > > 
>> > > > Probably beyond my capabilities for the moment. Any thoughts?
>> > > > 
>> > > > > > > > > ...
>> > > > > > > > > We will at least need about half a GB which should be a
>> > > > > > > > > sensible
>> > > > > > > > > requirement
>> > > > > > > > > for a build system any ways.
>> > > > 
>> > > > I'm afraid '500 MiB' are not enough...
>> > > 
>> > > No, so we will now make 1G the minimum for a build.
>> > > 
>> > > I tried to build the OS on an ARM board with only 512MB of RAM the other
>> > > day
>> > > and
>> > > even with a lot of swap it doesn't work. So we are past this any way.
>> > > 
>> > > > 
>> > > > > > > Agreed. Because of this I'm testing with '500Mib'.
>> > > > > > > 
>> > > > > > > > > Will you work on this bringing it into the build system?!
>> > > > > > > 
>> > > > > > > "I'll do my very best!" :-))
>> > > > > > 
>> > > > > > Very well Miss Sophie...
>> > > > > > ...
>> > > > 
>> > > > Best™,
>> > > > Matthias
>> > > 
>> > > -Michael
>