Re: Compress images with XZ

[Michael Tremer <michael.tremer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 13865 bytes --]

Hi,

On Sat, 2018-02-03 at 22:09 +0100, Matthias Fischer wrote:
> Hi,
> 
> Small addition:
> 
> I found another possibility for calculating XZ memory and adding the
> needed suffix:
> 
> ...
> ((XZ_MEM=(($HOST_MEM)/10)*7)) && XZ_MEM=${XZ_MEM}MiB

You can write this even short:

XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"

I swapped the order of the multiplication and division so that you would have a
more exact result.

For example if you system has 1024 MB or RAM, you divide this by 10 (102) and
then multiply by 7 which gives you 714. The other order is 1024 * 7 = 7167 and
then divided by 10 is 716 which is closer to the actual result of 716.8.
Shouldn't matter here but just in general this is better :)

You can put this probably next under "HOST_MEM=$(system_memory)" and then export
this to the internal shell by adding the variable to lfsmake2 for the disk image
and ipfiredist for the packages. Then in lfs/cdrom and lfs/Config for the dist
target, you just need to add the parameters to the command line.

Maybe it is a good idea to have a variable XZ_OPT that adds everything together
and is then used in both places.

Best,
-Michael

> ...
> 
> Looks better.
> 
> Best,
> Matthias
> 
> On 03.02.2018 20:44, Matthias Fischer wrote:
> > Hi,
> > 
> > On 30.01.2018 21:03, Michael Tremer wrote:
> > > Hi,
> > > 
> > > ...
> > > > 
> > > > I'll consider how I would do this and send it in as soon as I get a grip
> > > > on it.
> > > 
> > > Well, I added some functions already somewhere. I suppose you just need to
> > > do
> > > the maths to set that to X% of the host system which is just a simple
> > > multiplication of the output of ${HOST_MEM} in make.sh.
> > 
> > Ok, I take my heart in my hand. Here are my first results - but remember,
> > I'm not familiar with this. I built a short test script which looks as if it
> > is working as wanted, but...
> >  
> > > And that needs to be passed to the lfs scripts that are supposed to make
> > > the
> > > images.
> > 
> > ...this is a problem I'm struggling with right now:
> > How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?
> > 
> > I think this can be done using 'export XZ_MEM' or something else but I'm not
> > sure.
> > 
> > This is the script I'm testing with:
> > 
> > ***SNIP***
> > #!/bin/bash
> > 
> > system_memory() {
> > 	local key val unit
> > 
> > 	while read -r key val unit; do
> > 		case "${key}" in
> > 			MemTotal:*)
> > 				# Convert to MB
> > 				echo "$(( ${val} / 1024 ))"
> > 				break
> > 				;;
> > 		esac
> > 	done < /proc/meminfo
> > }
> > 
> > HOST_MEM=$(system_memory)
> > 
> > # Checking host memory
> > if [ $HOST_MEM -ge 1024 ]; then
> > 	echo Host-Memory: $HOST_MEM'MiB'
> > 	echo "Host memory is OK (must be at least 1024MiB), starting build."
> > else
> > 	echo Host-Memory: $HOST_MEM'MiB'
> > 	echo "Host memory too low (less than 1024MiB), building stopped."
> > exit 1
> > fi
> > 
> > # Host memory is ok, calculating needed XZ memory to about 70% of host
> > memory
> > ((XZ_MEM=(($HOST_MEM)/10)*7))
> > 	XZ_MEM=$XZ_MEM'Mib'
> > 	echo XZ-Memory: $XZ_MEM
> > 	echo "XZ memory size is OK (must be at least 700MiB), starting build."
> > ***SNAP***
> > 
> > The 'echoes' are mostly for debugging.
> > 
> > Calculation looks ok, 'MiB' must added without blanks (XZ man page: "In most
> > places where an integer argument is expected, an optional suffix is
> > supported
> > to easily indicate large integers. There must be no space between the
> > integer
> > and the suffix.")
> > 
> > > Please send questions where ever you need help. There is no reason to do
> > > this
> > > alone.
> > 
> > Please see above:
> > 
> > 1. Is this code what you where thinking of? I would leave the other
> > parameters
> > as they are (--threads=0 -8).
> > 
> > 2. Where should I add such code in the first place? I'd like to start with
> > 'make.sh', so building stops as soon as possible if there is not enough
> > memory. Where's
> > the best place? Right after function 'system_memory() {...}'?
> > 
> > 3. How to I pass contents of 'XZ_MEM' to 'cdrom' and 'Config'?
> > I consider something like:
> > ...
> > export XZ_OPT = --threads=0 -8 --memory=$XZ_MEM
> > ...
> > 
> > Best,
> > Matthias
> >  
> > > Best,
> > > -Michael
> > > 
> > > > 
> > > > Best,
> > > > Matthias
> > > >  > Best,
> > > > > -Michael
> > > > > 
> > > > > On Wed, 2017-11-22 at 16:37 +0000, Michael Tremer wrote:
> > > > > > Hi,
> > > > > > 
> > > > > > On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer wrote:
> > > > > > > Hi,
> > > > > > > 
> > > > > > > On 21.11.2017 15:42, Michael Tremer wrote:
> > > > > > > > Hello,
> > > > > > > > 
> > > > > > > > any progress on this?
> > > > > > > 
> > > > > > > Not much, because I've been struggling with a flu for the past two
> > > > > > > weeks, and its not over yet, but:
> > > > > > > 
> > > > > > > In the meantime I ran several builds with different parameters.
> > > > > > > 
> > > > > > > IMHO we could use:
> > > > > > > 
> > > > > > > 'export XZ_OPT = --threads=0 -8 --memory=700MiB'
> > > > > > > 
> > > > > > > in 'lfs/cdrom'
> > > > > > > 
> > > > > > > and
> > > > > > > 
> > > > > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8 --
> > > > > > > memory=700MiB" tar
> > > > > > > -c
> > > > > > > -p
> > > > > > > --numeric-owner -J -f /install/packages/package/files.tar.xz *
> > > > > > > 
> > > > > > > in 'lfs/Config'.
> > > > > > > 
> > > > > > > Using '9' or '--extreme'-parameter brought no further
> > > > > > > improvements,
> > > > > > > on the other hand, build time and RAM consumption increased
> > > > > > > strongly. With '-8', minimal RAM consumption was about 680 MiB,
> > > > > > > using
> > > > > > > '700 Mib' was always safe. So this is the minimum I would
> > > > > > > recommend.
> > > > > > 
> > > > > > Yes, I observed the same thing. -9 compresses a little bit better
> > > > > > and
> > > > > > extreme
> > > > > > doesn't change a bit.
> > > > > > 
> > > > > > -8 seems to be optimal since it won't require too much memory to
> > > > > > decompress
> > > > > > either. So let's go with that.
> > > > > > 
> > > > > > > Results:
> > > > > > > 
> > > > > > > (Buildtime: 5 hours 20 minutes 59 seconds)
> > > > > > > 
> > > > > > > 255866764  ipfire-2.19.1gb-ext4.i586-full-core117.img.gz
> > > > > > > 252262006  ipfire-2.19.1gb-ext4-scon.i586-full-core117.img.gz
> > > > > > > 175112192  ipfire-2.19.i586-full-core117.iso
> > > > > > >  
> > > > > > > > -Michael
> > > > > > > > 
> > > > > > > > On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote:
> > > > > > > > > Hi,
> > > > > > > > > 
> > > > > > > > > On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer wrote:
> > > > > > > > > > Hi,
> > > > > > > > > > 
> > > > > > > > > > On 07.11.2017 16:05, Michael Tremer wrote:
> > > > > > > > > > > Hi,
> > > > > > > > > > > 
> > > > > > > > > > > maybe this function helps:
> > > > > > > > > > > 
> > > > > > > > > > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=51
> > > > > > > > > > > 90eea2
> > > > > > > > > > > 4f98
> > > > > > > > > > > 22
> > > > > > > > > > > a63d
> > > > > > > > > > > c5d06d214b48f973b14f29
> > > > > > > > > > 
> > > > > > > > > > Thanks. I'll take a look...
> > > > > > > 
> > > > > > > This would help, indeed, but how to implement(!?), and... (see
> > > > > > > below)
> > > > > > 
> > > > > > Just run it somewhere in the script, figure out what is about 80% of
> > > > > > that
> > > > > > and
> > > > > > if
> > > > > > that is smaller than 800 MB, we should just fail.
> > > > > > 
> > > > > > It would be good to have a function that determines the xz
> > > > > > parameters and
> > > > > > we
> > > > > > use
> > > > > > that where ever we call xz. We don't want any duplicated code.
> > > > > > 
> > > > > > > > > > > On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote:
> > > > > > > > > > > > On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > ...
> > > > > > > > > > > > > 
> > > > > > > > > > > > > I forgot: Tuning the XZ-parameters... ;-)
> > > > > > > > > > > > 
> > > > > > > > > > > > Good that you are bringing this up. So I think what we
> > > > > > > > > > > > should
> > > > > > > > > > > > do
> > > > > > > > > > > > here is
> > > > > > > > > > > > bringing back XZ compression with -8 so that we do not
> > > > > > > > > > > > waste
> > > > > > > > > > > > too
> > > > > > > > > > > > much
> > > > > > > > > > > > space
> > > > > > > > > > > > when
> > > > > > > > > > > > extracting the image again.
> > > > > > > > > > 
> > > > > > > > > > Right now I'm testing how much RAM I need.
> > > > > > > > > > 
> > > > > > > > > > Current 'next'-build is running with:
> > > > > > > > > > 
> > > > > > > > > > 'lfs/cdrom':
> > > > > > > > > > ...
> > > > > > > > > > export XZ_OPT = --threads=0 -8 --memory=500MiB
> > > > > > > > > > ...
> > > > > > > > > > 
> > > > > > > > > > 'lfs/Config':
> > > > > > > > > > ...
> > > > > > > > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8" ...
> > > > > > > > > > ...
> > > > > > > > > > 
> > > > > > > > > > I'm waiting for the results. Takes a while.
> > > > > > > > > 
> > > > > > > > > You can just run it like this and it will tell you:
> > > > > > > > > 
> > > > > > > > > [ms(a)hughes ~]$ xz -vv8 -T0
> > > > > > > > > xz: Filter chain: --
> > > > > > > > > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,dep
> > > > > > > > > th=0
> > > > > > > > > xz: Using up to 4 threads.
> > > > > > > > > xz: 2,629 MiB of memory is required. The limiter is disabled.
> > > > > > > > > xz: Decompression will need 33 MiB of memory.
> > > > > > > > > xz: Compressed data cannot be written to a terminal
> > > > > > > > > xz: Try `xz --help' for more information.
> > > > > > > > > 
> > > > > > > > > > > > To not run into memory limits we should detect how much
> > > > > > > > > > > > memory
> > > > > > > > > > > > the
> > > > > > > > > > > > build
> > > > > > > > > > > > host
> > > > > > > > > > > > has and set the memory limit to maybe 80% of that. xz
> > > > > > > > > > > > will
> > > > > > > > > > > > then
> > > > > > > > > > > > automatically
> > > > > > > > > > > > scale down to a number of parallel processes that it can
> > > > > > > > > > > > fit
> > > > > > > > > > > > into
> > > > > > > > > > > > memory.
> > > > > > > 
> > > > > > > This is something I couldn't get to work. First, I couldn't
> > > > > > > specify not
> > > > > > > less
> > > > > > > than '700 MiB'. If I specify '600 MiB', building stops and tells
> > > > > > > me it
> > > > > > > needs
> > > > > > > at least about 680 MiB. If I use '70%', building stops. And I
> > > > > > > never saw
> > > > > > > that
> > > > > > > 'xz' "automatically scaled down". It just stopped everytime.
> > > > > > 
> > > > > > Yes, ~700M is the minimum.
> > > > > > 
> > > > > > It will just reduce the number of threads again to make sure
> > > > > > everything
> > > > > > fits
> > > > > > into memory.
> > > > > > 
> > > > > > >  
> > > > > > > > > > I'm a bit puzzled about this: I tried using '--memory=70%'
> > > > > > > > > > as
> > > > > > > > > > mentioned
> > > > > > > > > > in the xz man pages, but is seems to have no effect. It
> > > > > > > > > > looks as
> > > > > > > > > > if
> > > > > > > > > > the
> > > > > > > > > > percent parameter isn't recognized and ignored. It always
> > > > > > > > > > ends up
> > > > > > > > > > in
> > > > > > > > > > "cannot allocate memory". Any hints on this? Do both
> > > > > > > > > > 'lfs/cdrom'
> > > > > > > > > > AND
> > > > > > > > > > 'lfs/Config' need the '--memory'-parameter?
> > > > > > > 
> > > > > > > This was the next problem: 'percent'-parameter does not work as
> > > > > > > expected.
> > > > > > 
> > > > > > No, don't use it. Just determine yourself what it is.
> > > > > > 
> > > > > > I think we could even try to set 100% of the memory, because as far
> > > > > > as I
> > > > > > can
> > > > > > see
> > > > > > this is worst case and xz won't use everything that we allow.
> > > > > > 
> > > > > > If that does't work, let's scale down to 90%, or 80%, etc...
> > > > > > 
> > > > > > > > > Yes, but it would be good to have one place where the
> > > > > > > > > parameters are
> > > > > > > > > being generated and then we just use them from a variable.
> > > > > > > 
> > > > > > > Probably beyond my capabilities for the moment. Any thoughts?
> > > > > > > 
> > > > > > > > > > > > ...
> > > > > > > > > > > > We will at least need about half a GB which should be a
> > > > > > > > > > > > sensible
> > > > > > > > > > > > requirement
> > > > > > > > > > > > for a build system any ways.
> > > > > > > 
> > > > > > > I'm afraid '500 MiB' are not enough...
> > > > > > 
> > > > > > No, so we will now make 1G the minimum for a build.
> > > > > > 
> > > > > > I tried to build the OS on an ARM board with only 512MB of RAM the
> > > > > > other
> > > > > > day
> > > > > > and
> > > > > > even with a lot of swap it doesn't work. So we are past this any
> > > > > > way.
> > > > > > 
> > > > > > > 
> > > > > > > > > > Agreed. Because of this I'm testing with '500Mib'.
> > > > > > > > > > 
> > > > > > > > > > > > Will you work on this bringing it into the build
> > > > > > > > > > > > system?!
> > > > > > > > > > 
> > > > > > > > > > "I'll do my very best!" :-))
> > > > > > > > > 
> > > > > > > > > Very well Miss Sophie...
> > > > > > > > > ...
> > > > > > > 
> > > > > > > Best™,
> > > > > > > Matthias
> > > > > > 
> > > > > > -Michael
> > 
> > 
> 
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]