Re: Compress images with XZ

[Matthias Fischer <matthias.fischer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 11173 bytes --]

Hi,

Small addition:

I found another possibility for calculating XZ memory and adding the
needed suffix:

...
((XZ_MEM=(($HOST_MEM)/10)*7)) && XZ_MEM=${XZ_MEM}MiB
...

Looks better.

Best,
Matthias

On 03.02.2018 20:44, Matthias Fischer wrote:
> Hi,
> 
> On 30.01.2018 21:03, Michael Tremer wrote:
>> Hi,
>> 
>> ...
>>> 
>>> I'll consider how I would do this and send it in as soon as I get a grip
>>> on it.
>> 
>> Well, I added some functions already somewhere. I suppose you just need to do
>> the maths to set that to X% of the host system which is just a simple
>> multiplication of the output of ${HOST_MEM} in make.sh.
> 
> Ok, I take my heart in my hand. Here are my first results - but remember,
> I'm not familiar with this. I built a short test script which looks as if it
> is working as wanted, but...
>  
>> And that needs to be passed to the lfs scripts that are supposed to make the
>> images.
> 
> ...this is a problem I'm struggling with right now:
> How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?
> 
> I think this can be done using 'export XZ_MEM' or something else but I'm not sure.
> 
> This is the script I'm testing with:
> 
> ***SNIP***
> #!/bin/bash
> 
> system_memory() {
> 	local key val unit
> 
> 	while read -r key val unit; do
> 		case "${key}" in
> 			MemTotal:*)
> 				# Convert to MB
> 				echo "$(( ${val} / 1024 ))"
> 				break
> 				;;
> 		esac
> 	done < /proc/meminfo
> }
> 
> HOST_MEM=$(system_memory)
> 
> # Checking host memory
> if [ $HOST_MEM -ge 1024 ]; then
> 	echo Host-Memory: $HOST_MEM'MiB'
> 	echo "Host memory is OK (must be at least 1024MiB), starting build."
> else
> 	echo Host-Memory: $HOST_MEM'MiB'
> 	echo "Host memory too low (less than 1024MiB), building stopped."
> exit 1
> fi
> 
> # Host memory is ok, calculating needed XZ memory to about 70% of host memory
> ((XZ_MEM=(($HOST_MEM)/10)*7))
> 	XZ_MEM=$XZ_MEM'Mib'
> 	echo XZ-Memory: $XZ_MEM
> 	echo "XZ memory size is OK (must be at least 700MiB), starting build."
> ***SNAP***
> 
> The 'echoes' are mostly for debugging.
> 
> Calculation looks ok, 'MiB' must added without blanks (XZ man page: "In most
> places where an integer argument is expected, an optional suffix is supported
> to easily indicate large integers. There must be no space between the integer
> and the suffix.")
> 
>> Please send questions where ever you need help. There is no reason to do this
>> alone.
> 
> Please see above:
> 
> 1. Is this code what you where thinking of? I would leave the other parameters
> as they are (--threads=0 -8).
> 
> 2. Where should I add such code in the first place? I'd like to start with
> 'make.sh', so building stops as soon as possible if there is not enough memory. Where's
> the best place? Right after function 'system_memory() {...}'?
> 
> 3. How to I pass contents of 'XZ_MEM' to 'cdrom' and 'Config'?
> I consider something like:
> ...
> export XZ_OPT = --threads=0 -8 --memory=$XZ_MEM
> ...
> 
> Best,
> Matthias
>  
>> Best,
>> -Michael
>> 
>>> 
>>> Best,
>>> Matthias
>>>  > Best,
>>> > -Michael
>>> > 
>>> > On Wed, 2017-11-22 at 16:37 +0000, Michael Tremer wrote:
>>> > > Hi,
>>> > > 
>>> > > On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer wrote:
>>> > > > Hi,
>>> > > > 
>>> > > > On 21.11.2017 15:42, Michael Tremer wrote:
>>> > > > > Hello,
>>> > > > > 
>>> > > > > any progress on this?
>>> > > > 
>>> > > > Not much, because I've been struggling with a flu for the past two
>>> > > > weeks, and its not over yet, but:
>>> > > > 
>>> > > > In the meantime I ran several builds with different parameters.
>>> > > > 
>>> > > > IMHO we could use:
>>> > > > 
>>> > > > 'export XZ_OPT = --threads=0 -8 --memory=700MiB'
>>> > > > 
>>> > > > in 'lfs/cdrom'
>>> > > > 
>>> > > > and
>>> > > > 
>>> > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8 --memory=700MiB" tar
>>> > > > -c
>>> > > > -p
>>> > > > --numeric-owner -J -f /install/packages/package/files.tar.xz *
>>> > > > 
>>> > > > in 'lfs/Config'.
>>> > > > 
>>> > > > Using '9' or '--extreme'-parameter brought no further improvements,
>>> > > > on the other hand, build time and RAM consumption increased
>>> > > > strongly. With '-8', minimal RAM consumption was about 680 MiB, using
>>> > > > '700 Mib' was always safe. So this is the minimum I would recommend.
>>> > > 
>>> > > Yes, I observed the same thing. -9 compresses a little bit better and
>>> > > extreme
>>> > > doesn't change a bit.
>>> > > 
>>> > > -8 seems to be optimal since it won't require too much memory to
>>> > > decompress
>>> > > either. So let's go with that.
>>> > > 
>>> > > > Results:
>>> > > > 
>>> > > > (Buildtime: 5 hours 20 minutes 59 seconds)
>>> > > > 
>>> > > > 255866764  ipfire-2.19.1gb-ext4.i586-full-core117.img.gz
>>> > > > 252262006  ipfire-2.19.1gb-ext4-scon.i586-full-core117.img.gz
>>> > > > 175112192  ipfire-2.19.i586-full-core117.iso
>>> > > >  
>>> > > > > -Michael
>>> > > > > 
>>> > > > > On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote:
>>> > > > > > Hi,
>>> > > > > > 
>>> > > > > > On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer wrote:
>>> > > > > > > Hi,
>>> > > > > > > 
>>> > > > > > > On 07.11.2017 16:05, Michael Tremer wrote:
>>> > > > > > > > Hi,
>>> > > > > > > > 
>>> > > > > > > > maybe this function helps:
>>> > > > > > > > 
>>> > > > > > > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=5190eea2
>>> > > > > > > > 4f98
>>> > > > > > > > 22
>>> > > > > > > > a63d
>>> > > > > > > > c5d06d214b48f973b14f29
>>> > > > > > > 
>>> > > > > > > Thanks. I'll take a look...
>>> > > > 
>>> > > > This would help, indeed, but how to implement(!?), and... (see below)
>>> > > 
>>> > > Just run it somewhere in the script, figure out what is about 80% of that
>>> > > and
>>> > > if
>>> > > that is smaller than 800 MB, we should just fail.
>>> > > 
>>> > > It would be good to have a function that determines the xz parameters and
>>> > > we
>>> > > use
>>> > > that where ever we call xz. We don't want any duplicated code.
>>> > > 
>>> > > > > > > > On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote:
>>> > > > > > > > > On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer wrote:
>>> > > > > > > > > ...
>>> > > > > > > > > > 
>>> > > > > > > > > > I forgot: Tuning the XZ-parameters... ;-)
>>> > > > > > > > > 
>>> > > > > > > > > Good that you are bringing this up. So I think what we should
>>> > > > > > > > > do
>>> > > > > > > > > here is
>>> > > > > > > > > bringing back XZ compression with -8 so that we do not waste
>>> > > > > > > > > too
>>> > > > > > > > > much
>>> > > > > > > > > space
>>> > > > > > > > > when
>>> > > > > > > > > extracting the image again.
>>> > > > > > > 
>>> > > > > > > Right now I'm testing how much RAM I need.
>>> > > > > > > 
>>> > > > > > > Current 'next'-build is running with:
>>> > > > > > > 
>>> > > > > > > 'lfs/cdrom':
>>> > > > > > > ...
>>> > > > > > > export XZ_OPT = --threads=0 -8 --memory=500MiB
>>> > > > > > > ...
>>> > > > > > > 
>>> > > > > > > 'lfs/Config':
>>> > > > > > > ...
>>> > > > > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8" ...
>>> > > > > > > ...
>>> > > > > > > 
>>> > > > > > > I'm waiting for the results. Takes a while.
>>> > > > > > 
>>> > > > > > You can just run it like this and it will tell you:
>>> > > > > > 
>>> > > > > > [ms(a)hughes ~]$ xz -vv8 -T0
>>> > > > > > xz: Filter chain: --
>>> > > > > > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
>>> > > > > > xz: Using up to 4 threads.
>>> > > > > > xz: 2,629 MiB of memory is required. The limiter is disabled.
>>> > > > > > xz: Decompression will need 33 MiB of memory.
>>> > > > > > xz: Compressed data cannot be written to a terminal
>>> > > > > > xz: Try `xz --help' for more information.
>>> > > > > > 
>>> > > > > > > > > To not run into memory limits we should detect how much memory
>>> > > > > > > > > the
>>> > > > > > > > > build
>>> > > > > > > > > host
>>> > > > > > > > > has and set the memory limit to maybe 80% of that. xz will
>>> > > > > > > > > then
>>> > > > > > > > > automatically
>>> > > > > > > > > scale down to a number of parallel processes that it can fit
>>> > > > > > > > > into
>>> > > > > > > > > memory.
>>> > > > 
>>> > > > This is something I couldn't get to work. First, I couldn't specify not
>>> > > > less
>>> > > > than '700 MiB'. If I specify '600 MiB', building stops and tells me it
>>> > > > needs
>>> > > > at least about 680 MiB. If I use '70%', building stops. And I never saw
>>> > > > that
>>> > > > 'xz' "automatically scaled down". It just stopped everytime.
>>> > > 
>>> > > Yes, ~700M is the minimum.
>>> > > 
>>> > > It will just reduce the number of threads again to make sure everything
>>> > > fits
>>> > > into memory.
>>> > > 
>>> > > >  
>>> > > > > > > I'm a bit puzzled about this: I tried using '--memory=70%' as
>>> > > > > > > mentioned
>>> > > > > > > in the xz man pages, but is seems to have no effect. It looks as
>>> > > > > > > if
>>> > > > > > > the
>>> > > > > > > percent parameter isn't recognized and ignored. It always ends up
>>> > > > > > > in
>>> > > > > > > "cannot allocate memory". Any hints on this? Do both 'lfs/cdrom'
>>> > > > > > > AND
>>> > > > > > > 'lfs/Config' need the '--memory'-parameter?
>>> > > > 
>>> > > > This was the next problem: 'percent'-parameter does not work as
>>> > > > expected.
>>> > > 
>>> > > No, don't use it. Just determine yourself what it is.
>>> > > 
>>> > > I think we could even try to set 100% of the memory, because as far as I
>>> > > can
>>> > > see
>>> > > this is worst case and xz won't use everything that we allow.
>>> > > 
>>> > > If that does't work, let's scale down to 90%, or 80%, etc...
>>> > > 
>>> > > > > > Yes, but it would be good to have one place where the parameters are
>>> > > > > > being generated and then we just use them from a variable.
>>> > > > 
>>> > > > Probably beyond my capabilities for the moment. Any thoughts?
>>> > > > 
>>> > > > > > > > > ...
>>> > > > > > > > > We will at least need about half a GB which should be a
>>> > > > > > > > > sensible
>>> > > > > > > > > requirement
>>> > > > > > > > > for a build system any ways.
>>> > > > 
>>> > > > I'm afraid '500 MiB' are not enough...
>>> > > 
>>> > > No, so we will now make 1G the minimum for a build.
>>> > > 
>>> > > I tried to build the OS on an ARM board with only 512MB of RAM the other
>>> > > day
>>> > > and
>>> > > even with a lot of swap it doesn't work. So we are past this any way.
>>> > > 
>>> > > > 
>>> > > > > > > Agreed. Because of this I'm testing with '500Mib'.
>>> > > > > > > 
>>> > > > > > > > > Will you work on this bringing it into the build system?!
>>> > > > > > > 
>>> > > > > > > "I'll do my very best!" :-))
>>> > > > > > 
>>> > > > > > Very well Miss Sophie...
>>> > > > > > ...
>>> > > > 
>>> > > > Best™,
>>> > > > Matthias
>>> > > 
>>> > > -Michael
>> 
> 
>