From mboxrd@z Thu Jan 1 00:00:00 1970 From: Matthias Fischer To: development@lists.ipfire.org Subject: Re: Compress images with XZ Date: Sat, 03 Feb 2018 22:09:18 +0100 Message-ID: <06f38cee-e77c-2af2-f5be-90f68f7f2f21@ipfire.org> In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============3540988662854236501==" List-Id: --===============3540988662854236501== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi, Small addition: I found another possibility for calculating XZ memory and adding the needed suffix: ... ((XZ_MEM=3D(($HOST_MEM)/10)*7)) && XZ_MEM=3D${XZ_MEM}MiB ... Looks better. Best, Matthias On 03.02.2018 20:44, Matthias Fischer wrote: > Hi, >=20 > On 30.01.2018 21:03, Michael Tremer wrote: >> Hi, >>=20 >> ... >>>=20 >>> I'll consider how I would do this and send it in as soon as I get a grip >>> on it. >>=20 >> Well, I added some functions already somewhere. I suppose you just need to= do >> the maths to set that to X% of the host system which is just a simple >> multiplication of the output of ${HOST_MEM} in make.sh. >=20 > Ok, I take my heart in my hand. Here are my first results - but remember, > I'm not familiar with this. I built a short test script which looks as if it > is working as wanted, but... > =20 >> And that needs to be passed to the lfs scripts that are supposed to make t= he >> images. >=20 > ...this is a problem I'm struggling with right now: > How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'? >=20 > I think this can be done using 'export XZ_MEM' or something else but I'm no= t sure. >=20 > This is the script I'm testing with: >=20 > ***SNIP*** > #!/bin/bash >=20 > system_memory() { > local key val unit >=20 > while read -r key val unit; do > case "${key}" in > MemTotal:*) > # Convert to MB > echo "$(( ${val} / 1024 ))" > break > ;; > esac > done < /proc/meminfo > } >=20 > HOST_MEM=3D$(system_memory) >=20 > # Checking host memory > if [ $HOST_MEM -ge 1024 ]; then > echo Host-Memory: $HOST_MEM'MiB' > echo "Host memory is OK (must be at least 1024MiB), starting build." > else > echo Host-Memory: $HOST_MEM'MiB' > echo "Host memory too low (less than 1024MiB), building stopped." > exit 1 > fi >=20 > # Host memory is ok, calculating needed XZ memory to about 70% of host memo= ry > ((XZ_MEM=3D(($HOST_MEM)/10)*7)) > XZ_MEM=3D$XZ_MEM'Mib' > echo XZ-Memory: $XZ_MEM > echo "XZ memory size is OK (must be at least 700MiB), starting build." > ***SNAP*** >=20 > The 'echoes' are mostly for debugging. >=20 > Calculation looks ok, 'MiB' must added without blanks (XZ man page: "In most > places where an integer argument is expected, an optional suffix is support= ed > to easily indicate large integers. There must be no space between the integ= er > and the suffix.") >=20 >> Please send questions where ever you need help. There is no reason to do t= his >> alone. >=20 > Please see above: >=20 > 1. Is this code what you where thinking of? I would leave the other paramet= ers > as they are (--threads=3D0 -8). >=20 > 2. Where should I add such code in the first place? I'd like to start with > 'make.sh', so building stops as soon as possible if there is not enough mem= ory. Where's > the best place? Right after function 'system_memory() {...}'? >=20 > 3. How to I pass contents of 'XZ_MEM' to 'cdrom' and 'Config'? > I consider something like: > ... > export XZ_OPT =3D --threads=3D0 -8 --memory=3D$XZ_MEM > ... >=20 > Best, > Matthias > =20 >> Best, >> -Michael >>=20 >>>=20 >>> Best, >>> Matthias >>> > Best, >>> > -Michael >>> >=20 >>> > On Wed, 2017-11-22 at 16:37 +0000, Michael Tremer wrote: >>> > > Hi, >>> > >=20 >>> > > On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer wrote: >>> > > > Hi, >>> > > >=20 >>> > > > On 21.11.2017 15:42, Michael Tremer wrote: >>> > > > > Hello, >>> > > > >=20 >>> > > > > any progress on this? >>> > > >=20 >>> > > > Not much, because I've been struggling with a flu for the past two >>> > > > weeks, and its not over yet, but: >>> > > >=20 >>> > > > In the meantime I ran several builds with different parameters. >>> > > >=20 >>> > > > IMHO we could use: >>> > > >=20 >>> > > > 'export XZ_OPT =3D --threads=3D0 -8 --memory=3D700MiB' >>> > > >=20 >>> > > > in 'lfs/cdrom' >>> > > >=20 >>> > > > and >>> > > >=20 >>> > > > cd /install/packages/package/tmp/ && XZ_OPT=3D"-T0 -8 --memory=3D70= 0MiB" tar >>> > > > -c >>> > > > -p >>> > > > --numeric-owner -J -f /install/packages/package/files.tar.xz * >>> > > >=20 >>> > > > in 'lfs/Config'. >>> > > >=20 >>> > > > Using '9' or '--extreme'-parameter brought no further improvements, >>> > > > on the other hand, build time and RAM consumption increased >>> > > > strongly. With '-8', minimal RAM consumption was about 680 MiB, usi= ng >>> > > > '700 Mib' was always safe. So this is the minimum I would recommend. >>> > >=20 >>> > > Yes, I observed the same thing. -9 compresses a little bit better and >>> > > extreme >>> > > doesn't change a bit. >>> > >=20 >>> > > -8 seems to be optimal since it won't require too much memory to >>> > > decompress >>> > > either. So let's go with that. >>> > >=20 >>> > > > Results: >>> > > >=20 >>> > > > (Buildtime: 5 hours 20 minutes 59 seconds) >>> > > >=20 >>> > > > 255866764 ipfire-2.19.1gb-ext4.i586-full-core117.img.gz >>> > > > 252262006 ipfire-2.19.1gb-ext4-scon.i586-full-core117.img.gz >>> > > > 175112192 ipfire-2.19.i586-full-core117.iso >>> > > > =20 >>> > > > > -Michael >>> > > > >=20 >>> > > > > On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote: >>> > > > > > Hi, >>> > > > > >=20 >>> > > > > > On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer wrote: >>> > > > > > > Hi, >>> > > > > > >=20 >>> > > > > > > On 07.11.2017 16:05, Michael Tremer wrote: >>> > > > > > > > Hi, >>> > > > > > > >=20 >>> > > > > > > > maybe this function helps: >>> > > > > > > >=20 >>> > > > > > > > https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dcommitdiff;h= =3D5190eea2 >>> > > > > > > > 4f98 >>> > > > > > > > 22 >>> > > > > > > > a63d >>> > > > > > > > c5d06d214b48f973b14f29 >>> > > > > > >=20 >>> > > > > > > Thanks. I'll take a look... >>> > > >=20 >>> > > > This would help, indeed, but how to implement(!?), and... (see belo= w) >>> > >=20 >>> > > Just run it somewhere in the script, figure out what is about 80% of = that >>> > > and >>> > > if >>> > > that is smaller than 800 MB, we should just fail. >>> > >=20 >>> > > It would be good to have a function that determines the xz parameters= and >>> > > we >>> > > use >>> > > that where ever we call xz. We don't want any duplicated code. >>> > >=20 >>> > > > > > > > On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote: >>> > > > > > > > > On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer wrote: >>> > > > > > > > > ... >>> > > > > > > > > >=20 >>> > > > > > > > > > I forgot: Tuning the XZ-parameters... ;-) >>> > > > > > > > >=20 >>> > > > > > > > > Good that you are bringing this up. So I think what we sh= ould >>> > > > > > > > > do >>> > > > > > > > > here is >>> > > > > > > > > bringing back XZ compression with -8 so that we do not wa= ste >>> > > > > > > > > too >>> > > > > > > > > much >>> > > > > > > > > space >>> > > > > > > > > when >>> > > > > > > > > extracting the image again. >>> > > > > > >=20 >>> > > > > > > Right now I'm testing how much RAM I need. >>> > > > > > >=20 >>> > > > > > > Current 'next'-build is running with: >>> > > > > > >=20 >>> > > > > > > 'lfs/cdrom': >>> > > > > > > ... >>> > > > > > > export XZ_OPT =3D --threads=3D0 -8 --memory=3D500MiB >>> > > > > > > ... >>> > > > > > >=20 >>> > > > > > > 'lfs/Config': >>> > > > > > > ... >>> > > > > > > cd /install/packages/package/tmp/ && XZ_OPT=3D"-T0 -8" ... >>> > > > > > > ... >>> > > > > > >=20 >>> > > > > > > I'm waiting for the results. Takes a while. >>> > > > > >=20 >>> > > > > > You can just run it like this and it will tell you: >>> > > > > >=20 >>> > > > > > [ms(a)hughes ~]$ xz -vv8 -T0 >>> > > > > > xz: Filter chain: -- >>> > > > > > lzma2=3Ddict=3D32MiB,lc=3D3,lp=3D0,pb=3D2,mode=3Dnormal,nice=3D= 64,mf=3Dbt4,depth=3D0 >>> > > > > > xz: Using up to 4 threads. >>> > > > > > xz: 2,629 MiB of memory is required. The limiter is disabled. >>> > > > > > xz: Decompression will need 33 MiB of memory. >>> > > > > > xz: Compressed data cannot be written to a terminal >>> > > > > > xz: Try `xz --help' for more information. >>> > > > > >=20 >>> > > > > > > > > To not run into memory limits we should detect how much m= emory >>> > > > > > > > > the >>> > > > > > > > > build >>> > > > > > > > > host >>> > > > > > > > > has and set the memory limit to maybe 80% of that. xz will >>> > > > > > > > > then >>> > > > > > > > > automatically >>> > > > > > > > > scale down to a number of parallel processes that it can = fit >>> > > > > > > > > into >>> > > > > > > > > memory. >>> > > >=20 >>> > > > This is something I couldn't get to work. First, I couldn't specify= not >>> > > > less >>> > > > than '700 MiB'. If I specify '600 MiB', building stops and tells me= it >>> > > > needs >>> > > > at least about 680 MiB. If I use '70%', building stops. And I never= saw >>> > > > that >>> > > > 'xz' "automatically scaled down". It just stopped everytime. >>> > >=20 >>> > > Yes, ~700M is the minimum. >>> > >=20 >>> > > It will just reduce the number of threads again to make sure everythi= ng >>> > > fits >>> > > into memory. >>> > >=20 >>> > > > =20 >>> > > > > > > I'm a bit puzzled about this: I tried using '--memory=3D70%' = as >>> > > > > > > mentioned >>> > > > > > > in the xz man pages, but is seems to have no effect. It looks= as >>> > > > > > > if >>> > > > > > > the >>> > > > > > > percent parameter isn't recognized and ignored. It always end= s up >>> > > > > > > in >>> > > > > > > "cannot allocate memory". Any hints on this? Do both 'lfs/cdr= om' >>> > > > > > > AND >>> > > > > > > 'lfs/Config' need the '--memory'-parameter? >>> > > >=20 >>> > > > This was the next problem: 'percent'-parameter does not work as >>> > > > expected. >>> > >=20 >>> > > No, don't use it. Just determine yourself what it is. >>> > >=20 >>> > > I think we could even try to set 100% of the memory, because as far a= s I >>> > > can >>> > > see >>> > > this is worst case and xz won't use everything that we allow. >>> > >=20 >>> > > If that does't work, let's scale down to 90%, or 80%, etc... >>> > >=20 >>> > > > > > Yes, but it would be good to have one place where the parameter= s are >>> > > > > > being generated and then we just use them from a variable. >>> > > >=20 >>> > > > Probably beyond my capabilities for the moment. Any thoughts? >>> > > >=20 >>> > > > > > > > > ... >>> > > > > > > > > We will at least need about half a GB which should be a >>> > > > > > > > > sensible >>> > > > > > > > > requirement >>> > > > > > > > > for a build system any ways. >>> > > >=20 >>> > > > I'm afraid '500 MiB' are not enough... >>> > >=20 >>> > > No, so we will now make 1G the minimum for a build. >>> > >=20 >>> > > I tried to build the OS on an ARM board with only 512MB of RAM the ot= her >>> > > day >>> > > and >>> > > even with a lot of swap it doesn't work. So we are past this any way. >>> > >=20 >>> > > >=20 >>> > > > > > > Agreed. Because of this I'm testing with '500Mib'. >>> > > > > > >=20 >>> > > > > > > > > Will you work on this bringing it into the build system?! >>> > > > > > >=20 >>> > > > > > > "I'll do my very best!" :-)) >>> > > > > >=20 >>> > > > > > Very well Miss Sophie... >>> > > > > > ... >>> > > >=20 >>> > > > Best=E2=84=A2, >>> > > > Matthias >>> > >=20 >>> > > -Michael >>=20 >=20 >=20 --===============3540988662854236501==--