From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: Compress images with XZ Date: Fri, 11 May 2018 21:27:41 +0100 Message-ID: In-Reply-To: <1518379066.2498.13.camel@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============5396548742780571864==" List-Id: --===============5396548742780571864== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hey Matthias, are you still working on this? I am asking because I just noticed that the new next build is about 50MB larg= er than the old one due to the larger kernel and loads more of firmware files. Compressing it better would of course help to save space on disk. Best, -Michael On Sun, 2018-02-11 at 19:57 +0000, Michael Tremer wrote: > Hi, >=20 > On Sun, 2018-02-11 at 15:35 +0000, Matthias Fischer wrote: > > Hi, > >=20 > > On 06.02.2018 02:05, Michael Tremer wrote: > > > > ... > > > > ((XZ_MEM=3D(($HOST_MEM)/10)*7)) && XZ_MEM=3D${XZ_MEM}MiB > > >=20 > > > You can write this even short: > > >=20 > > > XZ_MEM=3D"$(( HOST_MEM * 7 / 10 ))MiB" > >=20 > > Oneliner! Yes. I was looking for something like that... ;-) > >=20 > > > I swapped the order of the multiplication and division so that you would > > > have > > > a > > > more exact result. > > >=20 > > > For example if you system has 1024 MB or RAM, you divide this by 10 (10= 2) > > > and > > > then multiply by 7 which gives you 714. The other order is 1024 * 7 =3D= 7167 > > > and > > > then divided by 10 is 716 which is closer to the actual result of 716.8. > > > Shouldn't matter here but just in general this is better :) > >=20 > > Thanks, looks better - noted for the next time! > >=20 > > Now it looks like this ('make.sh'): > >=20 > > ... > > # Host memory is ok, calculating XZ memory > > XZ_MEM=3D"$(( HOST_MEM * 7 / 10 ))MiB" > > XZ_OPT=3D"--threads=3D0 -8 $XZ_MEM" > > echo XZ-Memory: $XZ_MEM > > echo XZ-Options: $XZ_OPT > > echo "XZ memory size is OK (must be at least 700MiB), starting build..." > > ... > > ["echoes" for debugging!] > >=20 > > This is working. '$XZ_OPT' now gives me: --threads=3D0 -8 5450MiB >=20 > Will there be some checks in make.sh now that check if at least 1024MB > of memory are available for build? I think there should be a warning if > not enough memory is available, but we should still try to build. >=20 > > But. ;-) > >=20 > > > You can put this probably next under "HOST_MEM=3D$(system_memory)" and = then > > > export > > > this to the internal shell by adding the variable to lfsmake2 for the d= isk > > > image > > > and ipfiredist for the packages. > >=20 > > Sorry, I must confess that I don't know exactly how to do this. As I wrot= e: > > "How > > do > > I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?" Or in this case, > > XZ_OPT!? > >=20 > > > Then in lfs/cdrom and lfs/Config for the dist > > > target, you just need to add the parameters to the command line. > >=20 > > In 'Config' I changed: > >=20 > > ... > > cd /install/packages/package/tmp/ && XZ_OPT=3D-T0 tar -c -p --numeric-own= er -J > > -f > > /install/packages/package/files.tar.xz * > > ... > >=20 > > To: > > ... > > cd /install/packages/package/tmp/ && $XZ_OPT tar -c -p --numeric-owner -J= -f > > /install/packages/package/files.tar.xz * > > ... > >=20 > > > Maybe it is a good idea to have a variable XZ_OPT that adds everything > > > together > > > and is then used in both places. > >=20 > > I added the other xz-options and created XZ_OPT as shown above, but in > > 'cdrom' > > there > > is an 'export XZ_OPT =3D ...'-line and I don't know how to handle these l= ine. > > Is > > it still necessary? >=20 > No, that is no longer necessary and can be removed. We should just add > it to the tar command just like it is done in lfs/Config. >=20 > > Do I have to 'export' the XZ_OPT'-variable in 'make.sh'? >=20 > No, because the chroot environment will throw away all environment > variables from the host system and then set them again. That's why we > have this enterchroot() function that does this with the env command. >=20 > > And how do I "add" this variable to 'lfsmake2' and 'ipfiredist'(-function= )? >=20 > Just have a look at the functions. There should be loads of examples > where we pass other variables and you just need to add this one, too. >=20 > > Sorry if these questions sound simple, but I'm guessing a bit too much wh= ile > > trying > > to get a grip on this. >=20 > This is absolutely fine to sort this out first and then send the patch > :) That's what this list is for. >=20 > Best, > -Michael >=20 > >=20 > > Best, > > Matthias > >=20 > > >=20 > > > Best, > > > -Michael > > >=20 > > > > ... > > > >=20 > > > > Looks better. > > > >=20 > > > > Best, > > > > Matthias > > > >=20 > > > > On 03.02.2018 20:44, Matthias Fischer wrote: > > > > > Hi, > > > > >=20 > > > > > On 30.01.2018 21:03, Michael Tremer wrote: > > > > > > Hi, > > > > > >=20 > > > > > > ... > > > > > > >=20 > > > > > > > I'll consider how I would do this and send it in as soon as I g= et > > > > > > > a > > > > > > > grip > > > > > > > on it. > > > > > >=20 > > > > > > Well, I added some functions already somewhere. I suppose you just > > > > > > need > > > > > > to > > > > > > do > > > > > > the maths to set that to X% of the host system which is just a > > > > > > simple > > > > > > multiplication of the output of ${HOST_MEM} in make.sh. > > > > >=20 > > > > > Ok, I take my heart in my hand. Here are my first results - but > > > > > remember, > > > > > I'm not familiar with this. I built a short test script which looks= as > > > > > if > > > > > it > > > > > is working as wanted, but... > > > > > =20 > > > > > > And that needs to be passed to the lfs scripts that are supposed = to > > > > > > make > > > > > > the > > > > > > images. > > > > >=20 > > > > > ...this is a problem I'm struggling with right now: > > > > > How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'? > > > > >=20 > > > > > I think this can be done using 'export XZ_MEM' or something else but > > > > > I'm > > > > > not > > > > > sure. > > > > >=20 > > > > > This is the script I'm testing with: > > > > >=20 > > > > > ***SNIP*** > > > > > #!/bin/bash > > > > >=20 > > > > > system_memory() { > > > > > local key val unit > > > > >=20 > > > > > while read -r key val unit; do > > > > > case "${key}" in > > > > > MemTotal:*) > > > > > # Convert to MB > > > > > echo "$(( ${val} / 1024 ))" > > > > > break > > > > > ;; > > > > > esac > > > > > done < /proc/meminfo > > > > > } > > > > >=20 > > > > > HOST_MEM=3D$(system_memory) > > > > >=20 > > > > > # Checking host memory > > > > > if [ $HOST_MEM -ge 1024 ]; then > > > > > echo Host-Memory: $HOST_MEM'MiB' > > > > > echo "Host memory is OK (must be at least 1024MiB), starting > > > > > build." > > > > > else > > > > > echo Host-Memory: $HOST_MEM'MiB' > > > > > echo "Host memory too low (less than 1024MiB), building > > > > > stopped." > > > > > exit 1 > > > > > fi > > > > >=20 > > > > > # Host memory is ok, calculating needed XZ memory to about 70% of h= ost > > > > > memory > > > > > ((XZ_MEM=3D(($HOST_MEM)/10)*7)) > > > > > XZ_MEM=3D$XZ_MEM'Mib' > > > > > echo XZ-Memory: $XZ_MEM > > > > > echo "XZ memory size is OK (must be at least 700MiB), starting > > > > > build." > > > > > ***SNAP*** > > > > >=20 > > > > > The 'echoes' are mostly for debugging. > > > > >=20 > > > > > Calculation looks ok, 'MiB' must added without blanks (XZ man page: > > > > > "In > > > > > most > > > > > places where an integer argument is expected, an optional suffix is > > > > > supported > > > > > to easily indicate large integers. There must be no space between t= he > > > > > integer > > > > > and the suffix.") > > > > >=20 > > > > > > Please send questions where ever you need help. There is no reason > > > > > > to do > > > > > > this > > > > > > alone. > > > > >=20 > > > > > Please see above: > > > > >=20 > > > > > 1. Is this code what you where thinking of? I would leave the other > > > > > parameters > > > > > as they are (--threads=3D0 -8). > > > > >=20 > > > > > 2. Where should I add such code in the first place? I'd like to sta= rt > > > > > with > > > > > 'make.sh', so building stops as soon as possible if there is not > > > > > enough > > > > > memory. Where's > > > > > the best place? Right after function 'system_memory() {...}'? > > > > >=20 > > > > > 3. How to I pass contents of 'XZ_MEM' to 'cdrom' and 'Config'? > > > > > I consider something like: > > > > > ... > > > > > export XZ_OPT =3D --threads=3D0 -8 --memory=3D$XZ_MEM > > > > > ... > > > > >=20 > > > > > Best, > > > > > Matthias > > > > > =20 > > > > > > Best, > > > > > > -Michael > > > > > >=20 > > > > > > >=20 > > > > > > > Best, > > > > > > > Matthias > > > > > > > > Best, > > > > > > > > -Michael > > > > > > > >=20 > > > > > > > > On Wed, 2017-11-22 at 16:37 +0000, Michael Tremer wrote: > > > > > > > > > Hi, > > > > > > > > >=20 > > > > > > > > > On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer wrote: > > > > > > > > > > Hi, > > > > > > > > > >=20 > > > > > > > > > > On 21.11.2017 15:42, Michael Tremer wrote: > > > > > > > > > > > Hello, > > > > > > > > > > >=20 > > > > > > > > > > > any progress on this? > > > > > > > > > >=20 > > > > > > > > > > Not much, because I've been struggling with a flu for the > > > > > > > > > > past > > > > > > > > > > two > > > > > > > > > > weeks, and its not over yet, but: > > > > > > > > > >=20 > > > > > > > > > > In the meantime I ran several builds with different > > > > > > > > > > parameters. > > > > > > > > > >=20 > > > > > > > > > > IMHO we could use: > > > > > > > > > >=20 > > > > > > > > > > 'export XZ_OPT =3D --threads=3D0 -8 --memory=3D700MiB' > > > > > > > > > >=20 > > > > > > > > > > in 'lfs/cdrom' > > > > > > > > > >=20 > > > > > > > > > > and > > > > > > > > > >=20 > > > > > > > > > > cd /install/packages/package/tmp/ && XZ_OPT=3D"-T0 -8 -- > > > > > > > > > > memory=3D700MiB" tar > > > > > > > > > > -c > > > > > > > > > > -p > > > > > > > > > > --numeric-owner -J -f /install/packages/package/files.tar= .xz > > > > > > > > > > * > > > > > > > > > >=20 > > > > > > > > > > in 'lfs/Config'. > > > > > > > > > >=20 > > > > > > > > > > Using '9' or '--extreme'-parameter brought no further > > > > > > > > > > improvements, > > > > > > > > > > on the other hand, build time and RAM consumption increas= ed > > > > > > > > > > strongly. With '-8', minimal RAM consumption was about 680 > > > > > > > > > > MiB, > > > > > > > > > > using > > > > > > > > > > '700 Mib' was always safe. So this is the minimum I would > > > > > > > > > > recommend. > > > > > > > > >=20 > > > > > > > > > Yes, I observed the same thing. -9 compresses a little bit > > > > > > > > > better > > > > > > > > > and > > > > > > > > > extreme > > > > > > > > > doesn't change a bit. > > > > > > > > >=20 > > > > > > > > > -8 seems to be optimal since it won't require too much memo= ry > > > > > > > > > to > > > > > > > > > decompress > > > > > > > > > either. So let's go with that. > > > > > > > > >=20 > > > > > > > > > > Results: > > > > > > > > > >=20 > > > > > > > > > > (Buildtime: 5 hours 20 minutes 59 seconds) > > > > > > > > > >=20 > > > > > > > > > > 255866764 ipfire-2.19.1gb-ext4.i586-full-core117.img.gz > > > > > > > > > > 252262006 ipfire-2.19.1gb-ext4-scon.i586-full- > > > > > > > > > > core117.img.gz > > > > > > > > > > 175112192 ipfire-2.19.i586-full-core117.iso > > > > > > > > > > =20 > > > > > > > > > > > -Michael > > > > > > > > > > >=20 > > > > > > > > > > > On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote: > > > > > > > > > > > > Hi, > > > > > > > > > > > >=20 > > > > > > > > > > > > On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > >=20 > > > > > > > > > > > > > On 07.11.2017 16:05, Michael Tremer wrote: > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > >=20 > > > > > > > > > > > > > > maybe this function helps: > > > > > > > > > > > > > >=20 > > > > > > > > > > > > > > https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dco= mmitdif > > > > > > > > > > > > > > f;h=3D > > > > > > > > > > > > > > 51 > > > > > > > > > > > > > > 90eea2 > > > > > > > > > > > > > > 4f98 > > > > > > > > > > > > > > 22 > > > > > > > > > > > > > > a63d > > > > > > > > > > > > > > c5d06d214b48f973b14f29 > > > > > > > > > > > > >=20 > > > > > > > > > > > > > Thanks. I'll take a look... > > > > > > > > > >=20 > > > > > > > > > > This would help, indeed, but how to implement(!?), and... > > > > > > > > > > (see > > > > > > > > > > below) > > > > > > > > >=20 > > > > > > > > > Just run it somewhere in the script, figure out what is abo= ut > > > > > > > > > 80% > > > > > > > > > of > > > > > > > > > that > > > > > > > > > and > > > > > > > > > if > > > > > > > > > that is smaller than 800 MB, we should just fail. > > > > > > > > >=20 > > > > > > > > > It would be good to have a function that determines the xz > > > > > > > > > parameters and > > > > > > > > > we > > > > > > > > > use > > > > > > > > > that where ever we call xz. We don't want any duplicated co= de. > > > > > > > > >=20 > > > > > > > > > > > > > > On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > On Mon, 2017-11-06 at 22:09 +0100, Matthias > > > > > > > > > > > > > > > Fischer > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > ... > > > > > > > > > > > > > > > >=20 > > > > > > > > > > > > > > > > I forgot: Tuning the XZ-parameters... ;-) > > > > > > > > > > > > > > >=20 > > > > > > > > > > > > > > > Good that you are bringing this up. So I think > > > > > > > > > > > > > > > what we > > > > > > > > > > > > > > > should > > > > > > > > > > > > > > > do > > > > > > > > > > > > > > > here is > > > > > > > > > > > > > > > bringing back XZ compression with -8 so that we= do > > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > waste > > > > > > > > > > > > > > > too > > > > > > > > > > > > > > > much > > > > > > > > > > > > > > > space > > > > > > > > > > > > > > > when > > > > > > > > > > > > > > > extracting the image again. > > > > > > > > > > > > >=20 > > > > > > > > > > > > > Right now I'm testing how much RAM I need. > > > > > > > > > > > > >=20 > > > > > > > > > > > > > Current 'next'-build is running with: > > > > > > > > > > > > >=20 > > > > > > > > > > > > > 'lfs/cdrom': > > > > > > > > > > > > > ... > > > > > > > > > > > > > export XZ_OPT =3D --threads=3D0 -8 --memory=3D500MiB > > > > > > > > > > > > > ... > > > > > > > > > > > > >=20 > > > > > > > > > > > > > 'lfs/Config': > > > > > > > > > > > > > ... > > > > > > > > > > > > > cd /install/packages/package/tmp/ && XZ_OPT=3D"-T0 = -8" > > > > > > > > > > > > > ... > > > > > > > > > > > > > ... > > > > > > > > > > > > >=20 > > > > > > > > > > > > > I'm waiting for the results. Takes a while. > > > > > > > > > > > >=20 > > > > > > > > > > > > You can just run it like this and it will tell you: > > > > > > > > > > > >=20 > > > > > > > > > > > > [ms(a)hughes ~]$ xz -vv8 -T0 > > > > > > > > > > > > xz: Filter chain: -- > > > > > > > > > > > > lzma2=3Ddict=3D32MiB,lc=3D3,lp=3D0,pb=3D2,mode=3Dnorm= al,nice=3D64,mf=3Db > > > > > > > > > > > > t4,d > > > > > > > > > > > > ep > > > > > > > > > > > > th=3D0 > > > > > > > > > > > > xz: Using up to 4 threads. > > > > > > > > > > > > xz: 2,629 MiB of memory is required. The limiter is > > > > > > > > > > > > disabled. > > > > > > > > > > > > xz: Decompression will need 33 MiB of memory. > > > > > > > > > > > > xz: Compressed data cannot be written to a terminal > > > > > > > > > > > > xz: Try `xz --help' for more information. > > > > > > > > > > > >=20 > > > > > > > > > > > > > > > To not run into memory limits we should detect = how > > > > > > > > > > > > > > > much > > > > > > > > > > > > > > > memory > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > build > > > > > > > > > > > > > > > host > > > > > > > > > > > > > > > has and set the memory limit to maybe 80% of th= at. > > > > > > > > > > > > > > > xz > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > then > > > > > > > > > > > > > > > automatically > > > > > > > > > > > > > > > scale down to a number of parallel processes th= at > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > fit > > > > > > > > > > > > > > > into > > > > > > > > > > > > > > > memory. > > > > > > > > > >=20 > > > > > > > > > > This is something I couldn't get to work. First, I couldn= 't > > > > > > > > > > specify not > > > > > > > > > > less > > > > > > > > > > than '700 MiB'. If I specify '600 MiB', building stops and > > > > > > > > > > tells > > > > > > > > > > me it > > > > > > > > > > needs > > > > > > > > > > at least about 680 MiB. If I use '70%', building stops. A= nd > > > > > > > > > > I > > > > > > > > > > never saw > > > > > > > > > > that > > > > > > > > > > 'xz' "automatically scaled down". It just stopped everyti= me. > > > > > > > > >=20 > > > > > > > > > Yes, ~700M is the minimum. > > > > > > > > >=20 > > > > > > > > > It will just reduce the number of threads again to make sure > > > > > > > > > everything > > > > > > > > > fits > > > > > > > > > into memory. > > > > > > > > >=20 > > > > > > > > > > =20 > > > > > > > > > > > > > I'm a bit puzzled about this: I tried using ' > > > > > > > > > > > > > --memory=3D70%' > > > > > > > > > > > > > as > > > > > > > > > > > > > mentioned > > > > > > > > > > > > > in the xz man pages, but is seems to have no effect. > > > > > > > > > > > > > It > > > > > > > > > > > > > looks as > > > > > > > > > > > > > if > > > > > > > > > > > > > the > > > > > > > > > > > > > percent parameter isn't recognized and ignored. It > > > > > > > > > > > > > always > > > > > > > > > > > > > ends up > > > > > > > > > > > > > in > > > > > > > > > > > > > "cannot allocate memory". Any hints on this? Do both > > > > > > > > > > > > > 'lfs/cdrom' > > > > > > > > > > > > > AND > > > > > > > > > > > > > 'lfs/Config' need the '--memory'-parameter? > > > > > > > > > >=20 > > > > > > > > > > This was the next problem: 'percent'-parameter does not w= ork > > > > > > > > > > as > > > > > > > > > > expected. > > > > > > > > >=20 > > > > > > > > > No, don't use it. Just determine yourself what it is. > > > > > > > > >=20 > > > > > > > > > I think we could even try to set 100% of the memory, because > > > > > > > > > as > > > > > > > > > far > > > > > > > > > as I > > > > > > > > > can > > > > > > > > > see > > > > > > > > > this is worst case and xz won't use everything that we allo= w. > > > > > > > > >=20 > > > > > > > > > If that does't work, let's scale down to 90%, or 80%, etc... > > > > > > > > >=20 > > > > > > > > > > > > Yes, but it would be good to have one place where the > > > > > > > > > > > > parameters are > > > > > > > > > > > > being generated and then we just use them from a > > > > > > > > > > > > variable. > > > > > > > > > >=20 > > > > > > > > > > Probably beyond my capabilities for the moment. Any > > > > > > > > > > thoughts? > > > > > > > > > >=20 > > > > > > > > > > > > > > > ... > > > > > > > > > > > > > > > We will at least need about half a GB which sho= uld > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > sensible > > > > > > > > > > > > > > > requirement > > > > > > > > > > > > > > > for a build system any ways. > > > > > > > > > >=20 > > > > > > > > > > I'm afraid '500 MiB' are not enough... > > > > > > > > >=20 > > > > > > > > > No, so we will now make 1G the minimum for a build. > > > > > > > > >=20 > > > > > > > > > I tried to build the OS on an ARM board with only 512MB of = RAM > > > > > > > > > the > > > > > > > > > other > > > > > > > > > day > > > > > > > > > and > > > > > > > > > even with a lot of swap it doesn't work. So we are past this > > > > > > > > > any > > > > > > > > > way. > > > > > > > > >=20 > > > > > > > > > >=20 > > > > > > > > > > > > > Agreed. Because of this I'm testing with '500Mib'. > > > > > > > > > > > > >=20 > > > > > > > > > > > > > > > Will you work on this bringing it into the build > > > > > > > > > > > > > > > system?! > > > > > > > > > > > > >=20 > > > > > > > > > > > > > "I'll do my very best!" :-)) > > > > > > > > > > > >=20 > > > > > > > > > > > > Very well Miss Sophie... > > > > > > > > > > > > ... > > > > > > > > > >=20 > > > > > > > > > > Best=E2=84=A2, > > > > > > > > > > Matthias > > > > > > > > >=20 > > > > > > > > > -Michael --===============5396548742780571864==--