From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Compress images with XZ (was: Planning out Core Update 117) Date: Mon, 29 Jan 2018 13:23:06 +0000 Message-ID: <1517232186.2586.59.camel@ipfire.org> In-Reply-To: <1511368653.4838.612.camel@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============4322081698130017205==" List-Id: --===============4322081698130017205== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello Matthias, any update on this? Are you still on this? Best, -Michael On Wed, 2017-11-22 at 16:37 +0000, Michael Tremer wrote: > Hi, >=20 > On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer wrote: > > Hi, > >=20 > > On 21.11.2017 15:42, Michael Tremer wrote: > > > Hello, > > >=20 > > > any progress on this? > >=20 > > Not much, because I've been struggling with a flu for the past two > > weeks, and its not over yet, but: > >=20 > > In the meantime I ran several builds with different parameters. > >=20 > > IMHO we could use: > >=20 > > 'export XZ_OPT =3D --threads=3D0 -8 --memory=3D700MiB' > >=20 > > in 'lfs/cdrom' > >=20 > > and > >=20 > > cd /install/packages/package/tmp/ && XZ_OPT=3D"-T0 -8 --memory=3D700MiB" = tar -c > > -p > > --numeric-owner -J -f /install/packages/package/files.tar.xz * > >=20 > > in 'lfs/Config'. > >=20 > > Using '9' or '--extreme'-parameter brought no further improvements, > > on the other hand, build time and RAM consumption increased > > strongly. With '-8', minimal RAM consumption was about 680 MiB, using > > '700 Mib' was always safe. So this is the minimum I would recommend. >=20 > Yes, I observed the same thing. -9 compresses a little bit better and extre= me > doesn't change a bit. >=20 > -8 seems to be optimal since it won't require too much memory to decompress > either. So let's go with that. >=20 > > Results: > >=20 > > (Buildtime: 5 hours 20 minutes 59 seconds) > >=20 > > 255866764 ipfire-2.19.1gb-ext4.i586-full-core117.img.gz > > 252262006 ipfire-2.19.1gb-ext4-scon.i586-full-core117.img.gz > > 175112192 ipfire-2.19.i586-full-core117.iso > > =20 > > > -Michael > > >=20 > > > On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote: > > > > Hi, > > > >=20 > > > > On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer wrote: > > > > > Hi, > > > > >=20 > > > > > On 07.11.2017 16:05, Michael Tremer wrote: > > > > > > Hi, > > > > > >=20 > > > > > > maybe this function helps: > > > > > >=20 > > > > > > https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dcommitdiff;h=3D519= 0eea24f98 > > > > > > 22 > > > > > > a63d > > > > > > c5d06d214b48f973b14f29 > > > > >=20 > > > > > Thanks. I'll take a look... > >=20 > > This would help, indeed, but how to implement(!?), and... (see below) >=20 > Just run it somewhere in the script, figure out what is about 80% of that a= nd > if > that is smaller than 800 MB, we should just fail. >=20 > It would be good to have a function that determines the xz parameters and we > use > that where ever we call xz. We don't want any duplicated code. >=20 > > > > > > On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote: > > > > > > > On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer wrote: > > > > > > > ... > > > > > > > >=20 > > > > > > > > I forgot: Tuning the XZ-parameters... ;-) > > > > > > >=20 > > > > > > > Good that you are bringing this up. So I think what we should do > > > > > > > here is > > > > > > > bringing back XZ compression with -8 so that we do not waste too > > > > > > > much > > > > > > > space > > > > > > > when > > > > > > > extracting the image again. > > > > >=20 > > > > > Right now I'm testing how much RAM I need. > > > > >=20 > > > > > Current 'next'-build is running with: > > > > >=20 > > > > > 'lfs/cdrom': > > > > > ... > > > > > export XZ_OPT =3D --threads=3D0 -8 --memory=3D500MiB > > > > > ... > > > > >=20 > > > > > 'lfs/Config': > > > > > ... > > > > > cd /install/packages/package/tmp/ && XZ_OPT=3D"-T0 -8" ... > > > > > ... > > > > >=20 > > > > > I'm waiting for the results. Takes a while. > > > >=20 > > > > You can just run it like this and it will tell you: > > > >=20 > > > > [ms(a)hughes ~]$ xz -vv8 -T0 > > > > xz: Filter chain: -- > > > > lzma2=3Ddict=3D32MiB,lc=3D3,lp=3D0,pb=3D2,mode=3Dnormal,nice=3D64,mf= =3Dbt4,depth=3D0 > > > > xz: Using up to 4 threads. > > > > xz: 2,629 MiB of memory is required. The limiter is disabled. > > > > xz: Decompression will need 33 MiB of memory. > > > > xz: Compressed data cannot be written to a terminal > > > > xz: Try `xz --help' for more information. > > > >=20 > > > > > > > To not run into memory limits we should detect how much memory = the > > > > > > > build > > > > > > > host > > > > > > > has and set the memory limit to maybe 80% of that. xz will then > > > > > > > automatically > > > > > > > scale down to a number of parallel processes that it can fit in= to > > > > > > > memory. > >=20 > > This is something I couldn't get to work. First, I couldn't specify not l= ess > > than '700 MiB'. If I specify '600 MiB', building stops and tells me it ne= eds > > at least about 680 MiB. If I use '70%', building stops. And I never saw t= hat > > 'xz' "automatically scaled down". It just stopped everytime. >=20 > Yes, ~700M is the minimum. >=20 > It will just reduce the number of threads again to make sure everything fits > into memory. >=20 > > =20 > > > > > I'm a bit puzzled about this: I tried using '--memory=3D70%' as > > > > > mentioned > > > > > in the xz man pages, but is seems to have no effect. It looks as if > > > > > the > > > > > percent parameter isn't recognized and ignored. It always ends up in > > > > > "cannot allocate memory". Any hints on this? Do both 'lfs/cdrom' AND > > > > > 'lfs/Config' need the '--memory'-parameter? > >=20 > > This was the next problem: 'percent'-parameter does not work as expected. >=20 > No, don't use it. Just determine yourself what it is. >=20 > I think we could even try to set 100% of the memory, because as far as I can > see > this is worst case and xz won't use everything that we allow. >=20 > If that does't work, let's scale down to 90%, or 80%, etc... >=20 > > > > Yes, but it would be good to have one place where the parameters are > > > > being generated and then we just use them from a variable. > >=20 > > Probably beyond my capabilities for the moment. Any thoughts? > >=20 > > > > > > > ... > > > > > > > We will at least need about half a GB which should be a sensible > > > > > > > requirement > > > > > > > for a build system any ways. > >=20 > > I'm afraid '500 MiB' are not enough... >=20 > No, so we will now make 1G the minimum for a build. >=20 > I tried to build the OS on an ARM board with only 512MB of RAM the other day > and > even with a lot of swap it doesn't work. So we are past this any way. >=20 > >=20 > > > > > Agreed. Because of this I'm testing with '500Mib'. > > > > >=20 > > > > > > > Will you work on this bringing it into the build system?! > > > > >=20 > > > > > "I'll do my very best!" :-)) > > > >=20 > > > > Very well Miss Sophie... > > > > ... > >=20 > > Best=E2=84=A2, > > Matthias >=20 > -Michael --===============4322081698130017205== Content-Type: application/pgp-signature Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="signature.asc" MIME-Version: 1.0 LS0tLS1CRUdJTiBQR1AgU0lHTkFUVVJFLS0tLS0KCmlRSXpCQUFCQ2dBZEZpRUU1L3JXNWwzR0dl Mnlwa3R4Z0hudy8yK1FDUWNGQWxwdklEb0FDZ2tRZ0hudy8yK1EKQ1FlOEhRLytPY3pqMnBpdTQ4 QnpHblFhdmVXSnI0NUVoUTlEV216QWdzdmNTQmVlZlJDUHJVQk5CSTBkWjVHVwpkbXpZSFk3V0hV RHREVmtoZnZ0bGlGUEgwTXBiL21RVDB3RUJBSU1jVjk4UmVHVVpJaEdXUllmcjVRdnY4M1YxCjh4 QURic2t6ams3R0tlWHhhMk53SWpyem03ODRkb0EvRjZNMTNPdVJRQmJrcTJ4ODFHcnZzdmhrRVJY eDI5azEKbFV2d3djWUJXNzdBeHZQblFrY3l2OFNqcWJxWkpKYTZ5eFc0Unl6azZOT1I1cDg5bytZ Q2h1YVJrU2RmM3owdgpIUjNTbTlOZGI3cHdtZjZlSmVSZEowTTFmYnNLSFlXK3A4VER0SzlaMDEr YjhnNlFMOGFtQkwwYkFTdmlJeHNTCjl6LzJqTUlEUld1dVVMcURocVd0N3VKSjJDYmFJYmxmY1Jq NVkzajVJMXdoYi95QWdwbUxtYys5c0lIMDVJQTMKNktZNGdCUzY2UVp1eUhsTEh2bSt0eXFBenNu TUdNWHB5TDVFZW0rdUs4MS9icUJzSTBNdStrNmZIWkJzNXBxYgozYU1mWGs2UUJ0NXZnV0ZWU1Ja MWJJV3MrTTNaZTYvYVNzVTR5emIrL3kzQWFpSWJiTnVMUytMV0syZitXT0w1CkN4RXNLQWZ5Ung2 Sm1JakJQeHMxK1JpOFpja0lHblMyUk5SVEk5ZVlza2ZQSG96QlFYeTk0V1lpQmgyL21XN0QKaVAy TVFucWJxYWlidFA2TENzQjhrY3lNRXdUWm1vZ3RQOTREdXgwKzRoMzM1dWpobFkrUER0RFJFRFNU Y3VlYwpkbVJDRm9IcW9HT2Z0ek5FWDBXaDNDY3Z0eHlYQmV3RURmQ2xFcTZOeFRzTjFNZXpPVXc9 Cj01clZxCi0tLS0tRU5EIFBHUCBTSUdOQVRVUkUtLS0tLQo= --===============4322081698130017205==--