From mboxrd@z Thu Jan  1 00:00:00 1970
From: Michael Tremer <michael.tremer@ipfire.org>
To: development@lists.ipfire.org
Subject: Re: Compress images with XZ
Date: Tue, 06 Feb 2018 01:05:57 +0000
Message-ID: <1517879157.21272.77.camel@ipfire.org>
In-Reply-To: <06f38cee-e77c-2af2-f5be-90f68f7f2f21@ipfire.org>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============0319849236148878644=="
List-Id: <development.lists.ipfire.org>

--===============0319849236148878644==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

Hi,

On Sat, 2018-02-03 at 22:09 +0100, Matthias Fischer wrote:
> Hi,
>=20
> Small addition:
>=20
> I found another possibility for calculating XZ memory and adding the
> needed suffix:
>=20
> ...
> ((XZ_MEM=3D(($HOST_MEM)/10)*7)) && XZ_MEM=3D${XZ_MEM}MiB

You can write this even short:

XZ_MEM=3D"$(( HOST_MEM * 7 / 10 ))MiB"

I swapped the order of the multiplication and division so that you would have=
 a
more exact result.

For example if you system has 1024 MB or RAM, you divide this by 10 (102) and
then multiply by 7 which gives you 714. The other order is 1024 * 7 =3D 7167 =
and
then divided by 10 is 716 which is closer to the actual result of 716.8.
Shouldn't matter here but just in general this is better :)

You can put this probably next under "HOST_MEM=3D$(system_memory)" and then e=
xport
this to the internal shell by adding the variable to lfsmake2 for the disk im=
age
and ipfiredist for the packages. Then in lfs/cdrom and lfs/Config for the dist
target, you just need to add the parameters to the command line.

Maybe it is a good idea to have a variable XZ_OPT that adds everything togeth=
er
and is then used in both places.

Best,
-Michael

> ...
>=20
> Looks better.
>=20
> Best,
> Matthias
>=20
> On 03.02.2018 20:44, Matthias Fischer wrote:
> > Hi,
> >=20
> > On 30.01.2018 21:03, Michael Tremer wrote:
> > > Hi,
> > >=20
> > > ...
> > > >=20
> > > > I'll consider how I would do this and send it in as soon as I get a g=
rip
> > > > on it.
> > >=20
> > > Well, I added some functions already somewhere. I suppose you just need=
 to
> > > do
> > > the maths to set that to X% of the host system which is just a simple
> > > multiplication of the output of ${HOST_MEM} in make.sh.
> >=20
> > Ok, I take my heart in my hand. Here are my first results - but remember,
> > I'm not familiar with this. I built a short test script which looks as if=
 it
> > is working as wanted, but...
> > =20
> > > And that needs to be passed to the lfs scripts that are supposed to make
> > > the
> > > images.
> >=20
> > ...this is a problem I'm struggling with right now:
> > How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?
> >=20
> > I think this can be done using 'export XZ_MEM' or something else but I'm =
not
> > sure.
> >=20
> > This is the script I'm testing with:
> >=20
> > ***SNIP***
> > #!/bin/bash
> >=20
> > system_memory() {
> > 	local key val unit
> >=20
> > 	while read -r key val unit; do
> > 		case "${key}" in
> > 			MemTotal:*)
> > 				# Convert to MB
> > 				echo "$(( ${val} / 1024 ))"
> > 				break
> > 				;;
> > 		esac
> > 	done < /proc/meminfo
> > }
> >=20
> > HOST_MEM=3D$(system_memory)
> >=20
> > # Checking host memory
> > if [ $HOST_MEM -ge 1024 ]; then
> > 	echo Host-Memory: $HOST_MEM'MiB'
> > 	echo "Host memory is OK (must be at least 1024MiB), starting build."
> > else
> > 	echo Host-Memory: $HOST_MEM'MiB'
> > 	echo "Host memory too low (less than 1024MiB), building stopped."
> > exit 1
> > fi
> >=20
> > # Host memory is ok, calculating needed XZ memory to about 70% of host
> > memory
> > ((XZ_MEM=3D(($HOST_MEM)/10)*7))
> > 	XZ_MEM=3D$XZ_MEM'Mib'
> > 	echo XZ-Memory: $XZ_MEM
> > 	echo "XZ memory size is OK (must be at least 700MiB), starting build."
> > ***SNAP***
> >=20
> > The 'echoes' are mostly for debugging.
> >=20
> > Calculation looks ok, 'MiB' must added without blanks (XZ man page: "In m=
ost
> > places where an integer argument is expected, an optional suffix is
> > supported
> > to easily indicate large integers. There must be no space between the
> > integer
> > and the suffix.")
> >=20
> > > Please send questions where ever you need help. There is no reason to do
> > > this
> > > alone.
> >=20
> > Please see above:
> >=20
> > 1. Is this code what you where thinking of? I would leave the other
> > parameters
> > as they are (--threads=3D0 -8).
> >=20
> > 2. Where should I add such code in the first place? I'd like to start with
> > 'make.sh', so building stops as soon as possible if there is not enough
> > memory. Where's
> > the best place? Right after function 'system_memory() {...}'?
> >=20
> > 3. How to I pass contents of 'XZ_MEM' to 'cdrom' and 'Config'?
> > I consider something like:
> > ...
> > export XZ_OPT =3D --threads=3D0 -8 --memory=3D$XZ_MEM
> > ...
> >=20
> > Best,
> > Matthias
> > =20
> > > Best,
> > > -Michael
> > >=20
> > > >=20
> > > > Best,
> > > > Matthias
> > > >  > Best,
> > > > > -Michael
> > > > >=20
> > > > > On Wed, 2017-11-22 at 16:37 +0000, Michael Tremer wrote:
> > > > > > Hi,
> > > > > >=20
> > > > > > On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer wrote:
> > > > > > > Hi,
> > > > > > >=20
> > > > > > > On 21.11.2017 15:42, Michael Tremer wrote:
> > > > > > > > Hello,
> > > > > > > >=20
> > > > > > > > any progress on this?
> > > > > > >=20
> > > > > > > Not much, because I've been struggling with a flu for the past =
two
> > > > > > > weeks, and its not over yet, but:
> > > > > > >=20
> > > > > > > In the meantime I ran several builds with different parameters.
> > > > > > >=20
> > > > > > > IMHO we could use:
> > > > > > >=20
> > > > > > > 'export XZ_OPT =3D --threads=3D0 -8 --memory=3D700MiB'
> > > > > > >=20
> > > > > > > in 'lfs/cdrom'
> > > > > > >=20
> > > > > > > and
> > > > > > >=20
> > > > > > > cd /install/packages/package/tmp/ && XZ_OPT=3D"-T0 -8 --
> > > > > > > memory=3D700MiB" tar
> > > > > > > -c
> > > > > > > -p
> > > > > > > --numeric-owner -J -f /install/packages/package/files.tar.xz *
> > > > > > >=20
> > > > > > > in 'lfs/Config'.
> > > > > > >=20
> > > > > > > Using '9' or '--extreme'-parameter brought no further
> > > > > > > improvements,
> > > > > > > on the other hand, build time and RAM consumption increased
> > > > > > > strongly. With '-8', minimal RAM consumption was about 680 MiB,
> > > > > > > using
> > > > > > > '700 Mib' was always safe. So this is the minimum I would
> > > > > > > recommend.
> > > > > >=20
> > > > > > Yes, I observed the same thing. -9 compresses a little bit better
> > > > > > and
> > > > > > extreme
> > > > > > doesn't change a bit.
> > > > > >=20
> > > > > > -8 seems to be optimal since it won't require too much memory to
> > > > > > decompress
> > > > > > either. So let's go with that.
> > > > > >=20
> > > > > > > Results:
> > > > > > >=20
> > > > > > > (Buildtime: 5 hours 20 minutes 59 seconds)
> > > > > > >=20
> > > > > > > 255866764  ipfire-2.19.1gb-ext4.i586-full-core117.img.gz
> > > > > > > 252262006  ipfire-2.19.1gb-ext4-scon.i586-full-core117.img.gz
> > > > > > > 175112192  ipfire-2.19.i586-full-core117.iso
> > > > > > > =20
> > > > > > > > -Michael
> > > > > > > >=20
> > > > > > > > On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote:
> > > > > > > > > Hi,
> > > > > > > > >=20
> > > > > > > > > On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer wrote:
> > > > > > > > > > Hi,
> > > > > > > > > >=20
> > > > > > > > > > On 07.11.2017 16:05, Michael Tremer wrote:
> > > > > > > > > > > Hi,
> > > > > > > > > > >=20
> > > > > > > > > > > maybe this function helps:
> > > > > > > > > > >=20
> > > > > > > > > > > https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dcommitdi=
ff;h=3D51
> > > > > > > > > > > 90eea2
> > > > > > > > > > > 4f98
> > > > > > > > > > > 22
> > > > > > > > > > > a63d
> > > > > > > > > > > c5d06d214b48f973b14f29
> > > > > > > > > >=20
> > > > > > > > > > Thanks. I'll take a look...
> > > > > > >=20
> > > > > > > This would help, indeed, but how to implement(!?), and... (see
> > > > > > > below)
> > > > > >=20
> > > > > > Just run it somewhere in the script, figure out what is about 80%=
 of
> > > > > > that
> > > > > > and
> > > > > > if
> > > > > > that is smaller than 800 MB, we should just fail.
> > > > > >=20
> > > > > > It would be good to have a function that determines the xz
> > > > > > parameters and
> > > > > > we
> > > > > > use
> > > > > > that where ever we call xz. We don't want any duplicated code.
> > > > > >=20
> > > > > > > > > > > On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote:
> > > > > > > > > > > > On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > ...
> > > > > > > > > > > > >=20
> > > > > > > > > > > > > I forgot: Tuning the XZ-parameters... ;-)
> > > > > > > > > > > >=20
> > > > > > > > > > > > Good that you are bringing this up. So I think what we
> > > > > > > > > > > > should
> > > > > > > > > > > > do
> > > > > > > > > > > > here is
> > > > > > > > > > > > bringing back XZ compression with -8 so that we do not
> > > > > > > > > > > > waste
> > > > > > > > > > > > too
> > > > > > > > > > > > much
> > > > > > > > > > > > space
> > > > > > > > > > > > when
> > > > > > > > > > > > extracting the image again.
> > > > > > > > > >=20
> > > > > > > > > > Right now I'm testing how much RAM I need.
> > > > > > > > > >=20
> > > > > > > > > > Current 'next'-build is running with:
> > > > > > > > > >=20
> > > > > > > > > > 'lfs/cdrom':
> > > > > > > > > > ...
> > > > > > > > > > export XZ_OPT =3D --threads=3D0 -8 --memory=3D500MiB
> > > > > > > > > > ...
> > > > > > > > > >=20
> > > > > > > > > > 'lfs/Config':
> > > > > > > > > > ...
> > > > > > > > > > cd /install/packages/package/tmp/ && XZ_OPT=3D"-T0 -8" ...
> > > > > > > > > > ...
> > > > > > > > > >=20
> > > > > > > > > > I'm waiting for the results. Takes a while.
> > > > > > > > >=20
> > > > > > > > > You can just run it like this and it will tell you:
> > > > > > > > >=20
> > > > > > > > > [ms(a)hughes ~]$ xz -vv8 -T0
> > > > > > > > > xz: Filter chain: --
> > > > > > > > > lzma2=3Ddict=3D32MiB,lc=3D3,lp=3D0,pb=3D2,mode=3Dnormal,nic=
e=3D64,mf=3Dbt4,dep
> > > > > > > > > th=3D0
> > > > > > > > > xz: Using up to 4 threads.
> > > > > > > > > xz: 2,629 MiB of memory is required. The limiter is disable=
d.
> > > > > > > > > xz: Decompression will need 33 MiB of memory.
> > > > > > > > > xz: Compressed data cannot be written to a terminal
> > > > > > > > > xz: Try `xz --help' for more information.
> > > > > > > > >=20
> > > > > > > > > > > > To not run into memory limits we should detect how mu=
ch
> > > > > > > > > > > > memory
> > > > > > > > > > > > the
> > > > > > > > > > > > build
> > > > > > > > > > > > host
> > > > > > > > > > > > has and set the memory limit to maybe 80% of that. xz
> > > > > > > > > > > > will
> > > > > > > > > > > > then
> > > > > > > > > > > > automatically
> > > > > > > > > > > > scale down to a number of parallel processes that it =
can
> > > > > > > > > > > > fit
> > > > > > > > > > > > into
> > > > > > > > > > > > memory.
> > > > > > >=20
> > > > > > > This is something I couldn't get to work. First, I couldn't
> > > > > > > specify not
> > > > > > > less
> > > > > > > than '700 MiB'. If I specify '600 MiB', building stops and tells
> > > > > > > me it
> > > > > > > needs
> > > > > > > at least about 680 MiB. If I use '70%', building stops. And I
> > > > > > > never saw
> > > > > > > that
> > > > > > > 'xz' "automatically scaled down". It just stopped everytime.
> > > > > >=20
> > > > > > Yes, ~700M is the minimum.
> > > > > >=20
> > > > > > It will just reduce the number of threads again to make sure
> > > > > > everything
> > > > > > fits
> > > > > > into memory.
> > > > > >=20
> > > > > > > =20
> > > > > > > > > > I'm a bit puzzled about this: I tried using '--memory=3D7=
0%'
> > > > > > > > > > as
> > > > > > > > > > mentioned
> > > > > > > > > > in the xz man pages, but is seems to have no effect. It
> > > > > > > > > > looks as
> > > > > > > > > > if
> > > > > > > > > > the
> > > > > > > > > > percent parameter isn't recognized and ignored. It always
> > > > > > > > > > ends up
> > > > > > > > > > in
> > > > > > > > > > "cannot allocate memory". Any hints on this? Do both
> > > > > > > > > > 'lfs/cdrom'
> > > > > > > > > > AND
> > > > > > > > > > 'lfs/Config' need the '--memory'-parameter?
> > > > > > >=20
> > > > > > > This was the next problem: 'percent'-parameter does not work as
> > > > > > > expected.
> > > > > >=20
> > > > > > No, don't use it. Just determine yourself what it is.
> > > > > >=20
> > > > > > I think we could even try to set 100% of the memory, because as f=
ar
> > > > > > as I
> > > > > > can
> > > > > > see
> > > > > > this is worst case and xz won't use everything that we allow.
> > > > > >=20
> > > > > > If that does't work, let's scale down to 90%, or 80%, etc...
> > > > > >=20
> > > > > > > > > Yes, but it would be good to have one place where the
> > > > > > > > > parameters are
> > > > > > > > > being generated and then we just use them from a variable.
> > > > > > >=20
> > > > > > > Probably beyond my capabilities for the moment. Any thoughts?
> > > > > > >=20
> > > > > > > > > > > > ...
> > > > > > > > > > > > We will at least need about half a GB which should be=
 a
> > > > > > > > > > > > sensible
> > > > > > > > > > > > requirement
> > > > > > > > > > > > for a build system any ways.
> > > > > > >=20
> > > > > > > I'm afraid '500 MiB' are not enough...
> > > > > >=20
> > > > > > No, so we will now make 1G the minimum for a build.
> > > > > >=20
> > > > > > I tried to build the OS on an ARM board with only 512MB of RAM the
> > > > > > other
> > > > > > day
> > > > > > and
> > > > > > even with a lot of swap it doesn't work. So we are past this any
> > > > > > way.
> > > > > >=20
> > > > > > >=20
> > > > > > > > > > Agreed. Because of this I'm testing with '500Mib'.
> > > > > > > > > >=20
> > > > > > > > > > > > Will you work on this bringing it into the build
> > > > > > > > > > > > system?!
> > > > > > > > > >=20
> > > > > > > > > > "I'll do my very best!" :-))
> > > > > > > > >=20
> > > > > > > > > Very well Miss Sophie...
> > > > > > > > > ...
> > > > > > >=20
> > > > > > > Best=E2=84=A2,
> > > > > > > Matthias
> > > > > >=20
> > > > > > -Michael
> >=20
> >=20
>=20
>=20

--===============0319849236148878644==
Content-Type: application/pgp-signature
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="signature.asc"
MIME-Version: 1.0

LS0tLS1CRUdJTiBQR1AgU0lHTkFUVVJFLS0tLS0KCmlRSXpCQUFCQ2dBZEZpRUU1L3JXNWwzR0dl
Mnlwa3R4Z0hudy8yK1FDUWNGQWxwNC8zVUFDZ2tRZ0hudy8yK1EKQ1FlQytCQUFqTWtlSTRWVDdI
ZzU2Rmd3c3ZRVzFES1ZtTnM0enBHWC9vMXVOOWJONCtGWUJoTmJhWEFuRGhBRgo5MzltaWhPQURt
dzc2TjY4QWVJY2d5VUpaTlg4ZTAyOEVZNWpxTXczWlZWVXhYZk1SYzluYUd4SHZkOXpMaWJnCmdU
OUJLanJoaGtTczA0THR4aGh3SEZ1L0JaOUpQTzQ4QXZYQi9CZThOaGJzRFpnOXRNS1phWU00ejVH
RDhGdloKYURpTnUwUHVWK2xpVkVEQ09JeGlBV2JKRk1kY1hOZ0tLTk01YlVwbVNacUk5TGduVGZF
SU1CQnVMZHVnUDNLMgpXaDBqb1lLL25wWFE4SU4wVFdXVHlidG9GUEhnN09Kd1hjSmErRTVKYmNK
bVdkamhVeEc2ZlQrVldNakp5OFk4ClM1a1RKbG9kNHc1SnNZVjgrVTZYNHVtN1NUK09JZDJBOHY4
VlRzZThxb0U5aFZVVTVyallpeGt4OVhXNVE2YTIKZ1FtMGg1cllneHRlY2YyMHdvbXZPd2M4dGh6
OUl4L0M0c0N4UisrSGZLRkRyQlZlSlVGQjZWMFBlMHZZc1lKdQpkRlNYclVxRmRPVTZiL0JDRlFn
clg4TFdKb3FxSklnK0p4ZXpqN0EzdUNLV0NTQk1jSHZBd2tMb3lmanpYdDM0CmxJZWRnNFFuQjYv
NzEzcSt1ZXpyRTA3WWY0RzgyVTIwd2x4MzRTYUQ2ZGFiL1pJZlE0aURBOVdTbWVSYlVIVHoKSmJp
cEdCV3JOSGFqd0NaVnRvK1BkUVU1cFpuTWhNOXJaNkM2dUoxV2pJa1dFclhVUjlRbzlLVHB3N2I4
elBVUQpudkludUJXVkkvZk1wNElWa3FIK2R3UHdQdW05MXdmUnB5RzdvcDY2Yk9JWE4xbHFvbDQ9
Cj1udjFvCi0tLS0tRU5EIFBHUCBTSUdOQVRVUkUtLS0tLQo=

--===============0319849236148878644==--