Re: Compress images with XZ

[Matthias Fischer <matthias.fischer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 6894 bytes --]

HI,

On 11.05.2018 22:27, Michael Tremer wrote:
> Hey Matthias,
> 
> are you still working on this?

Its still on my list, but I must confess: I didn't get a grip on this yet.
 
> I am asking because I just noticed that the new next build is about 50MB larger
> than the old one due to the larger kernel and loads more of firmware files.

Yes, I saw this too. I was waiting for you to ask... ;-)

I'll try to explain what I did and where I'm stuck:

> ...
>> > 
>> > On 06.02.2018 02:05, Michael Tremer wrote:
>> > > > ...
>> > > > ((XZ_MEM=(($HOST_MEM)/10)*7)) && XZ_MEM=${XZ_MEM}MiB
>> > > 
>> > > You can write this even short:
>> > > 
>> > > XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
>> > 
>> > Oneliner! Yes. I was looking for something like that... ;-)

This did work, no problem here.

>> > ...
>> > # Host memory is ok, calculating XZ memory
>> > 	XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
>> > 	XZ_OPT="--threads=0 -8 $XZ_MEM"
>> > 	echo XZ-Memory: $XZ_MEM
>> > 	echo XZ-Options: $XZ_OPT
>> > 	echo "XZ memory size is OK (must be at least 700MiB), starting build..."
>> > ...
>> > ["echoes" for debugging!]
>> > 
>> > This is working. '$XZ_OPT' now gives me: --threads=0 -8 5450MiB
>> 
>> Will there be some checks in make.sh now that check if at least 1024MB
>> of memory are available for build?
>> ...

For this, I changed the above ("read -p..."-lines are just for debugging):

***SNIP***
...
# Get the amount of memory in this build system
HOST_MEM=$(system_memory)

# XZ-TUNING BEGIN
# Checking host memory

if [ $HOST_MEM -ge 1024 ]; then
	print_build_stage "Host-Memory: $HOST_MEM MiB"
	print_build_stage "Host memory is OK (must be at least 1024 MiB), calculating XZ memory..."

# Host memory is ok, calculating XZ memory
echo

	XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
	XZ_OPT="--threads=0 -8 $XZ_MEM"
	print_build_stage "XZ-Memory:" $XZ_MEM
	print_build_stage "XZ-Options:" $XZ_OPT
	print_build_stage "XZ memory size is OK (must be at least 700 MiB), moving on..."

echo
read -p "Press [Enter] key to continue..."

else

	print_build_stage "Host-Memory: $HOST_MEM MiB"
	print_build_stage "Not enough host memory (less than 1024 MiB, consider upgrading),"
	print_build_stage "building will use standard XZ options."
	XZ_OPT="-T0" # alte Standardparameter einsetzen

echo
read -p "Press [Enter] key to continue..."

fi
...
# XZ-TUNING END

***SNAP***

>> ...
>>  I think there should be a warning if
>>  not enough memory is available, but we should still try to build.

As far as I can see, this could be done by replacing the above 'echoes'
with 'print_build_stage'(?).

E.g.:
print_build_stage "Not enough host memory (less than 1024MiB, consider upgrading),"
print_build_stage "building will use standard XZ options."

>> ...
>> > But. ;-)
>> > 
>> > > You can put this probably next under "HOST_MEM=$(system_memory)" and then
>> > > export this to the internal shell by adding the variable to lfsmake2 for
>> > > the disk image and ipfiredist for the packages.
>> > 
>> > Sorry, I must confess that I don't know exactly how to do this. As I wrote:
>> > "How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?" Or in this case,
>> > XZ_OPT!?

And this is still my problem! I don't get this to work. How do I "pass" this variable?

>> > > Then in lfs/cdrom and lfs/Config for the dist
>> > > target, you just need to add the parameters to the command line.
>> > 
>> > In 'Config' I changed:
>> > 
>> > ...
>> > cd /install/packages/package/tmp/ && XZ_OPT=-T0 tar -c -p --numeric-owner -J
>> > -f
>> > /install/packages/package/files.tar.xz *
>> > ...
>> > 
>> > To:
>> > ...
>> > cd /install/packages/package/tmp/ && $XZ_OPT tar -c -p --numeric-owner -J -f
>> > /install/packages/package/files.tar.xz *
>> > ...
>> > 
>> > > Maybe it is a good idea to have a variable XZ_OPT that adds everything
>> > > together and is then used in both places.

I tried that with no luck. I added the variable '$XZ_OPT' to 'lfs/Config' in line
212:

cd /install/packages/package/tmp/ && $XZ_OPT tar -c -p --numeric-owner...
 
But this usually ends - like today - in '_build_packages.log' with:

***SNIP***
...
#Change xxxKVERxxx to Kernelversion
sed -e "s/xxxKVERxxx/4.14.40/g" -i /install/packages/package/update.sh
cd /install/packages/package && tar cf ../core-upgrade-2.19-$(basename core/121).ipfire \
	update.sh files.tar.xz ROOTFILES
rm -rf /install/packages/package
sed -e "s/NAME/core-upgrade/g" \
	-e "s/VER/2.19/g" \
	-e "s/RELEASE/$(basename core/121)/g" \
	-e "s/DEPS//g" \
	-e "s/SIZE/`ls -l \/install\/packages\/core-upgrade-2.19-$(basename core/121).ipfire | awk '{ print $5 }'`/g" \
	< /usr/src/src/pakfire/meta > /install/packages/meta-core-upgrade-$(basename core/121)
May 12 13:39:49: Building directfb DirectFB-1.7.7.tar.gz checksum OK
+ cd /usr/src/lfs
+ make -f directfb LFS_BASEDIR=/usr/src dist
'/usr/src/config/rootfiles/packages/i586/directfb' -> '/install/packages/package/ROOTFILES'
/bin/sh: Z_OPT: command not found
make: *** [directfb:60: dist] Error 127
...
***SNAP***

Why "Z_OPT"?

>> > 
>> > I added the other xz-options and created XZ_OPT as shown above, but in
>> > 'cdrom' there is an 'export XZ_OPT = ...'-line and I don't know how to
>> > handle these line.
>> > Is it still necessary?
>> 
>> No, that is no longer necessary and can be removed. We should just add
>> it to the tar command just like it is done in lfs/Config.

Understanding problem:
What do mean with "adding it to the tar command"? For me, this sounds as if
we need an "export $XZ_OPT"-line here...

>> > Do I have to 'export' the XZ_OPT'-variable in 'make.sh'?
>> 
>> No, because the chroot environment will throw away all environment
>> variables from the host system and then set them again. That's why we
>> have this enterchroot() function that does this with the env command.
>> 
>> > And how do I "add" this variable to 'lfsmake2' and 'ipfiredist'(-function)?
>> 
>> Just have a look at the functions. There should be loads of examples
>> where we pass other variables and you just need to add this one, too.

I think I've found these "examples" and tried several "local XZ_OPT='$XZ_OPT'"
placements and versions in 'lfsmake2' and 'ipfiredist' but none seemed to work
as I wanted.

>> > Sorry if these questions sound simple, but I'm guessing a bit too much while
>> > trying to get a grip on this.
>> > 
>> This is absolutely fine to sort this out first and then send the patch
>> :) That's what this list is for.

After several attempts I stopped guessing and concentrated on something
I could handle. Somehow I've got the feeling I'm making some really simple
mistakes - for reviewing I added my current test patches as attachments.

Please advise... ;-)

Best,
Matthias



[-- Attachment #2: 2018-05-12_xz-tuning.patch --]
[-- Type: text/plain, Size: 3470 bytes --]

--- cdrom	Sun May 06 20:15:10 2018
+++ cdrom	Mon Feb 19 20:35:38 2018
@@ -35,9 +35,6 @@
 	TAR_OPTIONS = --xz
 endif
 
-# Enable multi-threaded compression for LZMA
-export XZ_OPT = --threads=0
-
 ###############################################################################
 # Top-level Rules
 ###############################################################################
--- Config	Sun May 06 20:15:10 2018
+++ Config	Sun Feb 18 14:12:48 2018
@@ -209,7 +209,7 @@
 	tar xf /install/packages/package/files.tmp -C /install/packages/package/tmp/ \
 		-p --numeric-owner
 	rm -f /install/packages/package/files.tmp
-	cd /install/packages/package/tmp/ && XZ_OPT=-T0 tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz *
+	cd /install/packages/package/tmp/ && $XZ_OPT tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz *
 	rm -r /install/packages/package/tmp
 	-cat /install/packages/package/ROOTFILES | grep -v "#" > /install/packages/package/ROOTFILES.tmp
 	mv /install/packages/package/ROOTFILES.tmp /install/packages/package/ROOTFILES
--- make.sh	Thu May 10 09:53:08 2018
+++ make.sh	Sat May 12 16:03:55 2018
@@ -632,6 +632,10 @@
 
 	local PS1='\u:\w$ '
 
+# XZ-TUNING BEGIN
+	local XZ_OPT='$XZ_OPT'
+# XZ-TUNING END
+
 	enterchroot \
 		${EXTRA_PATH}bash -x -c "cd /usr/src/lfs && \
 			MAKETUNING=${MAKETUNING} \
@@ -653,6 +657,10 @@
 
 	local PS1='\u:\w$ '
 
+# XZ-TUNING BEGIN
+	local XZ_OPT='$XZ_OPT'
+# XZ-TUNING END
+
 	enterchroot \
 		bash -x -c "cd /usr/src/lfs && make -f $* LFS_BASEDIR=/usr/src dist" \
 		>> ${LOGFILE} 2>&1 &
@@ -860,6 +868,38 @@
 # Get the amount of memory in this build system
 HOST_MEM=$(system_memory)
 
+# XZ-TUNING BEGIN
+# Checking host memory
+
+if [ $HOST_MEM -ge 1024 ]; then
+	print_build_stage "Host-Memory: $HOST_MEM MiB"
+	print_build_stage "Host memory is OK (must be at least 1024 MiB), calculating XZ memory..."
+
+# Host memory is ok, calculating XZ memory
+echo
+
+	XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
+	XZ_OPT="--threads=0 -8 $XZ_MEM"
+	print_build_stage "XZ-Memory:" $XZ_MEM
+	print_build_stage "XZ-Options:" $XZ_OPT
+	print_build_stage "XZ memory size is OK (must be at least 700 MiB), moving on..."
+
+echo
+read -p "Press [Enter] key to continue..."
+
+else
+
+	print_build_stage "Host-Memory: $HOST_MEM MiB"
+	print_build_stage "Not enough host memory (less than 1024 MiB, consider upgrading),"
+	print_build_stage "building will use standard XZ options."
+	XZ_OPT="-T0" # alte Standardparameter einsetzen
+
+echo
+read -p "Press [Enter] key to continue..."
+
+fi
+# XZ-TUNING END
+
 if [ -n "${BUILD_ARCH}" ]; then
 	configure_build "${BUILD_ARCH}"
 else
@@ -1760,7 +1800,7 @@
 	buildtoolchain
 	echo "`date -u '+%b %e %T'`: Create toolchain image for ${BUILD_ARCH}" | tee -a $LOGFILE
 	test -d $BASEDIR/cache/toolchains || mkdir -p $BASEDIR/cache/toolchains
-	cd $BASEDIR && XZ_OPT="-T0 -8" tar -Jc --exclude='log/_build.*.log' -f cache/toolchains/$SNAME-$VERSION-toolchain-$TOOLCHAINVER-${BUILD_ARCH}.tar.xz \
+	cd $BASEDIR && $XZ_OPT tar -Jc --exclude='log/_build.*.log' -f cache/toolchains/$SNAME-$VERSION-toolchain-$TOOLCHAINVER-${BUILD_ARCH}.tar.xz \
 		build/${TOOLS_DIR} build/bin/sh log >> $LOGFILE
 	md5sum cache/toolchains/$SNAME-$VERSION-toolchain-$TOOLCHAINVER-${BUILD_ARCH}.tar.xz \
 		> cache/toolchains/$SNAME-$VERSION-toolchain-$TOOLCHAINVER-${BUILD_ARCH}.md5