From mboxrd@z Thu Jan 1 00:00:00 1970 From: Tom Rymes To: development@lists.ipfire.org Subject: Re: Compress images with XZ Date: Fri, 11 May 2018 20:35:15 -0400 Message-ID: In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============5282642694086760312==" List-Id: --===============5282642694086760312== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Would it make sense to make the firmware files a separate download, as only t= hose who use wireless Red or hostAPD need them? Tom > On May 11, 2018, at 4:28 PM, Michael Tremer w= rote: >=20 > Hey Matthias, >=20 > are you still working on this? >=20 > I am asking because I just noticed that the new next build is about 50MB la= rger > than the old one due to the larger kernel and loads more of firmware files. >=20 > Compressing it better would of course help to save space on disk. >=20 > Best, > -Michael >=20 >> On Sun, 2018-02-11 at 19:57 +0000, Michael Tremer wrote: >> Hi, >>=20 >>> On Sun, 2018-02-11 at 15:35 +0000, Matthias Fischer wrote: >>> Hi, >>>=20 >>> On 06.02.2018 02:05, Michael Tremer wrote: >>>>> ... >>>>> ((XZ_MEM=3D(($HOST_MEM)/10)*7)) && XZ_MEM=3D${XZ_MEM}MiB >>>>=20 >>>> You can write this even short: >>>>=20 >>>> XZ_MEM=3D"$(( HOST_MEM * 7 / 10 ))MiB" >>>=20 >>> Oneliner! Yes. I was looking for something like that... ;-) >>>=20 >>>> I swapped the order of the multiplication and division so that you would >>>> have >>>> a >>>> more exact result. >>>>=20 >>>> For example if you system has 1024 MB or RAM, you divide this by 10 (102) >>>> and >>>> then multiply by 7 which gives you 714. The other order is 1024 * 7 =3D = 7167 >>>> and >>>> then divided by 10 is 716 which is closer to the actual result of 716.8. >>>> Shouldn't matter here but just in general this is better :) >>>=20 >>> Thanks, looks better - noted for the next time! >>>=20 >>> Now it looks like this ('make.sh'): >>>=20 >>> ... >>> # Host memory is ok, calculating XZ memory >>> XZ_MEM=3D"$(( HOST_MEM * 7 / 10 ))MiB" >>> XZ_OPT=3D"--threads=3D0 -8 $XZ_MEM" >>> echo XZ-Memory: $XZ_MEM >>> echo XZ-Options: $XZ_OPT >>> echo "XZ memory size is OK (must be at least 700MiB), starting build..= ." >>> ... >>> ["echoes" for debugging!] >>>=20 >>> This is working. '$XZ_OPT' now gives me: --threads=3D0 -8 5450MiB >>=20 >> Will there be some checks in make.sh now that check if at least 1024MB >> of memory are available for build? I think there should be a warning if >> not enough memory is available, but we should still try to build. >>=20 >>> But. ;-) >>>=20 >>>> You can put this probably next under "HOST_MEM=3D$(system_memory)" and t= hen >>>> export >>>> this to the internal shell by adding the variable to lfsmake2 for the di= sk >>>> image >>>> and ipfiredist for the packages. >>>=20 >>> Sorry, I must confess that I don't know exactly how to do this. As I wrot= e: >>> "How >>> do >>> I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?" Or in this case, >>> XZ_OPT!? >>>=20 >>>> Then in lfs/cdrom and lfs/Config for the dist >>>> target, you just need to add the parameters to the command line. >>>=20 >>> In 'Config' I changed: >>>=20 >>> ... >>> cd /install/packages/package/tmp/ && XZ_OPT=3D-T0 tar -c -p --numeric-own= er -J >>> -f >>> /install/packages/package/files.tar.xz * >>> ... >>>=20 >>> To: >>> ... >>> cd /install/packages/package/tmp/ && $XZ_OPT tar -c -p --numeric-owner -J= -f >>> /install/packages/package/files.tar.xz * >>> ... >>>=20 >>>> Maybe it is a good idea to have a variable XZ_OPT that adds everything >>>> together >>>> and is then used in both places. >>>=20 >>> I added the other xz-options and created XZ_OPT as shown above, but in >>> 'cdrom' >>> there >>> is an 'export XZ_OPT =3D ...'-line and I don't know how to handle these l= ine. >>> Is >>> it still necessary? >>=20 >> No, that is no longer necessary and can be removed. We should just add >> it to the tar command just like it is done in lfs/Config. >>=20 >>> Do I have to 'export' the XZ_OPT'-variable in 'make.sh'? >>=20 >> No, because the chroot environment will throw away all environment >> variables from the host system and then set them again. That's why we >> have this enterchroot() function that does this with the env command. >>=20 >>> And how do I "add" this variable to 'lfsmake2' and 'ipfiredist'(-function= )? >>=20 >> Just have a look at the functions. There should be loads of examples >> where we pass other variables and you just need to add this one, too. >>=20 >>> Sorry if these questions sound simple, but I'm guessing a bit too much wh= ile >>> trying >>> to get a grip on this. >>=20 >> This is absolutely fine to sort this out first and then send the patch >> :) That's what this list is for. >>=20 >> Best, >> -Michael >>=20 >>>=20 >>> Best, >>> Matthias >>>=20 >>>>=20 >>>> Best, >>>> -Michael >>>>=20 >>>>> ... >>>>>=20 >>>>> Looks better. >>>>>=20 >>>>> Best, >>>>> Matthias >>>>>=20 >>>>>> On 03.02.2018 20:44, Matthias Fischer wrote: >>>>>> Hi, >>>>>>=20 >>>>>>> On 30.01.2018 21:03, Michael Tremer wrote: >>>>>>> Hi, >>>>>>>=20 >>>>>>> ... >>>>>>>>=20 >>>>>>>> I'll consider how I would do this and send it in as soon as I get >>>>>>>> a >>>>>>>> grip >>>>>>>> on it. >>>>>>>=20 >>>>>>> Well, I added some functions already somewhere. I suppose you just >>>>>>> need >>>>>>> to >>>>>>> do >>>>>>> the maths to set that to X% of the host system which is just a >>>>>>> simple >>>>>>> multiplication of the output of ${HOST_MEM} in make.sh. >>>>>>=20 >>>>>> Ok, I take my heart in my hand. Here are my first results - but >>>>>> remember, >>>>>> I'm not familiar with this. I built a short test script which looks as >>>>>> if >>>>>> it >>>>>> is working as wanted, but... >>>>>>=20 >>>>>>> And that needs to be passed to the lfs scripts that are supposed to >>>>>>> make >>>>>>> the >>>>>>> images. >>>>>>=20 >>>>>> ...this is a problem I'm struggling with right now: >>>>>> How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'? >>>>>>=20 >>>>>> I think this can be done using 'export XZ_MEM' or something else but >>>>>> I'm >>>>>> not >>>>>> sure. >>>>>>=20 >>>>>> This is the script I'm testing with: >>>>>>=20 >>>>>> ***SNIP*** >>>>>> #!/bin/bash >>>>>>=20 >>>>>> system_memory() { >>>>>> local key val unit >>>>>>=20 >>>>>> while read -r key val unit; do >>>>>> case "${key}" in >>>>>> MemTotal:*) >>>>>> # Convert to MB >>>>>> echo "$(( ${val} / 1024 ))" >>>>>> break >>>>>> ;; >>>>>> esac >>>>>> done < /proc/meminfo >>>>>> } >>>>>>=20 >>>>>> HOST_MEM=3D$(system_memory) >>>>>>=20 >>>>>> # Checking host memory >>>>>> if [ $HOST_MEM -ge 1024 ]; then >>>>>> echo Host-Memory: $HOST_MEM'MiB' >>>>>> echo "Host memory is OK (must be at least 1024MiB), starting >>>>>> build." >>>>>> else >>>>>> echo Host-Memory: $HOST_MEM'MiB' >>>>>> echo "Host memory too low (less than 1024MiB), building >>>>>> stopped." >>>>>> exit 1 >>>>>> fi >>>>>>=20 >>>>>> # Host memory is ok, calculating needed XZ memory to about 70% of host >>>>>> memory >>>>>> ((XZ_MEM=3D(($HOST_MEM)/10)*7)) >>>>>> XZ_MEM=3D$XZ_MEM'Mib' >>>>>> echo XZ-Memory: $XZ_MEM >>>>>> echo "XZ memory size is OK (must be at least 700MiB), starting >>>>>> build." >>>>>> ***SNAP*** >>>>>>=20 >>>>>> The 'echoes' are mostly for debugging. >>>>>>=20 >>>>>> Calculation looks ok, 'MiB' must added without blanks (XZ man page: >>>>>> "In >>>>>> most >>>>>> places where an integer argument is expected, an optional suffix is >>>>>> supported >>>>>> to easily indicate large integers. There must be no space between the >>>>>> integer >>>>>> and the suffix.") >>>>>>=20 >>>>>>> Please send questions where ever you need help. There is no reason >>>>>>> to do >>>>>>> this >>>>>>> alone. >>>>>>=20 >>>>>> Please see above: >>>>>>=20 >>>>>> 1. Is this code what you where thinking of? I would leave the other >>>>>> parameters >>>>>> as they are (--threads=3D0 -8). >>>>>>=20 >>>>>> 2. Where should I add such code in the first place? I'd like to start >>>>>> with >>>>>> 'make.sh', so building stops as soon as possible if there is not >>>>>> enough >>>>>> memory. Where's >>>>>> the best place? Right after function 'system_memory() {...}'? >>>>>>=20 >>>>>> 3. How to I pass contents of 'XZ_MEM' to 'cdrom' and 'Config'? >>>>>> I consider something like: >>>>>> ... >>>>>> export XZ_OPT =3D --threads=3D0 -8 --memory=3D$XZ_MEM >>>>>> ... >>>>>>=20 >>>>>> Best, >>>>>> Matthias >>>>>>=20 >>>>>>> Best, >>>>>>> -Michael >>>>>>>=20 >>>>>>>>=20 >>>>>>>> Best, >>>>>>>> Matthias >>>>>>>>> Best, >>>>>>>>> -Michael >>>>>>>>>=20 >>>>>>>>>> On Wed, 2017-11-22 at 16:37 +0000, Michael Tremer wrote: >>>>>>>>>> Hi, >>>>>>>>>>=20 >>>>>>>>>>> On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer wrote: >>>>>>>>>>> Hi, >>>>>>>>>>>=20 >>>>>>>>>>>> On 21.11.2017 15:42, Michael Tremer wrote: >>>>>>>>>>>> Hello, >>>>>>>>>>>>=20 >>>>>>>>>>>> any progress on this? >>>>>>>>>>>=20 >>>>>>>>>>> Not much, because I've been struggling with a flu for the >>>>>>>>>>> past >>>>>>>>>>> two >>>>>>>>>>> weeks, and its not over yet, but: >>>>>>>>>>>=20 >>>>>>>>>>> In the meantime I ran several builds with different >>>>>>>>>>> parameters. >>>>>>>>>>>=20 >>>>>>>>>>> IMHO we could use: >>>>>>>>>>>=20 >>>>>>>>>>> 'export XZ_OPT =3D --threads=3D0 -8 --memory=3D700MiB' >>>>>>>>>>>=20 >>>>>>>>>>> in 'lfs/cdrom' >>>>>>>>>>>=20 >>>>>>>>>>> and >>>>>>>>>>>=20 >>>>>>>>>>> cd /install/packages/package/tmp/ && XZ_OPT=3D"-T0 -8 -- >>>>>>>>>>> memory=3D700MiB" tar >>>>>>>>>>> -c >>>>>>>>>>> -p >>>>>>>>>>> --numeric-owner -J -f /install/packages/package/files.tar.xz >>>>>>>>>>> * >>>>>>>>>>>=20 >>>>>>>>>>> in 'lfs/Config'. >>>>>>>>>>>=20 >>>>>>>>>>> Using '9' or '--extreme'-parameter brought no further >>>>>>>>>>> improvements, >>>>>>>>>>> on the other hand, build time and RAM consumption increased >>>>>>>>>>> strongly. With '-8', minimal RAM consumption was about 680 >>>>>>>>>>> MiB, >>>>>>>>>>> using >>>>>>>>>>> '700 Mib' was always safe. So this is the minimum I would >>>>>>>>>>> recommend. >>>>>>>>>>=20 >>>>>>>>>> Yes, I observed the same thing. -9 compresses a little bit >>>>>>>>>> better >>>>>>>>>> and >>>>>>>>>> extreme >>>>>>>>>> doesn't change a bit. >>>>>>>>>>=20 >>>>>>>>>> -8 seems to be optimal since it won't require too much memory >>>>>>>>>> to >>>>>>>>>> decompress >>>>>>>>>> either. So let's go with that. >>>>>>>>>>=20 >>>>>>>>>>> Results: >>>>>>>>>>>=20 >>>>>>>>>>> (Buildtime: 5 hours 20 minutes 59 seconds) >>>>>>>>>>>=20 >>>>>>>>>>> 255866764 ipfire-2.19.1gb-ext4.i586-full-core117.img.gz >>>>>>>>>>> 252262006 ipfire-2.19.1gb-ext4-scon.i586-full- >>>>>>>>>>> core117.img.gz >>>>>>>>>>> 175112192 ipfire-2.19.i586-full-core117.iso >>>>>>>>>>>=20 >>>>>>>>>>>> -Michael >>>>>>>>>>>>=20 >>>>>>>>>>>>> On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote: >>>>>>>>>>>>> Hi, >>>>>>>>>>>>>=20 >>>>>>>>>>>>> On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> On 07.11.2017 16:05, Michael Tremer wrote: >>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> maybe this function helps: >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dcommitdif >>>>>>>>>>>>>>> f;h=3D >>>>>>>>>>>>>>> 51 >>>>>>>>>>>>>>> 90eea2 >>>>>>>>>>>>>>> 4f98 >>>>>>>>>>>>>>> 22 >>>>>>>>>>>>>>> a63d >>>>>>>>>>>>>>> c5d06d214b48f973b14f29 >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> Thanks. I'll take a look... >>>>>>>>>>>=20 >>>>>>>>>>> This would help, indeed, but how to implement(!?), and... >>>>>>>>>>> (see >>>>>>>>>>> below) >>>>>>>>>>=20 >>>>>>>>>> Just run it somewhere in the script, figure out what is about >>>>>>>>>> 80% >>>>>>>>>> of >>>>>>>>>> that >>>>>>>>>> and >>>>>>>>>> if >>>>>>>>>> that is smaller than 800 MB, we should just fail. >>>>>>>>>>=20 >>>>>>>>>> It would be good to have a function that determines the xz >>>>>>>>>> parameters and >>>>>>>>>> we >>>>>>>>>> use >>>>>>>>>> that where ever we call xz. We don't want any duplicated code. >>>>>>>>>>=20 >>>>>>>>>>>>>>> On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> On Mon, 2017-11-06 at 22:09 +0100, Matthias >>>>>>>>>>>>>>>> Fischer >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>> I forgot: Tuning the XZ-parameters... ;-) >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> Good that you are bringing this up. So I think >>>>>>>>>>>>>>>> what we >>>>>>>>>>>>>>>> should >>>>>>>>>>>>>>>> do >>>>>>>>>>>>>>>> here is >>>>>>>>>>>>>>>> bringing back XZ compression with -8 so that we do >>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>> waste >>>>>>>>>>>>>>>> too >>>>>>>>>>>>>>>> much >>>>>>>>>>>>>>>> space >>>>>>>>>>>>>>>> when >>>>>>>>>>>>>>>> extracting the image again. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> Right now I'm testing how much RAM I need. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> Current 'next'-build is running with: >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> 'lfs/cdrom': >>>>>>>>>>>>>> ... >>>>>>>>>>>>>> export XZ_OPT =3D --threads=3D0 -8 --memory=3D500MiB >>>>>>>>>>>>>> ... >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> 'lfs/Config': >>>>>>>>>>>>>> ... >>>>>>>>>>>>>> cd /install/packages/package/tmp/ && XZ_OPT=3D"-T0 -8" >>>>>>>>>>>>>> ... >>>>>>>>>>>>>> ... >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> I'm waiting for the results. Takes a while. >>>>>>>>>>>>>=20 >>>>>>>>>>>>> You can just run it like this and it will tell you: >>>>>>>>>>>>>=20 >>>>>>>>>>>>> [ms(a)hughes ~]$ xz -vv8 -T0 >>>>>>>>>>>>> xz: Filter chain: -- >>>>>>>>>>>>> lzma2=3Ddict=3D32MiB,lc=3D3,lp=3D0,pb=3D2,mode=3Dnormal,nice=3D= 64,mf=3Db >>>>>>>>>>>>> t4,d >>>>>>>>>>>>> ep >>>>>>>>>>>>> th=3D0 >>>>>>>>>>>>> xz: Using up to 4 threads. >>>>>>>>>>>>> xz: 2,629 MiB of memory is required. The limiter is >>>>>>>>>>>>> disabled. >>>>>>>>>>>>> xz: Decompression will need 33 MiB of memory. >>>>>>>>>>>>> xz: Compressed data cannot be written to a terminal >>>>>>>>>>>>> xz: Try `xz --help' for more information. >>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> To not run into memory limits we should detect how >>>>>>>>>>>>>>>> much >>>>>>>>>>>>>>>> memory >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> build >>>>>>>>>>>>>>>> host >>>>>>>>>>>>>>>> has and set the memory limit to maybe 80% of that. >>>>>>>>>>>>>>>> xz >>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>> then >>>>>>>>>>>>>>>> automatically >>>>>>>>>>>>>>>> scale down to a number of parallel processes that >>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>> fit >>>>>>>>>>>>>>>> into >>>>>>>>>>>>>>>> memory. >>>>>>>>>>>=20 >>>>>>>>>>> This is something I couldn't get to work. First, I couldn't >>>>>>>>>>> specify not >>>>>>>>>>> less >>>>>>>>>>> than '700 MiB'. If I specify '600 MiB', building stops and >>>>>>>>>>> tells >>>>>>>>>>> me it >>>>>>>>>>> needs >>>>>>>>>>> at least about 680 MiB. If I use '70%', building stops. And >>>>>>>>>>> I >>>>>>>>>>> never saw >>>>>>>>>>> that >>>>>>>>>>> 'xz' "automatically scaled down". It just stopped everytime. >>>>>>>>>>=20 >>>>>>>>>> Yes, ~700M is the minimum. >>>>>>>>>>=20 >>>>>>>>>> It will just reduce the number of threads again to make sure >>>>>>>>>> everything >>>>>>>>>> fits >>>>>>>>>> into memory. >>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>>>>> I'm a bit puzzled about this: I tried using ' >>>>>>>>>>>>>> --memory=3D70%' >>>>>>>>>>>>>> as >>>>>>>>>>>>>> mentioned >>>>>>>>>>>>>> in the xz man pages, but is seems to have no effect. >>>>>>>>>>>>>> It >>>>>>>>>>>>>> looks as >>>>>>>>>>>>>> if >>>>>>>>>>>>>> the >>>>>>>>>>>>>> percent parameter isn't recognized and ignored. It >>>>>>>>>>>>>> always >>>>>>>>>>>>>> ends up >>>>>>>>>>>>>> in >>>>>>>>>>>>>> "cannot allocate memory". Any hints on this? Do both >>>>>>>>>>>>>> 'lfs/cdrom' >>>>>>>>>>>>>> AND >>>>>>>>>>>>>> 'lfs/Config' need the '--memory'-parameter? >>>>>>>>>>>=20 >>>>>>>>>>> This was the next problem: 'percent'-parameter does not work >>>>>>>>>>> as >>>>>>>>>>> expected. >>>>>>>>>>=20 >>>>>>>>>> No, don't use it. Just determine yourself what it is. >>>>>>>>>>=20 >>>>>>>>>> I think we could even try to set 100% of the memory, because >>>>>>>>>> as >>>>>>>>>> far >>>>>>>>>> as I >>>>>>>>>> can >>>>>>>>>> see >>>>>>>>>> this is worst case and xz won't use everything that we allow. >>>>>>>>>>=20 >>>>>>>>>> If that does't work, let's scale down to 90%, or 80%, etc... >>>>>>>>>>=20 >>>>>>>>>>>>> Yes, but it would be good to have one place where the >>>>>>>>>>>>> parameters are >>>>>>>>>>>>> being generated and then we just use them from a >>>>>>>>>>>>> variable. >>>>>>>>>>>=20 >>>>>>>>>>> Probably beyond my capabilities for the moment. Any >>>>>>>>>>> thoughts? >>>>>>>>>>>=20 >>>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>>> We will at least need about half a GB which should >>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>> sensible >>>>>>>>>>>>>>>> requirement >>>>>>>>>>>>>>>> for a build system any ways. >>>>>>>>>>>=20 >>>>>>>>>>> I'm afraid '500 MiB' are not enough... >>>>>>>>>>=20 >>>>>>>>>> No, so we will now make 1G the minimum for a build. >>>>>>>>>>=20 >>>>>>>>>> I tried to build the OS on an ARM board with only 512MB of RAM >>>>>>>>>> the >>>>>>>>>> other >>>>>>>>>> day >>>>>>>>>> and >>>>>>>>>> even with a lot of swap it doesn't work. So we are past this >>>>>>>>>> any >>>>>>>>>> way. >>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>>>>> Agreed. Because of this I'm testing with '500Mib'. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> Will you work on this bringing it into the build >>>>>>>>>>>>>>>> system?! >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> "I'll do my very best!" :-)) >>>>>>>>>>>>>=20 >>>>>>>>>>>>> Very well Miss Sophie... >>>>>>>>>>>>> ... >>>>>>>>>>>=20 >>>>>>>>>>> Best=E2=84=A2, >>>>>>>>>>> Matthias >>>>>>>>>>=20 >>>>>>>>>> -Michael --===============5282642694086760312==--