Re: Compress images with XZ

[Tom Rymes <trymes@rymes.com>]

[-- Attachment #1: Type: text/plain, Size: 17510 bytes --]

Would it make sense to make the firmware files a separate download, as only those who use wireless Red or hostAPD need them?

Tom

> On May 11, 2018, at 4:28 PM, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
> 
> Hey Matthias,
> 
> are you still working on this?
> 
> I am asking because I just noticed that the new next build is about 50MB larger
> than the old one due to the larger kernel and loads more of firmware files.
> 
> Compressing it better would of course help to save space on disk.
> 
> Best,
> -Michael
> 
>> On Sun, 2018-02-11 at 19:57 +0000, Michael Tremer wrote:
>> Hi,
>> 
>>> On Sun, 2018-02-11 at 15:35 +0000, Matthias Fischer wrote:
>>> Hi,
>>> 
>>> On 06.02.2018 02:05, Michael Tremer wrote:
>>>>> ...
>>>>> ((XZ_MEM=(($HOST_MEM)/10)*7)) && XZ_MEM=${XZ_MEM}MiB
>>>> 
>>>> You can write this even short:
>>>> 
>>>> XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
>>> 
>>> Oneliner! Yes. I was looking for something like that... ;-)
>>> 
>>>> I swapped the order of the multiplication and division so that you would
>>>> have
>>>> a
>>>> more exact result.
>>>> 
>>>> For example if you system has 1024 MB or RAM, you divide this by 10 (102)
>>>> and
>>>> then multiply by 7 which gives you 714. The other order is 1024 * 7 = 7167
>>>> and
>>>> then divided by 10 is 716 which is closer to the actual result of 716.8.
>>>> Shouldn't matter here but just in general this is better :)
>>> 
>>> Thanks, looks better - noted for the next time!
>>> 
>>> Now it looks like this ('make.sh'):
>>> 
>>> ...
>>> # Host memory is ok, calculating XZ memory
>>>    XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
>>>    XZ_OPT="--threads=0 -8 $XZ_MEM"
>>>    echo XZ-Memory: $XZ_MEM
>>>    echo XZ-Options: $XZ_OPT
>>>    echo "XZ memory size is OK (must be at least 700MiB), starting build..."
>>> ...
>>> ["echoes" for debugging!]
>>> 
>>> This is working. '$XZ_OPT' now gives me: --threads=0 -8 5450MiB
>> 
>> Will there be some checks in make.sh now that check if at least 1024MB
>> of memory are available for build? I think there should be a warning if
>> not enough memory is available, but we should still try to build.
>> 
>>> But. ;-)
>>> 
>>>> You can put this probably next under "HOST_MEM=$(system_memory)" and then
>>>> export
>>>> this to the internal shell by adding the variable to lfsmake2 for the disk
>>>> image
>>>> and ipfiredist for the packages.
>>> 
>>> Sorry, I must confess that I don't know exactly how to do this. As I wrote:
>>> "How
>>> do
>>> I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?" Or in this case,
>>> XZ_OPT!?
>>> 
>>>> Then in lfs/cdrom and lfs/Config for the dist
>>>> target, you just need to add the parameters to the command line.
>>> 
>>> In 'Config' I changed:
>>> 
>>> ...
>>> cd /install/packages/package/tmp/ && XZ_OPT=-T0 tar -c -p --numeric-owner -J
>>> -f
>>> /install/packages/package/files.tar.xz *
>>> ...
>>> 
>>> To:
>>> ...
>>> cd /install/packages/package/tmp/ && $XZ_OPT tar -c -p --numeric-owner -J -f
>>> /install/packages/package/files.tar.xz *
>>> ...
>>> 
>>>> Maybe it is a good idea to have a variable XZ_OPT that adds everything
>>>> together
>>>> and is then used in both places.
>>> 
>>> I added the other xz-options and created XZ_OPT as shown above, but in
>>> 'cdrom'
>>> there
>>> is an 'export XZ_OPT = ...'-line and I don't know how to handle these line.
>>> Is
>>> it still necessary?
>> 
>> No, that is no longer necessary and can be removed. We should just add
>> it to the tar command just like it is done in lfs/Config.
>> 
>>> Do I have to 'export' the XZ_OPT'-variable in 'make.sh'?
>> 
>> No, because the chroot environment will throw away all environment
>> variables from the host system and then set them again. That's why we
>> have this enterchroot() function that does this with the env command.
>> 
>>> And how do I "add" this variable to 'lfsmake2' and 'ipfiredist'(-function)?
>> 
>> Just have a look at the functions. There should be loads of examples
>> where we pass other variables and you just need to add this one, too.
>> 
>>> Sorry if these questions sound simple, but I'm guessing a bit too much while
>>> trying
>>> to get a grip on this.
>> 
>> This is absolutely fine to sort this out first and then send the patch
>> :) That's what this list is for.
>> 
>> Best,
>> -Michael
>> 
>>> 
>>> Best,
>>> Matthias
>>> 
>>>> 
>>>> Best,
>>>> -Michael
>>>> 
>>>>> ...
>>>>> 
>>>>> Looks better.
>>>>> 
>>>>> Best,
>>>>> Matthias
>>>>> 
>>>>>> On 03.02.2018 20:44, Matthias Fischer wrote:
>>>>>> Hi,
>>>>>> 
>>>>>>> On 30.01.2018 21:03, Michael Tremer wrote:
>>>>>>> Hi,
>>>>>>> 
>>>>>>> ...
>>>>>>>> 
>>>>>>>> I'll consider how I would do this and send it in as soon as I get
>>>>>>>> a
>>>>>>>> grip
>>>>>>>> on it.
>>>>>>> 
>>>>>>> Well, I added some functions already somewhere. I suppose you just
>>>>>>> need
>>>>>>> to
>>>>>>> do
>>>>>>> the maths to set that to X% of the host system which is just a
>>>>>>> simple
>>>>>>> multiplication of the output of ${HOST_MEM} in make.sh.
>>>>>> 
>>>>>> Ok, I take my heart in my hand. Here are my first results - but
>>>>>> remember,
>>>>>> I'm not familiar with this. I built a short test script which looks as
>>>>>> if
>>>>>> it
>>>>>> is working as wanted, but...
>>>>>> 
>>>>>>> And that needs to be passed to the lfs scripts that are supposed to
>>>>>>> make
>>>>>>> the
>>>>>>> images.
>>>>>> 
>>>>>> ...this is a problem I'm struggling with right now:
>>>>>> How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?
>>>>>> 
>>>>>> I think this can be done using 'export XZ_MEM' or something else but
>>>>>> I'm
>>>>>> not
>>>>>> sure.
>>>>>> 
>>>>>> This is the script I'm testing with:
>>>>>> 
>>>>>> ***SNIP***
>>>>>> #!/bin/bash
>>>>>> 
>>>>>> system_memory() {
>>>>>>    local key val unit
>>>>>> 
>>>>>>    while read -r key val unit; do
>>>>>>        case "${key}" in
>>>>>>            MemTotal:*)
>>>>>>                # Convert to MB
>>>>>>                echo "$(( ${val} / 1024 ))"
>>>>>>                break
>>>>>>                ;;
>>>>>>        esac
>>>>>>    done < /proc/meminfo
>>>>>> }
>>>>>> 
>>>>>> HOST_MEM=$(system_memory)
>>>>>> 
>>>>>> # Checking host memory
>>>>>> if [ $HOST_MEM -ge 1024 ]; then
>>>>>>    echo Host-Memory: $HOST_MEM'MiB'
>>>>>>    echo "Host memory is OK (must be at least 1024MiB), starting
>>>>>> build."
>>>>>> else
>>>>>>    echo Host-Memory: $HOST_MEM'MiB'
>>>>>>    echo "Host memory too low (less than 1024MiB), building
>>>>>> stopped."
>>>>>> exit 1
>>>>>> fi
>>>>>> 
>>>>>> # Host memory is ok, calculating needed XZ memory to about 70% of host
>>>>>> memory
>>>>>> ((XZ_MEM=(($HOST_MEM)/10)*7))
>>>>>>    XZ_MEM=$XZ_MEM'Mib'
>>>>>>    echo XZ-Memory: $XZ_MEM
>>>>>>    echo "XZ memory size is OK (must be at least 700MiB), starting
>>>>>> build."
>>>>>> ***SNAP***
>>>>>> 
>>>>>> The 'echoes' are mostly for debugging.
>>>>>> 
>>>>>> Calculation looks ok, 'MiB' must added without blanks (XZ man page:
>>>>>> "In
>>>>>> most
>>>>>> places where an integer argument is expected, an optional suffix is
>>>>>> supported
>>>>>> to easily indicate large integers. There must be no space between the
>>>>>> integer
>>>>>> and the suffix.")
>>>>>> 
>>>>>>> Please send questions where ever you need help. There is no reason
>>>>>>> to do
>>>>>>> this
>>>>>>> alone.
>>>>>> 
>>>>>> Please see above:
>>>>>> 
>>>>>> 1. Is this code what you where thinking of? I would leave the other
>>>>>> parameters
>>>>>> as they are (--threads=0 -8).
>>>>>> 
>>>>>> 2. Where should I add such code in the first place? I'd like to start
>>>>>> with
>>>>>> 'make.sh', so building stops as soon as possible if there is not
>>>>>> enough
>>>>>> memory. Where's
>>>>>> the best place? Right after function 'system_memory() {...}'?
>>>>>> 
>>>>>> 3. How to I pass contents of 'XZ_MEM' to 'cdrom' and 'Config'?
>>>>>> I consider something like:
>>>>>> ...
>>>>>> export XZ_OPT = --threads=0 -8 --memory=$XZ_MEM
>>>>>> ...
>>>>>> 
>>>>>> Best,
>>>>>> Matthias
>>>>>> 
>>>>>>> Best,
>>>>>>> -Michael
>>>>>>> 
>>>>>>>> 
>>>>>>>> Best,
>>>>>>>> Matthias
>>>>>>>>> Best,
>>>>>>>>> -Michael
>>>>>>>>> 
>>>>>>>>>> On Wed, 2017-11-22 at 16:37 +0000, Michael Tremer wrote:
>>>>>>>>>> Hi,
>>>>>>>>>> 
>>>>>>>>>>> On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>> 
>>>>>>>>>>>> On 21.11.2017 15:42, Michael Tremer wrote:
>>>>>>>>>>>> Hello,
>>>>>>>>>>>> 
>>>>>>>>>>>> any progress on this?
>>>>>>>>>>> 
>>>>>>>>>>> Not much, because I've been struggling with a flu for the
>>>>>>>>>>> past
>>>>>>>>>>> two
>>>>>>>>>>> weeks, and its not over yet, but:
>>>>>>>>>>> 
>>>>>>>>>>> In the meantime I ran several builds with different
>>>>>>>>>>> parameters.
>>>>>>>>>>> 
>>>>>>>>>>> IMHO we could use:
>>>>>>>>>>> 
>>>>>>>>>>> 'export XZ_OPT = --threads=0 -8 --memory=700MiB'
>>>>>>>>>>> 
>>>>>>>>>>> in 'lfs/cdrom'
>>>>>>>>>>> 
>>>>>>>>>>> and
>>>>>>>>>>> 
>>>>>>>>>>> cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8 --
>>>>>>>>>>> memory=700MiB" tar
>>>>>>>>>>> -c
>>>>>>>>>>> -p
>>>>>>>>>>> --numeric-owner -J -f /install/packages/package/files.tar.xz
>>>>>>>>>>> *
>>>>>>>>>>> 
>>>>>>>>>>> in 'lfs/Config'.
>>>>>>>>>>> 
>>>>>>>>>>> Using '9' or '--extreme'-parameter brought no further
>>>>>>>>>>> improvements,
>>>>>>>>>>> on the other hand, build time and RAM consumption increased
>>>>>>>>>>> strongly. With '-8', minimal RAM consumption was about 680
>>>>>>>>>>> MiB,
>>>>>>>>>>> using
>>>>>>>>>>> '700 Mib' was always safe. So this is the minimum I would
>>>>>>>>>>> recommend.
>>>>>>>>>> 
>>>>>>>>>> Yes, I observed the same thing. -9 compresses a little bit
>>>>>>>>>> better
>>>>>>>>>> and
>>>>>>>>>> extreme
>>>>>>>>>> doesn't change a bit.
>>>>>>>>>> 
>>>>>>>>>> -8 seems to be optimal since it won't require too much memory
>>>>>>>>>> to
>>>>>>>>>> decompress
>>>>>>>>>> either. So let's go with that.
>>>>>>>>>> 
>>>>>>>>>>> Results:
>>>>>>>>>>> 
>>>>>>>>>>> (Buildtime: 5 hours 20 minutes 59 seconds)
>>>>>>>>>>> 
>>>>>>>>>>> 255866764  ipfire-2.19.1gb-ext4.i586-full-core117.img.gz
>>>>>>>>>>> 252262006  ipfire-2.19.1gb-ext4-scon.i586-full-
>>>>>>>>>>> core117.img.gz
>>>>>>>>>>> 175112192  ipfire-2.19.i586-full-core117.iso
>>>>>>>>>>> 
>>>>>>>>>>>> -Michael
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote:
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 07.11.2017 16:05, Michael Tremer wrote:
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> maybe this function helps:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdif
>>>>>>>>>>>>>>> f;h=
>>>>>>>>>>>>>>> 51
>>>>>>>>>>>>>>> 90eea2
>>>>>>>>>>>>>>> 4f98
>>>>>>>>>>>>>>> 22
>>>>>>>>>>>>>>> a63d
>>>>>>>>>>>>>>> c5d06d214b48f973b14f29
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks. I'll take a look...
>>>>>>>>>>> 
>>>>>>>>>>> This would help, indeed, but how to implement(!?), and...
>>>>>>>>>>> (see
>>>>>>>>>>> below)
>>>>>>>>>> 
>>>>>>>>>> Just run it somewhere in the script, figure out what is about
>>>>>>>>>> 80%
>>>>>>>>>> of
>>>>>>>>>> that
>>>>>>>>>> and
>>>>>>>>>> if
>>>>>>>>>> that is smaller than 800 MB, we should just fail.
>>>>>>>>>> 
>>>>>>>>>> It would be good to have a function that determines the xz
>>>>>>>>>> parameters and
>>>>>>>>>> we
>>>>>>>>>> use
>>>>>>>>>> that where ever we call xz. We don't want any duplicated code.
>>>>>>>>>> 
>>>>>>>>>>>>>>> On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> On Mon, 2017-11-06 at 22:09 +0100, Matthias
>>>>>>>>>>>>>>>> Fischer
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I forgot: Tuning the XZ-parameters... ;-)
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Good that you are bringing this up. So I think
>>>>>>>>>>>>>>>> what we
>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>> here is
>>>>>>>>>>>>>>>> bringing back XZ compression with -8 so that we do
>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>> waste
>>>>>>>>>>>>>>>> too
>>>>>>>>>>>>>>>> much
>>>>>>>>>>>>>>>> space
>>>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>>> extracting the image again.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Right now I'm testing how much RAM I need.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Current 'next'-build is running with:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 'lfs/cdrom':
>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>> export XZ_OPT = --threads=0 -8 --memory=500MiB
>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 'lfs/Config':
>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>> cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8"
>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I'm waiting for the results. Takes a while.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> You can just run it like this and it will tell you:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> [ms(a)hughes ~]$ xz -vv8 -T0
>>>>>>>>>>>>> xz: Filter chain: --
>>>>>>>>>>>>> lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=b
>>>>>>>>>>>>> t4,d
>>>>>>>>>>>>> ep
>>>>>>>>>>>>> th=0
>>>>>>>>>>>>> xz: Using up to 4 threads.
>>>>>>>>>>>>> xz: 2,629 MiB of memory is required. The limiter is
>>>>>>>>>>>>> disabled.
>>>>>>>>>>>>> xz: Decompression will need 33 MiB of memory.
>>>>>>>>>>>>> xz: Compressed data cannot be written to a terminal
>>>>>>>>>>>>> xz: Try `xz --help' for more information.
>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> To not run into memory limits we should detect how
>>>>>>>>>>>>>>>> much
>>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> build
>>>>>>>>>>>>>>>> host
>>>>>>>>>>>>>>>> has and set the memory limit to maybe 80% of that.
>>>>>>>>>>>>>>>> xz
>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>> automatically
>>>>>>>>>>>>>>>> scale down to a number of parallel processes that
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>> fit
>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>> memory.
>>>>>>>>>>> 
>>>>>>>>>>> This is something I couldn't get to work. First, I couldn't
>>>>>>>>>>> specify not
>>>>>>>>>>> less
>>>>>>>>>>> than '700 MiB'. If I specify '600 MiB', building stops and
>>>>>>>>>>> tells
>>>>>>>>>>> me it
>>>>>>>>>>> needs
>>>>>>>>>>> at least about 680 MiB. If I use '70%', building stops. And
>>>>>>>>>>> I
>>>>>>>>>>> never saw
>>>>>>>>>>> that
>>>>>>>>>>> 'xz' "automatically scaled down". It just stopped everytime.
>>>>>>>>>> 
>>>>>>>>>> Yes, ~700M is the minimum.
>>>>>>>>>> 
>>>>>>>>>> It will just reduce the number of threads again to make sure
>>>>>>>>>> everything
>>>>>>>>>> fits
>>>>>>>>>> into memory.
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>>>> I'm a bit puzzled about this: I tried using '
>>>>>>>>>>>>>> --memory=70%'
>>>>>>>>>>>>>> as
>>>>>>>>>>>>>> mentioned
>>>>>>>>>>>>>> in the xz man pages, but is seems to have no effect.
>>>>>>>>>>>>>> It
>>>>>>>>>>>>>> looks as
>>>>>>>>>>>>>> if
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> percent parameter isn't recognized and ignored. It
>>>>>>>>>>>>>> always
>>>>>>>>>>>>>> ends up
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>> "cannot allocate memory". Any hints on this? Do both
>>>>>>>>>>>>>> 'lfs/cdrom'
>>>>>>>>>>>>>> AND
>>>>>>>>>>>>>> 'lfs/Config' need the '--memory'-parameter?
>>>>>>>>>>> 
>>>>>>>>>>> This was the next problem: 'percent'-parameter does not work
>>>>>>>>>>> as
>>>>>>>>>>> expected.
>>>>>>>>>> 
>>>>>>>>>> No, don't use it. Just determine yourself what it is.
>>>>>>>>>> 
>>>>>>>>>> I think we could even try to set 100% of the memory, because
>>>>>>>>>> as
>>>>>>>>>> far
>>>>>>>>>> as I
>>>>>>>>>> can
>>>>>>>>>> see
>>>>>>>>>> this is worst case and xz won't use everything that we allow.
>>>>>>>>>> 
>>>>>>>>>> If that does't work, let's scale down to 90%, or 80%, etc...
>>>>>>>>>> 
>>>>>>>>>>>>> Yes, but it would be good to have one place where the
>>>>>>>>>>>>> parameters are
>>>>>>>>>>>>> being generated and then we just use them from a
>>>>>>>>>>>>> variable.
>>>>>>>>>>> 
>>>>>>>>>>> Probably beyond my capabilities for the moment. Any
>>>>>>>>>>> thoughts?
>>>>>>>>>>> 
>>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>> We will at least need about half a GB which should
>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> sensible
>>>>>>>>>>>>>>>> requirement
>>>>>>>>>>>>>>>> for a build system any ways.
>>>>>>>>>>> 
>>>>>>>>>>> I'm afraid '500 MiB' are not enough...
>>>>>>>>>> 
>>>>>>>>>> No, so we will now make 1G the minimum for a build.
>>>>>>>>>> 
>>>>>>>>>> I tried to build the OS on an ARM board with only 512MB of RAM
>>>>>>>>>> the
>>>>>>>>>> other
>>>>>>>>>> day
>>>>>>>>>> and
>>>>>>>>>> even with a lot of swap it doesn't work. So we are past this
>>>>>>>>>> any
>>>>>>>>>> way.
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>>>> Agreed. Because of this I'm testing with '500Mib'.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Will you work on this bringing it into the build
>>>>>>>>>>>>>>>> system?!
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> "I'll do my very best!" :-))
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Very well Miss Sophie...
>>>>>>>>>>>>> ...
>>>>>>>>>>> 
>>>>>>>>>>> Best™,
>>>>>>>>>>> Matthias
>>>>>>>>>> 
>>>>>>>>>> -Michael