Hi,

On 22.05.2018 20:03, Michael Tremer wrote:
> Hi,
> 
> I thought xz was now limited to lower memory.

Thats what I thought too, but it never worked as I wanted.

> Could you add -v to see what xz  has decided to do?

Adding '-v', I've got some weird results.

1. Added '-v' in 'make.sh':
 => "-T$(system_processors) -8 --memory=${XZ_MEM} -v"

Output:

***SNIP***
...
 tar -x -C /tmp -f /ipfire.tar
    rm -f /ipfire.tar
    cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB -v > /install/cdrom/distro.img && rm -rf *
    xz: Filter chain: --lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
    xz: Using up to 8 threads.
    xz: 5257 MiB of memory is required. The limit is 5450 MiB.
    xz: Decompression will need 33 MiB of memory.
    (stdin): 120.1 KiB / 512.0 KiB = 0.234
    xz: (stdin): Cannot allocate memory
    (stdin): 120.1 KiB / 512.0 KiB = 0.234
    make: *** [cdrom:66: /usr/src/log/cdrom] Error 1...
***SNAP***

Memory limit is higher than required memory but it still says
'cannot allocate memory'!?
And how to handle this (stdin)-message?

2. Changed line: XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"

to

XZ_MEM="$(( HOST_MEM * 9 / 10 ))MiB"

Output:

***SNIP***
...
 rm -f /ipfire.tar
    cd /tmp && tar cf - * | xz -T8 -8 --memory=7007MiB -v > /install/cdrom/distro.img && rm -rf *
    xz: Filter chain: --lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
    xz: Using up to 8 threads.
    xz: 5257 MiB of memory is required. The limit is 7007 MiB.
    xz: Decompression will need 33 MiB of memory.
    (stdin): 120.1 KiB / 512.0 KiB = 0.234
    xz: (stdin): Cannot allocate memory
    (stdin): 120.1 KiB / 512.0 KiB = 0.234
    make: *** [cdrom:66: /usr/src/log/cdrom] Error 1
...***SNAP***

3. Using '-T4' and XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"

All is well, log says:

***SNIP***
...
tar -x -C /tmp -f /ipfire.tar
rm -f /ipfire.tar
cd /tmp && tar cf - * | xz -T4 -8 --memory=5450MiB -v > /install/cdrom/distro.img && rm -rf *
xz: Filter chain: --lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
xz: Using up to 4 threads.
xz: 2629 MiB of memory is required. The limit is 5450 MiB.
xz: Decompression will need 33 MiB of memory.
...
***SNAP***

Some more infos (helpful?):

root(a)Devel: /home/matz/ipfire-2.x # xz --info-memory
Total amount of physical memory (RAM):  7,787 MiB (8,165,179,392 B)
Memory usage limit for compression:     Disabled
Memory usage limit for decompression:   Disabled

root(a)Devel: /home/matz/ipfire-2.x # ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 62187
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 62187
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

In the meantime I've found an interesting thread about this problem:
https://www.mail-archive.com/misc(a)openbsd.org/msg144710.html

"A closer reading of the man page reveals that memory consumption
is even higher in multi-threaded mode.
In multi-threaded mode about three times _size_ bytes will be
allocated in each thread for buffering input and output.
...
A quick check with top(1) confirms that xz in
multi-threaded mode allocates 1250 MB per thread at compression
level 9."

'-8' should be lower per thread, but too much for 8 threads.

Best,
Matthias

> -Michael
> 
> On Tue, 2018-05-22 at 18:31 +0200, Matthias Fischer wrote:
>> Hi!
>> 
>> Sorry, but last commit "make.sh: Use all processor cores for
>> compression" causes the build to break.
>> 
>> See
>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=c8453e87599fe66f4d18d901bc
>> 9dc431306afa2d
>> 
>> Build stops with:
>> 
>> ***SNIP***
>> ...
>> rm -f /tmp/ROOTFILES
>> tar -x -C /tmp -f /ipfire.tar
>> rm -f /ipfire.tar
>> cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB >
>> /install/cdrom/distro.img && rm -rf *
>> xz: (stdin): Cannot allocate memory
>> make: *** [cdrom:66: /usr/src/log/cdrom] Error 1
>> ...
>> ***SNAP***
>> 
>> On my machine, 'lsf/cdrom' doesn't like '-T8'. It needed '-T4'.
>> I experienced this already during my tests.
>> 
>> Can anyone confirm?
>> 
>> Best,
>> Matthias
>> 
>> P.S.: Hardware is i7/2600 with HT, 8 GB RAM.
>