From mboxrd@z Thu Jan  1 00:00:00 1970
From: Matthias Fischer <matthias.fischer@ipfire.org>
To: development@lists.ipfire.org
Subject: Re: Current 'next' won't build => 'cannot allocate memory'
Date: Tue, 22 May 2018 21:05:10 +0200
Message-ID: <0651d5fd-bd22-bba4-a835-d15fba8f104d@ipfire.org>
In-Reply-To: <06a7aa500aa06e9a3dbb692a1759fc76c7e650f9.camel@ipfire.org>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============5744022485081628223=="
List-Id: <development.lists.ipfire.org>

--===============5744022485081628223==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

Hi,

On 22.05.2018 20:03, Michael Tremer wrote:
> Hi,
>=20
> I thought xz was now limited to lower memory.

Thats what I thought too, but it never worked as I wanted.

> Could you add -v to see what xz  has decided to do?

Adding '-v', I've got some weird results.

1. Added '-v' in 'make.sh':
 =3D> "-T$(system_processors) -8 --memory=3D${XZ_MEM} -v"

Output:

***SNIP***
...
 tar -x -C /tmp -f /ipfire.tar
    rm -f /ipfire.tar
    cd /tmp && tar cf - * | xz -T8 -8 --memory=3D5450MiB -v > /install/cdrom/=
distro.img && rm -rf *
    xz: Filter chain: --lzma2=3Ddict=3D32MiB,lc=3D3,lp=3D0,pb=3D2,mode=3Dnorm=
al,nice=3D64,mf=3Dbt4,depth=3D0
    xz: Using up to 8 threads.
    xz: 5257 MiB of memory is required. The limit is 5450 MiB.
    xz: Decompression will need 33 MiB of memory.
    (stdin): 120.1 KiB / 512.0 KiB =3D 0.234
    xz: (stdin): Cannot allocate memory
    (stdin): 120.1 KiB / 512.0 KiB =3D 0.234
    make: *** [cdrom:66: /usr/src/log/cdrom] Error 1...
***SNAP***

Memory limit is higher than required memory but it still says
'cannot allocate memory'!?
And how to handle this (stdin)-message?

2. Changed line: XZ_MEM=3D"$(( HOST_MEM * 7 / 10 ))MiB"

to

XZ_MEM=3D"$(( HOST_MEM * 9 / 10 ))MiB"

Output:

***SNIP***
...
 rm -f /ipfire.tar
    cd /tmp && tar cf - * | xz -T8 -8 --memory=3D7007MiB -v > /install/cdrom/=
distro.img && rm -rf *
    xz: Filter chain: --lzma2=3Ddict=3D32MiB,lc=3D3,lp=3D0,pb=3D2,mode=3Dnorm=
al,nice=3D64,mf=3Dbt4,depth=3D0
    xz: Using up to 8 threads.
    xz: 5257 MiB of memory is required. The limit is 7007 MiB.
    xz: Decompression will need 33 MiB of memory.
    (stdin): 120.1 KiB / 512.0 KiB =3D 0.234
    xz: (stdin): Cannot allocate memory
    (stdin): 120.1 KiB / 512.0 KiB =3D 0.234
    make: *** [cdrom:66: /usr/src/log/cdrom] Error 1
...***SNAP***

3. Using '-T4' and XZ_MEM=3D"$(( HOST_MEM * 7 / 10 ))MiB"

All is well, log says:

***SNIP***
...
tar -x -C /tmp -f /ipfire.tar
rm -f /ipfire.tar
cd /tmp && tar cf - * | xz -T4 -8 --memory=3D5450MiB -v > /install/cdrom/dist=
ro.img && rm -rf *
xz: Filter chain: --lzma2=3Ddict=3D32MiB,lc=3D3,lp=3D0,pb=3D2,mode=3Dnormal,n=
ice=3D64,mf=3Dbt4,depth=3D0
xz: Using up to 4 threads.
xz: 2629 MiB of memory is required. The limit is 5450 MiB.
xz: Decompression will need 33 MiB of memory.
...
***SNAP***

Some more infos (helpful?):

root(a)Devel: /home/matz/ipfire-2.x # xz --info-memory
Total amount of physical memory (RAM):  7,787 MiB (8,165,179,392 B)
Memory usage limit for compression:     Disabled
Memory usage limit for decompression:   Disabled

root(a)Devel: /home/matz/ipfire-2.x # ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 62187
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 62187
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

In the meantime I've found an interesting thread about this problem:
https://www.mail-archive.com/misc(a)openbsd.org/msg144710.html

"A closer reading of the man page reveals that memory consumption
is even higher in multi-threaded mode.
In multi-threaded mode about three times _size_ bytes will be
allocated in each thread for buffering input and output.
...
A quick check with top(1) confirms that xz in
multi-threaded mode allocates 1250 MB per thread at compression
level 9."

'-8' should be lower per thread, but too much for 8 threads.

Best,
Matthias

> -Michael
>=20
> On Tue, 2018-05-22 at 18:31 +0200, Matthias Fischer wrote:
>> Hi!
>>=20
>> Sorry, but last commit "make.sh: Use all processor cores for
>> compression" causes the build to break.
>>=20
>> See
>> https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dcommit;h=3Dc8453e87599fe66f=
4d18d901bc
>> 9dc431306afa2d
>>=20
>> Build stops with:
>>=20
>> ***SNIP***
>> ...
>> rm -f /tmp/ROOTFILES
>> tar -x -C /tmp -f /ipfire.tar
>> rm -f /ipfire.tar
>> cd /tmp && tar cf - * | xz -T8 -8 --memory=3D5450MiB >
>> /install/cdrom/distro.img && rm -rf *
>> xz: (stdin): Cannot allocate memory
>> make: *** [cdrom:66: /usr/src/log/cdrom] Error 1
>> ...
>> ***SNAP***
>>=20
>> On my machine, 'lsf/cdrom' doesn't like '-T8'. It needed '-T4'.
>> I experienced this already during my tests.
>>=20
>> Can anyone confirm?
>>=20
>> Best,
>> Matthias
>>=20
>> P.S.: Hardware is i7/2600 with HT, 8 GB RAM.
>=20


--===============5744022485081628223==--