From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: Current 'next' won't build => 'cannot allocate memory' Date: Tue, 22 May 2018 20:57:31 +0100 Message-ID: <50d8e31cbc2aa82c09545e41d031e27fa5c43dbe.camel@ipfire.org> In-Reply-To: <0651d5fd-bd22-bba4-a835-d15fba8f104d@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2177524427125054936==" List-Id: --===============2177524427125054936== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi, yes that was me who broke that then. I just pushed another commit that set this back to 0. Then XZ will automatica= lly run as many processes as it has memory for. https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dcommitdiff;h=3Df03e254f39ddb= 7cacea7a39ec6b98dce77018b0d In addition to that I had to limit memory to 2GB on 32 bit systems. They cann= ot allocate more than that due to their limited address space. https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dcommitdiff;h=3Da92fb74d1aeb9= 229044c1a9fbc54abafca50da58 Let me know if that works for you as well. Best, -Michael On Tue, 2018-05-22 at 21:05 +0200, Matthias Fischer wrote: > Hi, >=20 > On 22.05.2018 20:03, Michael Tremer wrote: > > Hi, > >=20 > > I thought xz was now limited to lower memory. >=20 > Thats what I thought too, but it never worked as I wanted. >=20 > > Could you add -v to see what xz has decided to do? >=20 > Adding '-v', I've got some weird results. >=20 > 1. Added '-v' in 'make.sh': > =3D> "-T$(system_processors) -8 --memory=3D${XZ_MEM} -v" >=20 > Output: >=20 > ***SNIP*** > ... > tar -x -C /tmp -f /ipfire.tar > rm -f /ipfire.tar > cd /tmp && tar cf - * | xz -T8 -8 --memory=3D5450MiB -v > > /install/cdrom/distro.img && rm -rf * > xz: Filter chain: -- > lzma2=3Ddict=3D32MiB,lc=3D3,lp=3D0,pb=3D2,mode=3Dnormal,nice=3D64,mf=3Dbt4,= depth=3D0 > xz: Using up to 8 threads. > xz: 5257 MiB of memory is required. The limit is 5450 MiB. > xz: Decompression will need 33 MiB of memory. > (stdin): 120.1 KiB / 512.0 KiB =3D 0.234 > xz: (stdin): Cannot allocate memory > (stdin): 120.1 KiB / 512.0 KiB =3D 0.234 > make: *** [cdrom:66: /usr/src/log/cdrom] Error 1... > ***SNAP*** >=20 > Memory limit is higher than required memory but it still says > 'cannot allocate memory'!? > And how to handle this (stdin)-message? >=20 > 2. Changed line: XZ_MEM=3D"$(( HOST_MEM * 7 / 10 ))MiB" >=20 > to >=20 > XZ_MEM=3D"$(( HOST_MEM * 9 / 10 ))MiB" >=20 > Output: >=20 > ***SNIP*** > ... > rm -f /ipfire.tar > cd /tmp && tar cf - * | xz -T8 -8 --memory=3D7007MiB -v > > /install/cdrom/distro.img && rm -rf * > xz: Filter chain: -- > lzma2=3Ddict=3D32MiB,lc=3D3,lp=3D0,pb=3D2,mode=3Dnormal,nice=3D64,mf=3Dbt4,= depth=3D0 > xz: Using up to 8 threads. > xz: 5257 MiB of memory is required. The limit is 7007 MiB. > xz: Decompression will need 33 MiB of memory. > (stdin): 120.1 KiB / 512.0 KiB =3D 0.234 > xz: (stdin): Cannot allocate memory > (stdin): 120.1 KiB / 512.0 KiB =3D 0.234 > make: *** [cdrom:66: /usr/src/log/cdrom] Error 1 > ...***SNAP*** >=20 > 3. Using '-T4' and XZ_MEM=3D"$(( HOST_MEM * 7 / 10 ))MiB" >=20 > All is well, log says: >=20 > ***SNIP*** > ... > tar -x -C /tmp -f /ipfire.tar > rm -f /ipfire.tar > cd /tmp && tar cf - * | xz -T4 -8 --memory=3D5450MiB -v > > /install/cdrom/distro.img && rm -rf * > xz: Filter chain: -- > lzma2=3Ddict=3D32MiB,lc=3D3,lp=3D0,pb=3D2,mode=3Dnormal,nice=3D64,mf=3Dbt4,= depth=3D0 > xz: Using up to 4 threads. > xz: 2629 MiB of memory is required. The limit is 5450 MiB. > xz: Decompression will need 33 MiB of memory. > ... > ***SNAP*** >=20 > Some more infos (helpful?): >=20 > root(a)Devel: /home/matz/ipfire-2.x # xz --info-memory > Total amount of physical memory (RAM): 7,787 MiB (8,165,179,392 B) > Memory usage limit for compression: Disabled > Memory usage limit for decompression: Disabled >=20 > root(a)Devel: /home/matz/ipfire-2.x # ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) unlimited > pending signals (-i) 62187 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 1024 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 62187 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited >=20 > In the meantime I've found an interesting thread about this problem: > https://www.mail-archive.com/misc(a)openbsd.org/msg144710.html >=20 > "A closer reading of the man page reveals that memory consumption > is even higher in multi-threaded mode. > In multi-threaded mode about three times _size_ bytes will be > allocated in each thread for buffering input and output. > ... > A quick check with top(1) confirms that xz in > multi-threaded mode allocates 1250 MB per thread at compression > level 9." >=20 > '-8' should be lower per thread, but too much for 8 threads. >=20 > Best, > Matthias >=20 > > -Michael > >=20 > > On Tue, 2018-05-22 at 18:31 +0200, Matthias Fischer wrote: > > > Hi! > > >=20 > > > Sorry, but last commit "make.sh: Use all processor cores for > > > compression" causes the build to break. > > >=20 > > > See > > > https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dcommit;h=3Dc8453e87599fe= 66f4d18d9 > > > 01bc > > > 9dc431306afa2d > > >=20 > > > Build stops with: > > >=20 > > > ***SNIP*** > > > ... > > > rm -f /tmp/ROOTFILES > > > tar -x -C /tmp -f /ipfire.tar > > > rm -f /ipfire.tar > > > cd /tmp && tar cf - * | xz -T8 -8 --memory=3D5450MiB > > > > /install/cdrom/distro.img && rm -rf * > > > xz: (stdin): Cannot allocate memory > > > make: *** [cdrom:66: /usr/src/log/cdrom] Error 1 > > > ... > > > ***SNAP*** > > >=20 > > > On my machine, 'lsf/cdrom' doesn't like '-T8'. It needed '-T4'. > > > I experienced this already during my tests. > > >=20 > > > Can anyone confirm? > > >=20 > > > Best, > > > Matthias > > >=20 > > > P.S.: Hardware is i7/2600 with HT, 8 GB RAM. >=20 >=20 --===============2177524427125054936==--