From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: Current 'next' won't build => 'cannot allocate memory' Date: Wed, 23 May 2018 12:18:20 +0100 Message-ID: <8161725de8ec481a1f6e10f8bab8bde6dbc9e71a.camel@ipfire.org> In-Reply-To: <50d8e31cbc2aa82c09545e41d031e27fa5c43dbe.camel@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============6153821705401143297==" List-Id: --===============6153821705401143297== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable I fucked it up again. Sent another patch for builds on 32 bit architectures. https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dcommitdiff;h=3D212f17c1145bb= 6978b4797eb9c805f603ecb1f29 But if this is now running through, I suppose we are ready to call this closed. About the planet post: Has anything changed here? Can we release this? https://planet.ipfire.org/post/increasing-download-installation-speed-benef= its-of-a-smaller-iso-image -Michael On Tue, 2018-05-22 at 20:57 +0100, Michael Tremer wrote: > Hi, >=20 > yes that was me who broke that then. >=20 > I just pushed another commit that set this back to 0. Then XZ will automati= cally > run as many processes as it has memory for. >=20 > https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dcommitdiff;h=3Df03e254f39d= db7cacea7a39ec6b98dce77018b0d >=20 > In addition to that I had to limit memory to 2GB on 32 bit systems. They ca= nnot > allocate more than that due to their limited address space. >=20 > https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dcommitdiff;h=3Da92fb74d1ae= b9229044c1a9fbc54abafca50da58 >=20 > Let me know if that works for you as well. >=20 > Best, > -Michael >=20 > On Tue, 2018-05-22 at 21:05 +0200, Matthias Fischer wrote: > > Hi, > >=20 > > On 22.05.2018 20:03, Michael Tremer wrote: > > > Hi, > > >=20 > > > I thought xz was now limited to lower memory. > >=20 > > Thats what I thought too, but it never worked as I wanted. > >=20 > > > Could you add -v to see what xz has decided to do? > >=20 > > Adding '-v', I've got some weird results. > >=20 > > 1. Added '-v' in 'make.sh': > > =3D> "-T$(system_processors) -8 --memory=3D${XZ_MEM} -v" > >=20 > > Output: > >=20 > > ***SNIP*** > > ... > > tar -x -C /tmp -f /ipfire.tar > > rm -f /ipfire.tar > > cd /tmp && tar cf - * | xz -T8 -8 --memory=3D5450MiB -v > > > /install/cdrom/distro.img && rm -rf * > > xz: Filter chain: -- > > lzma2=3Ddict=3D32MiB,lc=3D3,lp=3D0,pb=3D2,mode=3Dnormal,nice=3D64,mf=3Dbt= 4,depth=3D0 > > xz: Using up to 8 threads. > > xz: 5257 MiB of memory is required. The limit is 5450 MiB. > > xz: Decompression will need 33 MiB of memory. > > (stdin): 120.1 KiB / 512.0 KiB =3D 0.234 > > xz: (stdin): Cannot allocate memory > > (stdin): 120.1 KiB / 512.0 KiB =3D 0.234 > > make: *** [cdrom:66: /usr/src/log/cdrom] Error 1... > > ***SNAP*** > >=20 > > Memory limit is higher than required memory but it still says > > 'cannot allocate memory'!? > > And how to handle this (stdin)-message? > >=20 > > 2. Changed line: XZ_MEM=3D"$(( HOST_MEM * 7 / 10 ))MiB" > >=20 > > to > >=20 > > XZ_MEM=3D"$(( HOST_MEM * 9 / 10 ))MiB" > >=20 > > Output: > >=20 > > ***SNIP*** > > ... > > rm -f /ipfire.tar > > cd /tmp && tar cf - * | xz -T8 -8 --memory=3D7007MiB -v > > > /install/cdrom/distro.img && rm -rf * > > xz: Filter chain: -- > > lzma2=3Ddict=3D32MiB,lc=3D3,lp=3D0,pb=3D2,mode=3Dnormal,nice=3D64,mf=3Dbt= 4,depth=3D0 > > xz: Using up to 8 threads. > > xz: 5257 MiB of memory is required. The limit is 7007 MiB. > > xz: Decompression will need 33 MiB of memory. > > (stdin): 120.1 KiB / 512.0 KiB =3D 0.234 > > xz: (stdin): Cannot allocate memory > > (stdin): 120.1 KiB / 512.0 KiB =3D 0.234 > > make: *** [cdrom:66: /usr/src/log/cdrom] Error 1 > > ...***SNAP*** > >=20 > > 3. Using '-T4' and XZ_MEM=3D"$(( HOST_MEM * 7 / 10 ))MiB" > >=20 > > All is well, log says: > >=20 > > ***SNIP*** > > ... > > tar -x -C /tmp -f /ipfire.tar > > rm -f /ipfire.tar > > cd /tmp && tar cf - * | xz -T4 -8 --memory=3D5450MiB -v > > > /install/cdrom/distro.img && rm -rf * > > xz: Filter chain: -- > > lzma2=3Ddict=3D32MiB,lc=3D3,lp=3D0,pb=3D2,mode=3Dnormal,nice=3D64,mf=3Dbt= 4,depth=3D0 > > xz: Using up to 4 threads. > > xz: 2629 MiB of memory is required. The limit is 5450 MiB. > > xz: Decompression will need 33 MiB of memory. > > ... > > ***SNAP*** > >=20 > > Some more infos (helpful?): > >=20 > > root(a)Devel: /home/matz/ipfire-2.x # xz --info-memory > > Total amount of physical memory (RAM): 7,787 MiB (8,165,179,392 B) > > Memory usage limit for compression: Disabled > > Memory usage limit for decompression: Disabled > >=20 > > root(a)Devel: /home/matz/ipfire-2.x # ulimit -a > > core file size (blocks, -c) 0 > > data seg size (kbytes, -d) unlimited > > scheduling priority (-e) 0 > > file size (blocks, -f) unlimited > > pending signals (-i) 62187 > > max locked memory (kbytes, -l) 64 > > max memory size (kbytes, -m) unlimited > > open files (-n) 1024 > > pipe size (512 bytes, -p) 8 > > POSIX message queues (bytes, -q) 819200 > > real-time priority (-r) 0 > > stack size (kbytes, -s) 8192 > > cpu time (seconds, -t) unlimited > > max user processes (-u) 62187 > > virtual memory (kbytes, -v) unlimited > > file locks (-x) unlimited > >=20 > > In the meantime I've found an interesting thread about this problem: > > https://www.mail-archive.com/misc(a)openbsd.org/msg144710.html > >=20 > > "A closer reading of the man page reveals that memory consumption > > is even higher in multi-threaded mode. > > In multi-threaded mode about three times _size_ bytes will be > > allocated in each thread for buffering input and output. > > ... > > A quick check with top(1) confirms that xz in > > multi-threaded mode allocates 1250 MB per thread at compression > > level 9." > >=20 > > '-8' should be lower per thread, but too much for 8 threads. > >=20 > > Best, > > Matthias > >=20 > > > -Michael > > >=20 > > > On Tue, 2018-05-22 at 18:31 +0200, Matthias Fischer wrote: > > > > Hi! > > > >=20 > > > > Sorry, but last commit "make.sh: Use all processor cores for > > > > compression" causes the build to break. > > > >=20 > > > > See > > > > https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dcommit;h=3Dc8453e87599= fe66f4d18d9 > > > > 01bc > > > > 9dc431306afa2d > > > >=20 > > > > Build stops with: > > > >=20 > > > > ***SNIP*** > > > > ... > > > > rm -f /tmp/ROOTFILES > > > > tar -x -C /tmp -f /ipfire.tar > > > > rm -f /ipfire.tar > > > > cd /tmp && tar cf - * | xz -T8 -8 --memory=3D5450MiB > > > > > /install/cdrom/distro.img && rm -rf * > > > > xz: (stdin): Cannot allocate memory > > > > make: *** [cdrom:66: /usr/src/log/cdrom] Error 1 > > > > ... > > > > ***SNAP*** > > > >=20 > > > > On my machine, 'lsf/cdrom' doesn't like '-T8'. It needed '-T4'. > > > > I experienced this already during my tests. > > > >=20 > > > > Can anyone confirm? > > > >=20 > > > > Best, > > > > Matthias > > > >=20 > > > > P.S.: Hardware is i7/2600 with HT, 8 GB RAM. --===============6153821705401143297==--