From mboxrd@z Thu Jan 1 00:00:00 1970 From: Matthias Fischer To: development@lists.ipfire.org Subject: Re: Current 'next' won't build => 'cannot allocate memory' Date: Thu, 24 May 2018 18:54:08 +0200 Message-ID: <0766a019-41cf-52f2-ef81-ed66f5945e13@ipfire.org> In-Reply-To: <8161725de8ec481a1f6e10f8bab8bde6dbc9e71a.camel@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============6194206445765564724==" List-Id: --===============6194206445765564724== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi, On 23.05.2018 13:18, Michael Tremer wrote: > I fucked it up again. Sent another patch for builds on 32 bit > architectures. Don't worry - things happen. I just looked and waited for what would come out in the end... ;-) And during testing I learned a few things about programming, too. > https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dcommitdiff;h=3D212f17c1145= bb6978b4797eb9c805f603ecb1f29 >=20 > But if this is now running through, I suppose we are ready to call this > closed. Arne pushed 4.14.43 yesterday, while my 'Devel' was still building 'next' from 23. May (https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dcommit;h=3D6b4174133a4cad58db= 3570d1cd1a16b243d4da84) and I hadn't the time to look at the results until now. For now, its looking good. Last build went down from ~319 MB (ipfire-2.19.2gb-ext4.i586-full-core121.img.gz) to ~183 MB (ipfire-2.19.2gb-ext4.i586-full-core121.img.xz). Minus -136 MB. Not bad. One thing I noticed: building on a 32bit system led to ~553 additional lines in '_build.packages.log', telling me that "xz: Adjusted the number of threads from 8 to 3..." Doesn't hurt me yet. I just noticed. > About the planet post: Has anything changed here? Can we release this? >=20 > https://planet.ipfire.org/post/increasing-download-installation-speed-ben= efits-of-a-smaller-iso-image For me, its ok. If you need the latest sizes: I'll build the current 'next' with 4.14.43 tonight (https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dcommit;h=3D6c9651f6208db27d0d= aad7a822c9a511e509fc9c) Best, Matthias > -Michael >=20 > On Tue, 2018-05-22 at 20:57 +0100, Michael Tremer wrote: >> Hi, >>=20 >> yes that was me who broke that then. >>=20 >> I just pushed another commit that set this back to 0. Then XZ will automat= ically >> run as many processes as it has memory for. >>=20 >> https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dcommitdiff;h=3Df03e254f39= ddb7cacea7a39ec6b98dce77018b0d >>=20 >> In addition to that I had to limit memory to 2GB on 32 bit systems. They c= annot >> allocate more than that due to their limited address space. >>=20 >> https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dcommitdiff;h=3Da92fb74d1a= eb9229044c1a9fbc54abafca50da58 >>=20 >> Let me know if that works for you as well. >>=20 >> Best, >> -Michael >>=20 >> On Tue, 2018-05-22 at 21:05 +0200, Matthias Fischer wrote: >> > Hi, >> >=20 >> > On 22.05.2018 20:03, Michael Tremer wrote: >> > > Hi, >> > >=20 >> > > I thought xz was now limited to lower memory. >> >=20 >> > Thats what I thought too, but it never worked as I wanted. >> >=20 >> > > Could you add -v to see what xz has decided to do? >> >=20 >> > Adding '-v', I've got some weird results. >> >=20 >> > 1. Added '-v' in 'make.sh': >> > =3D> "-T$(system_processors) -8 --memory=3D${XZ_MEM} -v" >> >=20 >> > Output: >> >=20 >> > ***SNIP*** >> > ... >> > tar -x -C /tmp -f /ipfire.tar >> > rm -f /ipfire.tar >> > cd /tmp && tar cf - * | xz -T8 -8 --memory=3D5450MiB -v > >> > /install/cdrom/distro.img && rm -rf * >> > xz: Filter chain: -- >> > lzma2=3Ddict=3D32MiB,lc=3D3,lp=3D0,pb=3D2,mode=3Dnormal,nice=3D64,mf=3Db= t4,depth=3D0 >> > xz: Using up to 8 threads. >> > xz: 5257 MiB of memory is required. The limit is 5450 MiB. >> > xz: Decompression will need 33 MiB of memory. >> > (stdin): 120.1 KiB / 512.0 KiB =3D 0.234 >> > xz: (stdin): Cannot allocate memory >> > (stdin): 120.1 KiB / 512.0 KiB =3D 0.234 >> > make: *** [cdrom:66: /usr/src/log/cdrom] Error 1... >> > ***SNAP*** >> >=20 >> > Memory limit is higher than required memory but it still says >> > 'cannot allocate memory'!? >> > And how to handle this (stdin)-message? >> >=20 >> > 2. Changed line: XZ_MEM=3D"$(( HOST_MEM * 7 / 10 ))MiB" >> >=20 >> > to >> >=20 >> > XZ_MEM=3D"$(( HOST_MEM * 9 / 10 ))MiB" >> >=20 >> > Output: >> >=20 >> > ***SNIP*** >> > ... >> > rm -f /ipfire.tar >> > cd /tmp && tar cf - * | xz -T8 -8 --memory=3D7007MiB -v > >> > /install/cdrom/distro.img && rm -rf * >> > xz: Filter chain: -- >> > lzma2=3Ddict=3D32MiB,lc=3D3,lp=3D0,pb=3D2,mode=3Dnormal,nice=3D64,mf=3Db= t4,depth=3D0 >> > xz: Using up to 8 threads. >> > xz: 5257 MiB of memory is required. The limit is 7007 MiB. >> > xz: Decompression will need 33 MiB of memory. >> > (stdin): 120.1 KiB / 512.0 KiB =3D 0.234 >> > xz: (stdin): Cannot allocate memory >> > (stdin): 120.1 KiB / 512.0 KiB =3D 0.234 >> > make: *** [cdrom:66: /usr/src/log/cdrom] Error 1 >> > ...***SNAP*** >> >=20 >> > 3. Using '-T4' and XZ_MEM=3D"$(( HOST_MEM * 7 / 10 ))MiB" >> >=20 >> > All is well, log says: >> >=20 >> > ***SNIP*** >> > ... >> > tar -x -C /tmp -f /ipfire.tar >> > rm -f /ipfire.tar >> > cd /tmp && tar cf - * | xz -T4 -8 --memory=3D5450MiB -v > >> > /install/cdrom/distro.img && rm -rf * >> > xz: Filter chain: -- >> > lzma2=3Ddict=3D32MiB,lc=3D3,lp=3D0,pb=3D2,mode=3Dnormal,nice=3D64,mf=3Db= t4,depth=3D0 >> > xz: Using up to 4 threads. >> > xz: 2629 MiB of memory is required. The limit is 5450 MiB. >> > xz: Decompression will need 33 MiB of memory. >> > ... >> > ***SNAP*** >> >=20 >> > Some more infos (helpful?): >> >=20 >> > root(a)Devel: /home/matz/ipfire-2.x # xz --info-memory >> > Total amount of physical memory (RAM): 7,787 MiB (8,165,179,392 B) >> > Memory usage limit for compression: Disabled >> > Memory usage limit for decompression: Disabled >> >=20 >> > root(a)Devel: /home/matz/ipfire-2.x # ulimit -a >> > core file size (blocks, -c) 0 >> > data seg size (kbytes, -d) unlimited >> > scheduling priority (-e) 0 >> > file size (blocks, -f) unlimited >> > pending signals (-i) 62187 >> > max locked memory (kbytes, -l) 64 >> > max memory size (kbytes, -m) unlimited >> > open files (-n) 1024 >> > pipe size (512 bytes, -p) 8 >> > POSIX message queues (bytes, -q) 819200 >> > real-time priority (-r) 0 >> > stack size (kbytes, -s) 8192 >> > cpu time (seconds, -t) unlimited >> > max user processes (-u) 62187 >> > virtual memory (kbytes, -v) unlimited >> > file locks (-x) unlimited >> >=20 >> > In the meantime I've found an interesting thread about this problem: >> > https://www.mail-archive.com/misc(a)openbsd.org/msg144710.html >> >=20 >> > "A closer reading of the man page reveals that memory consumption >> > is even higher in multi-threaded mode. >> > In multi-threaded mode about three times _size_ bytes will be >> > allocated in each thread for buffering input and output. >> > ... >> > A quick check with top(1) confirms that xz in >> > multi-threaded mode allocates 1250 MB per thread at compression >> > level 9." >> >=20 >> > '-8' should be lower per thread, but too much for 8 threads. >> >=20 >> > Best, >> > Matthias >> >=20 >> > > -Michael >> > >=20 >> > > On Tue, 2018-05-22 at 18:31 +0200, Matthias Fischer wrote: >> > > > Hi! >> > > >=20 >> > > > Sorry, but last commit "make.sh: Use all processor cores for >> > > > compression" causes the build to break. >> > > >=20 >> > > > See >> > > > https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dcommit;h=3Dc8453e8759= 9fe66f4d18d9 >> > > > 01bc >> > > > 9dc431306afa2d >> > > >=20 >> > > > Build stops with: >> > > >=20 >> > > > ***SNIP*** >> > > > ... >> > > > rm -f /tmp/ROOTFILES >> > > > tar -x -C /tmp -f /ipfire.tar >> > > > rm -f /ipfire.tar >> > > > cd /tmp && tar cf - * | xz -T8 -8 --memory=3D5450MiB > >> > > > /install/cdrom/distro.img && rm -rf * >> > > > xz: (stdin): Cannot allocate memory >> > > > make: *** [cdrom:66: /usr/src/log/cdrom] Error 1 >> > > > ... >> > > > ***SNAP*** >> > > >=20 >> > > > On my machine, 'lsf/cdrom' doesn't like '-T8'. It needed '-T4'. >> > > > I experienced this already during my tests. >> > > >=20 >> > > > Can anyone confirm? >> > > >=20 >> > > > Best, >> > > > Matthias >> > > >=20 >> > > > P.S.: Hardware is i7/2600 with HT, 8 GB RAM. >=20 --===============6194206445765564724==--