Re: Current 'next' won't build => 'cannot allocate memory'

[Michael Tremer <michael.tremer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 5810 bytes --]

Hi,

yes that was me who broke that then.

I just pushed another commit that set this back to 0. Then XZ will automatically
run as many processes as it has memory for.

  https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=f03e254f39ddb7cacea7a39ec6b98dce77018b0d

In addition to that I had to limit memory to 2GB on 32 bit systems. They cannot
allocate more than that due to their limited address space.

  https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=a92fb74d1aeb9229044c1a9fbc54abafca50da58

Let me know if that works for you as well.

Best,
-Michael

On Tue, 2018-05-22 at 21:05 +0200, Matthias Fischer wrote:
> Hi,
> 
> On 22.05.2018 20:03, Michael Tremer wrote:
> > Hi,
> > 
> > I thought xz was now limited to lower memory.
> 
> Thats what I thought too, but it never worked as I wanted.
> 
> > Could you add -v to see what xz  has decided to do?
> 
> Adding '-v', I've got some weird results.
> 
> 1. Added '-v' in 'make.sh':
>  => "-T$(system_processors) -8 --memory=${XZ_MEM} -v"
> 
> Output:
> 
> ***SNIP***
> ...
>  tar -x -C /tmp -f /ipfire.tar
>     rm -f /ipfire.tar
>     cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB -v >
> /install/cdrom/distro.img && rm -rf *
>     xz: Filter chain: --
> lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
>     xz: Using up to 8 threads.
>     xz: 5257 MiB of memory is required. The limit is 5450 MiB.
>     xz: Decompression will need 33 MiB of memory.
>     (stdin): 120.1 KiB / 512.0 KiB = 0.234
>     xz: (stdin): Cannot allocate memory
>     (stdin): 120.1 KiB / 512.0 KiB = 0.234
>     make: *** [cdrom:66: /usr/src/log/cdrom] Error 1...
> ***SNAP***
> 
> Memory limit is higher than required memory but it still says
> 'cannot allocate memory'!?
> And how to handle this (stdin)-message?
> 
> 2. Changed line: XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
> 
> to
> 
> XZ_MEM="$(( HOST_MEM * 9 / 10 ))MiB"
> 
> Output:
> 
> ***SNIP***
> ...
>  rm -f /ipfire.tar
>     cd /tmp && tar cf - * | xz -T8 -8 --memory=7007MiB -v >
> /install/cdrom/distro.img && rm -rf *
>     xz: Filter chain: --
> lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
>     xz: Using up to 8 threads.
>     xz: 5257 MiB of memory is required. The limit is 7007 MiB.
>     xz: Decompression will need 33 MiB of memory.
>     (stdin): 120.1 KiB / 512.0 KiB = 0.234
>     xz: (stdin): Cannot allocate memory
>     (stdin): 120.1 KiB / 512.0 KiB = 0.234
>     make: *** [cdrom:66: /usr/src/log/cdrom] Error 1
> ...***SNAP***
> 
> 3. Using '-T4' and XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
> 
> All is well, log says:
> 
> ***SNIP***
> ...
> tar -x -C /tmp -f /ipfire.tar
> rm -f /ipfire.tar
> cd /tmp && tar cf - * | xz -T4 -8 --memory=5450MiB -v >
> /install/cdrom/distro.img && rm -rf *
> xz: Filter chain: --
> lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
> xz: Using up to 4 threads.
> xz: 2629 MiB of memory is required. The limit is 5450 MiB.
> xz: Decompression will need 33 MiB of memory.
> ...
> ***SNAP***
> 
> Some more infos (helpful?):
> 
> root(a)Devel: /home/matz/ipfire-2.x # xz --info-memory
> Total amount of physical memory (RAM):  7,787 MiB (8,165,179,392 B)
> Memory usage limit for compression:     Disabled
> Memory usage limit for decompression:   Disabled
> 
> root(a)Devel: /home/matz/ipfire-2.x # ulimit -a
> core file size          (blocks, -c) 0
> data seg size           (kbytes, -d) unlimited
> scheduling priority             (-e) 0
> file size               (blocks, -f) unlimited
> pending signals                 (-i) 62187
> max locked memory       (kbytes, -l) 64
> max memory size         (kbytes, -m) unlimited
> open files                      (-n) 1024
> pipe size            (512 bytes, -p) 8
> POSIX message queues     (bytes, -q) 819200
> real-time priority              (-r) 0
> stack size              (kbytes, -s) 8192
> cpu time               (seconds, -t) unlimited
> max user processes              (-u) 62187
> virtual memory          (kbytes, -v) unlimited
> file locks                      (-x) unlimited
> 
> In the meantime I've found an interesting thread about this problem:
> https://www.mail-archive.com/misc(a)openbsd.org/msg144710.html
> 
> "A closer reading of the man page reveals that memory consumption
> is even higher in multi-threaded mode.
> In multi-threaded mode about three times _size_ bytes will be
> allocated in each thread for buffering input and output.
> ...
> A quick check with top(1) confirms that xz in
> multi-threaded mode allocates 1250 MB per thread at compression
> level 9."
> 
> '-8' should be lower per thread, but too much for 8 threads.
> 
> Best,
> Matthias
> 
> > -Michael
> > 
> > On Tue, 2018-05-22 at 18:31 +0200, Matthias Fischer wrote:
> > > Hi!
> > > 
> > > Sorry, but last commit "make.sh: Use all processor cores for
> > > compression" causes the build to break.
> > > 
> > > See
> > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=c8453e87599fe66f4d18d9
> > > 01bc
> > > 9dc431306afa2d
> > > 
> > > Build stops with:
> > > 
> > > ***SNIP***
> > > ...
> > > rm -f /tmp/ROOTFILES
> > > tar -x -C /tmp -f /ipfire.tar
> > > rm -f /ipfire.tar
> > > cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB >
> > > /install/cdrom/distro.img && rm -rf *
> > > xz: (stdin): Cannot allocate memory
> > > make: *** [cdrom:66: /usr/src/log/cdrom] Error 1
> > > ...
> > > ***SNAP***
> > > 
> > > On my machine, 'lsf/cdrom' doesn't like '-T8'. It needed '-T4'.
> > > I experienced this already during my tests.
> > > 
> > > Can anyone confirm?
> > > 
> > > Best,
> > > Matthias
> > > 
> > > P.S.: Hardware is i7/2600 with HT, 8 GB RAM.
> 
>