Re: Current 'next' won't build => 'cannot allocate memory'

[Matthias Fischer <matthias.fischer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 7986 bytes --]

Hi,

On 23.05.2018 13:18, Michael Tremer wrote:
> I fucked it up again. Sent another patch for builds on 32 bit
> architectures.

Don't worry - things happen.

I just looked and waited for what would come out in the end... ;-)

And during testing I learned a few things about programming, too.

>   https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=212f17c1145bb6978b4797eb9c805f603ecb1f29
> 
> But if this is now running through, I suppose we are ready to call this
> closed.

Arne pushed 4.14.43 yesterday, while my 'Devel' was still building
'next' from 23. May
(https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=6b4174133a4cad58db3570d1cd1a16b243d4da84)
and I hadn't the time to look at the results until now.

For now, its looking good.

Last build went down from ~319 MB
(ipfire-2.19.2gb-ext4.i586-full-core121.img.gz) to ~183 MB
(ipfire-2.19.2gb-ext4.i586-full-core121.img.xz). Minus -136 MB. Not bad.

One thing I noticed: building on a 32bit system led to ~553 additional
lines in '_build.packages.log', telling me that "xz: Adjusted the number
of threads from 8 to 3..." Doesn't hurt me yet. I just noticed.

> About the planet post: Has anything changed here? Can we release this?
> 
>   https://planet.ipfire.org/post/increasing-download-installation-speed-benefits-of-a-smaller-iso-image

For me, its ok.

If you need the latest sizes: I'll build the current 'next' with 4.14.43
tonight
(https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=6c9651f6208db27d0daad7a822c9a511e509fc9c)

Best,
Matthias

> -Michael
> 
> On Tue, 2018-05-22 at 20:57 +0100, Michael Tremer wrote:
>> Hi,
>> 
>> yes that was me who broke that then.
>> 
>> I just pushed another commit that set this back to 0. Then XZ will automatically
>> run as many processes as it has memory for.
>> 
>>   https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=f03e254f39ddb7cacea7a39ec6b98dce77018b0d
>> 
>> In addition to that I had to limit memory to 2GB on 32 bit systems. They cannot
>> allocate more than that due to their limited address space.
>> 
>>   https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=a92fb74d1aeb9229044c1a9fbc54abafca50da58
>> 
>> Let me know if that works for you as well.
>> 
>> Best,
>> -Michael
>> 
>> On Tue, 2018-05-22 at 21:05 +0200, Matthias Fischer wrote:
>> > Hi,
>> > 
>> > On 22.05.2018 20:03, Michael Tremer wrote:
>> > > Hi,
>> > > 
>> > > I thought xz was now limited to lower memory.
>> > 
>> > Thats what I thought too, but it never worked as I wanted.
>> > 
>> > > Could you add -v to see what xz  has decided to do?
>> > 
>> > Adding '-v', I've got some weird results.
>> > 
>> > 1. Added '-v' in 'make.sh':
>> >  => "-T$(system_processors) -8 --memory=${XZ_MEM} -v"
>> > 
>> > Output:
>> > 
>> > ***SNIP***
>> > ...
>> >  tar -x -C /tmp -f /ipfire.tar
>> >     rm -f /ipfire.tar
>> >     cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB -v >
>> > /install/cdrom/distro.img && rm -rf *
>> >     xz: Filter chain: --
>> > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
>> >     xz: Using up to 8 threads.
>> >     xz: 5257 MiB of memory is required. The limit is 5450 MiB.
>> >     xz: Decompression will need 33 MiB of memory.
>> >     (stdin): 120.1 KiB / 512.0 KiB = 0.234
>> >     xz: (stdin): Cannot allocate memory
>> >     (stdin): 120.1 KiB / 512.0 KiB = 0.234
>> >     make: *** [cdrom:66: /usr/src/log/cdrom] Error 1...
>> > ***SNAP***
>> > 
>> > Memory limit is higher than required memory but it still says
>> > 'cannot allocate memory'!?
>> > And how to handle this (stdin)-message?
>> > 
>> > 2. Changed line: XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
>> > 
>> > to
>> > 
>> > XZ_MEM="$(( HOST_MEM * 9 / 10 ))MiB"
>> > 
>> > Output:
>> > 
>> > ***SNIP***
>> > ...
>> >  rm -f /ipfire.tar
>> >     cd /tmp && tar cf - * | xz -T8 -8 --memory=7007MiB -v >
>> > /install/cdrom/distro.img && rm -rf *
>> >     xz: Filter chain: --
>> > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
>> >     xz: Using up to 8 threads.
>> >     xz: 5257 MiB of memory is required. The limit is 7007 MiB.
>> >     xz: Decompression will need 33 MiB of memory.
>> >     (stdin): 120.1 KiB / 512.0 KiB = 0.234
>> >     xz: (stdin): Cannot allocate memory
>> >     (stdin): 120.1 KiB / 512.0 KiB = 0.234
>> >     make: *** [cdrom:66: /usr/src/log/cdrom] Error 1
>> > ...***SNAP***
>> > 
>> > 3. Using '-T4' and XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
>> > 
>> > All is well, log says:
>> > 
>> > ***SNIP***
>> > ...
>> > tar -x -C /tmp -f /ipfire.tar
>> > rm -f /ipfire.tar
>> > cd /tmp && tar cf - * | xz -T4 -8 --memory=5450MiB -v >
>> > /install/cdrom/distro.img && rm -rf *
>> > xz: Filter chain: --
>> > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
>> > xz: Using up to 4 threads.
>> > xz: 2629 MiB of memory is required. The limit is 5450 MiB.
>> > xz: Decompression will need 33 MiB of memory.
>> > ...
>> > ***SNAP***
>> > 
>> > Some more infos (helpful?):
>> > 
>> > root(a)Devel: /home/matz/ipfire-2.x # xz --info-memory
>> > Total amount of physical memory (RAM):  7,787 MiB (8,165,179,392 B)
>> > Memory usage limit for compression:     Disabled
>> > Memory usage limit for decompression:   Disabled
>> > 
>> > root(a)Devel: /home/matz/ipfire-2.x # ulimit -a
>> > core file size          (blocks, -c) 0
>> > data seg size           (kbytes, -d) unlimited
>> > scheduling priority             (-e) 0
>> > file size               (blocks, -f) unlimited
>> > pending signals                 (-i) 62187
>> > max locked memory       (kbytes, -l) 64
>> > max memory size         (kbytes, -m) unlimited
>> > open files                      (-n) 1024
>> > pipe size            (512 bytes, -p) 8
>> > POSIX message queues     (bytes, -q) 819200
>> > real-time priority              (-r) 0
>> > stack size              (kbytes, -s) 8192
>> > cpu time               (seconds, -t) unlimited
>> > max user processes              (-u) 62187
>> > virtual memory          (kbytes, -v) unlimited
>> > file locks                      (-x) unlimited
>> > 
>> > In the meantime I've found an interesting thread about this problem:
>> > https://www.mail-archive.com/misc(a)openbsd.org/msg144710.html
>> > 
>> > "A closer reading of the man page reveals that memory consumption
>> > is even higher in multi-threaded mode.
>> > In multi-threaded mode about three times _size_ bytes will be
>> > allocated in each thread for buffering input and output.
>> > ...
>> > A quick check with top(1) confirms that xz in
>> > multi-threaded mode allocates 1250 MB per thread at compression
>> > level 9."
>> > 
>> > '-8' should be lower per thread, but too much for 8 threads.
>> > 
>> > Best,
>> > Matthias
>> > 
>> > > -Michael
>> > > 
>> > > On Tue, 2018-05-22 at 18:31 +0200, Matthias Fischer wrote:
>> > > > Hi!
>> > > > 
>> > > > Sorry, but last commit "make.sh: Use all processor cores for
>> > > > compression" causes the build to break.
>> > > > 
>> > > > See
>> > > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=c8453e87599fe66f4d18d9
>> > > > 01bc
>> > > > 9dc431306afa2d
>> > > > 
>> > > > Build stops with:
>> > > > 
>> > > > ***SNIP***
>> > > > ...
>> > > > rm -f /tmp/ROOTFILES
>> > > > tar -x -C /tmp -f /ipfire.tar
>> > > > rm -f /ipfire.tar
>> > > > cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB >
>> > > > /install/cdrom/distro.img && rm -rf *
>> > > > xz: (stdin): Cannot allocate memory
>> > > > make: *** [cdrom:66: /usr/src/log/cdrom] Error 1
>> > > > ...
>> > > > ***SNAP***
>> > > > 
>> > > > On my machine, 'lsf/cdrom' doesn't like '-T8'. It needed '-T4'.
>> > > > I experienced this already during my tests.
>> > > > 
>> > > > Can anyone confirm?
>> > > > 
>> > > > Best,
>> > > > Matthias
>> > > > 
>> > > > P.S.: Hardware is i7/2600 with HT, 8 GB RAM.
>