Current 'next' won't build => 'cannot allocate memory'

Current 'next' won't build => 'cannot allocate memory'

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 722 bytes --]

Hi!

Sorry, but last commit "make.sh: Use all processor cores for
compression" causes the build to break.

See
https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=c8453e87599fe66f4d18d901bc9dc431306afa2d

Build stops with:

***SNIP***
...
rm -f /tmp/ROOTFILES
tar -x -C /tmp -f /ipfire.tar
rm -f /ipfire.tar
cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB >
/install/cdrom/distro.img && rm -rf *
xz: (stdin): Cannot allocate memory
make: *** [cdrom:66: /usr/src/log/cdrom] Error 1
...
***SNAP***

On my machine, 'lsf/cdrom' doesn't like '-T8'. It needed '-T4'.
I experienced this already during my tests.

Can anyone confirm?

Best,
Matthias

P.S.: Hardware is i7/2600 with HT, 8 GB RAM.

Re: Current 'next' won't build => 'cannot allocate memory'

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 968 bytes --]

Hi,

I thought xz was now limited to lower memory. Could you add -v to see what xz
has decided to do?

-Michael

On Tue, 2018-05-22 at 18:31 +0200, Matthias Fischer wrote:
> Hi!
> 
> Sorry, but last commit "make.sh: Use all processor cores for
> compression" causes the build to break.
> 
> See
> https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=c8453e87599fe66f4d18d901bc
> 9dc431306afa2d
> 
> Build stops with:
> 
> ***SNIP***
> ...
> rm -f /tmp/ROOTFILES
> tar -x -C /tmp -f /ipfire.tar
> rm -f /ipfire.tar
> cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB >
> /install/cdrom/distro.img && rm -rf *
> xz: (stdin): Cannot allocate memory
> make: *** [cdrom:66: /usr/src/log/cdrom] Error 1
> ...
> ***SNAP***
> 
> On my machine, 'lsf/cdrom' doesn't like '-T8'. It needed '-T4'.
> I experienced this already during my tests.
> 
> Can anyone confirm?
> 
> Best,
> Matthias
> 
> P.S.: Hardware is i7/2600 with HT, 8 GB RAM.

Re: Current 'next' won't build => 'cannot allocate memory'

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 4794 bytes --]

Hi,

On 22.05.2018 20:03, Michael Tremer wrote:
> Hi,
> 
> I thought xz was now limited to lower memory.

Thats what I thought too, but it never worked as I wanted.

> Could you add -v to see what xz  has decided to do?

Adding '-v', I've got some weird results.

1. Added '-v' in 'make.sh':
 => "-T$(system_processors) -8 --memory=${XZ_MEM} -v"

Output:

***SNIP***
...
 tar -x -C /tmp -f /ipfire.tar
    rm -f /ipfire.tar
    cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB -v > /install/cdrom/distro.img && rm -rf *
    xz: Filter chain: --lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
    xz: Using up to 8 threads.
    xz: 5257 MiB of memory is required. The limit is 5450 MiB.
    xz: Decompression will need 33 MiB of memory.
    (stdin): 120.1 KiB / 512.0 KiB = 0.234
    xz: (stdin): Cannot allocate memory
    (stdin): 120.1 KiB / 512.0 KiB = 0.234
    make: *** [cdrom:66: /usr/src/log/cdrom] Error 1...
***SNAP***

Memory limit is higher than required memory but it still says
'cannot allocate memory'!?
And how to handle this (stdin)-message?

2. Changed line: XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"

to

XZ_MEM="$(( HOST_MEM * 9 / 10 ))MiB"

Output:

***SNIP***
...
 rm -f /ipfire.tar
    cd /tmp && tar cf - * | xz -T8 -8 --memory=7007MiB -v > /install/cdrom/distro.img && rm -rf *
    xz: Filter chain: --lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
    xz: Using up to 8 threads.
    xz: 5257 MiB of memory is required. The limit is 7007 MiB.
    xz: Decompression will need 33 MiB of memory.
    (stdin): 120.1 KiB / 512.0 KiB = 0.234
    xz: (stdin): Cannot allocate memory
    (stdin): 120.1 KiB / 512.0 KiB = 0.234
    make: *** [cdrom:66: /usr/src/log/cdrom] Error 1
...***SNAP***

3. Using '-T4' and XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"

All is well, log says:

***SNIP***
...
tar -x -C /tmp -f /ipfire.tar
rm -f /ipfire.tar
cd /tmp && tar cf - * | xz -T4 -8 --memory=5450MiB -v > /install/cdrom/distro.img && rm -rf *
xz: Filter chain: --lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
xz: Using up to 4 threads.
xz: 2629 MiB of memory is required. The limit is 5450 MiB.
xz: Decompression will need 33 MiB of memory.
...
***SNAP***

Some more infos (helpful?):

root(a)Devel: /home/matz/ipfire-2.x # xz --info-memory
Total amount of physical memory (RAM):  7,787 MiB (8,165,179,392 B)
Memory usage limit for compression:     Disabled
Memory usage limit for decompression:   Disabled

root(a)Devel: /home/matz/ipfire-2.x # ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 62187
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 62187
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

In the meantime I've found an interesting thread about this problem:
https://www.mail-archive.com/misc(a)openbsd.org/msg144710.html

"A closer reading of the man page reveals that memory consumption
is even higher in multi-threaded mode.
In multi-threaded mode about three times _size_ bytes will be
allocated in each thread for buffering input and output.
...
A quick check with top(1) confirms that xz in
multi-threaded mode allocates 1250 MB per thread at compression
level 9."

'-8' should be lower per thread, but too much for 8 threads.

Best,
Matthias

> -Michael
> 
> On Tue, 2018-05-22 at 18:31 +0200, Matthias Fischer wrote:
>> Hi!
>> 
>> Sorry, but last commit "make.sh: Use all processor cores for
>> compression" causes the build to break.
>> 
>> See
>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=c8453e87599fe66f4d18d901bc
>> 9dc431306afa2d
>> 
>> Build stops with:
>> 
>> ***SNIP***
>> ...
>> rm -f /tmp/ROOTFILES
>> tar -x -C /tmp -f /ipfire.tar
>> rm -f /ipfire.tar
>> cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB >
>> /install/cdrom/distro.img && rm -rf *
>> xz: (stdin): Cannot allocate memory
>> make: *** [cdrom:66: /usr/src/log/cdrom] Error 1
>> ...
>> ***SNAP***
>> 
>> On my machine, 'lsf/cdrom' doesn't like '-T8'. It needed '-T4'.
>> I experienced this already during my tests.
>> 
>> Can anyone confirm?
>> 
>> Best,
>> Matthias
>> 
>> P.S.: Hardware is i7/2600 with HT, 8 GB RAM.
> 

Re: Current 'next' won't build => 'cannot allocate memory'

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 5810 bytes --]

Hi,

yes that was me who broke that then.

I just pushed another commit that set this back to 0. Then XZ will automatically
run as many processes as it has memory for.

  https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=f03e254f39ddb7cacea7a39ec6b98dce77018b0d

In addition to that I had to limit memory to 2GB on 32 bit systems. They cannot
allocate more than that due to their limited address space.

  https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=a92fb74d1aeb9229044c1a9fbc54abafca50da58

Let me know if that works for you as well.

Best,
-Michael

On Tue, 2018-05-22 at 21:05 +0200, Matthias Fischer wrote:
> Hi,
> 
> On 22.05.2018 20:03, Michael Tremer wrote:
> > Hi,
> > 
> > I thought xz was now limited to lower memory.
> 
> Thats what I thought too, but it never worked as I wanted.
> 
> > Could you add -v to see what xz  has decided to do?
> 
> Adding '-v', I've got some weird results.
> 
> 1. Added '-v' in 'make.sh':
>  => "-T$(system_processors) -8 --memory=${XZ_MEM} -v"
> 
> Output:
> 
> ***SNIP***
> ...
>  tar -x -C /tmp -f /ipfire.tar
>     rm -f /ipfire.tar
>     cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB -v >
> /install/cdrom/distro.img && rm -rf *
>     xz: Filter chain: --
> lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
>     xz: Using up to 8 threads.
>     xz: 5257 MiB of memory is required. The limit is 5450 MiB.
>     xz: Decompression will need 33 MiB of memory.
>     (stdin): 120.1 KiB / 512.0 KiB = 0.234
>     xz: (stdin): Cannot allocate memory
>     (stdin): 120.1 KiB / 512.0 KiB = 0.234
>     make: *** [cdrom:66: /usr/src/log/cdrom] Error 1...
> ***SNAP***
> 
> Memory limit is higher than required memory but it still says
> 'cannot allocate memory'!?
> And how to handle this (stdin)-message?
> 
> 2. Changed line: XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
> 
> to
> 
> XZ_MEM="$(( HOST_MEM * 9 / 10 ))MiB"
> 
> Output:
> 
> ***SNIP***
> ...
>  rm -f /ipfire.tar
>     cd /tmp && tar cf - * | xz -T8 -8 --memory=7007MiB -v >
> /install/cdrom/distro.img && rm -rf *
>     xz: Filter chain: --
> lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
>     xz: Using up to 8 threads.
>     xz: 5257 MiB of memory is required. The limit is 7007 MiB.
>     xz: Decompression will need 33 MiB of memory.
>     (stdin): 120.1 KiB / 512.0 KiB = 0.234
>     xz: (stdin): Cannot allocate memory
>     (stdin): 120.1 KiB / 512.0 KiB = 0.234
>     make: *** [cdrom:66: /usr/src/log/cdrom] Error 1
> ...***SNAP***
> 
> 3. Using '-T4' and XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
> 
> All is well, log says:
> 
> ***SNIP***
> ...
> tar -x -C /tmp -f /ipfire.tar
> rm -f /ipfire.tar
> cd /tmp && tar cf - * | xz -T4 -8 --memory=5450MiB -v >
> /install/cdrom/distro.img && rm -rf *
> xz: Filter chain: --
> lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
> xz: Using up to 4 threads.
> xz: 2629 MiB of memory is required. The limit is 5450 MiB.
> xz: Decompression will need 33 MiB of memory.
> ...
> ***SNAP***
> 
> Some more infos (helpful?):
> 
> root(a)Devel: /home/matz/ipfire-2.x # xz --info-memory
> Total amount of physical memory (RAM):  7,787 MiB (8,165,179,392 B)
> Memory usage limit for compression:     Disabled
> Memory usage limit for decompression:   Disabled
> 
> root(a)Devel: /home/matz/ipfire-2.x # ulimit -a
> core file size          (blocks, -c) 0
> data seg size           (kbytes, -d) unlimited
> scheduling priority             (-e) 0
> file size               (blocks, -f) unlimited
> pending signals                 (-i) 62187
> max locked memory       (kbytes, -l) 64
> max memory size         (kbytes, -m) unlimited
> open files                      (-n) 1024
> pipe size            (512 bytes, -p) 8
> POSIX message queues     (bytes, -q) 819200
> real-time priority              (-r) 0
> stack size              (kbytes, -s) 8192
> cpu time               (seconds, -t) unlimited
> max user processes              (-u) 62187
> virtual memory          (kbytes, -v) unlimited
> file locks                      (-x) unlimited
> 
> In the meantime I've found an interesting thread about this problem:
> https://www.mail-archive.com/misc(a)openbsd.org/msg144710.html
> 
> "A closer reading of the man page reveals that memory consumption
> is even higher in multi-threaded mode.
> In multi-threaded mode about three times _size_ bytes will be
> allocated in each thread for buffering input and output.
> ...
> A quick check with top(1) confirms that xz in
> multi-threaded mode allocates 1250 MB per thread at compression
> level 9."
> 
> '-8' should be lower per thread, but too much for 8 threads.
> 
> Best,
> Matthias
> 
> > -Michael
> > 
> > On Tue, 2018-05-22 at 18:31 +0200, Matthias Fischer wrote:
> > > Hi!
> > > 
> > > Sorry, but last commit "make.sh: Use all processor cores for
> > > compression" causes the build to break.
> > > 
> > > See
> > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=c8453e87599fe66f4d18d9
> > > 01bc
> > > 9dc431306afa2d
> > > 
> > > Build stops with:
> > > 
> > > ***SNIP***
> > > ...
> > > rm -f /tmp/ROOTFILES
> > > tar -x -C /tmp -f /ipfire.tar
> > > rm -f /ipfire.tar
> > > cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB >
> > > /install/cdrom/distro.img && rm -rf *
> > > xz: (stdin): Cannot allocate memory
> > > make: *** [cdrom:66: /usr/src/log/cdrom] Error 1
> > > ...
> > > ***SNAP***
> > > 
> > > On my machine, 'lsf/cdrom' doesn't like '-T8'. It needed '-T4'.
> > > I experienced this already during my tests.
> > > 
> > > Can anyone confirm?
> > > 
> > > Best,
> > > Matthias
> > > 
> > > P.S.: Hardware is i7/2600 with HT, 8 GB RAM.
> 
> 

Re: Current 'next' won't build => 'cannot allocate memory'

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 6678 bytes --]

I fucked it up again. Sent another patch for builds on 32 bit
architectures.

  https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=212f17c1145bb6978b4797eb9c805f603ecb1f29

But if this is now running through, I suppose we are ready to call this
closed.

About the planet post: Has anything changed here? Can we release this?

  https://planet.ipfire.org/post/increasing-download-installation-speed-benefits-of-a-smaller-iso-image

-Michael

On Tue, 2018-05-22 at 20:57 +0100, Michael Tremer wrote:
> Hi,
> 
> yes that was me who broke that then.
> 
> I just pushed another commit that set this back to 0. Then XZ will automatically
> run as many processes as it has memory for.
> 
>   https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=f03e254f39ddb7cacea7a39ec6b98dce77018b0d
> 
> In addition to that I had to limit memory to 2GB on 32 bit systems. They cannot
> allocate more than that due to their limited address space.
> 
>   https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=a92fb74d1aeb9229044c1a9fbc54abafca50da58
> 
> Let me know if that works for you as well.
> 
> Best,
> -Michael
> 
> On Tue, 2018-05-22 at 21:05 +0200, Matthias Fischer wrote:
> > Hi,
> > 
> > On 22.05.2018 20:03, Michael Tremer wrote:
> > > Hi,
> > > 
> > > I thought xz was now limited to lower memory.
> > 
> > Thats what I thought too, but it never worked as I wanted.
> > 
> > > Could you add -v to see what xz  has decided to do?
> > 
> > Adding '-v', I've got some weird results.
> > 
> > 1. Added '-v' in 'make.sh':
> >  => "-T$(system_processors) -8 --memory=${XZ_MEM} -v"
> > 
> > Output:
> > 
> > ***SNIP***
> > ...
> >  tar -x -C /tmp -f /ipfire.tar
> >     rm -f /ipfire.tar
> >     cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB -v >
> > /install/cdrom/distro.img && rm -rf *
> >     xz: Filter chain: --
> > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
> >     xz: Using up to 8 threads.
> >     xz: 5257 MiB of memory is required. The limit is 5450 MiB.
> >     xz: Decompression will need 33 MiB of memory.
> >     (stdin): 120.1 KiB / 512.0 KiB = 0.234
> >     xz: (stdin): Cannot allocate memory
> >     (stdin): 120.1 KiB / 512.0 KiB = 0.234
> >     make: *** [cdrom:66: /usr/src/log/cdrom] Error 1...
> > ***SNAP***
> > 
> > Memory limit is higher than required memory but it still says
> > 'cannot allocate memory'!?
> > And how to handle this (stdin)-message?
> > 
> > 2. Changed line: XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
> > 
> > to
> > 
> > XZ_MEM="$(( HOST_MEM * 9 / 10 ))MiB"
> > 
> > Output:
> > 
> > ***SNIP***
> > ...
> >  rm -f /ipfire.tar
> >     cd /tmp && tar cf - * | xz -T8 -8 --memory=7007MiB -v >
> > /install/cdrom/distro.img && rm -rf *
> >     xz: Filter chain: --
> > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
> >     xz: Using up to 8 threads.
> >     xz: 5257 MiB of memory is required. The limit is 7007 MiB.
> >     xz: Decompression will need 33 MiB of memory.
> >     (stdin): 120.1 KiB / 512.0 KiB = 0.234
> >     xz: (stdin): Cannot allocate memory
> >     (stdin): 120.1 KiB / 512.0 KiB = 0.234
> >     make: *** [cdrom:66: /usr/src/log/cdrom] Error 1
> > ...***SNAP***
> > 
> > 3. Using '-T4' and XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
> > 
> > All is well, log says:
> > 
> > ***SNIP***
> > ...
> > tar -x -C /tmp -f /ipfire.tar
> > rm -f /ipfire.tar
> > cd /tmp && tar cf - * | xz -T4 -8 --memory=5450MiB -v >
> > /install/cdrom/distro.img && rm -rf *
> > xz: Filter chain: --
> > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
> > xz: Using up to 4 threads.
> > xz: 2629 MiB of memory is required. The limit is 5450 MiB.
> > xz: Decompression will need 33 MiB of memory.
> > ...
> > ***SNAP***
> > 
> > Some more infos (helpful?):
> > 
> > root(a)Devel: /home/matz/ipfire-2.x # xz --info-memory
> > Total amount of physical memory (RAM):  7,787 MiB (8,165,179,392 B)
> > Memory usage limit for compression:     Disabled
> > Memory usage limit for decompression:   Disabled
> > 
> > root(a)Devel: /home/matz/ipfire-2.x # ulimit -a
> > core file size          (blocks, -c) 0
> > data seg size           (kbytes, -d) unlimited
> > scheduling priority             (-e) 0
> > file size               (blocks, -f) unlimited
> > pending signals                 (-i) 62187
> > max locked memory       (kbytes, -l) 64
> > max memory size         (kbytes, -m) unlimited
> > open files                      (-n) 1024
> > pipe size            (512 bytes, -p) 8
> > POSIX message queues     (bytes, -q) 819200
> > real-time priority              (-r) 0
> > stack size              (kbytes, -s) 8192
> > cpu time               (seconds, -t) unlimited
> > max user processes              (-u) 62187
> > virtual memory          (kbytes, -v) unlimited
> > file locks                      (-x) unlimited
> > 
> > In the meantime I've found an interesting thread about this problem:
> > https://www.mail-archive.com/misc(a)openbsd.org/msg144710.html
> > 
> > "A closer reading of the man page reveals that memory consumption
> > is even higher in multi-threaded mode.
> > In multi-threaded mode about three times _size_ bytes will be
> > allocated in each thread for buffering input and output.
> > ...
> > A quick check with top(1) confirms that xz in
> > multi-threaded mode allocates 1250 MB per thread at compression
> > level 9."
> > 
> > '-8' should be lower per thread, but too much for 8 threads.
> > 
> > Best,
> > Matthias
> > 
> > > -Michael
> > > 
> > > On Tue, 2018-05-22 at 18:31 +0200, Matthias Fischer wrote:
> > > > Hi!
> > > > 
> > > > Sorry, but last commit "make.sh: Use all processor cores for
> > > > compression" causes the build to break.
> > > > 
> > > > See
> > > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=c8453e87599fe66f4d18d9
> > > > 01bc
> > > > 9dc431306afa2d
> > > > 
> > > > Build stops with:
> > > > 
> > > > ***SNIP***
> > > > ...
> > > > rm -f /tmp/ROOTFILES
> > > > tar -x -C /tmp -f /ipfire.tar
> > > > rm -f /ipfire.tar
> > > > cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB >
> > > > /install/cdrom/distro.img && rm -rf *
> > > > xz: (stdin): Cannot allocate memory
> > > > make: *** [cdrom:66: /usr/src/log/cdrom] Error 1
> > > > ...
> > > > ***SNAP***
> > > > 
> > > > On my machine, 'lsf/cdrom' doesn't like '-T8'. It needed '-T4'.
> > > > I experienced this already during my tests.
> > > > 
> > > > Can anyone confirm?
> > > > 
> > > > Best,
> > > > Matthias
> > > > 
> > > > P.S.: Hardware is i7/2600 with HT, 8 GB RAM.

Re: Current 'next' won't build => 'cannot allocate memory'

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 7986 bytes --]

Hi,

On 23.05.2018 13:18, Michael Tremer wrote:
> I fucked it up again. Sent another patch for builds on 32 bit
> architectures.

Don't worry - things happen.

I just looked and waited for what would come out in the end... ;-)

And during testing I learned a few things about programming, too.

>   https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=212f17c1145bb6978b4797eb9c805f603ecb1f29
> 
> But if this is now running through, I suppose we are ready to call this
> closed.

Arne pushed 4.14.43 yesterday, while my 'Devel' was still building
'next' from 23. May
(https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=6b4174133a4cad58db3570d1cd1a16b243d4da84)
and I hadn't the time to look at the results until now.

For now, its looking good.

Last build went down from ~319 MB
(ipfire-2.19.2gb-ext4.i586-full-core121.img.gz) to ~183 MB
(ipfire-2.19.2gb-ext4.i586-full-core121.img.xz). Minus -136 MB. Not bad.

One thing I noticed: building on a 32bit system led to ~553 additional
lines in '_build.packages.log', telling me that "xz: Adjusted the number
of threads from 8 to 3..." Doesn't hurt me yet. I just noticed.

> About the planet post: Has anything changed here? Can we release this?
> 
>   https://planet.ipfire.org/post/increasing-download-installation-speed-benefits-of-a-smaller-iso-image

For me, its ok.

If you need the latest sizes: I'll build the current 'next' with 4.14.43
tonight
(https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=6c9651f6208db27d0daad7a822c9a511e509fc9c)

Best,
Matthias

> -Michael
> 
> On Tue, 2018-05-22 at 20:57 +0100, Michael Tremer wrote:
>> Hi,
>> 
>> yes that was me who broke that then.
>> 
>> I just pushed another commit that set this back to 0. Then XZ will automatically
>> run as many processes as it has memory for.
>> 
>>   https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=f03e254f39ddb7cacea7a39ec6b98dce77018b0d
>> 
>> In addition to that I had to limit memory to 2GB on 32 bit systems. They cannot
>> allocate more than that due to their limited address space.
>> 
>>   https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=a92fb74d1aeb9229044c1a9fbc54abafca50da58
>> 
>> Let me know if that works for you as well.
>> 
>> Best,
>> -Michael
>> 
>> On Tue, 2018-05-22 at 21:05 +0200, Matthias Fischer wrote:
>> > Hi,
>> > 
>> > On 22.05.2018 20:03, Michael Tremer wrote:
>> > > Hi,
>> > > 
>> > > I thought xz was now limited to lower memory.
>> > 
>> > Thats what I thought too, but it never worked as I wanted.
>> > 
>> > > Could you add -v to see what xz  has decided to do?
>> > 
>> > Adding '-v', I've got some weird results.
>> > 
>> > 1. Added '-v' in 'make.sh':
>> >  => "-T$(system_processors) -8 --memory=${XZ_MEM} -v"
>> > 
>> > Output:
>> > 
>> > ***SNIP***
>> > ...
>> >  tar -x -C /tmp -f /ipfire.tar
>> >     rm -f /ipfire.tar
>> >     cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB -v >
>> > /install/cdrom/distro.img && rm -rf *
>> >     xz: Filter chain: --
>> > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
>> >     xz: Using up to 8 threads.
>> >     xz: 5257 MiB of memory is required. The limit is 5450 MiB.
>> >     xz: Decompression will need 33 MiB of memory.
>> >     (stdin): 120.1 KiB / 512.0 KiB = 0.234
>> >     xz: (stdin): Cannot allocate memory
>> >     (stdin): 120.1 KiB / 512.0 KiB = 0.234
>> >     make: *** [cdrom:66: /usr/src/log/cdrom] Error 1...
>> > ***SNAP***
>> > 
>> > Memory limit is higher than required memory but it still says
>> > 'cannot allocate memory'!?
>> > And how to handle this (stdin)-message?
>> > 
>> > 2. Changed line: XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
>> > 
>> > to
>> > 
>> > XZ_MEM="$(( HOST_MEM * 9 / 10 ))MiB"
>> > 
>> > Output:
>> > 
>> > ***SNIP***
>> > ...
>> >  rm -f /ipfire.tar
>> >     cd /tmp && tar cf - * | xz -T8 -8 --memory=7007MiB -v >
>> > /install/cdrom/distro.img && rm -rf *
>> >     xz: Filter chain: --
>> > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
>> >     xz: Using up to 8 threads.
>> >     xz: 5257 MiB of memory is required. The limit is 7007 MiB.
>> >     xz: Decompression will need 33 MiB of memory.
>> >     (stdin): 120.1 KiB / 512.0 KiB = 0.234
>> >     xz: (stdin): Cannot allocate memory
>> >     (stdin): 120.1 KiB / 512.0 KiB = 0.234
>> >     make: *** [cdrom:66: /usr/src/log/cdrom] Error 1
>> > ...***SNAP***
>> > 
>> > 3. Using '-T4' and XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
>> > 
>> > All is well, log says:
>> > 
>> > ***SNIP***
>> > ...
>> > tar -x -C /tmp -f /ipfire.tar
>> > rm -f /ipfire.tar
>> > cd /tmp && tar cf - * | xz -T4 -8 --memory=5450MiB -v >
>> > /install/cdrom/distro.img && rm -rf *
>> > xz: Filter chain: --
>> > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
>> > xz: Using up to 4 threads.
>> > xz: 2629 MiB of memory is required. The limit is 5450 MiB.
>> > xz: Decompression will need 33 MiB of memory.
>> > ...
>> > ***SNAP***
>> > 
>> > Some more infos (helpful?):
>> > 
>> > root(a)Devel: /home/matz/ipfire-2.x # xz --info-memory
>> > Total amount of physical memory (RAM):  7,787 MiB (8,165,179,392 B)
>> > Memory usage limit for compression:     Disabled
>> > Memory usage limit for decompression:   Disabled
>> > 
>> > root(a)Devel: /home/matz/ipfire-2.x # ulimit -a
>> > core file size          (blocks, -c) 0
>> > data seg size           (kbytes, -d) unlimited
>> > scheduling priority             (-e) 0
>> > file size               (blocks, -f) unlimited
>> > pending signals                 (-i) 62187
>> > max locked memory       (kbytes, -l) 64
>> > max memory size         (kbytes, -m) unlimited
>> > open files                      (-n) 1024
>> > pipe size            (512 bytes, -p) 8
>> > POSIX message queues     (bytes, -q) 819200
>> > real-time priority              (-r) 0
>> > stack size              (kbytes, -s) 8192
>> > cpu time               (seconds, -t) unlimited
>> > max user processes              (-u) 62187
>> > virtual memory          (kbytes, -v) unlimited
>> > file locks                      (-x) unlimited
>> > 
>> > In the meantime I've found an interesting thread about this problem:
>> > https://www.mail-archive.com/misc(a)openbsd.org/msg144710.html
>> > 
>> > "A closer reading of the man page reveals that memory consumption
>> > is even higher in multi-threaded mode.
>> > In multi-threaded mode about three times _size_ bytes will be
>> > allocated in each thread for buffering input and output.
>> > ...
>> > A quick check with top(1) confirms that xz in
>> > multi-threaded mode allocates 1250 MB per thread at compression
>> > level 9."
>> > 
>> > '-8' should be lower per thread, but too much for 8 threads.
>> > 
>> > Best,
>> > Matthias
>> > 
>> > > -Michael
>> > > 
>> > > On Tue, 2018-05-22 at 18:31 +0200, Matthias Fischer wrote:
>> > > > Hi!
>> > > > 
>> > > > Sorry, but last commit "make.sh: Use all processor cores for
>> > > > compression" causes the build to break.
>> > > > 
>> > > > See
>> > > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=c8453e87599fe66f4d18d9
>> > > > 01bc
>> > > > 9dc431306afa2d
>> > > > 
>> > > > Build stops with:
>> > > > 
>> > > > ***SNIP***
>> > > > ...
>> > > > rm -f /tmp/ROOTFILES
>> > > > tar -x -C /tmp -f /ipfire.tar
>> > > > rm -f /ipfire.tar
>> > > > cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB >
>> > > > /install/cdrom/distro.img && rm -rf *
>> > > > xz: (stdin): Cannot allocate memory
>> > > > make: *** [cdrom:66: /usr/src/log/cdrom] Error 1
>> > > > ...
>> > > > ***SNAP***
>> > > > 
>> > > > On my machine, 'lsf/cdrom' doesn't like '-T8'. It needed '-T4'.
>> > > > I experienced this already during my tests.
>> > > > 
>> > > > Can anyone confirm?
>> > > > 
>> > > > Best,
>> > > > Matthias
>> > > > 
>> > > > P.S.: Hardware is i7/2600 with HT, 8 GB RAM.
> 

Re: Current 'next' won't build => 'cannot allocate memory'

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 9216 bytes --]

Hi,

On Thu, 2018-05-24 at 18:54 +0200, Matthias Fischer wrote:
> Hi,
> 
> On 23.05.2018 13:18, Michael Tremer wrote:
> > I fucked it up again. Sent another patch for builds on 32 bit
> > architectures.
> 
> Don't worry - things happen.
> 
> I just looked and waited for what would come out in the end... ;-)

I think it is working fine now.

Can you remind me again why we picked 70%? Was there a specific reason? Should
we go up to 80 or 90%?

> And during testing I learned a few things about programming, too.
> 
> >   https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=212f17c1145bb6978b
> > 4797eb9c805f603ecb1f29
> > 
> > But if this is now running through, I suppose we are ready to call this
> > closed.
> 
> Arne pushed 4.14.43 yesterday, while my 'Devel' was still building
> 'next' from 23. May
> (https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=6b4174133a4cad58db3570d1c
> d1a16b243d4da84)
> and I hadn't the time to look at the results until now.
> 
> For now, its looking good.
> 
> Last build went down from ~319 MB
> (ipfire-2.19.2gb-ext4.i586-full-core121.img.gz) to ~183 MB
> (ipfire-2.19.2gb-ext4.i586-full-core121.img.xz). Minus -136 MB. Not bad.

Yes indeed. There is also no serial image any more. That saves extra space :)

> One thing I noticed: building on a 32bit system led to ~553 additional
> lines in '_build.packages.log', telling me that "xz: Adjusted the number
> of threads from 8 to 3..." Doesn't hurt me yet. I just noticed.

Yes that is expected since the systems cannot use too much memory. But we are
using as many cores as we can which I think is the best.

> > About the planet post: Has anything changed here? Can we release this?
> > 
> >   https://planet.ipfire.org/post/increasing-download-installation-speed-bene
> > fits-of-a-smaller-iso-image
> 
> For me, its ok.

It needs some values in there because I didn't know how much we would save.

When should we release this?

> If you need the latest sizes: I'll build the current 'next' with 4.14.43
> tonight
> (https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=6c9651f6208db27d0daad7a82
> 2c9a511e509fc9c)
> 
> Best,
> Matthias
> 
> > -Michael
> > 
> > On Tue, 2018-05-22 at 20:57 +0100, Michael Tremer wrote:
> > > Hi,
> > > 
> > > yes that was me who broke that then.
> > > 
> > > I just pushed another commit that set this back to 0. Then XZ will
> > > automatically
> > > run as many processes as it has memory for.
> > > 
> > >   https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=f03e254f39ddb7ca
> > > cea7a39ec6b98dce77018b0d
> > > 
> > > In addition to that I had to limit memory to 2GB on 32 bit systems. They
> > > cannot
> > > allocate more than that due to their limited address space.
> > > 
> > >   https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=a92fb74d1aeb9229
> > > 044c1a9fbc54abafca50da58
> > > 
> > > Let me know if that works for you as well.
> > > 
> > > Best,
> > > -Michael
> > > 
> > > On Tue, 2018-05-22 at 21:05 +0200, Matthias Fischer wrote:
> > > > Hi,
> > > > 
> > > > On 22.05.2018 20:03, Michael Tremer wrote:
> > > > > Hi,
> > > > > 
> > > > > I thought xz was now limited to lower memory.
> > > > 
> > > > Thats what I thought too, but it never worked as I wanted.
> > > > 
> > > > > Could you add -v to see what xz  has decided to do?
> > > > 
> > > > Adding '-v', I've got some weird results.
> > > > 
> > > > 1. Added '-v' in 'make.sh':
> > > >  => "-T$(system_processors) -8 --memory=${XZ_MEM} -v"
> > > > 
> > > > Output:
> > > > 
> > > > ***SNIP***
> > > > ...
> > > >  tar -x -C /tmp -f /ipfire.tar
> > > >     rm -f /ipfire.tar
> > > >     cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB -v >
> > > > /install/cdrom/distro.img && rm -rf *
> > > >     xz: Filter chain: --
> > > > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
> > > >     xz: Using up to 8 threads.
> > > >     xz: 5257 MiB of memory is required. The limit is 5450 MiB.
> > > >     xz: Decompression will need 33 MiB of memory.
> > > >     (stdin): 120.1 KiB / 512.0 KiB = 0.234
> > > >     xz: (stdin): Cannot allocate memory
> > > >     (stdin): 120.1 KiB / 512.0 KiB = 0.234
> > > >     make: *** [cdrom:66: /usr/src/log/cdrom] Error 1...
> > > > ***SNAP***
> > > > 
> > > > Memory limit is higher than required memory but it still says
> > > > 'cannot allocate memory'!?
> > > > And how to handle this (stdin)-message?
> > > > 
> > > > 2. Changed line: XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
> > > > 
> > > > to
> > > > 
> > > > XZ_MEM="$(( HOST_MEM * 9 / 10 ))MiB"
> > > > 
> > > > Output:
> > > > 
> > > > ***SNIP***
> > > > ...
> > > >  rm -f /ipfire.tar
> > > >     cd /tmp && tar cf - * | xz -T8 -8 --memory=7007MiB -v >
> > > > /install/cdrom/distro.img && rm -rf *
> > > >     xz: Filter chain: --
> > > > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
> > > >     xz: Using up to 8 threads.
> > > >     xz: 5257 MiB of memory is required. The limit is 7007 MiB.
> > > >     xz: Decompression will need 33 MiB of memory.
> > > >     (stdin): 120.1 KiB / 512.0 KiB = 0.234
> > > >     xz: (stdin): Cannot allocate memory
> > > >     (stdin): 120.1 KiB / 512.0 KiB = 0.234
> > > >     make: *** [cdrom:66: /usr/src/log/cdrom] Error 1
> > > > ...***SNAP***
> > > > 
> > > > 3. Using '-T4' and XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
> > > > 
> > > > All is well, log says:
> > > > 
> > > > ***SNIP***
> > > > ...
> > > > tar -x -C /tmp -f /ipfire.tar
> > > > rm -f /ipfire.tar
> > > > cd /tmp && tar cf - * | xz -T4 -8 --memory=5450MiB -v >
> > > > /install/cdrom/distro.img && rm -rf *
> > > > xz: Filter chain: --
> > > > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
> > > > xz: Using up to 4 threads.
> > > > xz: 2629 MiB of memory is required. The limit is 5450 MiB.
> > > > xz: Decompression will need 33 MiB of memory.
> > > > ...
> > > > ***SNAP***
> > > > 
> > > > Some more infos (helpful?):
> > > > 
> > > > root(a)Devel: /home/matz/ipfire-2.x # xz --info-memory
> > > > Total amount of physical memory (RAM):  7,787 MiB (8,165,179,392 B)
> > > > Memory usage limit for compression:     Disabled
> > > > Memory usage limit for decompression:   Disabled
> > > > 
> > > > root(a)Devel: /home/matz/ipfire-2.x # ulimit -a
> > > > core file size          (blocks, -c) 0
> > > > data seg size           (kbytes, -d) unlimited
> > > > scheduling priority             (-e) 0
> > > > file size               (blocks, -f) unlimited
> > > > pending signals                 (-i) 62187
> > > > max locked memory       (kbytes, -l) 64
> > > > max memory size         (kbytes, -m) unlimited
> > > > open files                      (-n) 1024
> > > > pipe size            (512 bytes, -p) 8
> > > > POSIX message queues     (bytes, -q) 819200
> > > > real-time priority              (-r) 0
> > > > stack size              (kbytes, -s) 8192
> > > > cpu time               (seconds, -t) unlimited
> > > > max user processes              (-u) 62187
> > > > virtual memory          (kbytes, -v) unlimited
> > > > file locks                      (-x) unlimited
> > > > 
> > > > In the meantime I've found an interesting thread about this problem:
> > > > https://www.mail-archive.com/misc(a)openbsd.org/msg144710.html
> > > > 
> > > > "A closer reading of the man page reveals that memory consumption
> > > > is even higher in multi-threaded mode.
> > > > In multi-threaded mode about three times _size_ bytes will be
> > > > allocated in each thread for buffering input and output.
> > > > ...
> > > > A quick check with top(1) confirms that xz in
> > > > multi-threaded mode allocates 1250 MB per thread at compression
> > > > level 9."
> > > > 
> > > > '-8' should be lower per thread, but too much for 8 threads.
> > > > 
> > > > Best,
> > > > Matthias
> > > > 
> > > > > -Michael
> > > > > 
> > > > > On Tue, 2018-05-22 at 18:31 +0200, Matthias Fischer wrote:
> > > > > > Hi!
> > > > > > 
> > > > > > Sorry, but last commit "make.sh: Use all processor cores for
> > > > > > compression" causes the build to break.
> > > > > > 
> > > > > > See
> > > > > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=c8453e87599fe66f
> > > > > > 4d18d9
> > > > > > 01bc
> > > > > > 9dc431306afa2d
> > > > > > 
> > > > > > Build stops with:
> > > > > > 
> > > > > > ***SNIP***
> > > > > > ...
> > > > > > rm -f /tmp/ROOTFILES
> > > > > > tar -x -C /tmp -f /ipfire.tar
> > > > > > rm -f /ipfire.tar
> > > > > > cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB >
> > > > > > /install/cdrom/distro.img && rm -rf *
> > > > > > xz: (stdin): Cannot allocate memory
> > > > > > make: *** [cdrom:66: /usr/src/log/cdrom] Error 1
> > > > > > ...
> > > > > > ***SNAP***
> > > > > > 
> > > > > > On my machine, 'lsf/cdrom' doesn't like '-T8'. It needed '-T4'.
> > > > > > I experienced this already during my tests.
> > > > > > 
> > > > > > Can anyone confirm?
> > > > > > 
> > > > > > Best,
> > > > > > Matthias
> > > > > > 
> > > > > > P.S.: Hardware is i7/2600 with HT, 8 GB RAM.
> 
> 

Re: Current 'next' won't build => 'cannot allocate memory'

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 2483 bytes --]

Hi,

On 25.05.2018 15:13, Michael Tremer wrote:
> Hi,
> ...
>> I just looked and waited for what would come out in the end... ;-)
> 
> I think it is working fine now.
> 
> Can you remind me again why we picked 70%? Was there a specific reason? Should
> we go up to 80 or 90%?

7.11.2017 (You):
To not run into memory limits we should detect how much memory the build
host has and set the memory limit to maybe 80% of that.

22.11.2017 (Me):
> Using '9' or '--extreme'-parameter brought no further improvements,
> on the other hand, build time and RAM consumption increased
> strongly. With '-8', minimal RAM consumption was about 680 MiB, using
> '700 Mib' was always safe. So this is the minimum I would recommend.

We set 1024 MB as system requirement and I found that I needed about 700
MB (680 MB) during my first tests.

Right now, 'Devel' is running with the the latest 'squid 4.0.24' patches
and "local xz_memory=$(( HOST_MEM * 9 / 10 ))" => I've set it to 90% -
we'll see how much cores will come into place.

>> ...
>> 
>> Last build went down from ~319 MB
>> (ipfire-2.19.2gb-ext4.i586-full-core121.img.gz) to ~183 MB
>> (ipfire-2.19.2gb-ext4.i586-full-core121.img.xz). Minus -136 MB. Not bad.
> 
> Yes indeed. There is also no serial image any more. That saves extra space :)

I was a bit puzzled as how small the *.img.xz image went. Fascinating.

>> One thing I noticed: building on a 32bit system led to ~553 additional
>> lines in '_build.packages.log', telling me that "xz: Adjusted the number
>> of threads from 8 to 3..." Doesn't hurt me yet. I just noticed.
> 
> Yes that is expected since the systems cannot use too much memory. But we are
> using as many cores as we can which I think is the best.

I'm curious what the 90% might bring.

>> > About the planet post: Has anything changed here? Can we release this?
>> > 
>> >   https://planet.ipfire.org/post/increasing-download-installation-speed-bene
>> > fits-of-a-smaller-iso-image
>> 
>> For me, its ok.
> 
> It needs some values in there because I didn't know how much we would save.
> 
> When should we release this?

WIR. ;-)

Good question. Is it running stable? Will it build on
other/smaller/bigger systems than mine? If we can answer this with a big
"YES", we should be able to release it.

I'll wait until the current building with 90% is finished and report.

But I can't speak for other development machines...

Best,
Matthias

Re: Current 'next' won't build => 'cannot allocate memory'

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 362 bytes --]

Hi,
On 25.05.2018 19:21, Matthias Fischer wrote:
> Right now, 'Devel' is running with the the latest 'squid 4.0.24' patches
> and "local xz_memory=$(( HOST_MEM * 9 / 10 ))" => I've set it to 90% -
> we'll see how much cores will come into place.

Ups. I forgot: on MY system (32bit) it'll stop at 2048 MB... ;-(

Can anyone thest this on 64bit!?

Best,
Matthias

Re: Current 'next' won't build => 'cannot allocate memory'

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 369 bytes --]

Hi,

On 25.05.2018 19:21, Matthias Fischer wrote:
> I'll wait until the current building with 90% is finished and report.

In short - results for last 'next'-build (32bit):

ipfire-2.19.2gb-ext4.i586-full-core121.img.xz => 176 MB
ipfire-2.19.i586-full-core121.iso => 207 MB

>From me: 32bit seems to be ok.

Still curious about results for 64bit... ;-)

Best,
Matthias

Re: Current 'next' won't build => 'cannot allocate memory'

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 728 bytes --]

On Sat, 2018-05-26 at 18:39 +0200, Matthias Fischer wrote:
> Hi,
> 
> On 25.05.2018 19:21, Matthias Fischer wrote:
> > I'll wait until the current building with 90% is finished and report.
> 
> In short - results for last 'next'-build (32bit):
> 
> ipfire-2.19.2gb-ext4.i586-full-core121.img.xz => 176 MB
> ipfire-2.19.i586-full-core121.iso => 207 MB
> 
> From me: 32bit seems to be ok.
> 
> Still curious about results for 64bit... ;-)

The compressed output will be exactly the same. It won't compress any better. It
will just use more memory and CPU cores and therefore will be faster.

It is running through on all build systems and even on the ARM machines.

Best,
-Michael

> 
> Best,
> Matthias