Hi!
Sorry, but last commit "make.sh: Use all processor cores for compression" causes the build to break.
See https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=c8453e87599fe66f4d18d901...
Build stops with:
***SNIP*** ... rm -f /tmp/ROOTFILES tar -x -C /tmp -f /ipfire.tar rm -f /ipfire.tar cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB > /install/cdrom/distro.img && rm -rf * xz: (stdin): Cannot allocate memory make: *** [cdrom:66: /usr/src/log/cdrom] Error 1 ... ***SNAP***
On my machine, 'lsf/cdrom' doesn't like '-T8'. It needed '-T4'. I experienced this already during my tests.
Can anyone confirm?
Best, Matthias
P.S.: Hardware is i7/2600 with HT, 8 GB RAM.
Hi,
I thought xz was now limited to lower memory. Could you add -v to see what xz has decided to do?
-Michael
On Tue, 2018-05-22 at 18:31 +0200, Matthias Fischer wrote:
Hi!
Sorry, but last commit "make.sh: Use all processor cores for compression" causes the build to break.
See https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=c8453e87599fe66f4d18d901... 9dc431306afa2d
Build stops with:
***SNIP*** ... rm -f /tmp/ROOTFILES tar -x -C /tmp -f /ipfire.tar rm -f /ipfire.tar cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB > /install/cdrom/distro.img && rm -rf * xz: (stdin): Cannot allocate memory make: *** [cdrom:66: /usr/src/log/cdrom] Error 1 ... ***SNAP***
On my machine, 'lsf/cdrom' doesn't like '-T8'. It needed '-T4'. I experienced this already during my tests.
Can anyone confirm?
Best, Matthias
P.S.: Hardware is i7/2600 with HT, 8 GB RAM.
Hi,
On 22.05.2018 20:03, Michael Tremer wrote:
Hi,
I thought xz was now limited to lower memory.
Thats what I thought too, but it never worked as I wanted.
Could you add -v to see what xz has decided to do?
Adding '-v', I've got some weird results.
1. Added '-v' in 'make.sh': => "-T$(system_processors) -8 --memory=${XZ_MEM} -v"
Output:
***SNIP*** ... tar -x -C /tmp -f /ipfire.tar rm -f /ipfire.tar cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB -v > /install/cdrom/distro.img && rm -rf * xz: Filter chain: --lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0 xz: Using up to 8 threads. xz: 5257 MiB of memory is required. The limit is 5450 MiB. xz: Decompression will need 33 MiB of memory. (stdin): 120.1 KiB / 512.0 KiB = 0.234 xz: (stdin): Cannot allocate memory (stdin): 120.1 KiB / 512.0 KiB = 0.234 make: *** [cdrom:66: /usr/src/log/cdrom] Error 1... ***SNAP***
Memory limit is higher than required memory but it still says 'cannot allocate memory'!? And how to handle this (stdin)-message?
2. Changed line: XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
to
XZ_MEM="$(( HOST_MEM * 9 / 10 ))MiB"
Output:
***SNIP*** ... rm -f /ipfire.tar cd /tmp && tar cf - * | xz -T8 -8 --memory=7007MiB -v > /install/cdrom/distro.img && rm -rf * xz: Filter chain: --lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0 xz: Using up to 8 threads. xz: 5257 MiB of memory is required. The limit is 7007 MiB. xz: Decompression will need 33 MiB of memory. (stdin): 120.1 KiB / 512.0 KiB = 0.234 xz: (stdin): Cannot allocate memory (stdin): 120.1 KiB / 512.0 KiB = 0.234 make: *** [cdrom:66: /usr/src/log/cdrom] Error 1 ...***SNAP***
3. Using '-T4' and XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
All is well, log says:
***SNIP*** ... tar -x -C /tmp -f /ipfire.tar rm -f /ipfire.tar cd /tmp && tar cf - * | xz -T4 -8 --memory=5450MiB -v > /install/cdrom/distro.img && rm -rf * xz: Filter chain: --lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0 xz: Using up to 4 threads. xz: 2629 MiB of memory is required. The limit is 5450 MiB. xz: Decompression will need 33 MiB of memory. ... ***SNAP***
Some more infos (helpful?):
root@Devel: /home/matz/ipfire-2.x # xz --info-memory Total amount of physical memory (RAM): 7,787 MiB (8,165,179,392 B) Memory usage limit for compression: Disabled Memory usage limit for decompression: Disabled
root@Devel: /home/matz/ipfire-2.x # ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 62187 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 62187 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
In the meantime I've found an interesting thread about this problem: https://www.mail-archive.com/misc@openbsd.org/msg144710.html
"A closer reading of the man page reveals that memory consumption is even higher in multi-threaded mode. In multi-threaded mode about three times _size_ bytes will be allocated in each thread for buffering input and output. ... A quick check with top(1) confirms that xz in multi-threaded mode allocates 1250 MB per thread at compression level 9."
'-8' should be lower per thread, but too much for 8 threads.
Best, Matthias
-Michael
On Tue, 2018-05-22 at 18:31 +0200, Matthias Fischer wrote:
Hi!
Sorry, but last commit "make.sh: Use all processor cores for compression" causes the build to break.
See https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=c8453e87599fe66f4d18d901... 9dc431306afa2d
Build stops with:
***SNIP*** ... rm -f /tmp/ROOTFILES tar -x -C /tmp -f /ipfire.tar rm -f /ipfire.tar cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB > /install/cdrom/distro.img && rm -rf * xz: (stdin): Cannot allocate memory make: *** [cdrom:66: /usr/src/log/cdrom] Error 1 ... ***SNAP***
On my machine, 'lsf/cdrom' doesn't like '-T8'. It needed '-T4'. I experienced this already during my tests.
Can anyone confirm?
Best, Matthias
P.S.: Hardware is i7/2600 with HT, 8 GB RAM.
Hi,
yes that was me who broke that then.
I just pushed another commit that set this back to 0. Then XZ will automatically run as many processes as it has memory for.
https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=f03e254f39ddb7cacea7...
In addition to that I had to limit memory to 2GB on 32 bit systems. They cannot allocate more than that due to their limited address space.
https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=a92fb74d1aeb9229044c...
Let me know if that works for you as well.
Best, -Michael
On Tue, 2018-05-22 at 21:05 +0200, Matthias Fischer wrote:
Hi,
On 22.05.2018 20:03, Michael Tremer wrote:
Hi,
I thought xz was now limited to lower memory.
Thats what I thought too, but it never worked as I wanted.
Could you add -v to see what xz has decided to do?
Adding '-v', I've got some weird results.
- Added '-v' in 'make.sh':
=> "-T$(system_processors) -8 --memory=${XZ_MEM} -v"
Output:
***SNIP*** ... tar -x -C /tmp -f /ipfire.tar rm -f /ipfire.tar cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB -v > /install/cdrom/distro.img && rm -rf * xz: Filter chain: -- lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0 xz: Using up to 8 threads. xz: 5257 MiB of memory is required. The limit is 5450 MiB. xz: Decompression will need 33 MiB of memory. (stdin): 120.1 KiB / 512.0 KiB = 0.234 xz: (stdin): Cannot allocate memory (stdin): 120.1 KiB / 512.0 KiB = 0.234 make: *** [cdrom:66: /usr/src/log/cdrom] Error 1... ***SNAP***
Memory limit is higher than required memory but it still says 'cannot allocate memory'!? And how to handle this (stdin)-message?
- Changed line: XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
to
XZ_MEM="$(( HOST_MEM * 9 / 10 ))MiB"
Output:
***SNIP*** ... rm -f /ipfire.tar cd /tmp && tar cf - * | xz -T8 -8 --memory=7007MiB -v > /install/cdrom/distro.img && rm -rf * xz: Filter chain: -- lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0 xz: Using up to 8 threads. xz: 5257 MiB of memory is required. The limit is 7007 MiB. xz: Decompression will need 33 MiB of memory. (stdin): 120.1 KiB / 512.0 KiB = 0.234 xz: (stdin): Cannot allocate memory (stdin): 120.1 KiB / 512.0 KiB = 0.234 make: *** [cdrom:66: /usr/src/log/cdrom] Error 1 ...***SNAP***
- Using '-T4' and XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
All is well, log says:
***SNIP*** ... tar -x -C /tmp -f /ipfire.tar rm -f /ipfire.tar cd /tmp && tar cf - * | xz -T4 -8 --memory=5450MiB -v > /install/cdrom/distro.img && rm -rf * xz: Filter chain: -- lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0 xz: Using up to 4 threads. xz: 2629 MiB of memory is required. The limit is 5450 MiB. xz: Decompression will need 33 MiB of memory. ... ***SNAP***
Some more infos (helpful?):
root@Devel: /home/matz/ipfire-2.x # xz --info-memory Total amount of physical memory (RAM): 7,787 MiB (8,165,179,392 B) Memory usage limit for compression: Disabled Memory usage limit for decompression: Disabled
root@Devel: /home/matz/ipfire-2.x # ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 62187 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 62187 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
In the meantime I've found an interesting thread about this problem: https://www.mail-archive.com/misc@openbsd.org/msg144710.html
"A closer reading of the man page reveals that memory consumption is even higher in multi-threaded mode. In multi-threaded mode about three times _size_ bytes will be allocated in each thread for buffering input and output. ... A quick check with top(1) confirms that xz in multi-threaded mode allocates 1250 MB per thread at compression level 9."
'-8' should be lower per thread, but too much for 8 threads.
Best, Matthias
-Michael
On Tue, 2018-05-22 at 18:31 +0200, Matthias Fischer wrote:
Hi!
Sorry, but last commit "make.sh: Use all processor cores for compression" causes the build to break.
See https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=c8453e87599fe66f4d18d9 01bc 9dc431306afa2d
Build stops with:
***SNIP*** ... rm -f /tmp/ROOTFILES tar -x -C /tmp -f /ipfire.tar rm -f /ipfire.tar cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB > /install/cdrom/distro.img && rm -rf * xz: (stdin): Cannot allocate memory make: *** [cdrom:66: /usr/src/log/cdrom] Error 1 ... ***SNAP***
On my machine, 'lsf/cdrom' doesn't like '-T8'. It needed '-T4'. I experienced this already during my tests.
Can anyone confirm?
Best, Matthias
P.S.: Hardware is i7/2600 with HT, 8 GB RAM.
I fucked it up again. Sent another patch for builds on 32 bit architectures.
https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=212f17c1145bb6978b47...
But if this is now running through, I suppose we are ready to call this closed.
About the planet post: Has anything changed here? Can we release this?
https://planet.ipfire.org/post/increasing-download-installation-speed-benefi...
-Michael
On Tue, 2018-05-22 at 20:57 +0100, Michael Tremer wrote:
Hi,
yes that was me who broke that then.
I just pushed another commit that set this back to 0. Then XZ will automatically run as many processes as it has memory for.
https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=f03e254f39ddb7cacea7...
In addition to that I had to limit memory to 2GB on 32 bit systems. They cannot allocate more than that due to their limited address space.
https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=a92fb74d1aeb9229044c...
Let me know if that works for you as well.
Best, -Michael
On Tue, 2018-05-22 at 21:05 +0200, Matthias Fischer wrote:
Hi,
On 22.05.2018 20:03, Michael Tremer wrote:
Hi,
I thought xz was now limited to lower memory.
Thats what I thought too, but it never worked as I wanted.
Could you add -v to see what xz has decided to do?
Adding '-v', I've got some weird results.
- Added '-v' in 'make.sh':
=> "-T$(system_processors) -8 --memory=${XZ_MEM} -v"
Output:
***SNIP*** ... tar -x -C /tmp -f /ipfire.tar rm -f /ipfire.tar cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB -v > /install/cdrom/distro.img && rm -rf * xz: Filter chain: -- lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0 xz: Using up to 8 threads. xz: 5257 MiB of memory is required. The limit is 5450 MiB. xz: Decompression will need 33 MiB of memory. (stdin): 120.1 KiB / 512.0 KiB = 0.234 xz: (stdin): Cannot allocate memory (stdin): 120.1 KiB / 512.0 KiB = 0.234 make: *** [cdrom:66: /usr/src/log/cdrom] Error 1... ***SNAP***
Memory limit is higher than required memory but it still says 'cannot allocate memory'!? And how to handle this (stdin)-message?
- Changed line: XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
to
XZ_MEM="$(( HOST_MEM * 9 / 10 ))MiB"
Output:
***SNIP*** ... rm -f /ipfire.tar cd /tmp && tar cf - * | xz -T8 -8 --memory=7007MiB -v > /install/cdrom/distro.img && rm -rf * xz: Filter chain: -- lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0 xz: Using up to 8 threads. xz: 5257 MiB of memory is required. The limit is 7007 MiB. xz: Decompression will need 33 MiB of memory. (stdin): 120.1 KiB / 512.0 KiB = 0.234 xz: (stdin): Cannot allocate memory (stdin): 120.1 KiB / 512.0 KiB = 0.234 make: *** [cdrom:66: /usr/src/log/cdrom] Error 1 ...***SNAP***
- Using '-T4' and XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
All is well, log says:
***SNIP*** ... tar -x -C /tmp -f /ipfire.tar rm -f /ipfire.tar cd /tmp && tar cf - * | xz -T4 -8 --memory=5450MiB -v > /install/cdrom/distro.img && rm -rf * xz: Filter chain: -- lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0 xz: Using up to 4 threads. xz: 2629 MiB of memory is required. The limit is 5450 MiB. xz: Decompression will need 33 MiB of memory. ... ***SNAP***
Some more infos (helpful?):
root@Devel: /home/matz/ipfire-2.x # xz --info-memory Total amount of physical memory (RAM): 7,787 MiB (8,165,179,392 B) Memory usage limit for compression: Disabled Memory usage limit for decompression: Disabled
root@Devel: /home/matz/ipfire-2.x # ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 62187 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 62187 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
In the meantime I've found an interesting thread about this problem: https://www.mail-archive.com/misc@openbsd.org/msg144710.html
"A closer reading of the man page reveals that memory consumption is even higher in multi-threaded mode. In multi-threaded mode about three times _size_ bytes will be allocated in each thread for buffering input and output. ... A quick check with top(1) confirms that xz in multi-threaded mode allocates 1250 MB per thread at compression level 9."
'-8' should be lower per thread, but too much for 8 threads.
Best, Matthias
-Michael
On Tue, 2018-05-22 at 18:31 +0200, Matthias Fischer wrote:
Hi!
Sorry, but last commit "make.sh: Use all processor cores for compression" causes the build to break.
See https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=c8453e87599fe66f4d18d9 01bc 9dc431306afa2d
Build stops with:
***SNIP*** ... rm -f /tmp/ROOTFILES tar -x -C /tmp -f /ipfire.tar rm -f /ipfire.tar cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB > /install/cdrom/distro.img && rm -rf * xz: (stdin): Cannot allocate memory make: *** [cdrom:66: /usr/src/log/cdrom] Error 1 ... ***SNAP***
On my machine, 'lsf/cdrom' doesn't like '-T8'. It needed '-T4'. I experienced this already during my tests.
Can anyone confirm?
Best, Matthias
P.S.: Hardware is i7/2600 with HT, 8 GB RAM.
Hi,
On 23.05.2018 13:18, Michael Tremer wrote:
I fucked it up again. Sent another patch for builds on 32 bit architectures.
Don't worry - things happen.
I just looked and waited for what would come out in the end... ;-)
And during testing I learned a few things about programming, too.
https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=212f17c1145bb6978b47...
But if this is now running through, I suppose we are ready to call this closed.
Arne pushed 4.14.43 yesterday, while my 'Devel' was still building 'next' from 23. May (https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=6b4174133a4cad58db3570d1...) and I hadn't the time to look at the results until now.
For now, its looking good.
Last build went down from ~319 MB (ipfire-2.19.2gb-ext4.i586-full-core121.img.gz) to ~183 MB (ipfire-2.19.2gb-ext4.i586-full-core121.img.xz). Minus -136 MB. Not bad.
One thing I noticed: building on a 32bit system led to ~553 additional lines in '_build.packages.log', telling me that "xz: Adjusted the number of threads from 8 to 3..." Doesn't hurt me yet. I just noticed.
About the planet post: Has anything changed here? Can we release this?
https://planet.ipfire.org/post/increasing-download-installation-speed-benefi...
For me, its ok.
If you need the latest sizes: I'll build the current 'next' with 4.14.43 tonight (https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=6c9651f6208db27d0daad7a8...)
Best, Matthias
-Michael
On Tue, 2018-05-22 at 20:57 +0100, Michael Tremer wrote:
Hi,
yes that was me who broke that then.
I just pushed another commit that set this back to 0. Then XZ will automatically run as many processes as it has memory for.
https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=f03e254f39ddb7cacea7...
In addition to that I had to limit memory to 2GB on 32 bit systems. They cannot allocate more than that due to their limited address space.
https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=a92fb74d1aeb9229044c...
Let me know if that works for you as well.
Best, -Michael
On Tue, 2018-05-22 at 21:05 +0200, Matthias Fischer wrote:
Hi,
On 22.05.2018 20:03, Michael Tremer wrote:
Hi,
I thought xz was now limited to lower memory.
Thats what I thought too, but it never worked as I wanted.
Could you add -v to see what xz has decided to do?
Adding '-v', I've got some weird results.
- Added '-v' in 'make.sh':
=> "-T$(system_processors) -8 --memory=${XZ_MEM} -v"
Output:
***SNIP*** ... tar -x -C /tmp -f /ipfire.tar rm -f /ipfire.tar cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB -v > /install/cdrom/distro.img && rm -rf * xz: Filter chain: -- lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0 xz: Using up to 8 threads. xz: 5257 MiB of memory is required. The limit is 5450 MiB. xz: Decompression will need 33 MiB of memory. (stdin): 120.1 KiB / 512.0 KiB = 0.234 xz: (stdin): Cannot allocate memory (stdin): 120.1 KiB / 512.0 KiB = 0.234 make: *** [cdrom:66: /usr/src/log/cdrom] Error 1... ***SNAP***
Memory limit is higher than required memory but it still says 'cannot allocate memory'!? And how to handle this (stdin)-message?
- Changed line: XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
to
XZ_MEM="$(( HOST_MEM * 9 / 10 ))MiB"
Output:
***SNIP*** ... rm -f /ipfire.tar cd /tmp && tar cf - * | xz -T8 -8 --memory=7007MiB -v > /install/cdrom/distro.img && rm -rf * xz: Filter chain: -- lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0 xz: Using up to 8 threads. xz: 5257 MiB of memory is required. The limit is 7007 MiB. xz: Decompression will need 33 MiB of memory. (stdin): 120.1 KiB / 512.0 KiB = 0.234 xz: (stdin): Cannot allocate memory (stdin): 120.1 KiB / 512.0 KiB = 0.234 make: *** [cdrom:66: /usr/src/log/cdrom] Error 1 ...***SNAP***
- Using '-T4' and XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
All is well, log says:
***SNIP*** ... tar -x -C /tmp -f /ipfire.tar rm -f /ipfire.tar cd /tmp && tar cf - * | xz -T4 -8 --memory=5450MiB -v > /install/cdrom/distro.img && rm -rf * xz: Filter chain: -- lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0 xz: Using up to 4 threads. xz: 2629 MiB of memory is required. The limit is 5450 MiB. xz: Decompression will need 33 MiB of memory. ... ***SNAP***
Some more infos (helpful?):
root@Devel: /home/matz/ipfire-2.x # xz --info-memory Total amount of physical memory (RAM): 7,787 MiB (8,165,179,392 B) Memory usage limit for compression: Disabled Memory usage limit for decompression: Disabled
root@Devel: /home/matz/ipfire-2.x # ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 62187 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 62187 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
In the meantime I've found an interesting thread about this problem: https://www.mail-archive.com/misc@openbsd.org/msg144710.html
"A closer reading of the man page reveals that memory consumption is even higher in multi-threaded mode. In multi-threaded mode about three times _size_ bytes will be allocated in each thread for buffering input and output. ... A quick check with top(1) confirms that xz in multi-threaded mode allocates 1250 MB per thread at compression level 9."
'-8' should be lower per thread, but too much for 8 threads.
Best, Matthias
-Michael
On Tue, 2018-05-22 at 18:31 +0200, Matthias Fischer wrote:
Hi!
Sorry, but last commit "make.sh: Use all processor cores for compression" causes the build to break.
See https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=c8453e87599fe66f4d18d9 01bc 9dc431306afa2d
Build stops with:
***SNIP*** ... rm -f /tmp/ROOTFILES tar -x -C /tmp -f /ipfire.tar rm -f /ipfire.tar cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB > /install/cdrom/distro.img && rm -rf * xz: (stdin): Cannot allocate memory make: *** [cdrom:66: /usr/src/log/cdrom] Error 1 ... ***SNAP***
On my machine, 'lsf/cdrom' doesn't like '-T8'. It needed '-T4'. I experienced this already during my tests.
Can anyone confirm?
Best, Matthias
P.S.: Hardware is i7/2600 with HT, 8 GB RAM.
Hi,
On Thu, 2018-05-24 at 18:54 +0200, Matthias Fischer wrote:
Hi,
On 23.05.2018 13:18, Michael Tremer wrote:
I fucked it up again. Sent another patch for builds on 32 bit architectures.
Don't worry - things happen.
I just looked and waited for what would come out in the end... ;-)
I think it is working fine now.
Can you remind me again why we picked 70%? Was there a specific reason? Should we go up to 80 or 90%?
And during testing I learned a few things about programming, too.
https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=212f17c1145bb6978b 4797eb9c805f603ecb1f29
But if this is now running through, I suppose we are ready to call this closed.
Arne pushed 4.14.43 yesterday, while my 'Devel' was still building 'next' from 23. May (https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=6b4174133a4cad58db3570d1... d1a16b243d4da84) and I hadn't the time to look at the results until now.
For now, its looking good.
Last build went down from ~319 MB (ipfire-2.19.2gb-ext4.i586-full-core121.img.gz) to ~183 MB (ipfire-2.19.2gb-ext4.i586-full-core121.img.xz). Minus -136 MB. Not bad.
Yes indeed. There is also no serial image any more. That saves extra space :)
One thing I noticed: building on a 32bit system led to ~553 additional lines in '_build.packages.log', telling me that "xz: Adjusted the number of threads from 8 to 3..." Doesn't hurt me yet. I just noticed.
Yes that is expected since the systems cannot use too much memory. But we are using as many cores as we can which I think is the best.
About the planet post: Has anything changed here? Can we release this?
https://planet.ipfire.org/post/increasing-download-installation-speed-bene fits-of-a-smaller-iso-image
For me, its ok.
It needs some values in there because I didn't know how much we would save.
When should we release this?
If you need the latest sizes: I'll build the current 'next' with 4.14.43 tonight (https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=6c9651f6208db27d0daad7a8... 2c9a511e509fc9c)
Best, Matthias
-Michael
On Tue, 2018-05-22 at 20:57 +0100, Michael Tremer wrote:
Hi,
yes that was me who broke that then.
I just pushed another commit that set this back to 0. Then XZ will automatically run as many processes as it has memory for.
https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=f03e254f39ddb7ca cea7a39ec6b98dce77018b0d
In addition to that I had to limit memory to 2GB on 32 bit systems. They cannot allocate more than that due to their limited address space.
https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=a92fb74d1aeb9229 044c1a9fbc54abafca50da58
Let me know if that works for you as well.
Best, -Michael
On Tue, 2018-05-22 at 21:05 +0200, Matthias Fischer wrote:
Hi,
On 22.05.2018 20:03, Michael Tremer wrote:
Hi,
I thought xz was now limited to lower memory.
Thats what I thought too, but it never worked as I wanted.
Could you add -v to see what xz has decided to do?
Adding '-v', I've got some weird results.
- Added '-v' in 'make.sh':
=> "-T$(system_processors) -8 --memory=${XZ_MEM} -v"
Output:
***SNIP*** ... tar -x -C /tmp -f /ipfire.tar rm -f /ipfire.tar cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB -v > /install/cdrom/distro.img && rm -rf * xz: Filter chain: -- lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0 xz: Using up to 8 threads. xz: 5257 MiB of memory is required. The limit is 5450 MiB. xz: Decompression will need 33 MiB of memory. (stdin): 120.1 KiB / 512.0 KiB = 0.234 xz: (stdin): Cannot allocate memory (stdin): 120.1 KiB / 512.0 KiB = 0.234 make: *** [cdrom:66: /usr/src/log/cdrom] Error 1... ***SNAP***
Memory limit is higher than required memory but it still says 'cannot allocate memory'!? And how to handle this (stdin)-message?
- Changed line: XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
to
XZ_MEM="$(( HOST_MEM * 9 / 10 ))MiB"
Output:
***SNIP*** ... rm -f /ipfire.tar cd /tmp && tar cf - * | xz -T8 -8 --memory=7007MiB -v > /install/cdrom/distro.img && rm -rf * xz: Filter chain: -- lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0 xz: Using up to 8 threads. xz: 5257 MiB of memory is required. The limit is 7007 MiB. xz: Decompression will need 33 MiB of memory. (stdin): 120.1 KiB / 512.0 KiB = 0.234 xz: (stdin): Cannot allocate memory (stdin): 120.1 KiB / 512.0 KiB = 0.234 make: *** [cdrom:66: /usr/src/log/cdrom] Error 1 ...***SNAP***
- Using '-T4' and XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
All is well, log says:
***SNIP*** ... tar -x -C /tmp -f /ipfire.tar rm -f /ipfire.tar cd /tmp && tar cf - * | xz -T4 -8 --memory=5450MiB -v > /install/cdrom/distro.img && rm -rf * xz: Filter chain: -- lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0 xz: Using up to 4 threads. xz: 2629 MiB of memory is required. The limit is 5450 MiB. xz: Decompression will need 33 MiB of memory. ... ***SNAP***
Some more infos (helpful?):
root@Devel: /home/matz/ipfire-2.x # xz --info-memory Total amount of physical memory (RAM): 7,787 MiB (8,165,179,392 B) Memory usage limit for compression: Disabled Memory usage limit for decompression: Disabled
root@Devel: /home/matz/ipfire-2.x # ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 62187 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 62187 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
In the meantime I've found an interesting thread about this problem: https://www.mail-archive.com/misc@openbsd.org/msg144710.html
"A closer reading of the man page reveals that memory consumption is even higher in multi-threaded mode. In multi-threaded mode about three times _size_ bytes will be allocated in each thread for buffering input and output. ... A quick check with top(1) confirms that xz in multi-threaded mode allocates 1250 MB per thread at compression level 9."
'-8' should be lower per thread, but too much for 8 threads.
Best, Matthias
-Michael
On Tue, 2018-05-22 at 18:31 +0200, Matthias Fischer wrote:
Hi!
Sorry, but last commit "make.sh: Use all processor cores for compression" causes the build to break.
See https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=c8453e87599fe66f 4d18d9 01bc 9dc431306afa2d
Build stops with:
***SNIP*** ... rm -f /tmp/ROOTFILES tar -x -C /tmp -f /ipfire.tar rm -f /ipfire.tar cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB > /install/cdrom/distro.img && rm -rf * xz: (stdin): Cannot allocate memory make: *** [cdrom:66: /usr/src/log/cdrom] Error 1 ... ***SNAP***
On my machine, 'lsf/cdrom' doesn't like '-T8'. It needed '-T4'. I experienced this already during my tests.
Can anyone confirm?
Best, Matthias
P.S.: Hardware is i7/2600 with HT, 8 GB RAM.
Hi,
On 25.05.2018 15:13, Michael Tremer wrote:
Hi, ...
I just looked and waited for what would come out in the end... ;-)
I think it is working fine now.
Can you remind me again why we picked 70%? Was there a specific reason? Should we go up to 80 or 90%?
7.11.2017 (You): To not run into memory limits we should detect how much memory the build host has and set the memory limit to maybe 80% of that.
22.11.2017 (Me):
Using '9' or '--extreme'-parameter brought no further improvements, on the other hand, build time and RAM consumption increased strongly. With '-8', minimal RAM consumption was about 680 MiB, using '700 Mib' was always safe. So this is the minimum I would recommend.
We set 1024 MB as system requirement and I found that I needed about 700 MB (680 MB) during my first tests.
Right now, 'Devel' is running with the the latest 'squid 4.0.24' patches and "local xz_memory=$(( HOST_MEM * 9 / 10 ))" => I've set it to 90% - we'll see how much cores will come into place.
...
Last build went down from ~319 MB (ipfire-2.19.2gb-ext4.i586-full-core121.img.gz) to ~183 MB (ipfire-2.19.2gb-ext4.i586-full-core121.img.xz). Minus -136 MB. Not bad.
Yes indeed. There is also no serial image any more. That saves extra space :)
I was a bit puzzled as how small the *.img.xz image went. Fascinating.
One thing I noticed: building on a 32bit system led to ~553 additional lines in '_build.packages.log', telling me that "xz: Adjusted the number of threads from 8 to 3..." Doesn't hurt me yet. I just noticed.
Yes that is expected since the systems cannot use too much memory. But we are using as many cores as we can which I think is the best.
I'm curious what the 90% might bring.
About the planet post: Has anything changed here? Can we release this?
https://planet.ipfire.org/post/increasing-download-installation-speed-bene fits-of-a-smaller-iso-image
For me, its ok.
It needs some values in there because I didn't know how much we would save.
When should we release this?
WIR. ;-)
Good question. Is it running stable? Will it build on other/smaller/bigger systems than mine? If we can answer this with a big "YES", we should be able to release it.
I'll wait until the current building with 90% is finished and report.
But I can't speak for other development machines...
Best, Matthias
Hi, On 25.05.2018 19:21, Matthias Fischer wrote:
Right now, 'Devel' is running with the the latest 'squid 4.0.24' patches and "local xz_memory=$(( HOST_MEM * 9 / 10 ))" => I've set it to 90% - we'll see how much cores will come into place.
Ups. I forgot: on MY system (32bit) it'll stop at 2048 MB... ;-(
Can anyone thest this on 64bit!?
Best, Matthias
Hi,
On 25.05.2018 19:21, Matthias Fischer wrote:
I'll wait until the current building with 90% is finished and report.
In short - results for last 'next'-build (32bit):
ipfire-2.19.2gb-ext4.i586-full-core121.img.xz => 176 MB ipfire-2.19.i586-full-core121.iso => 207 MB
From me: 32bit seems to be ok.
Still curious about results for 64bit... ;-)
Best, Matthias
On Sat, 2018-05-26 at 18:39 +0200, Matthias Fischer wrote:
Hi,
On 25.05.2018 19:21, Matthias Fischer wrote:
I'll wait until the current building with 90% is finished and report.
In short - results for last 'next'-build (32bit):
ipfire-2.19.2gb-ext4.i586-full-core121.img.xz => 176 MB ipfire-2.19.i586-full-core121.iso => 207 MB
From me: 32bit seems to be ok.
Still curious about results for 64bit... ;-)
The compressed output will be exactly the same. It won't compress any better. It will just use more memory and CPU cores and therefore will be faster.
It is running through on all build systems and even on the ARM machines.
Best, -Michael
Best, Matthias
-
Matthias Fischer -
Michael Tremer