Hi,
On Thu, 2018-05-24 at 18:54 +0200, Matthias Fischer wrote:
Hi,
On 23.05.2018 13:18, Michael Tremer wrote:
I fucked it up again. Sent another patch for builds on 32 bit architectures.
Don't worry - things happen.
I just looked and waited for what would come out in the end... ;-)
I think it is working fine now.
Can you remind me again why we picked 70%? Was there a specific reason? Should we go up to 80 or 90%?
And during testing I learned a few things about programming, too.
https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=212f17c1145bb6978b 4797eb9c805f603ecb1f29
But if this is now running through, I suppose we are ready to call this closed.
Arne pushed 4.14.43 yesterday, while my 'Devel' was still building 'next' from 23. May (https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=6b4174133a4cad58db3570d1... d1a16b243d4da84) and I hadn't the time to look at the results until now.
For now, its looking good.
Last build went down from ~319 MB (ipfire-2.19.2gb-ext4.i586-full-core121.img.gz) to ~183 MB (ipfire-2.19.2gb-ext4.i586-full-core121.img.xz). Minus -136 MB. Not bad.
Yes indeed. There is also no serial image any more. That saves extra space :)
One thing I noticed: building on a 32bit system led to ~553 additional lines in '_build.packages.log', telling me that "xz: Adjusted the number of threads from 8 to 3..." Doesn't hurt me yet. I just noticed.
Yes that is expected since the systems cannot use too much memory. But we are using as many cores as we can which I think is the best.
About the planet post: Has anything changed here? Can we release this?
https://planet.ipfire.org/post/increasing-download-installation-speed-bene fits-of-a-smaller-iso-image
For me, its ok.
It needs some values in there because I didn't know how much we would save.
When should we release this?
If you need the latest sizes: I'll build the current 'next' with 4.14.43 tonight (https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=6c9651f6208db27d0daad7a8... 2c9a511e509fc9c)
Best, Matthias
-Michael
On Tue, 2018-05-22 at 20:57 +0100, Michael Tremer wrote:
Hi,
yes that was me who broke that then.
I just pushed another commit that set this back to 0. Then XZ will automatically run as many processes as it has memory for.
https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=f03e254f39ddb7ca cea7a39ec6b98dce77018b0d
In addition to that I had to limit memory to 2GB on 32 bit systems. They cannot allocate more than that due to their limited address space.
https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=a92fb74d1aeb9229 044c1a9fbc54abafca50da58
Let me know if that works for you as well.
Best, -Michael
On Tue, 2018-05-22 at 21:05 +0200, Matthias Fischer wrote:
Hi,
On 22.05.2018 20:03, Michael Tremer wrote:
Hi,
I thought xz was now limited to lower memory.
Thats what I thought too, but it never worked as I wanted.
Could you add -v to see what xz has decided to do?
Adding '-v', I've got some weird results.
- Added '-v' in 'make.sh':
=> "-T$(system_processors) -8 --memory=${XZ_MEM} -v"
Output:
***SNIP*** ... tar -x -C /tmp -f /ipfire.tar rm -f /ipfire.tar cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB -v > /install/cdrom/distro.img && rm -rf * xz: Filter chain: -- lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0 xz: Using up to 8 threads. xz: 5257 MiB of memory is required. The limit is 5450 MiB. xz: Decompression will need 33 MiB of memory. (stdin): 120.1 KiB / 512.0 KiB = 0.234 xz: (stdin): Cannot allocate memory (stdin): 120.1 KiB / 512.0 KiB = 0.234 make: *** [cdrom:66: /usr/src/log/cdrom] Error 1... ***SNAP***
Memory limit is higher than required memory but it still says 'cannot allocate memory'!? And how to handle this (stdin)-message?
- Changed line: XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
to
XZ_MEM="$(( HOST_MEM * 9 / 10 ))MiB"
Output:
***SNIP*** ... rm -f /ipfire.tar cd /tmp && tar cf - * | xz -T8 -8 --memory=7007MiB -v > /install/cdrom/distro.img && rm -rf * xz: Filter chain: -- lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0 xz: Using up to 8 threads. xz: 5257 MiB of memory is required. The limit is 7007 MiB. xz: Decompression will need 33 MiB of memory. (stdin): 120.1 KiB / 512.0 KiB = 0.234 xz: (stdin): Cannot allocate memory (stdin): 120.1 KiB / 512.0 KiB = 0.234 make: *** [cdrom:66: /usr/src/log/cdrom] Error 1 ...***SNAP***
- Using '-T4' and XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
All is well, log says:
***SNIP*** ... tar -x -C /tmp -f /ipfire.tar rm -f /ipfire.tar cd /tmp && tar cf - * | xz -T4 -8 --memory=5450MiB -v > /install/cdrom/distro.img && rm -rf * xz: Filter chain: -- lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0 xz: Using up to 4 threads. xz: 2629 MiB of memory is required. The limit is 5450 MiB. xz: Decompression will need 33 MiB of memory. ... ***SNAP***
Some more infos (helpful?):
root@Devel: /home/matz/ipfire-2.x # xz --info-memory Total amount of physical memory (RAM): 7,787 MiB (8,165,179,392 B) Memory usage limit for compression: Disabled Memory usage limit for decompression: Disabled
root@Devel: /home/matz/ipfire-2.x # ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 62187 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 62187 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
In the meantime I've found an interesting thread about this problem: https://www.mail-archive.com/misc@openbsd.org/msg144710.html
"A closer reading of the man page reveals that memory consumption is even higher in multi-threaded mode. In multi-threaded mode about three times _size_ bytes will be allocated in each thread for buffering input and output. ... A quick check with top(1) confirms that xz in multi-threaded mode allocates 1250 MB per thread at compression level 9."
'-8' should be lower per thread, but too much for 8 threads.
Best, Matthias
-Michael
On Tue, 2018-05-22 at 18:31 +0200, Matthias Fischer wrote:
Hi!
Sorry, but last commit "make.sh: Use all processor cores for compression" causes the build to break.
See https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=c8453e87599fe66f 4d18d9 01bc 9dc431306afa2d
Build stops with:
***SNIP*** ... rm -f /tmp/ROOTFILES tar -x -C /tmp -f /ipfire.tar rm -f /ipfire.tar cd /tmp && tar cf - * | xz -T8 -8 --memory=5450MiB > /install/cdrom/distro.img && rm -rf * xz: (stdin): Cannot allocate memory make: *** [cdrom:66: /usr/src/log/cdrom] Error 1 ... ***SNAP***
On my machine, 'lsf/cdrom' doesn't like '-T8'. It needed '-T4'. I experienced this already during my tests.
Can anyone confirm?
Best, Matthias
P.S.: Hardware is i7/2600 with HT, 8 GB RAM.