Hi,
Current 'next' always gives me:
***SNIP*** ... tar -c -C / --files-from=/tmp/ROOTFILES \ -f /ipfire.tar --exclude='#*' --exclude='dev/pts/*' \ --exclude='proc/*' --exclude='tmp/ROOTFILES' rm -f /tmp/ROOTFILES tar -x -C /tmp -f /ipfire.tar rm -f /ipfire.tar cd /tmp && tar cf /install/cdrom/distro.img --xz * && rm -rf * xz: (stdin): Cannot allocate memory make: *** [cdrom:75: /usr/src/log/cdrom] Error 141 ... ***SNAP***
Building on Ubuntu 16.04.3 LTS / 32bit.
Can anyone confirm?
Best, Matthias
Hi,
how much memory does your build system have?
I changed the compression algorithm and it will now use up to ~370MB of memory (probably times the number of processor cores you have).
If that is too much, we need to change this to something lower again. The benefit right now is, that the ISO image is about 15 MB smaller.
-Michael
On Wed, 2017-10-18 at 04:39 +0200, Matthias Fischer wrote:
Hi,
Current 'next' always gives me:
***SNIP*** ... tar -c -C / --files-from=/tmp/ROOTFILES \ -f /ipfire.tar --exclude='#*' --exclude='dev/pts/*' \ --exclude='proc/*' --exclude='tmp/ROOTFILES' rm -f /tmp/ROOTFILES tar -x -C /tmp -f /ipfire.tar rm -f /ipfire.tar cd /tmp && tar cf /install/cdrom/distro.img --xz * && rm -rf * xz: (stdin): Cannot allocate memory make: *** [cdrom:75: /usr/src/log/cdrom] Error 141 ... ***SNAP***
Building on Ubuntu 16.04.3 LTS / 32bit.
Can anyone confirm?
Best, Matthias
Hi,
On 18.10.2017 04:39, Michael Tremer wrote:
Hi,
how much memory does your build system have?
8GB, with 32bit Ubuntu.
'top' says:
KiB Mem : 7973868 total, 7739960 free, 44100 used, 189808 buff/cache KiB Swap: 1046524 total, 1046524 free, 0 used. 7347712 avail Mem
I changed the compression algorithm and it will now use up to ~370MB of memory (probably times the number of processor cores you have).
'lscpu' says:
Architecture: i686 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 42 Model name: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz Stepping: 7 CPU MHz: 1600.921 CPU max MHz: 3800.0000 CPU min MHz: 1600.0000 BogoMIPS: 6784.18 Virtualisation: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 8192K
If that is too much, we need to change this to something lower again. The benefit right now is, that the ISO image is about 15 MB smaller.
Anything I could test?
Best, Matthias
-Michael
On Wed, 2017-10-18 at 04:39 +0200, Matthias Fischer wrote:
Hi,
Current 'next' always gives me:
***SNIP*** ... tar -c -C / --files-from=/tmp/ROOTFILES \ -f /ipfire.tar --exclude='#*' --exclude='dev/pts/*' \ --exclude='proc/*' --exclude='tmp/ROOTFILES' rm -f /tmp/ROOTFILES tar -x -C /tmp -f /ipfire.tar rm -f /ipfire.tar cd /tmp && tar cf /install/cdrom/distro.img --xz * && rm -rf * xz: (stdin): Cannot allocate memory make: *** [cdrom:75: /usr/src/log/cdrom] Error 141 ... ***SNAP***
Building on Ubuntu 16.04.3 LTS / 32bit.
Can anyone confirm?
Best, Matthias
On 18.10.2017 15:56, Matthias Fischer wrote:
On 18.10.2017 04:39, Michael Tremer wrote:
Hi,
how much memory does your build system have? ...
Another info:
***SNIP*** root@Devel: /home/matz/ipfire-2.x # xz --info-memory Total amount of physical memory (RAM): 7,787 MiB (8,165,240,832 B) Memory usage limit for compression: Disabled Memory usage limit for decompression: Disabled ***SNAP***
8GB. So why is it telling me 'out of memory...?
Best, Matthias
Hi,
I have no idea why that is. It is supposed to use only 370MB for the compression.
However, the build failed on the nightly build server, too which also has 8GB. If xz uses that per core (of which this one has 8) this should still be well under the limit.
I reverted the commits and now it will only compress in parallel. Please pull from next.
-Michael
On Wed, 2017-10-18 at 16:22 +0200, Matthias Fischer wrote:
On 18.10.2017 15:56, Matthias Fischer wrote:
On 18.10.2017 04:39, Michael Tremer wrote:
Hi,
how much memory does your build system have? ...
Another info:
***SNIP*** root@Devel: /home/matz/ipfire-2.x # xz --info-memory Total amount of physical memory (RAM): 7,787 MiB (8,165,240,832 B) Memory usage limit for compression: Disabled Memory usage limit for decompression: Disabled ***SNAP***
8GB. So why is it telling me 'out of memory...?
Best, Matthias
Hi,
On 18.10.2017 16:50, Michael Tremer wrote:
I reverted the commits and now it will only compress in parallel. Please pull from next.
***SNIP*** ***Build is finished now and took 0 hour(s) 28 minute(s) 17 second(s)! root@Devel: /home/matz/ipfire-2.x # ***SNAP***
It worked.
I'll try a complete clean build with '--memory=70%" and will report.
Best, Matthias
P.S.: ... The limit can be specified as a percentage of physical RAM. Example: --memory=70% ...
Hi,
On 18.10.2017 17:54, Matthias Fischer wrote:
I'll try a complete clean build with '--memory=70%" and will report.
This *didn't* work, tried again with '--memlimit=70%', didn't work either.
Only working solution:
--memory=700MiB (or: 750MiB, to be sure)
Or:
--memory=700000000 (doesn't look good...)
Buildtime for cdrom: ~339-355 seconds.
Best, Matthias
Did you compare image size after? Is that being influenced by the memory parameter?
On Wed, 2017-10-18 at 19:03 +0200, Matthias Fischer wrote:
Hi,
On 18.10.2017 17:54, Matthias Fischer wrote:
I'll try a complete clean build with '--memory=70%" and will report.
This *didn't* work, tried again with '--memlimit=70%', didn't work either.
Only working solution:
--memory=700MiB (or: 750MiB, to be sure)
Or:
--memory=700000000 (doesn't look good...)
Buildtime for cdrom: ~339-355 seconds.
Best, Matthias
Hi,
On 18.10.2017 19:42, Michael Tremer wrote:
Did you compare image size after? Is that being influenced by the memory parameter?
Slightly.
Current 'next':
Buildtime 03:59:03
***SNIP*** -rw-r--r-- 1 root root 255029305 Oct 19 05:15 ipfire-2.19.1gb-ext4.i586-full-core115.img.gz -rw-r--r-- 1 root root 80 Oct 19 05:26 ipfire-2.19.1gb-ext4.i586-full-core115.img.gz.md5 -rw-r--r-- 1 root root 252170425 Oct 19 05:19 ipfire-2.19.1gb-ext4-scon.i586-full-core115.img.gz -rw-r--r-- 1 root root 85 Oct 19 05:26 ipfire-2.19.1gb-ext4-scon.i586-full-core115.img.gz.md5 -rw-r--r-- 1 root root 184549376 Oct 19 04:50 ipfire-2.19.i586-full-core115.iso -rw-r--r-- 1 root root 68 Oct 19 05:26 ipfire-2.19.i586-full-core115.iso.md5 -rw-r--r-- 1 root root 3522 Oct 19 05:26 ipfire-2.19.xen.i586-downloader-core115.tar.bz2 -rw-r--r-- 1 root root 82 Oct 19 05:26 ipfire-2.19.xen.i586-downloader-core115.tar.bz2.md5 ***SNAP***
Optimized ('... && XZ_OPT="-T0 -8' / '... export XZ_OPT = --threads=0 -8 --memory=750MiB'):
Buildtime 4:25:29
***SNIP*** -rw-r--r-- 1 root root 255221827 Oct 19 15:39 ipfire-2.19.1gb-ext4.i586-full-core115.img.gz -rw-r--r-- 1 root root 80 Oct 19 15:52 ipfire-2.19.1gb-ext4.i586-full-core115.img.gz.md5 -rw-r--r-- 1 root root 252146461 Oct 19 15:42 ipfire-2.19.1gb-ext4-scon.i586-full-core115.img.gz -rw-r--r-- 1 root root 85 Oct 19 15:52 ipfire-2.19.1gb-ext4-scon.i586-full-core115.img.gz.md5 -rw-r--r-- 1 root root 175112192 Oct 19 15:11 ipfire-2.19.i586-full-core115.iso -rw-r--r-- 1 root root 68 Oct 19 15:52 ipfire-2.19.i586-full-core115.iso.md5 -rw-r--r-- 1 root root 3537 Oct 19 15:52 ipfire-2.19.xen.i586-downloader-core115.tar.bz2 -rw-r--r-- 1 root root 82 Oct 19 15:52 ipfire-2.19.xen.i586-downloader-core115.tar.bz2.md5 ***SNAP***
Take your pick... ;-)
Best, Matthias
Hi,
On 18.10.2017 19:42, Michael Tremer wrote:
Did you compare image size after? Is that being influenced by the memory parameter?
For the records:
Current 'next', as in previous posting:
Buildtime 03:59:03
***SNIP*** -rw-r--r-- 1 root root 255029305 Oct 19 05:15 ipfire-2.19.1gb-ext4.i586-full-core115.img.gz -rw-r--r-- 1 root root 80 Oct 19 05:26 ipfire-2.19.1gb-ext4.i586-full-core115.img.gz.md5 -rw-r--r-- 1 root root 252170425 Oct 19 05:19 ipfire-2.19.1gb-ext4-scon.i586-full-core115.img.gz -rw-r--r-- 1 root root 85 Oct 19 05:26 ipfire-2.19.1gb-ext4-scon.i586-full-core115.img.gz.md5 -rw-r--r-- 1 root root 184549376 Oct 19 04:50 ipfire-2.19.i586-full-core115.iso -rw-r--r-- 1 root root 68 Oct 19 05:26 ipfire-2.19.i586-full-core115.iso.md5 -rw-r--r-- 1 root root 3522 Oct 19 05:26 ipfire-2.19.xen.i586-downloader-core115.tar.bz2 -rw-r--r-- 1 root root 82 Oct 19 05:26 ipfire-2.19.xen.i586-downloader-core115.tar.bz2.md5 ***SNAP***
'xz' updated to '5.2.3', "optimized":
Build time 3:30:05 ***SNIP***-rw-r--r-- 1 root root 255172283 Oct 19 20:41 ipfire-2.19.1gb-ext4.i586-full-core115.img.gz -rw-r--r-- 1 root root 80 Oct 19 20:54 ipfire-2.19.1gb-ext4.i586-full-core115.img.gz.md5 -rw-r--r-- 1 root root 252156393 Oct 19 20:45 ipfire-2.19.1gb-ext4-scon.i586-full-core115.img.gz -rw-r--r-- 1 root root 85 Oct 19 20:54 ipfire-2.19.1gb-ext4-scon.i586-full-core115.img.gz.md5 -rw-r--r-- 1 root root 175112192 Oct 19 20:14 ipfire-2.19.i586-full-core115.iso -rw-r--r-- 1 root root 68 Oct 19 20:54 ipfire-2.19.i586-full-core115.iso.md5 -rw-r--r-- 1 root root 3531 Oct 19 20:54 ipfire-2.19.xen.i586-downloader-core115.tar.bz2 -rw-r--r-- 1 root root 82 Oct 19 20:54 ipfire-2.19.xen.i586-downloader-core115.tar.bz2.md5 ***SNAP***
Looks interesting?
Best, Matthias
Hi,
did you test if this is a hard limit for all threads together?
If so that would be quite nice because we can use -9 then and it would just go up to the ceiling of memory usage and then just cap the size of the dictionary there (that's what I get from the man page).
I think we can even go up to 1GB since that is a minimum each build system should have. I would really like to see that extra 10-15MB shaven off the ISO image.
Would you like to send a patch with all of this? I reverted all previous commits again because the nightly builds failed.
Best, -Michael
On Thu, 2017-10-19 at 21:10 +0200, Matthias Fischer wrote:
Hi,
On 18.10.2017 19:42, Michael Tremer wrote:
Did you compare image size after? Is that being influenced by the memory parameter?
For the records:
Current 'next', as in previous posting:
Buildtime 03:59:03
***SNIP*** -rw-r--r-- 1 root root 255029305 Oct 19 05:15 ipfire-2.19.1gb-ext4.i586- full-core115.img.gz -rw-r--r-- 1 root root 80 Oct 19 05:26 ipfire-2.19.1gb-ext4.i586- full-core115.img.gz.md5 -rw-r--r-- 1 root root 252170425 Oct 19 05:19 ipfire-2.19.1gb-ext4- scon.i586-full-core115.img.gz -rw-r--r-- 1 root root 85 Oct 19 05:26 ipfire-2.19.1gb-ext4- scon.i586-full-core115.img.gz.md5 -rw-r--r-- 1 root root 184549376 Oct 19 04:50 ipfire-2.19.i586-full- core115.iso -rw-r--r-- 1 root root 68 Oct 19 05:26 ipfire-2.19.i586-full- core115.iso.md5 -rw-r--r-- 1 root root 3522 Oct 19 05:26 ipfire-2.19.xen.i586- downloader-core115.tar.bz2 -rw-r--r-- 1 root root 82 Oct 19 05:26 ipfire-2.19.xen.i586- downloader-core115.tar.bz2.md5 ***SNAP***
'xz' updated to '5.2.3', "optimized":
Build time 3:30:05 ***SNIP***-rw-r--r-- 1 root root 255172283 Oct 19 20:41 ipfire-2.19.1gb- ext4.i586-full-core115.img.gz -rw-r--r-- 1 root root 80 Oct 19 20:54 ipfire-2.19.1gb-ext4.i586- full-core115.img.gz.md5 -rw-r--r-- 1 root root 252156393 Oct 19 20:45 ipfire-2.19.1gb-ext4- scon.i586-full-core115.img.gz -rw-r--r-- 1 root root 85 Oct 19 20:54 ipfire-2.19.1gb-ext4- scon.i586-full-core115.img.gz.md5 -rw-r--r-- 1 root root 175112192 Oct 19 20:14 ipfire-2.19.i586-full- core115.iso -rw-r--r-- 1 root root 68 Oct 19 20:54 ipfire-2.19.i586-full- core115.iso.md5 -rw-r--r-- 1 root root 3531 Oct 19 20:54 ipfire-2.19.xen.i586- downloader-core115.tar.bz2 -rw-r--r-- 1 root root 82 Oct 19 20:54 ipfire-2.19.xen.i586- downloader-core115.tar.bz2.md5 ***SNAP***
Looks interesting?
Best, Matthias
-
Matthias Fischer -
Michael Tremer