Development May 2018

development@lists.ipfire.org

10 participants
46 discussions

[PATCH] tar: Update to 1.30
by Matthias Fischer 28 May '18

28 May '18

For details see: https://www.gnu.org/software/tar/ Best, Matthias Signed-off-by: Matthias Fischer <matthias.fischer(a)ipfire.org> --- lfs/tar | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lfs/tar b/lfs/tar index 6a949a003..4e914b535 100644 --- a/lfs/tar +++ b/lfs/tar @@ -1,7 +1,7 @@ ############################################################################### # # # IPFire.org - A linux based firewall # -# Copyright (C) 2007 Michael Tremer & Christian Schmidt # +# Copyright (C) 2018 Michael Tremer & Christian Schmidt # # # # This program is free software: you can redistribute it and/or modify # # it under the terms of the GNU General Public License as published by # @@ -24,7 +24,7 @@ include Config -VER = 1.28 +VER = 1.30 THISAPP = tar-$(VER) DL_FILE = $(THISAPP).tar.bz2 @@ -50,7 +50,7 @@ objects = $(DL_FILE) $(DL_FILE) = $(DL_FROM)/$(DL_FILE) -$(DL_FILE)_MD5 = 8f32b2bc1ed7ddf4cf4e4a39711341b0 +$(DL_FILE)_MD5 = 8404e4c1fc5a3000228ab2b8ad674a65 install : $(TARGET) -- 2.17.0

2 3

[PATCH] Add intel microcode
by Jonatan Schlag 28 May '18

28 May '18

This add just the binary blobs of the intel micro code. Signed-off-by: Jonatan Schlag <jonatan.schlag(a)ipfire.org> --- config/rootfiles/common/i586/intel-microcode | 97 ++++++++++++++++++++++++++ config/rootfiles/common/x86_64/intel-microcode | 97 ++++++++++++++++++++++++++ lfs/intel-microcode | 80 +++++++++++++++++++++ make.sh | 1 + 4 files changed, 275 insertions(+) create mode 100644 config/rootfiles/common/i586/intel-microcode create mode 100644 config/rootfiles/common/x86_64/intel-microcode create mode 100644 lfs/intel-microcode diff --git a/config/rootfiles/common/i586/intel-microcode b/config/rootfiles/common/i586/intel-microcode new file mode 100644 index 0000000..e6e62b7 --- /dev/null +++ b/config/rootfiles/common/i586/intel-microcode @@ -0,0 +1,97 @@ +#lib/firmware/intel-ucode +lib/firmware/intel-ucode/06-03-02 +lib/firmware/intel-ucode/06-05-00 +lib/firmware/intel-ucode/06-05-01 +lib/firmware/intel-ucode/06-05-02 +lib/firmware/intel-ucode/06-05-03 +lib/firmware/intel-ucode/06-06-00 +lib/firmware/intel-ucode/06-06-05 +lib/firmware/intel-ucode/06-06-0a +lib/firmware/intel-ucode/06-06-0d +lib/firmware/intel-ucode/06-07-01 +lib/firmware/intel-ucode/06-07-02 +lib/firmware/intel-ucode/06-07-03 +lib/firmware/intel-ucode/06-08-01 +lib/firmware/intel-ucode/06-08-03 +lib/firmware/intel-ucode/06-08-06 +lib/firmware/intel-ucode/06-08-0a +lib/firmware/intel-ucode/06-09-05 +lib/firmware/intel-ucode/06-0a-00 +lib/firmware/intel-ucode/06-0a-01 +lib/firmware/intel-ucode/06-0b-01 +lib/firmware/intel-ucode/06-0b-04 +lib/firmware/intel-ucode/06-0d-06 +lib/firmware/intel-ucode/06-0e-08 +lib/firmware/intel-ucode/06-0e-0c +lib/firmware/intel-ucode/06-0f-02 +lib/firmware/intel-ucode/06-0f-06 +lib/firmware/intel-ucode/06-0f-07 +lib/firmware/intel-ucode/06-0f-0a +lib/firmware/intel-ucode/06-0f-0b +lib/firmware/intel-ucode/06-0f-0d +lib/firmware/intel-ucode/06-16-01 +lib/firmware/intel-ucode/06-17-06 +lib/firmware/intel-ucode/06-17-07 +lib/firmware/intel-ucode/06-17-0a +lib/firmware/intel-ucode/06-1a-04 +lib/firmware/intel-ucode/06-1a-05 +lib/firmware/intel-ucode/06-1c-02 +lib/firmware/intel-ucode/06-1c-0a +lib/firmware/intel-ucode/06-1d-01 +lib/firmware/intel-ucode/06-1e-05 +lib/firmware/intel-ucode/06-25-02 +lib/firmware/intel-ucode/06-25-05 +lib/firmware/intel-ucode/06-26-01 +lib/firmware/intel-ucode/06-2a-07 +lib/firmware/intel-ucode/06-2d-06 +lib/firmware/intel-ucode/06-2d-07 +lib/firmware/intel-ucode/06-2f-02 +lib/firmware/intel-ucode/06-3a-09 +lib/firmware/intel-ucode/06-3c-03 +lib/firmware/intel-ucode/06-3d-04 +lib/firmware/intel-ucode/06-3e-04 +lib/firmware/intel-ucode/06-3e-06 +lib/firmware/intel-ucode/06-3e-07 +lib/firmware/intel-ucode/06-3f-02 +lib/firmware/intel-ucode/06-3f-04 +lib/firmware/intel-ucode/06-45-01 +lib/firmware/intel-ucode/06-46-01 +lib/firmware/intel-ucode/06-47-01 +lib/firmware/intel-ucode/06-4e-03 +lib/firmware/intel-ucode/06-55-03 +lib/firmware/intel-ucode/06-55-04 +lib/firmware/intel-ucode/06-56-02 +lib/firmware/intel-ucode/06-56-03 +lib/firmware/intel-ucode/06-56-04 +lib/firmware/intel-ucode/06-56-05 +lib/firmware/intel-ucode/06-5c-09 +lib/firmware/intel-ucode/06-5e-03 +lib/firmware/intel-ucode/06-7a-01 +lib/firmware/intel-ucode/06-8e-09 +lib/firmware/intel-ucode/06-8e-0a +lib/firmware/intel-ucode/06-9e-09 +lib/firmware/intel-ucode/06-9e-0a +lib/firmware/intel-ucode/06-9e-0b +lib/firmware/intel-ucode/0f-00-07 +lib/firmware/intel-ucode/0f-00-0a +lib/firmware/intel-ucode/0f-01-02 +lib/firmware/intel-ucode/0f-02-04 +lib/firmware/intel-ucode/0f-02-05 +lib/firmware/intel-ucode/0f-02-06 +lib/firmware/intel-ucode/0f-02-07 +lib/firmware/intel-ucode/0f-02-09 +lib/firmware/intel-ucode/0f-03-02 +lib/firmware/intel-ucode/0f-03-03 +lib/firmware/intel-ucode/0f-03-04 +lib/firmware/intel-ucode/0f-04-01 +lib/firmware/intel-ucode/0f-04-03 +lib/firmware/intel-ucode/0f-04-04 +lib/firmware/intel-ucode/0f-04-07 +lib/firmware/intel-ucode/0f-04-08 +lib/firmware/intel-ucode/0f-04-09 +lib/firmware/intel-ucode/0f-04-0a +lib/firmware/intel-ucode/0f-06-02 +lib/firmware/intel-ucode/0f-06-04 +lib/firmware/intel-ucode/0f-06-05 +lib/firmware/intel-ucode/0f-06-08 +lib/firmware/intel-ucode/list diff --git a/config/rootfiles/common/x86_64/intel-microcode b/config/rootfiles/common/x86_64/intel-microcode new file mode 100644 index 0000000..e6e62b7 --- /dev/null +++ b/config/rootfiles/common/x86_64/intel-microcode @@ -0,0 +1,97 @@ +#lib/firmware/intel-ucode +lib/firmware/intel-ucode/06-03-02 +lib/firmware/intel-ucode/06-05-00 +lib/firmware/intel-ucode/06-05-01 +lib/firmware/intel-ucode/06-05-02 +lib/firmware/intel-ucode/06-05-03 +lib/firmware/intel-ucode/06-06-00 +lib/firmware/intel-ucode/06-06-05 +lib/firmware/intel-ucode/06-06-0a +lib/firmware/intel-ucode/06-06-0d +lib/firmware/intel-ucode/06-07-01 +lib/firmware/intel-ucode/06-07-02 +lib/firmware/intel-ucode/06-07-03 +lib/firmware/intel-ucode/06-08-01 +lib/firmware/intel-ucode/06-08-03 +lib/firmware/intel-ucode/06-08-06 +lib/firmware/intel-ucode/06-08-0a +lib/firmware/intel-ucode/06-09-05 +lib/firmware/intel-ucode/06-0a-00 +lib/firmware/intel-ucode/06-0a-01 +lib/firmware/intel-ucode/06-0b-01 +lib/firmware/intel-ucode/06-0b-04 +lib/firmware/intel-ucode/06-0d-06 +lib/firmware/intel-ucode/06-0e-08 +lib/firmware/intel-ucode/06-0e-0c +lib/firmware/intel-ucode/06-0f-02 +lib/firmware/intel-ucode/06-0f-06 +lib/firmware/intel-ucode/06-0f-07 +lib/firmware/intel-ucode/06-0f-0a +lib/firmware/intel-ucode/06-0f-0b +lib/firmware/intel-ucode/06-0f-0d +lib/firmware/intel-ucode/06-16-01 +lib/firmware/intel-ucode/06-17-06 +lib/firmware/intel-ucode/06-17-07 +lib/firmware/intel-ucode/06-17-0a +lib/firmware/intel-ucode/06-1a-04 +lib/firmware/intel-ucode/06-1a-05 +lib/firmware/intel-ucode/06-1c-02 +lib/firmware/intel-ucode/06-1c-0a +lib/firmware/intel-ucode/06-1d-01 +lib/firmware/intel-ucode/06-1e-05 +lib/firmware/intel-ucode/06-25-02 +lib/firmware/intel-ucode/06-25-05 +lib/firmware/intel-ucode/06-26-01 +lib/firmware/intel-ucode/06-2a-07 +lib/firmware/intel-ucode/06-2d-06 +lib/firmware/intel-ucode/06-2d-07 +lib/firmware/intel-ucode/06-2f-02 +lib/firmware/intel-ucode/06-3a-09 +lib/firmware/intel-ucode/06-3c-03 +lib/firmware/intel-ucode/06-3d-04 +lib/firmware/intel-ucode/06-3e-04 +lib/firmware/intel-ucode/06-3e-06 +lib/firmware/intel-ucode/06-3e-07 +lib/firmware/intel-ucode/06-3f-02 +lib/firmware/intel-ucode/06-3f-04 +lib/firmware/intel-ucode/06-45-01 +lib/firmware/intel-ucode/06-46-01 +lib/firmware/intel-ucode/06-47-01 +lib/firmware/intel-ucode/06-4e-03 +lib/firmware/intel-ucode/06-55-03 +lib/firmware/intel-ucode/06-55-04 +lib/firmware/intel-ucode/06-56-02 +lib/firmware/intel-ucode/06-56-03 +lib/firmware/intel-ucode/06-56-04 +lib/firmware/intel-ucode/06-56-05 +lib/firmware/intel-ucode/06-5c-09 +lib/firmware/intel-ucode/06-5e-03 +lib/firmware/intel-ucode/06-7a-01 +lib/firmware/intel-ucode/06-8e-09 +lib/firmware/intel-ucode/06-8e-0a +lib/firmware/intel-ucode/06-9e-09 +lib/firmware/intel-ucode/06-9e-0a +lib/firmware/intel-ucode/06-9e-0b +lib/firmware/intel-ucode/0f-00-07 +lib/firmware/intel-ucode/0f-00-0a +lib/firmware/intel-ucode/0f-01-02 +lib/firmware/intel-ucode/0f-02-04 +lib/firmware/intel-ucode/0f-02-05 +lib/firmware/intel-ucode/0f-02-06 +lib/firmware/intel-ucode/0f-02-07 +lib/firmware/intel-ucode/0f-02-09 +lib/firmware/intel-ucode/0f-03-02 +lib/firmware/intel-ucode/0f-03-03 +lib/firmware/intel-ucode/0f-03-04 +lib/firmware/intel-ucode/0f-04-01 +lib/firmware/intel-ucode/0f-04-03 +lib/firmware/intel-ucode/0f-04-04 +lib/firmware/intel-ucode/0f-04-07 +lib/firmware/intel-ucode/0f-04-08 +lib/firmware/intel-ucode/0f-04-09 +lib/firmware/intel-ucode/0f-04-0a +lib/firmware/intel-ucode/0f-06-02 +lib/firmware/intel-ucode/0f-06-04 +lib/firmware/intel-ucode/0f-06-05 +lib/firmware/intel-ucode/0f-06-08 +lib/firmware/intel-ucode/list diff --git a/lfs/intel-microcode b/lfs/intel-microcode new file mode 100644 index 0000000..351bb76 --- /dev/null +++ b/lfs/intel-microcode @@ -0,0 +1,80 @@ +############################################################################### +# # +# IPFire.org - A linux based firewall # +# Copyright (C) 2007-2016 IPFire Team <info(a)ipfire.org> # +# # +# This program is free software: you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation, either version 3 of the License, or # +# (at your option) any later version. # +# # +# This program is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with this program. If not, see <http://www.gnu.org/licenses/>. # +# # +############################################################################### + +############################################################################### +# Definitions +############################################################################### + +include Config + +VER = 20180425 + +THISAPP = microcode-$(VER) +DL_FILE = $(THISAPP).tgz +DL_FROM = $(URL_IPFIRE) +DIR_APP = $(DIR_SRC)/$(THISAPP) +TARGET = $(DIR_INFO)/$(THISAPP) +SUP_ARCH = i586 x86_64 + +############################################################################### +# Top-level Rules +############################################################################### + +objects = $(DL_FILE) + +$(DL_FILE) = $(DL_FROM)/$(DL_FILE) + +$(DL_FILE)_MD5 = 99c80f9229554953a868127cda44e7e3 + +install : $(TARGET) +check : $(patsubst %,$(DIR_CHK)/%,$(objects)) + +download :$(patsubst %,$(DIR_DL)/%,$(objects)) + +md5 : $(subst %,%_MD5,$(objects)) + +dist:. + $(PAK) + +############################################################################### +# Downloading, checking, md5sum +############################################################################### + +$(patsubst %,$(DIR_CHK)/%,$(objects)) : + @$(CHECK) + +$(patsubst %,$(DIR_DL)/%,$(objects)) : + @$(LOAD) + +$(subst %,%_MD5,$(objects)) : + @$(MD5) + +############################################################################### +# Installation Details +############################################################################### + +$(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) + @$(PREBUILD) + @rm -rf $(DIR_APP) && cd $(DIR_SRC) && mkdir -p $(DIR_APP) && tar zxf $(DIR_DL)/$(DL_FILE) -C $(DIR_APP) + ls /usr/src + # Copy the firmware files into the right position + cd $(DIR_APP) && cp -R intel-ucode /lib/firmware/ + @rm -rf $(DIR_APP) + @$(POSTBUILD) diff --git a/make.sh b/make.sh index 1050eb0..95860b3 100755 --- a/make.sh +++ b/make.sh @@ -1142,6 +1142,7 @@ buildipfire() { lfsmake2 linux-initrd KCFG="-multi" ;; esac + lfsmake2 intel-microcode lfsmake2 xtables-addons USPACE="1" lfsmake2 openssl [ "${BUILD_ARCH}" = "i586" ] && lfsmake2 openssl KCFG='-sse2' -- 1.8.3.1

1 0

[PATCH] xz: Update to 5.2.4
by Matthias Fischer 26 May '18

26 May '18

For details see: https://git.tukaani.org/?p=xz.git;a=blob;f=NEWS;hb=HEAD Best, Matthias Signed-off-by: Matthias Fischer <matthias.fischer(a)ipfire.org> --- lfs/xz | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lfs/xz b/lfs/xz index f8940ab4e..bfee28038 100644 --- a/lfs/xz +++ b/lfs/xz @@ -1,7 +1,7 @@ ############################################################################### # # # IPFire.org - A linux based firewall # -# Copyright (C) 2007-2017 IPFire Team <info(a)ipfire.org> # +# Copyright (C) 2007-2018 IPFire Team <info(a)ipfire.org> # # # # This program is free software: you can redistribute it and/or modify # # it under the terms of the GNU General Public License as published by # @@ -24,7 +24,7 @@ include Config -VER = 5.2.3 +VER = 5.2.4 THISAPP = xz-$(VER) DL_FILE = $(THISAPP).tar.xz @@ -45,7 +45,7 @@ objects = $(DL_FILE) $(DL_FILE) = $(DL_FROM)/$(DL_FILE) -$(DL_FILE)_MD5 = 60fb79cab777e3f71ca43d298adacbd5 +$(DL_FILE)_MD5 = 003e4d0b1b1899fc6e3000b24feddf7c install : $(TARGET) -- 2.17.0

1 0

[PATCH] BUG11505: Captive Portal: no way to remove an uploaded logo
by Alexander Marx 24 May '18

24 May '18

added a delete button Signed-off-by: Alexander Marx <alexander.marx(a)ipfire.org> --- html/cgi-bin/captive.cgi | 9 ++++++++- langs/de/cgi-bin/de.pl | 1 + langs/en/cgi-bin/en.pl | 1 + 3 files changed, 10 insertions(+), 1 deletion(-) diff --git a/html/cgi-bin/captive.cgi b/html/cgi-bin/captive.cgi index 65cc349..02b2eb5 100755 --- a/html/cgi-bin/captive.cgi +++ b/html/cgi-bin/captive.cgi @@ -142,6 +142,10 @@ if ($cgiparams{'ACTION'} eq $Lang::tr{'save'}) { } } +if ($cgiparams{'ACTION'} eq "$Lang::tr{'Captive delete logo'}") { + unlink $logo; +} + if ($cgiparams{'ACTION'} eq "$Lang::tr{'Captive generate coupons'}") { #check valid remark if ($cgiparams{'REMARK'} ne '' && !&validremark($cgiparams{'REMARK'})){ @@ -381,7 +385,10 @@ if (-e $logo) { print <<END; <tr> <td>$Lang::tr{'Captive logo uploaded'}</td> - <td>$Lang::tr{'yes'}</td> + <td> + $Lang::tr{'yes'}  + <input type='submit' name='ACTION' value="$Lang::tr{'Captive delete logo'}"/> + </td> </tr> END } diff --git a/langs/de/cgi-bin/de.pl b/langs/de/cgi-bin/de.pl index edff684..f00a667 100644 --- a/langs/de/cgi-bin/de.pl +++ b/langs/de/cgi-bin/de.pl @@ -28,6 +28,7 @@ 'Captive clients' => 'Clients', 'Captive config' => 'Konfiguration', 'Captive coupon' => 'Gutschein', +'Captive delete logo' => 'Logo löschen', 'Captive err doublevoucher' => 'Ein Gutschein mit diesem Code ist bereits im Umlauf', 'Captive expire' => 'Ablauf', 'Captive expiry time' => 'Ablaufzeit', diff --git a/langs/en/cgi-bin/en.pl b/langs/en/cgi-bin/en.pl index 0c77ce4..ef141a3 100644 --- a/langs/en/cgi-bin/en.pl +++ b/langs/en/cgi-bin/en.pl @@ -27,6 +27,7 @@ 'Captive client session expiry time' => 'Session Expiry Time', 'Captive config' => 'Settings', 'Captive coupon' => 'Coupon', +'Captive delete logo' => 'Delete Logo', 'Captive err doublevoucher' => 'A coupon with this code already exists', 'Captive expire' => 'Expire', 'Captive expiry time' => 'Expiry Time', -- 2.7.4

1 0

ARM 64?
by Guy Ellis 23 May '18

23 May '18

Dear list, Just wondering if there is any interest in supporting ARM 64 hardware moving forward? We can assist with donated hardware and support if there is interest. https://traverse.com.au/products/ls1043s-router-board/ Regards, - Guy. -- Guy Ellis +61 419 398 234 www.traverse.com.au

2 1

[PATCH] xz: Performance and compression tuning (V2) - fine tuning
by Matthias Fischer 21 May '18

21 May '18

Changes since V1: - Tuned some more 'tar'-command lines - Included 'lfs/core-updates' - Some fine-tuning Hi, Current results with V2 (clean builds): 'next': 'packages' => 255 MB 'ipfire-2.19.2gb-ext4.i586-full-core121.img.gz => 319 MB 'ipfire-2.19.i586-full-core121.iso' => 218 MB Total => 792 MB 'xz-tuning': 'packages' => 228 MB 'ipfire-2.19.2gb-ext4.i586-full-core121.img.gz' => 318 MB 'ipfire-2.19.i586-full-core121.iso' => 207 MB Total => 753 MB (-39 MB) It would be nice if someone could review and test these. ;-) If necessary, I can upload a complete patch. Best, Matthias Signed-off-by: Matthias Fischer <matthias.fischer(a)ipfire.org> --- lfs/Config | 6 ++++-- lfs/cdrom | 4 ++-- lfs/core-updates | 9 +++++---- make.sh | 2 +- 4 files changed, 12 insertions(+), 9 deletions(-) diff --git a/lfs/Config b/lfs/Config index c667a3cc7..a726f171d 100644 --- a/lfs/Config +++ b/lfs/Config @@ -47,6 +47,8 @@ else PREFIX = /usr endif +TAR_OPTIONS = --xz + # URLs that are common sources of downloads. If you're having trouble with # a site you should change its URL to that of a suitable mirror site. # @@ -209,11 +211,11 @@ define PAK tar xf /install/packages/package/files.tmp -C /install/packages/package/tmp/ \ -p --numeric-owner rm -f /install/packages/package/files.tmp - cd /install/packages/package/tmp/ && XZ_OPT="$(XZ_OPT)" tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz * + cd /install/packages/package/tmp/ && XZ_OPT="$(XZ_OPT)" tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz $(TAR_OPTIONS) * rm -r /install/packages/package/tmp -cat /install/packages/package/ROOTFILES | grep -v "#" > /install/packages/package/ROOTFILES.tmp mv /install/packages/package/ROOTFILES.tmp /install/packages/package/ROOTFILES - cd /install/packages/package && tar cf ../$(PROG)-$(VER)-$(PAK_VER).ipfire --files-from=/usr/src/src/paks/files + cd /install/packages/package && XZ_OPT="$(XZ_OPT)" tar cf ../$(PROG)-$(VER)-$(PAK_VER).ipfire $(TAR_OPTIONS) --files-from=/usr/src/src/paks/files rm -rf /install/packages/package sed -e s/NAME/$(PROG)/g \ -e s/VER/$(VER)/g \ diff --git a/lfs/cdrom b/lfs/cdrom index 820c55b87..cdb7b07da 100644 --- a/lfs/cdrom +++ b/lfs/cdrom @@ -71,14 +71,14 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) # Compress root filesystem # Reason for this tar+untar+tar is removing of entries listed two or more in src/ROOTFILES - XZ_OPT="$(XZ_OPT)" tar -c -C / --files-from=/tmp/ROOTFILES \ + tar -c -C / --files-from=/tmp/ROOTFILES \ -f /$(SNAME).tar --exclude='#*' --exclude='dev/pts/*' \ --exclude='proc/*' --exclude='tmp/ROOTFILES' rm -f /tmp/ROOTFILES tar -x -C /tmp -f /$(SNAME).tar rm -f /$(SNAME).tar @mkdir /tmp/sys - cd /tmp && tar cf /install/cdrom/distro.img $(TAR_OPTIONS) * && rm -rf * + cd /tmp && XZ_OPT="$(XZ_OPT)" tar cf /install/cdrom/distro.img $(TAR_OPTIONS) * && rm -rf * # Other files touch /install/cdrom/$(SNAME)-$(VERSION)-core$(CORE).media diff --git a/lfs/core-updates b/lfs/core-updates index 3c847daf2..d34926967 100644 --- a/lfs/core-updates +++ b/lfs/core-updates @@ -26,8 +26,9 @@ include Config VER = ipfire -THISAPP = core-updates -TARGET = $(DIR_INFO)/$(THISAPP) +THISAPP = core-updates +TARGET = $(DIR_INFO)/$(THISAPP) +TAR_OPTIONS = --xz ############################################################################### # Top-level Rules @@ -78,7 +79,7 @@ core/%: mv -f /tmp/ROOTFILES /install/packages/package/ROOTFILES tar -p --numeric-owner -x -C /tmp -f /$(SNAME).tar rm -f /$(SNAME).tar - cd /tmp && tar cJf /install/packages/package/files.tar.xz * && rm -rf * + cd /tmp && XZ_OPT="$(XZ_OPT)" tar cJf /install/packages/package/files.tar.xz $(TAR_OPTIONS) * && rm -rf * cat /install/packages/package/ROOTFILES | grep -v "#" > /install/packages/package/ROOTFILES.tmp rm /install/packages/package/ROOTFILES mv /install/packages/package/ROOTFILES.tmp /install/packages/package/ROOTFILES @@ -89,7 +90,7 @@ core/%: #Change xxxKVERxxx to Kernelversion sed -e "s/xxxKVERxxx/$(KVER)/g" -i /install/packages/package/update.sh - cd /install/packages/package && tar cf ../core-upgrade-$(VERSION)-$$(basename $(a)).ipfire \ + cd /install/packages/package && XZ_OPT="$(XZ_OPT)" tar cf ../core-upgrade-$(VERSION)-$$(basename $(a)).ipfire $(TAR_OPTIONS) \ update.sh files.tar.xz ROOTFILES rm -rf /install/packages/package sed -e "s/NAME/core-upgrade/g" \ diff --git a/make.sh b/make.sh index 33ebc16bb..9893fbb20 100755 --- a/make.sh +++ b/make.sh @@ -1776,7 +1776,7 @@ toolchain) buildtoolchain echo "`date -u '+%b %e %T'`: Create toolchain image for ${BUILD_ARCH}" | tee -a $LOGFILE test -d $BASEDIR/cache/toolchains || mkdir -p $BASEDIR/cache/toolchains - cd $BASEDIR && XZ_OPT="$(XZ_OPT)" tar -Jc --exclude='log/_build.*.log' -f cache/toolchains/$SNAME-$VERSION-toolchain-$TOOLCHAINVER-${BUILD_ARCH}.tar.xz \ + cd $BASEDIR && XZ_OPT="$(XZ_OPT)" tar -Jc --exclude='log/_build.*.log' -f cache/toolchains/$SNAME-$VERSION-toolchain-$TOOLCHAINVER-${BUILD_ARCH}.tar.xz --xz \ build/${TOOLS_DIR} build/bin/sh log >> $LOGFILE md5sum cache/toolchains/$SNAME-$VERSION-toolchain-$TOOLCHAINVER-${BUILD_ARCH}.tar.xz \ > cache/toolchains/$SNAME-$VERSION-toolchain-$TOOLCHAINVER-${BUILD_ARCH}.md5 -- 2.17.0

1 0

[PATCH] cmake: Update to 3.11.2
by Matthias Fischer 20 May '18

20 May '18

For details see: https://cmake.org/cmake/help/v3.11/release/3.11.html#id2 Best, Matthias Signed-off-by: Matthias Fischer <matthias.fischer(a)ipfire.org> --- lfs/cmake | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lfs/cmake b/lfs/cmake index e5977b971..dd0f626a9 100644 --- a/lfs/cmake +++ b/lfs/cmake @@ -24,7 +24,7 @@ include Config -VER = 3.11.1 +VER = 3.11.2 THISAPP = cmake-$(VER) DL_FILE = $(THISAPP).tar.gz @@ -42,7 +42,7 @@ objects = $(DL_FILE) $(DL_FILE) = $(DL_FROM)/$(DL_FILE) -$(DL_FILE)_MD5 = 12a3177477e4e2c7bc514193d421dafe +$(DL_FILE)_MD5 = d2d554c05fc07cfae7846d2aa205f12a install : $(TARGET) -- 2.17.0

1 0

[PATCH] nmap: Update to 7.70
by Matthias Fischer 20 May '18

20 May '18

For details see: https://nmap.org/changelog.html Best, Matthias Signed-off-by: Matthias Fischer <matthias.fischer(a)ipfire.org> --- config/rootfiles/packages/nmap | 50 ++++++++++++++++++++++++++++++++-- lfs/nmap | 14 ++++++---- 2 files changed, 56 insertions(+), 8 deletions(-) diff --git a/config/rootfiles/packages/nmap b/config/rootfiles/packages/nmap index 111610a62..211a8e758 100644 --- a/config/rootfiles/packages/nmap +++ b/config/rootfiles/packages/nmap @@ -14,8 +14,6 @@ usr/lib/python2.7/site-packages/ndiff.pyc #usr/share/man/hu/man1 #usr/share/man/hu/man1/nmap.1 #usr/share/man/it/man1/nmap.1 -#usr/share/man/ja -#usr/share/man/ja/man1 #usr/share/man/ja/man1/nmap.1 #usr/share/man/man1/ndiff.1 #usr/share/man/man1/nmap.1 @@ -85,6 +83,7 @@ usr/share/nmap/nselib/data/http-fingerprints.lua usr/share/nmap/nselib/data/http-folders.txt usr/share/nmap/nselib/data/http-sql-errors.lst usr/share/nmap/nselib/data/http-web-files-extensions.lst +usr/share/nmap/nselib/data/idnaMappings.lua usr/share/nmap/nselib/data/ike-fingerprints.lua #usr/share/nmap/nselib/data/jdwp-class #usr/share/nmap/nselib/data/jdwp-class/JDWPExecCmd.class @@ -110,6 +109,7 @@ usr/share/nmap/nselib/data/psexec/network.lua usr/share/nmap/nselib/data/psexec/nmap_service.c usr/share/nmap/nselib/data/psexec/nmap_service.vcproj usr/share/nmap/nselib/data/psexec/pwdump.lua +usr/share/nmap/nselib/data/publickeydb usr/share/nmap/nselib/data/rtsp-urls.txt usr/share/nmap/nselib/data/snmpcommunities.lst usr/share/nmap/nselib/data/ssl-fingerprints @@ -138,18 +138,22 @@ usr/share/nmap/nselib/gps.lua usr/share/nmap/nselib/http.lua usr/share/nmap/nselib/httpspider.lua usr/share/nmap/nselib/iax2.lua +usr/share/nmap/nselib/idna.lua usr/share/nmap/nselib/ike.lua usr/share/nmap/nselib/imap.lua usr/share/nmap/nselib/informix.lua usr/share/nmap/nselib/ipOps.lua usr/share/nmap/nselib/ipmi.lua usr/share/nmap/nselib/ipp.lua +usr/share/nmap/nselib/irc.lua usr/share/nmap/nselib/iscsi.lua usr/share/nmap/nselib/isns.lua usr/share/nmap/nselib/jdwp.lua usr/share/nmap/nselib/json.lua usr/share/nmap/nselib/ldap.lua usr/share/nmap/nselib/lfs.luadoc +usr/share/nmap/nselib/libssh2-utility.lua +usr/share/nmap/nselib/libssh2.luadoc usr/share/nmap/nselib/listop.lua usr/share/nmap/nselib/lpeg-utility.lua usr/share/nmap/nselib/lpeg.luadoc @@ -166,6 +170,7 @@ usr/share/nmap/nselib/mssql.lua usr/share/nmap/nselib/multicast.lua usr/share/nmap/nselib/mysql.lua usr/share/nmap/nselib/natpmp.lua +usr/share/nmap/nselib/nbd.lua usr/share/nmap/nselib/ncp.lua usr/share/nmap/nselib/ndmp.lua usr/share/nmap/nselib/netbios.lua @@ -181,6 +186,7 @@ usr/share/nmap/nselib/pgsql.lua usr/share/nmap/nselib/pop3.lua usr/share/nmap/nselib/pppoe.lua usr/share/nmap/nselib/proxy.lua +usr/share/nmap/nselib/punycode.lua usr/share/nmap/nselib/rdp.lua usr/share/nmap/nselib/re.lua usr/share/nmap/nselib/redis.lua @@ -194,6 +200,7 @@ usr/share/nmap/nselib/shortport.lua usr/share/nmap/nselib/sip.lua usr/share/nmap/nselib/slaxml.lua usr/share/nmap/nselib/smb.lua +usr/share/nmap/nselib/smb2.lua usr/share/nmap/nselib/smbauth.lua usr/share/nmap/nselib/smtp.lua usr/share/nmap/nselib/snmp.lua @@ -225,6 +232,7 @@ usr/share/nmap/nselib/vuzedht.lua usr/share/nmap/nselib/wsdd.lua usr/share/nmap/nselib/xdmcp.lua usr/share/nmap/nselib/xmpp.lua +usr/share/nmap/nselib/zlib.luadoc #usr/share/nmap/scripts usr/share/nmap/scripts/acarsd-info.nse usr/share/nmap/scripts/address-info.nse @@ -267,6 +275,7 @@ usr/share/nmap/scripts/broadcast-ms-sql-discover.nse usr/share/nmap/scripts/broadcast-netbios-master-browser.nse usr/share/nmap/scripts/broadcast-networker-discover.nse usr/share/nmap/scripts/broadcast-novell-locate.nse +usr/share/nmap/scripts/broadcast-ospf2-discover.nse usr/share/nmap/scripts/broadcast-pc-anywhere.nse usr/share/nmap/scripts/broadcast-pc-duo.nse usr/share/nmap/scripts/broadcast-pim-discovery.nse @@ -287,6 +296,8 @@ usr/share/nmap/scripts/cassandra-brute.nse usr/share/nmap/scripts/cassandra-info.nse usr/share/nmap/scripts/cccam-version.nse usr/share/nmap/scripts/cics-enum.nse +usr/share/nmap/scripts/cics-info.nse +usr/share/nmap/scripts/cics-user-brute.nse usr/share/nmap/scripts/cics-user-enum.nse usr/share/nmap/scripts/citrix-brute-xml.nse usr/share/nmap/scripts/citrix-enum-apps-xml.nse @@ -306,6 +317,7 @@ usr/share/nmap/scripts/cvs-brute.nse usr/share/nmap/scripts/daap-get-library.nse usr/share/nmap/scripts/daytime.nse usr/share/nmap/scripts/db2-das-info.nse +usr/share/nmap/scripts/deluge-rpc-brute.nse usr/share/nmap/scripts/dhcp-discover.nse usr/share/nmap/scripts/dict-info.nse usr/share/nmap/scripts/distcc-cve2004-2687.nse @@ -352,6 +364,7 @@ usr/share/nmap/scripts/ftp-bounce.nse usr/share/nmap/scripts/ftp-brute.nse usr/share/nmap/scripts/ftp-libopie.nse usr/share/nmap/scripts/ftp-proftpd-backdoor.nse +usr/share/nmap/scripts/ftp-syst.nse usr/share/nmap/scripts/ftp-vsftpd-backdoor.nse usr/share/nmap/scripts/ftp-vuln-cve2010-4221.nse usr/share/nmap/scripts/ganglia-info.nse @@ -369,6 +382,7 @@ usr/share/nmap/scripts/hbase-region-info.nse usr/share/nmap/scripts/hddtemp-info.nse usr/share/nmap/scripts/hnap-info.nse usr/share/nmap/scripts/hostmap-bfk.nse +usr/share/nmap/scripts/hostmap-crtsh.nse usr/share/nmap/scripts/hostmap-ip2hosts.nse usr/share/nmap/scripts/hostmap-robtex.nse usr/share/nmap/scripts/http-adobe-coldfusion-apsa1301.nse @@ -383,6 +397,7 @@ usr/share/nmap/scripts/http-awstatstotals-exec.nse usr/share/nmap/scripts/http-axis2-dir-traversal.nse usr/share/nmap/scripts/http-backup-finder.nse usr/share/nmap/scripts/http-barracuda-dir-traversal.nse +usr/share/nmap/scripts/http-bigip-cookie.nse usr/share/nmap/scripts/http-brute.nse usr/share/nmap/scripts/http-cakephp-version.nse usr/share/nmap/scripts/http-chrono.nse @@ -390,6 +405,7 @@ usr/share/nmap/scripts/http-cisco-anyconnect.nse usr/share/nmap/scripts/http-coldfusion-subzero.nse usr/share/nmap/scripts/http-comments-displayer.nse usr/share/nmap/scripts/http-config-backup.nse +usr/share/nmap/scripts/http-cookie-flags.nse usr/share/nmap/scripts/http-cors.nse usr/share/nmap/scripts/http-cross-domain-policy.nse usr/share/nmap/scripts/http-csrf.nse @@ -424,6 +440,7 @@ usr/share/nmap/scripts/http-iis-short-name-brute.nse usr/share/nmap/scripts/http-iis-webdav-vuln.nse usr/share/nmap/scripts/http-internal-ip-disclosure.nse usr/share/nmap/scripts/http-joomla-brute.nse +usr/share/nmap/scripts/http-jsonp-detection.nse usr/share/nmap/scripts/http-litespeed-sourcecode-download.nse usr/share/nmap/scripts/http-ls.nse usr/share/nmap/scripts/http-majordomo2-dir-traversal.nse @@ -447,6 +464,7 @@ usr/share/nmap/scripts/http-rfi-spider.nse usr/share/nmap/scripts/http-robots.txt.nse usr/share/nmap/scripts/http-robtex-reverse-ip.nse usr/share/nmap/scripts/http-robtex-shared-ns.nse +usr/share/nmap/scripts/http-security-headers.nse usr/share/nmap/scripts/http-server-header.nse usr/share/nmap/scripts/http-shellshock.nse usr/share/nmap/scripts/http-sitemap-generator.nse @@ -460,6 +478,7 @@ usr/share/nmap/scripts/http-title.nse usr/share/nmap/scripts/http-tplink-dir-traversal.nse usr/share/nmap/scripts/http-trace.nse usr/share/nmap/scripts/http-traceroute.nse +usr/share/nmap/scripts/http-trane-info.nse usr/share/nmap/scripts/http-unsafe-output-escaping.nse usr/share/nmap/scripts/http-useragent-tester.nse usr/share/nmap/scripts/http-userdir-enum.nse @@ -485,6 +504,10 @@ usr/share/nmap/scripts/http-vuln-cve2014-3704.nse usr/share/nmap/scripts/http-vuln-cve2014-8877.nse usr/share/nmap/scripts/http-vuln-cve2015-1427.nse usr/share/nmap/scripts/http-vuln-cve2015-1635.nse +usr/share/nmap/scripts/http-vuln-cve2017-1001000.nse +usr/share/nmap/scripts/http-vuln-cve2017-5638.nse +usr/share/nmap/scripts/http-vuln-cve2017-5689.nse +usr/share/nmap/scripts/http-vuln-cve2017-8917.nse usr/share/nmap/scripts/http-vuln-misfortune-cookie.nse usr/share/nmap/scripts/http-vuln-wnr1000-creds.nse usr/share/nmap/scripts/http-waf-detect.nse @@ -497,10 +520,12 @@ usr/share/nmap/scripts/http-xssed.nse usr/share/nmap/scripts/iax2-brute.nse usr/share/nmap/scripts/iax2-version.nse usr/share/nmap/scripts/icap-info.nse +usr/share/nmap/scripts/iec-identify.nse usr/share/nmap/scripts/ike-version.nse usr/share/nmap/scripts/imap-brute.nse usr/share/nmap/scripts/imap-capabilities.nse usr/share/nmap/scripts/imap-ntlm-info.nse +usr/share/nmap/scripts/impress-remote-discover.nse usr/share/nmap/scripts/informix-brute.nse usr/share/nmap/scripts/informix-query.nse usr/share/nmap/scripts/informix-tables.nse @@ -585,6 +610,7 @@ usr/share/nmap/scripts/mysql-variables.nse usr/share/nmap/scripts/mysql-vuln-cve2012-2122.nse usr/share/nmap/scripts/nat-pmp-info.nse usr/share/nmap/scripts/nat-pmp-mapport.nse +usr/share/nmap/scripts/nbd-info.nse usr/share/nmap/scripts/nbstat.nse usr/share/nmap/scripts/ncp-enum-users.nse usr/share/nmap/scripts/ncp-serverinfo.nse @@ -612,6 +638,7 @@ usr/share/nmap/scripts/omp2-enum-targets.nse usr/share/nmap/scripts/omron-info.nse usr/share/nmap/scripts/openlookup-info.nse usr/share/nmap/scripts/openvas-otp-brute.nse +usr/share/nmap/scripts/openwebnet-discovery.nse usr/share/nmap/scripts/oracle-brute-stealth.nse usr/share/nmap/scripts/oracle-brute.nse usr/share/nmap/scripts/oracle-enum-users.nse @@ -628,6 +655,7 @@ usr/share/nmap/scripts/pop3-brute.nse usr/share/nmap/scripts/pop3-capabilities.nse usr/share/nmap/scripts/pop3-ntlm-info.nse usr/share/nmap/scripts/pptp-version.nse +usr/share/nmap/scripts/puppet-naivesigning.nse usr/share/nmap/scripts/qconn-exec.nse usr/share/nmap/scripts/qscan.nse usr/share/nmap/scripts/quake1-info.nse @@ -650,6 +678,7 @@ usr/share/nmap/scripts/rpc-grind.nse usr/share/nmap/scripts/rpcap-brute.nse usr/share/nmap/scripts/rpcap-info.nse usr/share/nmap/scripts/rpcinfo.nse +usr/share/nmap/scripts/rsa-vuln-roca.nse usr/share/nmap/scripts/rsync-brute.nse usr/share/nmap/scripts/rsync-list-modules.nse usr/share/nmap/scripts/rtsp-methods.nse @@ -666,9 +695,11 @@ usr/share/nmap/scripts/sip-enum-users.nse usr/share/nmap/scripts/sip-methods.nse usr/share/nmap/scripts/skypev2-version.nse usr/share/nmap/scripts/smb-brute.nse +usr/share/nmap/scripts/smb-double-pulsar-backdoor.nse usr/share/nmap/scripts/smb-enum-domains.nse usr/share/nmap/scripts/smb-enum-groups.nse usr/share/nmap/scripts/smb-enum-processes.nse +usr/share/nmap/scripts/smb-enum-services.nse usr/share/nmap/scripts/smb-enum-sessions.nse usr/share/nmap/scripts/smb-enum-shares.nse usr/share/nmap/scripts/smb-enum-users.nse @@ -677,19 +708,25 @@ usr/share/nmap/scripts/smb-ls.nse usr/share/nmap/scripts/smb-mbenum.nse usr/share/nmap/scripts/smb-os-discovery.nse usr/share/nmap/scripts/smb-print-text.nse +usr/share/nmap/scripts/smb-protocols.nse usr/share/nmap/scripts/smb-psexec.nse usr/share/nmap/scripts/smb-security-mode.nse usr/share/nmap/scripts/smb-server-stats.nse usr/share/nmap/scripts/smb-system-info.nse usr/share/nmap/scripts/smb-vuln-conficker.nse +usr/share/nmap/scripts/smb-vuln-cve-2017-7494.nse usr/share/nmap/scripts/smb-vuln-cve2009-3103.nse usr/share/nmap/scripts/smb-vuln-ms06-025.nse usr/share/nmap/scripts/smb-vuln-ms07-029.nse usr/share/nmap/scripts/smb-vuln-ms08-067.nse usr/share/nmap/scripts/smb-vuln-ms10-054.nse usr/share/nmap/scripts/smb-vuln-ms10-061.nse +usr/share/nmap/scripts/smb-vuln-ms17-010.nse usr/share/nmap/scripts/smb-vuln-regsvc-dos.nse -usr/share/nmap/scripts/smbv2-enabled.nse +usr/share/nmap/scripts/smb2-capabilities.nse +usr/share/nmap/scripts/smb2-security-mode.nse +usr/share/nmap/scripts/smb2-time.nse +usr/share/nmap/scripts/smb2-vuln-uptime.nse usr/share/nmap/scripts/smtp-brute.nse usr/share/nmap/scripts/smtp-commands.nse usr/share/nmap/scripts/smtp-enum-users.nse @@ -715,7 +752,11 @@ usr/share/nmap/scripts/snmp-win32-users.nse usr/share/nmap/scripts/socks-auth-info.nse usr/share/nmap/scripts/socks-brute.nse usr/share/nmap/scripts/socks-open-proxy.nse +usr/share/nmap/scripts/ssh-auth-methods.nse +usr/share/nmap/scripts/ssh-brute.nse usr/share/nmap/scripts/ssh-hostkey.nse +usr/share/nmap/scripts/ssh-publickey-acceptance.nse +usr/share/nmap/scripts/ssh-run.nse usr/share/nmap/scripts/ssh2-enum-algos.nse usr/share/nmap/scripts/sshv1.nse usr/share/nmap/scripts/ssl-ccs-injection.nse @@ -750,7 +791,9 @@ usr/share/nmap/scripts/telnet-brute.nse usr/share/nmap/scripts/telnet-encryption.nse usr/share/nmap/scripts/telnet-ntlm-info.nse usr/share/nmap/scripts/tftp-enum.nse +usr/share/nmap/scripts/tls-alpn.nse usr/share/nmap/scripts/tls-nextprotoneg.nse +usr/share/nmap/scripts/tls-ticketbleed.nse usr/share/nmap/scripts/tn3270-screen.nse usr/share/nmap/scripts/tor-consensus-checker.nse usr/share/nmap/scripts/traceroute-geolocation.nse @@ -763,6 +806,7 @@ usr/share/nmap/scripts/url-snarf.nse usr/share/nmap/scripts/ventrilo-info.nse usr/share/nmap/scripts/versant-info.nse usr/share/nmap/scripts/vmauthd-brute.nse +usr/share/nmap/scripts/vmware-version.nse usr/share/nmap/scripts/vnc-brute.nse usr/share/nmap/scripts/vnc-info.nse usr/share/nmap/scripts/vnc-title.nse diff --git a/lfs/nmap b/lfs/nmap index 1daeb89b0..1d9b33f21 100644 --- a/lfs/nmap +++ b/lfs/nmap @@ -1,7 +1,7 @@ ############################################################################### # # # IPFire.org - A linux based firewall # -# Copyright (C) 2017 IPFire Team <info(a)ipfire.org> # +# Copyright (C) 2018 IPFire Team <info(a)ipfire.org> # # # # This program is free software: you can redistribute it and/or modify # # it under the terms of the GNU General Public License as published by # @@ -24,7 +24,7 @@ include Config -VER = 7.40 +VER = 7.70 THISAPP = nmap-$(VER) DL_FILE = $(THISAPP).tar.bz2 @@ -32,7 +32,7 @@ DL_FROM = $(URL_IPFIRE) DIR_APP = $(DIR_SRC)/$(THISAPP) TARGET = $(DIR_INFO)/$(THISAPP) PROG = nmap -PAK_VER = 9 +PAK_VER = 10 DEPS = "" @@ -44,7 +44,7 @@ objects = $(DL_FILE) $(DL_FILE) = $(DL_FROM)/$(DL_FILE) -$(DL_FILE)_MD5 = 9c5a28bfb46228bade82b238408c065e +$(DL_FILE)_MD5 = 84eb6fbe788e0d4918c2b1e39421bf79 install : $(TARGET) @@ -77,7 +77,11 @@ $(subst %,%_MD5,$(objects)) : $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) @$(PREBUILD) @rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar jxf $(DIR_DL)/$(DL_FILE) - cd $(DIR_APP) && ./configure --prefix=/usr --without-nmapfe --without-zenmap --without-ncat + cd $(DIR_APP) && ./configure \ + --prefix=/usr \ + --without-nmapfe \ + --without-zenmap \ + --without-ncat cd $(DIR_APP) && make $(MAKETUNING) $(EXTRA_MAKE) cd $(DIR_APP) && make install @rm -rf $(DIR_APP) -- 2.17.0

1 0

[PATCH] xz: Performance and compression tuning (V1)
by Matthias Fischer 20 May '18

20 May '18

Hi, as mentioned in the "list", we're trying to make the archives as small as possible using 'xz'-compression. In order to achieve this, this patch tests the size of the memory available on the host system. It sets the xz-memory-limit (--memory=[N]Mib) to 70% of the available working memory, a maximum of four xz-'worker threads' (-T4) and a compression rate of '-8' (-8). These options are written to the 'XZ_OPT=' environment. For details see: https://linux.die.net/man/1/xz I have set the number of available xz-'worker threads' (-T) to four (-T4), because during the final tests the '-T0' parameter led to error messages snd stopped: 'cannot allocate memory'. It wouldn't even run with 90%. Furthermore, testing with '-T0' led to countless messages filling up '_build.packages.log'. E.g.: ... xz: Adjusted the number of threads from 8 to 2 to not exceed the memory usage limit of 1557 MiB ... Tests took place on a 32bit-Ubuntu 16.04.4-system with 8 GB RAM and an Intel I7-2600. Build time was about 04:30 hrs. Perhaps a 64bit-system would perform better (higher values), but my goal was to make this run on as many systems as possible, so I choosed these averages. If minimum requirements (1024 MB RAM) are not met, building stops. Current results: 'next', untuned: ipfire-2.19.2gb-ext4.i586-full-core121.img.gz => 332951687 Bytes ipfire-2.19.i586-full-core121.iso => 228589568 Bytes 'next', XZ_OPT: -T4 -8, 70% RAM: ipfire-2.19.2gb-ext4.i586-full-core121.img.gz => 329725723 Bytes ipfire-2.19.i586-full-core121.iso => 217055232 Bytes These two resulting archives are 14760300 Bytes smaller than before. Best, Matthias Signed-off-by: Matthias Fischer <matthias.fischer(a)ipfire.org> --- lfs/Config | 2 +- lfs/cdrom | 5 +---- make.sh | 18 +++++++++++++++++- 3 files changed, 19 insertions(+), 6 deletions(-) diff --git a/lfs/Config b/lfs/Config index d63a1dc9a..c667a3cc7 100644 --- a/lfs/Config +++ b/lfs/Config @@ -209,7 +209,7 @@ define PAK tar xf /install/packages/package/files.tmp -C /install/packages/package/tmp/ \ -p --numeric-owner rm -f /install/packages/package/files.tmp - cd /install/packages/package/tmp/ && XZ_OPT=-T0 tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz * + cd /install/packages/package/tmp/ && XZ_OPT="$(XZ_OPT)" tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz * rm -r /install/packages/package/tmp -cat /install/packages/package/ROOTFILES | grep -v "#" > /install/packages/package/ROOTFILES.tmp mv /install/packages/package/ROOTFILES.tmp /install/packages/package/ROOTFILES diff --git a/lfs/cdrom b/lfs/cdrom index 7a7fff166..820c55b87 100644 --- a/lfs/cdrom +++ b/lfs/cdrom @@ -35,9 +35,6 @@ else TAR_OPTIONS = --xz endif -# Enable multi-threaded compression for LZMA -export XZ_OPT = --threads=0 - ############################################################################### # Top-level Rules ############################################################################### @@ -74,7 +71,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) # Compress root filesystem # Reason for this tar+untar+tar is removing of entries listed two or more in src/ROOTFILES - tar -c -C / --files-from=/tmp/ROOTFILES \ + XZ_OPT="$(XZ_OPT)" tar -c -C / --files-from=/tmp/ROOTFILES \ -f /$(SNAME).tar --exclude='#*' --exclude='dev/pts/*' \ --exclude='proc/*' --exclude='tmp/ROOTFILES' rm -f /tmp/ROOTFILES diff --git a/make.sh b/make.sh index 419f9fa95..b6f3f37d2 100755 --- a/make.sh +++ b/make.sh @@ -526,6 +526,7 @@ enterchroot() { CCACHE_COMPRESS="${CCACHE_COMPRESS}" \ CCACHE_COMPILERCHECK="${CCACHE_COMPILERCHECK}" \ KVER="${KVER}" \ + XZ_OPT="${XZ_OPT}" \ $(fake_environ) \ $(qemu_environ) \ "$@" @@ -860,6 +861,21 @@ fi # Get the amount of memory in this build system HOST_MEM=$(system_memory) +# Checking host memory, tuning XZ_OPT + +if [ $HOST_MEM -lt 16384 ]; then + print_build_stage "Host-Memory: $HOST_MEM MiB" + print_build_stage "Not enough host memory (less than 1024 MiB, please consider upgrading)" + + exit 1 + +else + + XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB" + XZ_OPT="-T4 -8 --memory=$XZ_MEM" + +fi + if [ -n "${BUILD_ARCH}" ]; then configure_build "${BUILD_ARCH}" else @@ -1760,7 +1776,7 @@ toolchain) buildtoolchain echo "`date -u '+%b %e %T'`: Create toolchain image for ${BUILD_ARCH}" | tee -a $LOGFILE test -d $BASEDIR/cache/toolchains || mkdir -p $BASEDIR/cache/toolchains - cd $BASEDIR && XZ_OPT="-T0 -8" tar -Jc --exclude='log/_build.*.log' -f cache/toolchains/$SNAME-$VERSION-toolchain-$TOOLCHAINVER-${BUILD_ARCH}.tar.xz \ + cd $BASEDIR && XZ_OPT="$(XZ_OPT)" tar -Jc --exclude='log/_build.*.log' -f cache/toolchains/$SNAME-$VERSION-toolchain-$TOOLCHAINVER-${BUILD_ARCH}.tar.xz \ build/${TOOLS_DIR} build/bin/sh log >> $LOGFILE md5sum cache/toolchains/$SNAME-$VERSION-toolchain-$TOOLCHAINVER-${BUILD_ARCH}.tar.xz \ > cache/toolchains/$SNAME-$VERSION-toolchain-$TOOLCHAINVER-${BUILD_ARCH}.md5 -- 2.17.0

1 1

[PATCH] xz: Performance and compression tuning (V1)
by Matthias Fischer 20 May '18

20 May '18

Hi, as mentioned in the "list", we're trying to make the archives as small as possible using 'xz'-compression. In order to achieve this, this patch tests the size of the memory available on the host system. It sets the xz-memory-limit (--memory=[N]Mib) to 70% of the available working memory, a maximum of four xz-'worker threads' (-T4) and a compression rate of '-8' (-8). These options are written to the 'XZ_OPT=' environment. For details see: https://linux.die.net/man/1/xz I have set the number of available xz-'worker threads' (-T) to four (-T4), because during the final tests the '-T0' parameter led to error messages snd stopped: 'cannot allocate memory'. It wouldn't even run with 90%. Furthermore, testing with '-T0' led to countless messages filling up '_build.packages.log'. E.g.: ... xz: Adjusted the number of threads from 8 to 2 to not exceed the memory usage limit of 1557 MiB ... Tests took place on a 32bit-Ubuntu 16.04.4-system with 8 GB RAM and an Intel I7-2600. Build time was about 04:30 hrs. Perhaps a 64bit-system would perform better (higher values), but my goal was to make this run on as many systems as possible, so I choosed these averages. If minimum requirements (1024 MB RAM) are not met, building stops. Current results: 'next', untuned: ipfire-2.19.2gb-ext4.i586-full-core121.img.gz => 332951687 Bytes ipfire-2.19.i586-full-core121.iso => 228589568 Bytes 'next', XZ_OPT: -T4 -8, 70% RAM: ipfire-2.19.2gb-ext4.i586-full-core121.img.gz => 329725723 Bytes ipfire-2.19.i586-full-core121.iso => 217055232 Bytes These two resulting archives are 14760300 Bytes smaller than before. Best, Matthias Signed-off-by: Matthias Fischer <matthias.fischer(a)ipfire.org> --- lfs/Config | 2 +- lfs/cdrom | 5 +---- make.sh | 18 +++++++++++++++++- 3 files changed, 19 insertions(+), 6 deletions(-) diff --git a/lfs/Config b/lfs/Config index d63a1dc9a..c667a3cc7 100644 --- a/lfs/Config +++ b/lfs/Config @@ -209,7 +209,7 @@ define PAK tar xf /install/packages/package/files.tmp -C /install/packages/package/tmp/ \ -p --numeric-owner rm -f /install/packages/package/files.tmp - cd /install/packages/package/tmp/ && XZ_OPT=-T0 tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz * + cd /install/packages/package/tmp/ && XZ_OPT="$(XZ_OPT)" tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz * rm -r /install/packages/package/tmp -cat /install/packages/package/ROOTFILES | grep -v "#" > /install/packages/package/ROOTFILES.tmp mv /install/packages/package/ROOTFILES.tmp /install/packages/package/ROOTFILES diff --git a/lfs/cdrom b/lfs/cdrom index 7a7fff166..820c55b87 100644 --- a/lfs/cdrom +++ b/lfs/cdrom @@ -35,9 +35,6 @@ else TAR_OPTIONS = --xz endif -# Enable multi-threaded compression for LZMA -export XZ_OPT = --threads=0 - ############################################################################### # Top-level Rules ############################################################################### @@ -74,7 +71,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) # Compress root filesystem # Reason for this tar+untar+tar is removing of entries listed two or more in src/ROOTFILES - tar -c -C / --files-from=/tmp/ROOTFILES \ + XZ_OPT="$(XZ_OPT)" tar -c -C / --files-from=/tmp/ROOTFILES \ -f /$(SNAME).tar --exclude='#*' --exclude='dev/pts/*' \ --exclude='proc/*' --exclude='tmp/ROOTFILES' rm -f /tmp/ROOTFILES diff --git a/make.sh b/make.sh index 419f9fa95..33ebc16bb 100755 --- a/make.sh +++ b/make.sh @@ -526,6 +526,7 @@ enterchroot() { CCACHE_COMPRESS="${CCACHE_COMPRESS}" \ CCACHE_COMPILERCHECK="${CCACHE_COMPILERCHECK}" \ KVER="${KVER}" \ + XZ_OPT="${XZ_OPT}" \ $(fake_environ) \ $(qemu_environ) \ "$@" @@ -860,6 +861,21 @@ fi # Get the amount of memory in this build system HOST_MEM=$(system_memory) +# Checking host memory, tuning XZ_OPT + +if [ $HOST_MEM -lt 1024 ]; then + print_build_stage "Host-Memory: $HOST_MEM MiB" + print_build_stage "Not enough host memory (less than 1024 MiB, please consider upgrading)" + + exit 1 + +else + + XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB" + XZ_OPT="-T4 -8 --memory=$XZ_MEM" + +fi + if [ -n "${BUILD_ARCH}" ]; then configure_build "${BUILD_ARCH}" else @@ -1760,7 +1776,7 @@ toolchain) buildtoolchain echo "`date -u '+%b %e %T'`: Create toolchain image for ${BUILD_ARCH}" | tee -a $LOGFILE test -d $BASEDIR/cache/toolchains || mkdir -p $BASEDIR/cache/toolchains - cd $BASEDIR && XZ_OPT="-T0 -8" tar -Jc --exclude='log/_build.*.log' -f cache/toolchains/$SNAME-$VERSION-toolchain-$TOOLCHAINVER-${BUILD_ARCH}.tar.xz \ + cd $BASEDIR && XZ_OPT="$(XZ_OPT)" tar -Jc --exclude='log/_build.*.log' -f cache/toolchains/$SNAME-$VERSION-toolchain-$TOOLCHAINVER-${BUILD_ARCH}.tar.xz \ build/${TOOLS_DIR} build/bin/sh log >> $LOGFILE md5sum cache/toolchains/$SNAME-$VERSION-toolchain-$TOOLCHAINVER-${BUILD_ARCH}.tar.xz \ > cache/toolchains/$SNAME-$VERSION-toolchain-$TOOLCHAINVER-${BUILD_ARCH}.md5 -- 2.17.0

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

Development May 2018