With this patch, I tried to optimize the build (because it really takes a long time if I compile from scratch for testing). Nowadays, most packages are already optimizing for parallel build. I was testing the build on an 8 core system (i.e., MAKETUNING is -j9).
I also found out something else, the collectd package build crashes when libstatgrab is already built: https://github.com/collectd/collectd/issues/445
Someone else already noticed this on the list, without discovering the cause. An upgrade of collectd would be required to solve this problem. Since I am not sure for what collectd is actually used (and a major upgrade would be required), I did not perform this.
Speed-up: Package | Before | After boost | 1:20:41 | 13:39 cyrus-imapd | 3:50 | 0:48 dnsdist | 5:23 | 1:18 gcc | 46:15 | 11:29 gettext | 9:55 | 4:56 gutenprint | 3:14 | 1:22 ipfire-netboot | 9:33 | 2:07 krb5 | 5:28 | 1:27 lcd4linux | 1:04 | 0:27 netsnmpd | 8:00 | 3:33 nut | 1:59 | 0:21 openssl | 7:57 | 2:37 rrdtool | 1:31 | 0:32 samba | 5:03 | 0:55 slang | 1:04 | 0:19 snort | 5:11 | 0:56 xfsprogs | 5:41 | 1:46
I also marked packages that (in their current version) crash when being built in parallel. I performed multiple builds (and tested the ISO image), so I hope that this patch is reliable for everyone. Nevertheless, please test the patch on different build environments before applying it.
Wolfgang Apolinarski (1): Parallelized build for several packages
lfs/boost | 2 +- lfs/collectd | 2 +- lfs/cyrus-imapd | 2 +- lfs/dhcp | 2 +- lfs/dnsdist | 2 +- lfs/gcc | 4 ++-- lfs/gettext | 8 ++++---- lfs/groff | 2 +- lfs/gutenprint | 2 +- lfs/ipfire-netboot | 4 ++-- lfs/krb5 | 2 +- lfs/lcd4linux | 2 +- lfs/netpbm | 2 +- lfs/netsnmpd | 2 +- lfs/nut | 2 +- lfs/openssl | 2 +- lfs/rrdtool | 2 +- lfs/samba | 4 ++-- lfs/slang | 2 +- lfs/snort | 2 +- lfs/xfsprogs | 2 +- 21 files changed, 27 insertions(+), 27 deletions(-)
Added $(MAKETUNING) to several packages. Marked packages that do not support parallel build. --- lfs/boost | 2 +- lfs/collectd | 2 +- lfs/cyrus-imapd | 2 +- lfs/dhcp | 2 +- lfs/dnsdist | 2 +- lfs/gcc | 4 ++-- lfs/gettext | 8 ++++---- lfs/groff | 2 +- lfs/gutenprint | 2 +- lfs/ipfire-netboot | 4 ++-- lfs/krb5 | 2 +- lfs/lcd4linux | 2 +- lfs/netpbm | 2 +- lfs/netsnmpd | 2 +- lfs/nut | 2 +- lfs/openssl | 2 +- lfs/rrdtool | 2 +- lfs/samba | 4 ++-- lfs/slang | 2 +- lfs/snort | 2 +- lfs/xfsprogs | 2 +- 21 files changed, 27 insertions(+), 27 deletions(-)
diff --git a/lfs/boost b/lfs/boost index bed2b0cdb..3ce49a0a9 100644 --- a/lfs/boost +++ b/lfs/boost @@ -95,7 +95,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) @rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar axf $(DIR_DL)/$(DL_FILE)
cd $(DIR_APP) && ./bootstrap.sh --with-toolset=gcc - cd $(DIR_APP) && ./b2 -d+2 -q $(CONFIGURE_OPTIONS) stage + cd $(DIR_APP) && ./b2 -d+2 -q $(MAKETUNING) $(CONFIGURE_OPTIONS) stage cd $(DIR_APP) && ./b2 $(CONFIGURE_OPTIONS) install
@rm -rf $(DIR_APP) diff --git a/lfs/collectd b/lfs/collectd index 9592ec95a..e31324817 100644 --- a/lfs/collectd +++ b/lfs/collectd @@ -116,7 +116,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) --enable-{network,nfs,ntpd,ping,processes,rrdtool,sensors,swap,syslog} \ --enable-{tcpconns,unixsock,users,wireless} \ --with-librrd=/usr/share/rrdtool-1.2.30 - cd $(DIR_APP) && make install + cd $(DIR_APP) && make install #collectd-4 does not support parallel build cp -vf $(DIR_SRC)/config/collectd/collectd.* /etc/ mv /etc/collectd.vpn /var/ipfire/ovpn/collectd.vpn chown nobody.nobody /var/ipfire/ovpn/collectd.vpn diff --git a/lfs/cyrus-imapd b/lfs/cyrus-imapd index 1800a08ec..50cd66875 100644 --- a/lfs/cyrus-imapd +++ b/lfs/cyrus-imapd @@ -89,7 +89,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) --enable-autocreate \ --enable-idled
- cd $(DIR_APP) && make $(EXTRA_MAKE) + cd $(DIR_APP) && make $(MAKETUNING) $(EXTRA_MAKE) cd $(DIR_APP) && make install -mkdir /var/imap chown cyrus:mail /var/imap diff --git a/lfs/dhcp b/lfs/dhcp index a055d6081..4c01428f5 100644 --- a/lfs/dhcp +++ b/lfs/dhcp @@ -84,7 +84,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) --enable-early-chroot \ --disable-dhcpv6
- cd $(DIR_APP) && make + cd $(DIR_APP) && make #This package does not support parallel build cd $(DIR_APP) && make install
mkdir -pv /var/state/dhcp diff --git a/lfs/dnsdist b/lfs/dnsdist index 85a841fa5..e748004ef 100644 --- a/lfs/dnsdist +++ b/lfs/dnsdist @@ -79,7 +79,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) @$(PREBUILD) @rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar axf $(DIR_DL)/$(DL_FILE) cd $(DIR_APP) && ./configure --prefix=/usr --sysconfdir=/etc - cd $(DIR_APP) && make + cd $(DIR_APP) && make $(MAKETUNING) cd $(DIR_APP) && make install
#install initscripts diff --git a/lfs/gcc b/lfs/gcc index 6fdf2ea31..5aac2d382 100644 --- a/lfs/gcc +++ b/lfs/gcc @@ -264,7 +264,7 @@ ifeq "$(PASS)" "L" $(EXTRA_ENV) \ $(DIR_APP)/libstdc++-v3/configure \ $(EXTRA_CONFIG) - cd $(DIR_SRC)/gcc-build && make $(EXTRA_MAKE) + cd $(DIR_SRC)/gcc-build && make $(MAKETUNING) $(EXTRA_MAKE) cd $(DIR_SRC)/gcc-build && make $(EXTRA_INSTALL) install
else @@ -273,7 +273,7 @@ else $(EXTRA_ENV) \ $(DIR_APP)/configure \ $(EXTRA_CONFIG) - cd $(DIR_SRC)/gcc-build && make $(EXTRA_MAKE) + cd $(DIR_SRC)/gcc-build && make $(MAKETUNING) $(EXTRA_MAKE) cd $(DIR_SRC)/gcc-build && make $(EXTRA_INSTALL) install endif
diff --git a/lfs/gettext b/lfs/gettext index 632ecac43..b1d75ed2d 100644 --- a/lfs/gettext +++ b/lfs/gettext @@ -81,13 +81,13 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) @rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar axf $(DIR_DL)/$(DL_FILE) ifeq "$(ROOT)" "" cd $(DIR_APP) && ./configure $(EXTRA_CONFIG) - cd $(DIR_APP) && make + cd $(DIR_APP) && make $(MAKETUNING) cd $(DIR_APP) && make install else cd $(DIR_APP)/gettext-tools && ./configure $(EXTRA_CONFIG) - cd $(DIR_APP)/gettext-tools && make -C gnulib-lib - cd $(DIR_APP)/gettext-tools && make -C intl pluralx.c - cd $(DIR_APP)/gettext-tools && make -C src msgfmt + cd $(DIR_APP)/gettext-tools && make $(MAKETUNING) -C gnulib-lib + cd $(DIR_APP)/gettext-tools && make $(MAKETUNING) -C intl pluralx.c + cd $(DIR_APP)/gettext-tools && make $(MAKETUNING) -C src msgfmt cd $(DIR_APP)/gettext-tools && cp -v src/msgfmt $(TOOLS_DIR)/bin endif @rm -rf $(DIR_APP) diff --git a/lfs/groff b/lfs/groff index 4d5bf6373..1ed393c62 100644 --- a/lfs/groff +++ b/lfs/groff @@ -73,7 +73,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) cd $(DIR_APP) && PAGE=A4 ./configure \ --prefix=/usr
- cd $(DIR_APP) && make + cd $(DIR_APP) && make #This package does not support parallel build cd $(DIR_APP) && make install ln -svf eqn /usr/bin/geqn ln -svf tbl /usr/bin/gtbl diff --git a/lfs/gutenprint b/lfs/gutenprint index 7286b78a2..0a18aa73b 100644 --- a/lfs/gutenprint +++ b/lfs/gutenprint @@ -79,7 +79,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) @rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar jxf $(DIR_DL)/$(DL_FILE) $(UPDATE_AUTOMAKE) cd $(DIR_APP) && ./configure --prefix=/usr --sysconfdir=/etc - cd $(DIR_APP) && make + cd $(DIR_APP) && make $(MAKETUNING) cd $(DIR_APP) && make install @rm -rf $(DIR_APP) @$(POSTBUILD) diff --git a/lfs/ipfire-netboot b/lfs/ipfire-netboot index 9536c9514..b316c9bbd 100644 --- a/lfs/ipfire-netboot +++ b/lfs/ipfire-netboot @@ -78,9 +78,9 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) # Extract iPXE source cd $(DIR_APP) && tar axf $(DIR_DL)/ipxe-$(PXE_VER).tar.gz cd $(DIR_APP) && rm -rfv ipxe && ln -s ipxe-$(PXE_VER) ipxe - cd $(DIR_APP) && make bin/ipxe.lkrn + cd $(DIR_APP) && make $(MAKETUNING) bin/ipxe.lkrn ifeq "$(BUILD_ARCH)" "x86_64" - cd $(DIR_APP) && make bin-x86_64-efi/ipxe.efi + cd $(DIR_APP) && make $(MAKETUNING) bin-x86_64-efi/ipxe.efi endif
# Installation diff --git a/lfs/krb5 b/lfs/krb5 index b1ea0f76d..5a619ab84 100644 --- a/lfs/krb5 +++ b/lfs/krb5 @@ -92,7 +92,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) --enable-dns-for-realm \ CPPFLAGS="-I/usr/include/et"
- cd $(DIR_APP) && make #$(MAKETUNING) + cd $(DIR_APP) && make $(MAKETUNING) cd $(DIR_APP) && make install
for LIB in gssapi_krb5 gssrpc k5crypto kadm5clnt kadm5srv \ diff --git a/lfs/lcd4linux b/lfs/lcd4linux index 81f994621..052c905c3 100644 --- a/lfs/lcd4linux +++ b/lfs/lcd4linux @@ -79,7 +79,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) @rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar axf $(DIR_DL)/$(DL_FILE) $(UPDATE_AUTOMAKE) cd $(DIR_APP) && ./configure --with-plugins=all,!qnaplog,!dbus --prefix=/usr - cd $(DIR_APP) && make + cd $(DIR_APP) && make $(MAKETUNING) cd $(DIR_APP) && make install
#install initscripts diff --git a/lfs/netpbm b/lfs/netpbm index e22e0fbc8..58a6921a4 100644 --- a/lfs/netpbm +++ b/lfs/netpbm @@ -81,7 +81,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) @rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar zxf $(DIR_DL)/$(DL_FILE) rm -rf /usr/netpbm cp $(DIR_SRC)/config/netpbm/config.mk $(DIR_APP) - cd $(DIR_APP) && make CFLAGS="$(CFLAGS)" + cd $(DIR_APP) && make CFLAGS="$(CFLAGS)" #The build of this version cannot be parallelized cd $(DIR_APP) && make package PKGDIR=/usr/netpbm mkdir -pv /usr/include/netpbm mkdir -pv /usr/share/netpbm diff --git a/lfs/netsnmpd b/lfs/netsnmpd index 06233f3e9..4e402dd7f 100644 --- a/lfs/netsnmpd +++ b/lfs/netsnmpd @@ -95,7 +95,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) sctp-mib rmon-mib etherlike-mib ucd-snmp/lmsensorsMib" --libdir=/usr/lib \ --sysconfdir="/etc" - cd $(DIR_APP) && make + cd $(DIR_APP) && make $(MAKETUNING) cd $(DIR_APP) && make install install -v -m 644 $(DIR_SRC)/config/netsnmpd/snmpd.conf /etc/snmpd.conf install -v -m 644 $(DIR_SRC)/config/backup/includes/netsnmpd \ diff --git a/lfs/nut b/lfs/nut index 10e146c63..c1f019274 100644 --- a/lfs/nut +++ b/lfs/nut @@ -80,7 +80,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) cd $(DIR_APP) && ./configure --prefix=/usr --sysconfdir=/etc/nut \ --with-usb --with-user=root --with-group=nut \ --with-wrap=no --with-udev-dir=/etc/udev - cd $(DIR_APP) && make + cd $(DIR_APP) && make $(MAKETUNING) cd $(DIR_APP) && make install # sed -i -e "s|ATTR{|SYSFS{|g" /etc/udev/rules.d/52-nut-usbups.rules mkdir -p /var/state/ups diff --git a/lfs/openssl b/lfs/openssl index 05d28ef20..67e7365dc 100644 --- a/lfs/openssl +++ b/lfs/openssl @@ -130,7 +130,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) $(CFLAGS) $(LDFLAGS)
cd $(DIR_APP) && make depend - cd $(DIR_APP) && make + cd $(DIR_APP) && make $(MAKETUNING)
ifeq "$(KCFG)" "-sse2" -mkdir -pv /usr/lib/sse2 diff --git a/lfs/rrdtool b/lfs/rrdtool index ef67dc26d..36d373d2c 100644 --- a/lfs/rrdtool +++ b/lfs/rrdtool @@ -78,7 +78,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) --disable-tcl \ --disable-ruby \ --disable-python - cd $(DIR_APP) && make + cd $(DIR_APP) && make $(MAKETUNING) cd $(DIR_APP) && make install -mkdir -p /srv/web/ipfire/html/graphs/ chmod 777 /srv/web/ipfire/html/graphs/ diff --git a/lfs/samba b/lfs/samba index 44f4623c0..c437793d6 100644 --- a/lfs/samba +++ b/lfs/samba @@ -180,8 +180,8 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) --enable-cups \ --disable-avahi \ --with-syslog - cd $(DIR_APP)/source3 && make idl_full - cd $(DIR_APP)/source3 && make proto && make all $(MAKETUNING) $(EXTRA_MAKE) + cd $(DIR_APP)/source3 && make $(MAKETUNING) idl_full + cd $(DIR_APP)/source3 && make $(MAKETUNING) proto && make all $(MAKETUNING) $(EXTRA_MAKE) cd $(DIR_APP)/source3 && make install cd $(DIR_APP)/source3 && chmod -v 644 /usr/include/libsmbclient.h #cd $(DIR_APP)/source3 && install -v -m755 nsswitch/libnss_wins.so /lib diff --git a/lfs/slang b/lfs/slang index d6e685855..217e74c77 100644 --- a/lfs/slang +++ b/lfs/slang @@ -71,7 +71,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) @$(PREBUILD) @rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar axf $(DIR_DL)/$(DL_FILE) cd $(DIR_APP) && ./configure --prefix=/usr --sysconfdir=/etc - cd $(DIR_APP) && make #$(MAKETUNING) + cd $(DIR_APP) && make $(MAKETUNING) cd $(DIR_APP) && make install @rm -rf $(DIR_APP) @$(POSTBUILD) diff --git a/lfs/snort b/lfs/snort index e7b5a19f7..c66a0dd1a 100644 --- a/lfs/snort +++ b/lfs/snort @@ -88,7 +88,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) --enable-react \ --enable-flexresp3
- cd $(DIR_APP) && make + cd $(DIR_APP) && make $(MAKETUNING) cd $(DIR_APP) && make install mv /usr/bin/snort /usr/sbin/ -mkdir -p /etc/snort/rules diff --git a/lfs/xfsprogs b/lfs/xfsprogs index b68e1b138..af6ccce37 100644 --- a/lfs/xfsprogs +++ b/lfs/xfsprogs @@ -73,7 +73,7 @@ $(subst %,%_MD5,$(objects)) : $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) @$(PREBUILD) @rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar axf $(DIR_DL)/$(DL_FILE) - cd $(DIR_APP) && make DEBUG=-DNDEBUG INSTALL_USER=root INSTALL_GROUP=root \ + cd $(DIR_APP) && make $(MAKETUNING) DEBUG=-DNDEBUG INSTALL_USER=root INSTALL_GROUP=root \ LOCAL_CONFIGURE_OPTIONS="--enable-readline=yes" cd $(DIR_APP) && make install install-dev cd $(DIR_APP) && install -v -m755 -D libhandle/libhandle.la /usr/lib/libhandle.la
On the most of this packages (eg. boot and cmake) i have disabled parallel build because it fails on machines with low ram (less than 2GB) or on arm. So such changes need tested on an 1GB machine, on all architectures and build two times after deleting ccache. Because the build need much lower resources if it has the file already prebuilt in cache.
-
Arne Fitzenreiter -
Wolfgang Apolinarski