[PATCH 0/1] Parallelized build for several packages

public inbox for development@lists.ipfire.org
 help / color / mirror / Atom feed

* [PATCH 0/1] Parallelized build for several packages
@ 2019-02-20 19:18 Wolfgang Apolinarski
  2019-02-20 19:18 ` [PATCH 1/1] " Wolfgang Apolinarski
  2019-02-21  8:44 ` [PATCH 0/1] " Arne Fitzenreiter
  0 siblings, 2 replies; 3+ messages in thread
From: Wolfgang Apolinarski @ 2019-02-20 19:18 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 2096 bytes --]

With this patch, I tried to optimize the build (because it really takes
a long time if I compile from scratch for testing).
Nowadays, most packages are already optimizing for parallel build. I was
testing the build on an 8 core system (i.e., MAKETUNING is -j9).

I also found out something else, the collectd package build crashes when
libstatgrab is already built:
https://github.com/collectd/collectd/issues/445

Someone else already noticed this on the list, without discovering the
cause. An upgrade of collectd would be required to solve this problem.
Since I am not sure for what collectd is actually used (and a major
upgrade would be required), I did not perform this.

Speed-up:
Package | Before | After
boost | 1:20:41 | 13:39
cyrus-imapd | 3:50 | 0:48
dnsdist | 5:23 | 1:18
gcc | 46:15 | 11:29
gettext | 9:55 | 4:56
gutenprint | 3:14 | 1:22
ipfire-netboot | 9:33 | 2:07
krb5 | 5:28 | 1:27
lcd4linux | 1:04 | 0:27
netsnmpd | 8:00 | 3:33
nut | 1:59 | 0:21
openssl | 7:57 | 2:37
rrdtool | 1:31 | 0:32
samba | 5:03 | 0:55
slang | 1:04 | 0:19
snort | 5:11 | 0:56
xfsprogs | 5:41 | 1:46

I also marked packages that (in their current version) crash when being
built in parallel. I performed multiple builds (and tested the ISO
image), so I hope that this patch is reliable for everyone.
Nevertheless, please test the patch on different build environments
before applying it.

Wolfgang Apolinarski (1):
  Parallelized build for several packages

 lfs/boost          | 2 +-
 lfs/collectd       | 2 +-
 lfs/cyrus-imapd    | 2 +-
 lfs/dhcp           | 2 +-
 lfs/dnsdist        | 2 +-
 lfs/gcc            | 4 ++--
 lfs/gettext        | 8 ++++----
 lfs/groff          | 2 +-
 lfs/gutenprint     | 2 +-
 lfs/ipfire-netboot | 4 ++--
 lfs/krb5           | 2 +-
 lfs/lcd4linux      | 2 +-
 lfs/netpbm         | 2 +-
 lfs/netsnmpd       | 2 +-
 lfs/nut            | 2 +-
 lfs/openssl        | 2 +-
 lfs/rrdtool        | 2 +-
 lfs/samba          | 4 ++--
 lfs/slang          | 2 +-
 lfs/snort          | 2 +-
 lfs/xfsprogs       | 2 +-
 21 files changed, 27 insertions(+), 27 deletions(-)

-- 
2.19.2

^ permalink raw reply	[flat|nested] 3+ messages in thread

* [PATCH 1/1] Parallelized build for several packages
  2019-02-20 19:18 [PATCH 0/1] Parallelized build for several packages Wolfgang Apolinarski
@ 2019-02-20 19:18 ` Wolfgang Apolinarski
  2019-02-21  8:44 ` [PATCH 0/1] " Arne Fitzenreiter
  1 sibling, 0 replies; 3+ messages in thread
From: Wolfgang Apolinarski @ 2019-02-20 19:18 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 11859 bytes --]

Added $(MAKETUNING) to several packages.
Marked packages that do not support parallel build.
---
 lfs/boost          | 2 +-
 lfs/collectd       | 2 +-
 lfs/cyrus-imapd    | 2 +-
 lfs/dhcp           | 2 +-
 lfs/dnsdist        | 2 +-
 lfs/gcc            | 4 ++--
 lfs/gettext        | 8 ++++----
 lfs/groff          | 2 +-
 lfs/gutenprint     | 2 +-
 lfs/ipfire-netboot | 4 ++--
 lfs/krb5           | 2 +-
 lfs/lcd4linux      | 2 +-
 lfs/netpbm         | 2 +-
 lfs/netsnmpd       | 2 +-
 lfs/nut            | 2 +-
 lfs/openssl        | 2 +-
 lfs/rrdtool        | 2 +-
 lfs/samba          | 4 ++--
 lfs/slang          | 2 +-
 lfs/snort          | 2 +-
 lfs/xfsprogs       | 2 +-
 21 files changed, 27 insertions(+), 27 deletions(-)

diff --git a/lfs/boost b/lfs/boost
index bed2b0cdb..3ce49a0a9 100644
--- a/lfs/boost
+++ b/lfs/boost
@@ -95,7 +95,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
 	@rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar axf $(DIR_DL)/$(DL_FILE)
 
 	cd $(DIR_APP) && ./bootstrap.sh --with-toolset=gcc
-	cd $(DIR_APP) && ./b2 -d+2 -q $(CONFIGURE_OPTIONS) stage
+	cd $(DIR_APP) && ./b2 -d+2 -q $(MAKETUNING) $(CONFIGURE_OPTIONS) stage
 	cd $(DIR_APP) && ./b2 $(CONFIGURE_OPTIONS) install
 
 	@rm -rf $(DIR_APP)
diff --git a/lfs/collectd b/lfs/collectd
index 9592ec95a..e31324817 100644
--- a/lfs/collectd
+++ b/lfs/collectd
@@ -116,7 +116,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
 		--enable-{network,nfs,ntpd,ping,processes,rrdtool,sensors,swap,syslog} \
 		--enable-{tcpconns,unixsock,users,wireless} \
 		--with-librrd=/usr/share/rrdtool-1.2.30
-	cd $(DIR_APP) && make install
+	cd $(DIR_APP) && make install #collectd-4 does not support parallel build
 	cp -vf $(DIR_SRC)/config/collectd/collectd.* /etc/
 	mv /etc/collectd.vpn /var/ipfire/ovpn/collectd.vpn
 	chown nobody.nobody /var/ipfire/ovpn/collectd.vpn
diff --git a/lfs/cyrus-imapd b/lfs/cyrus-imapd
index 1800a08ec..50cd66875 100644
--- a/lfs/cyrus-imapd
+++ b/lfs/cyrus-imapd
@@ -89,7 +89,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
 		--enable-autocreate \
 		--enable-idled
 
-	cd $(DIR_APP) && make $(EXTRA_MAKE)
+	cd $(DIR_APP) && make $(MAKETUNING) $(EXTRA_MAKE)
 	cd $(DIR_APP) && make install
 	-mkdir /var/imap
 	chown cyrus:mail /var/imap
diff --git a/lfs/dhcp b/lfs/dhcp
index a055d6081..4c01428f5 100644
--- a/lfs/dhcp
+++ b/lfs/dhcp
@@ -84,7 +84,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
 			--enable-early-chroot \
 			--disable-dhcpv6
 
-	cd $(DIR_APP) && make
+	cd $(DIR_APP) && make #This package does not support parallel build
 	cd $(DIR_APP) && make install
 
 	mkdir -pv /var/state/dhcp
diff --git a/lfs/dnsdist b/lfs/dnsdist
index 85a841fa5..e748004ef 100644
--- a/lfs/dnsdist
+++ b/lfs/dnsdist
@@ -79,7 +79,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
 	@$(PREBUILD)
 	@rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar axf $(DIR_DL)/$(DL_FILE)
 	cd $(DIR_APP) && ./configure --prefix=/usr --sysconfdir=/etc
-	cd $(DIR_APP) && make
+	cd $(DIR_APP) && make $(MAKETUNING)
 	cd $(DIR_APP) && make install
 
 	#install initscripts
diff --git a/lfs/gcc b/lfs/gcc
index 6fdf2ea31..5aac2d382 100644
--- a/lfs/gcc
+++ b/lfs/gcc
@@ -264,7 +264,7 @@ ifeq "$(PASS)" "L"
 		$(EXTRA_ENV) \
 		$(DIR_APP)/libstdc++-v3/configure \
 			$(EXTRA_CONFIG)
-	cd $(DIR_SRC)/gcc-build && make $(EXTRA_MAKE)
+	cd $(DIR_SRC)/gcc-build && make $(MAKETUNING) $(EXTRA_MAKE)
 	cd $(DIR_SRC)/gcc-build && make $(EXTRA_INSTALL) install
 
 else
@@ -273,7 +273,7 @@ else
 		$(EXTRA_ENV) \
 		$(DIR_APP)/configure \
 			$(EXTRA_CONFIG)
-	cd $(DIR_SRC)/gcc-build && make $(EXTRA_MAKE)
+	cd $(DIR_SRC)/gcc-build && make $(MAKETUNING) $(EXTRA_MAKE)
 	cd $(DIR_SRC)/gcc-build && make $(EXTRA_INSTALL) install
 endif
 
diff --git a/lfs/gettext b/lfs/gettext
index 632ecac43..b1d75ed2d 100644
--- a/lfs/gettext
+++ b/lfs/gettext
@@ -81,13 +81,13 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
 	@rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar axf $(DIR_DL)/$(DL_FILE)
 ifeq "$(ROOT)" ""
 	cd $(DIR_APP) && ./configure $(EXTRA_CONFIG)
-	cd $(DIR_APP) && make
+	cd $(DIR_APP) && make $(MAKETUNING)
 	cd $(DIR_APP) && make install
 else
 	cd $(DIR_APP)/gettext-tools && ./configure $(EXTRA_CONFIG)
-	cd $(DIR_APP)/gettext-tools && make -C gnulib-lib
-	cd $(DIR_APP)/gettext-tools && make -C intl pluralx.c
-	cd $(DIR_APP)/gettext-tools && make -C src msgfmt
+	cd $(DIR_APP)/gettext-tools && make $(MAKETUNING) -C gnulib-lib
+	cd $(DIR_APP)/gettext-tools && make $(MAKETUNING) -C intl pluralx.c
+	cd $(DIR_APP)/gettext-tools && make $(MAKETUNING) -C src msgfmt
 	cd $(DIR_APP)/gettext-tools && cp -v src/msgfmt $(TOOLS_DIR)/bin
 endif
 	@rm -rf $(DIR_APP)
diff --git a/lfs/groff b/lfs/groff
index 4d5bf6373..1ed393c62 100644
--- a/lfs/groff
+++ b/lfs/groff
@@ -73,7 +73,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
 	cd $(DIR_APP) && PAGE=A4 ./configure \
 		--prefix=/usr
 
-	cd $(DIR_APP) && make
+	cd $(DIR_APP) && make #This package does not support parallel build
 	cd $(DIR_APP) && make install
 	ln -svf eqn /usr/bin/geqn
 	ln -svf tbl /usr/bin/gtbl
diff --git a/lfs/gutenprint b/lfs/gutenprint
index 7286b78a2..0a18aa73b 100644
--- a/lfs/gutenprint
+++ b/lfs/gutenprint
@@ -79,7 +79,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
 	@rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar jxf $(DIR_DL)/$(DL_FILE)
 	$(UPDATE_AUTOMAKE)
 	cd $(DIR_APP) && ./configure --prefix=/usr --sysconfdir=/etc
-	cd $(DIR_APP) && make
+	cd $(DIR_APP) && make $(MAKETUNING)
 	cd $(DIR_APP) && make install
 	@rm -rf $(DIR_APP)
 	@$(POSTBUILD)
diff --git a/lfs/ipfire-netboot b/lfs/ipfire-netboot
index 9536c9514..b316c9bbd 100644
--- a/lfs/ipfire-netboot
+++ b/lfs/ipfire-netboot
@@ -78,9 +78,9 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
 	# Extract iPXE source
 	cd $(DIR_APP) && tar axf $(DIR_DL)/ipxe-$(PXE_VER).tar.gz
 	cd $(DIR_APP) && rm -rfv ipxe && ln -s ipxe-$(PXE_VER) ipxe
-	cd $(DIR_APP) && make bin/ipxe.lkrn
+	cd $(DIR_APP) && make $(MAKETUNING) bin/ipxe.lkrn
 ifeq "$(BUILD_ARCH)" "x86_64"
-	cd $(DIR_APP) && make bin-x86_64-efi/ipxe.efi
+	cd $(DIR_APP) && make $(MAKETUNING) bin-x86_64-efi/ipxe.efi
 endif
 
 	# Installation
diff --git a/lfs/krb5 b/lfs/krb5
index b1ea0f76d..5a619ab84 100644
--- a/lfs/krb5
+++ b/lfs/krb5
@@ -92,7 +92,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
 		--enable-dns-for-realm \
 		CPPFLAGS="-I/usr/include/et"
 
-	cd $(DIR_APP) && make #$(MAKETUNING)
+	cd $(DIR_APP) && make $(MAKETUNING)
 	cd $(DIR_APP) && make install
 
 	for LIB in gssapi_krb5 gssrpc k5crypto kadm5clnt kadm5srv \
diff --git a/lfs/lcd4linux b/lfs/lcd4linux
index 81f994621..052c905c3 100644
--- a/lfs/lcd4linux
+++ b/lfs/lcd4linux
@@ -79,7 +79,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
 	@rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar axf $(DIR_DL)/$(DL_FILE)
 	$(UPDATE_AUTOMAKE)
 	cd $(DIR_APP) && ./configure  --with-plugins=all,!qnaplog,!dbus --prefix=/usr
-	cd $(DIR_APP) && make
+	cd $(DIR_APP) && make $(MAKETUNING)
 	cd $(DIR_APP) && make install
 
 	#install initscripts
diff --git a/lfs/netpbm b/lfs/netpbm
index e22e0fbc8..58a6921a4 100644
--- a/lfs/netpbm
+++ b/lfs/netpbm
@@ -81,7 +81,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
 	@rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar zxf $(DIR_DL)/$(DL_FILE)
 	rm -rf /usr/netpbm
 	cp $(DIR_SRC)/config/netpbm/config.mk $(DIR_APP)
-	cd $(DIR_APP) && make CFLAGS="$(CFLAGS)"
+	cd $(DIR_APP) && make CFLAGS="$(CFLAGS)" #The build of this version cannot be parallelized
 	cd $(DIR_APP) && make package PKGDIR=/usr/netpbm
 	mkdir -pv /usr/include/netpbm 
 	mkdir -pv /usr/share/netpbm
diff --git a/lfs/netsnmpd b/lfs/netsnmpd
index 06233f3e9..4e402dd7f 100644
--- a/lfs/netsnmpd
+++ b/lfs/netsnmpd
@@ -95,7 +95,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
 		sctp-mib rmon-mib etherlike-mib ucd-snmp/lmsensorsMib"
 		--libdir=/usr/lib \
 		--sysconfdir="/etc"
-	cd $(DIR_APP) && make 
+	cd $(DIR_APP) && make $(MAKETUNING)
 	cd $(DIR_APP) && make install
 	install -v -m 644 $(DIR_SRC)/config/netsnmpd/snmpd.conf /etc/snmpd.conf
 	install -v -m 644 $(DIR_SRC)/config/backup/includes/netsnmpd \
diff --git a/lfs/nut b/lfs/nut
index 10e146c63..c1f019274 100644
--- a/lfs/nut
+++ b/lfs/nut
@@ -80,7 +80,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
 	cd $(DIR_APP) && ./configure --prefix=/usr --sysconfdir=/etc/nut \
 			--with-usb --with-user=root --with-group=nut \
 			--with-wrap=no --with-udev-dir=/etc/udev
-	cd $(DIR_APP) && make
+	cd $(DIR_APP) && make $(MAKETUNING)
 	cd $(DIR_APP) && make install
 #	sed -i -e "s|ATTR{|SYSFS{|g" /etc/udev/rules.d/52-nut-usbups.rules
 	mkdir -p /var/state/ups
diff --git a/lfs/openssl b/lfs/openssl
index 05d28ef20..67e7365dc 100644
--- a/lfs/openssl
+++ b/lfs/openssl
@@ -130,7 +130,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
 		$(CFLAGS) $(LDFLAGS)
 
 	cd $(DIR_APP) && make depend
-	cd $(DIR_APP) && make
+	cd $(DIR_APP) && make $(MAKETUNING)
 
 ifeq "$(KCFG)" "-sse2"
 	-mkdir -pv /usr/lib/sse2
diff --git a/lfs/rrdtool b/lfs/rrdtool
index ef67dc26d..36d373d2c 100644
--- a/lfs/rrdtool
+++ b/lfs/rrdtool
@@ -78,7 +78,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
 		--disable-tcl \
 		--disable-ruby \
 		--disable-python
-	cd $(DIR_APP) && make
+	cd $(DIR_APP) && make $(MAKETUNING)
 	cd $(DIR_APP) && make install
 	-mkdir -p /srv/web/ipfire/html/graphs/
 	chmod 777 /srv/web/ipfire/html/graphs/
diff --git a/lfs/samba b/lfs/samba
index 44f4623c0..c437793d6 100644
--- a/lfs/samba
+++ b/lfs/samba
@@ -180,8 +180,8 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
 		--enable-cups \
 		--disable-avahi \
 		--with-syslog
-	cd $(DIR_APP)/source3 && make idl_full
-	cd $(DIR_APP)/source3 && make proto && make all $(MAKETUNING) $(EXTRA_MAKE)
+	cd $(DIR_APP)/source3 && make $(MAKETUNING) idl_full
+	cd $(DIR_APP)/source3 && make $(MAKETUNING) proto && make all $(MAKETUNING) $(EXTRA_MAKE)
 	cd $(DIR_APP)/source3 && make install
 	cd $(DIR_APP)/source3 && chmod -v 644 /usr/include/libsmbclient.h
 	#cd $(DIR_APP)/source3 && install -v -m755 nsswitch/libnss_wins.so /lib
diff --git a/lfs/slang b/lfs/slang
index d6e685855..217e74c77 100644
--- a/lfs/slang
+++ b/lfs/slang
@@ -71,7 +71,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
 	@$(PREBUILD)
 	@rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar axf $(DIR_DL)/$(DL_FILE)
 	cd $(DIR_APP) && ./configure --prefix=/usr --sysconfdir=/etc
-	cd $(DIR_APP) && make #$(MAKETUNING)
+	cd $(DIR_APP) && make $(MAKETUNING)
 	cd $(DIR_APP) && make install
 	@rm -rf $(DIR_APP)
 	@$(POSTBUILD)
diff --git a/lfs/snort b/lfs/snort
index e7b5a19f7..c66a0dd1a 100644
--- a/lfs/snort
+++ b/lfs/snort
@@ -88,7 +88,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
 		--enable-react \
 		--enable-flexresp3
 
-	cd $(DIR_APP) && make
+	cd $(DIR_APP) && make $(MAKETUNING)
 	cd $(DIR_APP) && make install
 	mv /usr/bin/snort /usr/sbin/
 	-mkdir -p /etc/snort/rules
diff --git a/lfs/xfsprogs b/lfs/xfsprogs
index b68e1b138..af6ccce37 100644
--- a/lfs/xfsprogs
+++ b/lfs/xfsprogs
@@ -73,7 +73,7 @@ $(subst %,%_MD5,$(objects)) :
 $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
 	@$(PREBUILD)
 	@rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar axf $(DIR_DL)/$(DL_FILE)
-	cd $(DIR_APP) && make DEBUG=-DNDEBUG INSTALL_USER=root INSTALL_GROUP=root \
+	cd $(DIR_APP) && make $(MAKETUNING) DEBUG=-DNDEBUG INSTALL_USER=root INSTALL_GROUP=root \
 		LOCAL_CONFIGURE_OPTIONS="--enable-readline=yes"
 	cd $(DIR_APP) && make install install-dev
 	cd $(DIR_APP) && install -v -m755 -D libhandle/libhandle.la /usr/lib/libhandle.la
-- 
2.19.2


^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [PATCH 0/1] Parallelized build for several packages
  2019-02-20 19:18 [PATCH 0/1] Parallelized build for several packages Wolfgang Apolinarski
  2019-02-20 19:18 ` [PATCH 1/1] " Wolfgang Apolinarski
@ 2019-02-21  8:44 ` Arne Fitzenreiter
  1 sibling, 0 replies; 3+ messages in thread
From: Arne Fitzenreiter @ 2019-02-21  8:44 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 355 bytes --]

On the most of this packages (eg. boot and cmake) i have disabled 
parallel build because it fails on machines with low ram (less than 2GB) 
or on arm.
So such changes need tested on an 1GB machine, on all architectures and 
build two times after deleting ccache. Because the build need much lower 
resources if it has the file already prebuilt in cache.

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2019-02-21  8:44 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-02-20 19:18 [PATCH 0/1] Parallelized build for several packages Wolfgang Apolinarski
2019-02-20 19:18 ` [PATCH 1/1] " Wolfgang Apolinarski
2019-02-21  8:44 ` [PATCH 0/1] " Arne Fitzenreiter

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox