Reducing I/O of the build process

public inbox for development@lists.ipfire.org
 help / color / mirror / Atom feed

* Reducing I/O of the build process
@ 2020-03-10 13:26 Michael Tremer
  2020-03-10 13:26 ` [PATCH 1/4] make.sh: Make /tmp a ramdisk if ramdisks are enabled Michael Tremer
                   ` (3 more replies)
  0 siblings, 4 replies; 6+ messages in thread
From: Michael Tremer @ 2020-03-10 13:26 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 908 bytes --]

Hello,

I have noticed that when packing the image, we put quite a lot of
pressure on the disks. Our main build VMs become quite slow and even
slow down other machines like the web servers.

To avoid that, I have made a few changes:

/tmp is now a ramdisk and ccache is putting temporary files in there.

This helps a slot with smaller files that are written to disk or at
least the journal.

When the OS tarball for the CDROM is generated, this is now done
in memory and the OS is only written to disk again once. Saves us
again a gigabyte or so in writes.

All other temporary files are moved into /tmp to take advantage of
the ramdisk.

I do not expect any (significant) changes in build time, but if
anything, it could only have gone up. There is no extra memory
needed and the ramdisk is only enabled when enough memory is
available.

But please let me know if you notice any changes.

Best,
-Michael

^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH 1/4] make.sh: Make /tmp a ramdisk if ramdisks are enabled
  2020-03-10 13:26 Reducing I/O of the build process Michael Tremer
@ 2020-03-10 13:26 ` Michael Tremer
  2020-03-10 13:28   ` [PATCH] make.sh: Umount /tmp when it is a ramdisk Michael Tremer
  2020-03-10 13:26 ` [PATCH 2/4] cdrom: Do not write the temporary tarball to disk Michael Tremer
                   ` (2 subsequent siblings)
  3 siblings, 1 reply; 6+ messages in thread
From: Michael Tremer @ 2020-03-10 13:26 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 835 bytes --]

The build system is writing a large amount of temporary file
systems that might land on disk or at least in the journal.

This change will speed up the build and remove a lot of I/O
usage.

Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org>
---
 make.sh | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/make.sh b/make.sh
index 984fc95b2..188355a0d 100755
--- a/make.sh
+++ b/make.sh
@@ -476,6 +476,9 @@ prepareenv() {
 	if [ "${ENABLE_RAMDISK}" = "on" ]; then
 		mkdir -p $BASEDIR/build/usr/src
 		mount -t tmpfs tmpfs -o size=4G,nr_inodes=1M,mode=1777 $BASEDIR/build/usr/src
+
+		mkdir -p ${BASEDIR}/build/tmp
+		mount -t tmpfs tmpfs -o size=4G,nr_inodes=1M,mode=1777 ${BASEDIR}/build/tmp
 	fi
 
 	mkdir -p $BASEDIR/build/usr/src/{cache,config,doc,html,langs,lfs,log,src,ccache}
-- 
2.20.1


^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH] make.sh: Umount /tmp when it is a ramdisk
  2020-03-10 13:26 ` [PATCH 1/4] make.sh: Make /tmp a ramdisk if ramdisks are enabled Michael Tremer
@ 2020-03-10 13:28   ` Michael Tremer
  0 siblings, 0 replies; 6+ messages in thread
From: Michael Tremer @ 2020-03-10 13:28 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 453 bytes --]

Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org>
---
 make.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/make.sh b/make.sh
index 9c88592da..756b27f7a 100755
--- a/make.sh
+++ b/make.sh
@@ -285,6 +285,7 @@ stdumount() {
 	umount $BASEDIR/build/usr/src/log		2>/dev/null;
 	umount $BASEDIR/build/usr/src/src		2>/dev/null;
 	umount $BASEDIR/build/usr/src		2>/dev/null;
+	umount $BASEDIR/build/tmp		2>/dev/null;
 }
 
 now() {
-- 
2.20.1


^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH 2/4] cdrom: Do not write the temporary tarball to disk
  2020-03-10 13:26 Reducing I/O of the build process Michael Tremer
  2020-03-10 13:26 ` [PATCH 1/4] make.sh: Make /tmp a ramdisk if ramdisks are enabled Michael Tremer
@ 2020-03-10 13:26 ` Michael Tremer
  2020-03-10 13:26 ` [PATCH 3/4] cdrom+flash-image: Move all temporary files to /tmp Michael Tremer
  2020-03-10 13:26 ` [PATCH 4/4] make.sh: Move ccache's temp directory into /tmp Michael Tremer
  3 siblings, 0 replies; 6+ messages in thread
From: Michael Tremer @ 2020-03-10 13:26 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 2295 bytes --]

In order to remove any duplicate files in the tarball, we
have to unpack it again. The whole filesystem is written
to disk twice which is unnecessary.

This patch removes that temporary step and reduces IO
during the build.

Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org>
---
 lfs/cdrom | 23 +++++++++++++----------
 1 file changed, 13 insertions(+), 10 deletions(-)

diff --git a/lfs/cdrom b/lfs/cdrom
index b26db754b..836cae981 100644
--- a/lfs/cdrom
+++ b/lfs/cdrom
@@ -29,6 +29,9 @@ VER = ipfire
 THISAPP    = cdrom
 TARGET     = $(DIR_INFO)/$(THISAPP)
 
+# Fail when there is an error in the tar pipe
+SHELL=/bin/bash -o pipefail
+
 ifeq "$(BUILD_PLATFORM)" "arm"
 	TAR_OPTIONS =
 else
@@ -147,7 +150,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
 	# since stage2 has been executed..
 	echo "$(SYSTEM_RELEASE)" > /etc/system-release
 
-	rm -rf /install/cdrom /tmp/*
+	rm -rf /install/cdrom
 	mkdir -p /install/cdrom/doc
 
 	# Clear mtab (prevents .journal problems)
@@ -158,18 +161,18 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
 	BUILDTARGET="$(BUILDTARGET)" BUILD_ARCH="$(BUILD_ARCH)" KVER="$(KVER)" \
 		$(DIR_SRC)/src/scripts/archive.files \
 		$(DIR_SRC)/config/rootfiles/common \
-		> /tmp/ROOTFILES
+		> $(DIR_TMP)/ROOTFILES
 
 	# Compress root filesystem
 	# Reason for this tar+untar+tar is removing of entries listed two or more in src/ROOTFILES
-	tar -c --exclude='#*' --exclude='proc/*' --exclude='dev/pts/*' \
-		--exclude='__pycache__' --exclude='tmp/ROOTFILES' \
-		-C / --files-from=/tmp/ROOTFILES -f /$(SNAME).tar
-	rm -f /tmp/ROOTFILES
-	tar -x -C /tmp -f /$(SNAME).tar
-	rm -f /$(SNAME).tar
-	@mkdir /tmp/sys
-	cd /tmp && tar cf - * | xz $(XZ_OPT) > /install/cdrom/distro.img && rm -rf *
+	rm -rf $(DIR_TMP)/root && mkdir -p $(DIR_TMP)/root
+	tar -c --exclude='#*' --exclude='proc/*' --exclude='dev/pts/*' --exclude='tmp/*' \
+		--exclude='__pycache__' \
+		-C / --files-from=$(DIR_TMP)/ROOTFILES | tar -x -C $(DIR_TMP)/root
+	rm -f $(DIR_TMP)/ROOTFILES
+	mkdir $(DIR_TMP)/root/sys
+	cd $(DIR_TMP)/root && tar cf - * | xz $(XZ_OPT) > /install/cdrom/distro.img
+	rm -rf $(DIR_TMP)/root
 
 	# Other files
 	touch /install/cdrom/$(SNAME)-$(VERSION)-core$(CORE).media
-- 
2.20.1


^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH 3/4] cdrom+flash-image: Move all temporary files to /tmp
  2020-03-10 13:26 Reducing I/O of the build process Michael Tremer
  2020-03-10 13:26 ` [PATCH 1/4] make.sh: Make /tmp a ramdisk if ramdisks are enabled Michael Tremer
  2020-03-10 13:26 ` [PATCH 2/4] cdrom: Do not write the temporary tarball to disk Michael Tremer
@ 2020-03-10 13:26 ` Michael Tremer
  2020-03-10 13:26 ` [PATCH 4/4] make.sh: Move ccache's temp directory into /tmp Michael Tremer
  3 siblings, 0 replies; 6+ messages in thread
From: Michael Tremer @ 2020-03-10 13:26 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 7401 bytes --]

Since /tmp is now a ramdisk, we move all temporary files into it.

Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org>
---
 lfs/cdrom        | 71 +++++++++++++++++++++++++-----------------------
 lfs/flash-images |  8 +++---
 2 files changed, 41 insertions(+), 38 deletions(-)

diff --git a/lfs/cdrom b/lfs/cdrom
index 836cae981..f18b80a03 100644
--- a/lfs/cdrom
+++ b/lfs/cdrom
@@ -150,8 +150,8 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
 	# since stage2 has been executed..
 	echo "$(SYSTEM_RELEASE)" > /etc/system-release
 
-	rm -rf /install/cdrom
-	mkdir -p /install/cdrom/doc
+	# Create a directory to authorise the CDROM in
+	rm -rf $(DIR_TMP)/cdrom && mkdir -p $(DIR_TMP)/cdrom
 
 	# Clear mtab (prevents .journal problems)
 	rm -vf /etc/mtab
@@ -171,40 +171,43 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
 		-C / --files-from=$(DIR_TMP)/ROOTFILES | tar -x -C $(DIR_TMP)/root
 	rm -f $(DIR_TMP)/ROOTFILES
 	mkdir $(DIR_TMP)/root/sys
-	cd $(DIR_TMP)/root && tar cf - * | xz $(XZ_OPT) > /install/cdrom/distro.img
+	cd $(DIR_TMP)/root && tar cf - * | xz $(XZ_OPT) > $(DIR_TMP)/cdrom/distro.img
 	rm -rf $(DIR_TMP)/root
 
 	# Other files
-	touch /install/cdrom/$(SNAME)-$(VERSION)-core$(CORE).media
-	sed 's/VERSION/$(VERSION)/' $(DIR_SRC)/config/cdrom/README.txt > /install/cdrom/README.txt
-	cp $(DIR_SRC)/doc/COPYING				/install/cdrom/
-	cp $(DIR_SRC)/doc/{ChangeLog,packages-list.txt}	/install/cdrom/doc
+	touch $(DIR_TMP)/cdrom/$(SNAME)-$(VERSION)-core$(CORE).media
+	sed 's/VERSION/$(VERSION)/' $(DIR_SRC)/config/cdrom/README.txt > $(DIR_TMP)/cdrom/README.txt
+	cp $(DIR_SRC)/doc/COPYING $(DIR_TMP)/cdrom/
 
-	mkdir -p /install/cdrom/boot/isolinux
+	# Install documentation
+	-mkdir -pv $(DIR_TMP)/cdrom/doc
+	cp $(DIR_SRC)/doc/{ChangeLog,packages-list.txt}	$(DIR_TMP)/cdrom/doc
+
+	mkdir -p $(DIR_TMP)/cdrom/boot/isolinux
 
 ifeq "$(HAS_KERNEL)" "1"
-	cp /boot/vmlinuz-$(KVER)-ipfire             /install/cdrom/boot/isolinux/vmlinuz
-	dracut --force --early-microcode -a "installer" --strip --xz /install/cdrom/boot/isolinux/instroot $(KVER)-ipfire
+	cp /boot/vmlinuz-$(KVER)-ipfire $(DIR_TMP)/cdrom/boot/isolinux/vmlinuz
+	dracut --force --early-microcode -a "installer" --strip --xz $(DIR_TMP)/cdrom/boot/isolinux/instroot $(KVER)-ipfire
 endif
 
 ifeq "$(HAS_ISOLINUX)" "1"
-	dd if=/dev/zero  bs=1k count=2            > /install/cdrom/boot/isolinux/boot.catalog
-	cp $(DIR_SRC)/config/syslinux/boot.png      /install/cdrom/boot/isolinux/boot.png
-	cp /usr/share/syslinux/isolinux.bin         /install/cdrom/boot/isolinux/isolinux.bin
-	cp /usr/share/hwdata/pci.ids                /install/cdrom/boot/isolinux/pci.ids
-	cp -vf /usr/share/syslinux/*.c32            /install/cdrom/boot/isolinux/
+	dd if=/dev/zero  bs=1k count=2            > $(DIR_TMP)/cdrom/boot/isolinux/boot.catalog
+	cp $(DIR_SRC)/config/syslinux/boot.png      $(DIR_TMP)/cdrom/boot/isolinux/boot.png
+	cp /usr/share/syslinux/isolinux.bin         $(DIR_TMP)/cdrom/boot/isolinux/isolinux.bin
+	cp /usr/share/hwdata/pci.ids                $(DIR_TMP)/cdrom/boot/isolinux/pci.ids
+	cp -vf /usr/share/syslinux/*.c32            $(DIR_TMP)/cdrom/boot/isolinux/
 	sed -e "s/VERSION/$(VERSION) - Core $(CORE)/g" \
 		$(DIR_SRC)/config/syslinux/syslinux.cfg \
-		> /install/cdrom/boot/isolinux/isolinux.cfg
+		> $(DIR_TMP)/cdrom/boot/isolinux/isolinux.cfg
 endif
 
 ifeq "$(HAS_MEMTEST)" "1"
 	# Install memtest
-	cp /usr/lib/memtest86+/memtest.bin          /install/cdrom/boot/isolinux/memtest
+	cp /usr/lib/memtest86+/memtest.bin $(DIR_TMP)/cdrom/boot/isolinux/memtest
 endif
 
 ifeq "$(HAS_IPXE)" "1"
-	cp /usr/share/ipfire-netboot/ipxe.lkrn      /install/cdrom/boot/isolinux/netboot
+	cp /usr/share/ipfire-netboot/ipxe.lkrn $(DIR_TMP)/cdrom/boot/isolinux/netboot
 endif
 
 ifeq "$(EFI)" "1"
@@ -213,43 +216,43 @@ ifeq "$(EFI)" "1"
 		$(DIR_SRC)/config/cdrom/grub-efi.cfg > /tmp/grub-efi.cfg
 
 	# Build a GRUB EFI image
-	mkdir -pv /install/cdrom/EFI/BOOT
+	mkdir -pv $(DIR_TMP)/cdrom/EFI/BOOT
 	grub-mkimage \
 		--format=$(GRUB_ARCH)-efi \
-		--output=/install/cdrom/EFI/BOOT/boot$(EFI_ARCH).efi \
-		--config=/tmp/grub-efi.cfg \
+		--output=$(DIR_TMP)/cdrom/EFI/BOOT/boot$(EFI_ARCH).efi \
+		--config=$(DIR_TMP)/grub-efi.cfg \
 		--compression=xz \
 		--prefix=/EFI/BOOT \
 		$$(for mod in $(GRUB_EFI_MODULES); do [ -f "/usr/lib/grub/$(GRUB_ARCH)-efi/$${mod}.mod" ] && echo "$${mod}"; done)
 
 	# Install GRUB configuration
-	mkdir -pv /install/cdrom/EFI/BOOT
+	mkdir -pv $(DIR_TMP)/cdrom/EFI/BOOT
 	sed -e "s/NAME/$(NAME)/g" -e "s/VERSION/$(VERSION)/g" -e "s/ARCH/$(BUILD_ARCH)/g" \
-		< $(DIR_SRC)/config/cdrom/grub.cfg > /install/cdrom/EFI/BOOT/grub.cfg
+		< $(DIR_SRC)/config/cdrom/grub.cfg > $(DIR_TMP)/cdrom/EFI/BOOT/grub.cfg
 
 	# Create the EFI Eltorito image
-	dd if=/dev/zero of=/install/cdrom/boot/isolinux/efiboot.img bs=1k count=1440
-	mkdosfs -F 12 -n "IPFIRE_EFI" /install/cdrom/boot/isolinux/efiboot.img
+	dd if=/dev/zero of=$(DIR_TMP)/cdrom/boot/isolinux/efiboot.img bs=1k count=1440
+	mkdosfs -F 12 -n "IPFIRE_EFI" $(DIR_TMP)/cdrom/boot/isolinux/efiboot.img
 
 	# Mount the EFI image
-	mkdir -pv /install/efiboot.img
-	mount -o loop /install/cdrom/boot/isolinux/efiboot.img /install/efiboot.img
+	mkdir -pv $(DIR_TMP)/efiboot.img
+	mount -o loop $(DIR_TMP)/cdrom/boot/isolinux/efiboot.img $(DIR_TMP)/efiboot.img
 
 	# Copy the bootloader into the image
-	mkdir -pv /install/efiboot.img/EFI/BOOT
-	cp -a /install/cdrom/EFI/BOOT/boot$(EFI_ARCH).efi \
-		/install/efiboot.img/EFI/BOOT/boot$(EFI_ARCH).efi
+	mkdir -p $(DIR_TMP)/efiboot.img/EFI/BOOT
+	cp -a $(DIR_TMP)/cdrom/EFI/BOOT/boot$(EFI_ARCH).efi \
+		$(DIR_TMP)/efiboot.img/EFI/BOOT/boot$(EFI_ARCH).efi
 
 	# Cleanup
-	umount /install/efiboot.img
-	rm -rf /install/efiboot.img /tmp/grub-efi.cfg
+	umount $(DIR_TMP)/efiboot.img
+	rm -rf $(DIR_TMP)/efiboot.img $(DIR_TMP)/grub-efi.cfg
 endif
 
-	cd /install/cdrom && find -type f ! -name md5sum.txt | grep -v "./boot" | \
+	cd $(DIR_TMP)/cdrom && find -type f ! -name md5sum.txt | grep -v "./boot" | \
 						xargs md5sum > md5sum.txt
 
 	mkdir -p /install/images
-	cd /install/cdrom && mkisofs $(ISO_ARGS) \
+	cd $(DIR_TMP)/cdrom && mkisofs $(ISO_ARGS) \
 		-o /install/images/$(SNAME)-$(VERSION).$(BUILD_ARCH)-full-core$(CORE).iso .
 
 ifeq "$(HAS_ISOLINUX)" "1"
diff --git a/lfs/flash-images b/lfs/flash-images
index e4af896f1..422acac22 100644
--- a/lfs/flash-images
+++ b/lfs/flash-images
@@ -55,8 +55,8 @@ md5 :
 ###############################################################################
 # Installation Details
 ###############################################################################
-MNThdd   := /install/harddisk
-IMG      := /install/images/image.img
+MNThdd   := $(DIR_TMP)/harddisk
+IMG      := $(DIR_TMP)/image.img
 
 # All sizes in blocks
 ifeq "$(BUILD_PLATFORM)" "arm"
@@ -152,9 +152,9 @@ endif
 
 	# Install IPFire
 ifneq "$(BUILD_PLATFORM)" "arm"
-	tar -x --xz -C $(MNThdd)/ -f /install/cdrom/distro.img
+	tar -x --xz -C $(MNThdd)/ -f $(DIR_TMP)/cdrom/distro.img
 else
-	tar -x -C $(MNThdd)/ -f /install/cdrom/distro.img
+	tar -x -C $(MNThdd)/ -f $(DIR_TMP)/cdrom/distro.img
 endif
 	-touch $(MNThdd)/lib/modules/$(KVER)-ipfire/modules.dep
 	mkdir $(MNThdd)/proc
-- 
2.20.1


^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH 4/4] make.sh: Move ccache's temp directory into /tmp
  2020-03-10 13:26 Reducing I/O of the build process Michael Tremer
                   ` (2 preceding siblings ...)
  2020-03-10 13:26 ` [PATCH 3/4] cdrom+flash-image: Move all temporary files to /tmp Michael Tremer
@ 2020-03-10 13:26 ` Michael Tremer
  3 siblings, 0 replies; 6+ messages in thread
From: Michael Tremer @ 2020-03-10 13:26 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 1106 bytes --]

Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org>
---
 make.sh | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/make.sh b/make.sh
index 188355a0d..9c88592da 100755
--- a/make.sh
+++ b/make.sh
@@ -504,6 +504,7 @@ prepareenv() {
 
 	# Run LFS static binary creation scripts one by one
 	export CCACHE_DIR=$BASEDIR/ccache
+	export CCACHE_TEMPDIR="/tmp"
 	export CCACHE_COMPRESS=1
 	export CCACHE_COMPILERCHECK="string:toolchain-${TOOLCHAINVER} ${BUILD_ARCH}"
 
@@ -575,6 +576,7 @@ enterchroot() {
 		BUILD_ARCH="${BUILD_ARCH}" \
 		BUILD_PLATFORM="${BUILD_PLATFORM}" \
 		CCACHE_DIR=/usr/src/ccache \
+		CCACHE_TEMPDIR="${CCACHE_TEMPDIR}" \
 		CCACHE_COMPRESS="${CCACHE_COMPRESS}" \
 		CCACHE_COMPILERCHECK="${CCACHE_COMPILERCHECK}" \
 		GOCACHE="/usr/src/ccache/go" \
@@ -658,6 +660,7 @@ lfsmake1() {
 	cd $BASEDIR/lfs && env -i \
 		PATH="${TOOLS_DIR}/ccache/bin:${TOOLS_DIR}/bin:$PATH" \
 		CCACHE_DIR="${CCACHE_DIR}" \
+		CCACHE_TEMPDIR="${CCACHE_TEMPDIR}" \
 		CCACHE_COMPRESS="${CCACHE_COMPRESS}" \
 		CCACHE_COMPILERCHECK="${CCACHE_COMPILERCHECK}" \
 		CFLAGS="${CFLAGS}" \
-- 
2.20.1


^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2020-03-10 13:28 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-03-10 13:26 Reducing I/O of the build process Michael Tremer
2020-03-10 13:26 ` [PATCH 1/4] make.sh: Make /tmp a ramdisk if ramdisks are enabled Michael Tremer
2020-03-10 13:28   ` [PATCH] make.sh: Umount /tmp when it is a ramdisk Michael Tremer
2020-03-10 13:26 ` [PATCH 2/4] cdrom: Do not write the temporary tarball to disk Michael Tremer
2020-03-10 13:26 ` [PATCH 3/4] cdrom+flash-image: Move all temporary files to /tmp Michael Tremer
2020-03-10 13:26 ` [PATCH 4/4] make.sh: Move ccache's temp directory into /tmp Michael Tremer

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox