* Reducing I/O of the build process
@ 2020-03-10 13:26 Michael Tremer
2020-03-10 13:26 ` [PATCH 1/4] make.sh: Make /tmp a ramdisk if ramdisks are enabled Michael Tremer
` (3 more replies)
0 siblings, 4 replies; 6+ messages in thread
From: Michael Tremer @ 2020-03-10 13:26 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 908 bytes --]
Hello,
I have noticed that when packing the image, we put quite a lot of
pressure on the disks. Our main build VMs become quite slow and even
slow down other machines like the web servers.
To avoid that, I have made a few changes:
/tmp is now a ramdisk and ccache is putting temporary files in there.
This helps a slot with smaller files that are written to disk or at
least the journal.
When the OS tarball for the CDROM is generated, this is now done
in memory and the OS is only written to disk again once. Saves us
again a gigabyte or so in writes.
All other temporary files are moved into /tmp to take advantage of
the ramdisk.
I do not expect any (significant) changes in build time, but if
anything, it could only have gone up. There is no extra memory
needed and the ramdisk is only enabled when enough memory is
available.
But please let me know if you notice any changes.
Best,
-Michael
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 1/4] make.sh: Make /tmp a ramdisk if ramdisks are enabled
2020-03-10 13:26 Reducing I/O of the build process Michael Tremer
@ 2020-03-10 13:26 ` Michael Tremer
2020-03-10 13:28 ` [PATCH] make.sh: Umount /tmp when it is a ramdisk Michael Tremer
2020-03-10 13:26 ` [PATCH 2/4] cdrom: Do not write the temporary tarball to disk Michael Tremer
` (2 subsequent siblings)
3 siblings, 1 reply; 6+ messages in thread
From: Michael Tremer @ 2020-03-10 13:26 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 835 bytes --]
The build system is writing a large amount of temporary file
systems that might land on disk or at least in the journal.
This change will speed up the build and remove a lot of I/O
usage.
Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org>
---
make.sh | 3 +++
1 file changed, 3 insertions(+)
diff --git a/make.sh b/make.sh
index 984fc95b2..188355a0d 100755
--- a/make.sh
+++ b/make.sh
@@ -476,6 +476,9 @@ prepareenv() {
if [ "${ENABLE_RAMDISK}" = "on" ]; then
mkdir -p $BASEDIR/build/usr/src
mount -t tmpfs tmpfs -o size=4G,nr_inodes=1M,mode=1777 $BASEDIR/build/usr/src
+
+ mkdir -p ${BASEDIR}/build/tmp
+ mount -t tmpfs tmpfs -o size=4G,nr_inodes=1M,mode=1777 ${BASEDIR}/build/tmp
fi
mkdir -p $BASEDIR/build/usr/src/{cache,config,doc,html,langs,lfs,log,src,ccache}
--
2.20.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 2/4] cdrom: Do not write the temporary tarball to disk
2020-03-10 13:26 Reducing I/O of the build process Michael Tremer
2020-03-10 13:26 ` [PATCH 1/4] make.sh: Make /tmp a ramdisk if ramdisks are enabled Michael Tremer
@ 2020-03-10 13:26 ` Michael Tremer
2020-03-10 13:26 ` [PATCH 3/4] cdrom+flash-image: Move all temporary files to /tmp Michael Tremer
2020-03-10 13:26 ` [PATCH 4/4] make.sh: Move ccache's temp directory into /tmp Michael Tremer
3 siblings, 0 replies; 6+ messages in thread
From: Michael Tremer @ 2020-03-10 13:26 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 2295 bytes --]
In order to remove any duplicate files in the tarball, we
have to unpack it again. The whole filesystem is written
to disk twice which is unnecessary.
This patch removes that temporary step and reduces IO
during the build.
Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org>
---
lfs/cdrom | 23 +++++++++++++----------
1 file changed, 13 insertions(+), 10 deletions(-)
diff --git a/lfs/cdrom b/lfs/cdrom
index b26db754b..836cae981 100644
--- a/lfs/cdrom
+++ b/lfs/cdrom
@@ -29,6 +29,9 @@ VER = ipfire
THISAPP = cdrom
TARGET = $(DIR_INFO)/$(THISAPP)
+# Fail when there is an error in the tar pipe
+SHELL=/bin/bash -o pipefail
+
ifeq "$(BUILD_PLATFORM)" "arm"
TAR_OPTIONS =
else
@@ -147,7 +150,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
# since stage2 has been executed..
echo "$(SYSTEM_RELEASE)" > /etc/system-release
- rm -rf /install/cdrom /tmp/*
+ rm -rf /install/cdrom
mkdir -p /install/cdrom/doc
# Clear mtab (prevents .journal problems)
@@ -158,18 +161,18 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
BUILDTARGET="$(BUILDTARGET)" BUILD_ARCH="$(BUILD_ARCH)" KVER="$(KVER)" \
$(DIR_SRC)/src/scripts/archive.files \
$(DIR_SRC)/config/rootfiles/common \
- > /tmp/ROOTFILES
+ > $(DIR_TMP)/ROOTFILES
# Compress root filesystem
# Reason for this tar+untar+tar is removing of entries listed two or more in src/ROOTFILES
- tar -c --exclude='#*' --exclude='proc/*' --exclude='dev/pts/*' \
- --exclude='__pycache__' --exclude='tmp/ROOTFILES' \
- -C / --files-from=/tmp/ROOTFILES -f /$(SNAME).tar
- rm -f /tmp/ROOTFILES
- tar -x -C /tmp -f /$(SNAME).tar
- rm -f /$(SNAME).tar
- @mkdir /tmp/sys
- cd /tmp && tar cf - * | xz $(XZ_OPT) > /install/cdrom/distro.img && rm -rf *
+ rm -rf $(DIR_TMP)/root && mkdir -p $(DIR_TMP)/root
+ tar -c --exclude='#*' --exclude='proc/*' --exclude='dev/pts/*' --exclude='tmp/*' \
+ --exclude='__pycache__' \
+ -C / --files-from=$(DIR_TMP)/ROOTFILES | tar -x -C $(DIR_TMP)/root
+ rm -f $(DIR_TMP)/ROOTFILES
+ mkdir $(DIR_TMP)/root/sys
+ cd $(DIR_TMP)/root && tar cf - * | xz $(XZ_OPT) > /install/cdrom/distro.img
+ rm -rf $(DIR_TMP)/root
# Other files
touch /install/cdrom/$(SNAME)-$(VERSION)-core$(CORE).media
--
2.20.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 3/4] cdrom+flash-image: Move all temporary files to /tmp
2020-03-10 13:26 Reducing I/O of the build process Michael Tremer
2020-03-10 13:26 ` [PATCH 1/4] make.sh: Make /tmp a ramdisk if ramdisks are enabled Michael Tremer
2020-03-10 13:26 ` [PATCH 2/4] cdrom: Do not write the temporary tarball to disk Michael Tremer
@ 2020-03-10 13:26 ` Michael Tremer
2020-03-10 13:26 ` [PATCH 4/4] make.sh: Move ccache's temp directory into /tmp Michael Tremer
3 siblings, 0 replies; 6+ messages in thread
From: Michael Tremer @ 2020-03-10 13:26 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 7401 bytes --]
Since /tmp is now a ramdisk, we move all temporary files into it.
Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org>
---
lfs/cdrom | 71 +++++++++++++++++++++++++-----------------------
lfs/flash-images | 8 +++---
2 files changed, 41 insertions(+), 38 deletions(-)
diff --git a/lfs/cdrom b/lfs/cdrom
index 836cae981..f18b80a03 100644
--- a/lfs/cdrom
+++ b/lfs/cdrom
@@ -150,8 +150,8 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
# since stage2 has been executed..
echo "$(SYSTEM_RELEASE)" > /etc/system-release
- rm -rf /install/cdrom
- mkdir -p /install/cdrom/doc
+ # Create a directory to authorise the CDROM in
+ rm -rf $(DIR_TMP)/cdrom && mkdir -p $(DIR_TMP)/cdrom
# Clear mtab (prevents .journal problems)
rm -vf /etc/mtab
@@ -171,40 +171,43 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
-C / --files-from=$(DIR_TMP)/ROOTFILES | tar -x -C $(DIR_TMP)/root
rm -f $(DIR_TMP)/ROOTFILES
mkdir $(DIR_TMP)/root/sys
- cd $(DIR_TMP)/root && tar cf - * | xz $(XZ_OPT) > /install/cdrom/distro.img
+ cd $(DIR_TMP)/root && tar cf - * | xz $(XZ_OPT) > $(DIR_TMP)/cdrom/distro.img
rm -rf $(DIR_TMP)/root
# Other files
- touch /install/cdrom/$(SNAME)-$(VERSION)-core$(CORE).media
- sed 's/VERSION/$(VERSION)/' $(DIR_SRC)/config/cdrom/README.txt > /install/cdrom/README.txt
- cp $(DIR_SRC)/doc/COPYING /install/cdrom/
- cp $(DIR_SRC)/doc/{ChangeLog,packages-list.txt} /install/cdrom/doc
+ touch $(DIR_TMP)/cdrom/$(SNAME)-$(VERSION)-core$(CORE).media
+ sed 's/VERSION/$(VERSION)/' $(DIR_SRC)/config/cdrom/README.txt > $(DIR_TMP)/cdrom/README.txt
+ cp $(DIR_SRC)/doc/COPYING $(DIR_TMP)/cdrom/
- mkdir -p /install/cdrom/boot/isolinux
+ # Install documentation
+ -mkdir -pv $(DIR_TMP)/cdrom/doc
+ cp $(DIR_SRC)/doc/{ChangeLog,packages-list.txt} $(DIR_TMP)/cdrom/doc
+
+ mkdir -p $(DIR_TMP)/cdrom/boot/isolinux
ifeq "$(HAS_KERNEL)" "1"
- cp /boot/vmlinuz-$(KVER)-ipfire /install/cdrom/boot/isolinux/vmlinuz
- dracut --force --early-microcode -a "installer" --strip --xz /install/cdrom/boot/isolinux/instroot $(KVER)-ipfire
+ cp /boot/vmlinuz-$(KVER)-ipfire $(DIR_TMP)/cdrom/boot/isolinux/vmlinuz
+ dracut --force --early-microcode -a "installer" --strip --xz $(DIR_TMP)/cdrom/boot/isolinux/instroot $(KVER)-ipfire
endif
ifeq "$(HAS_ISOLINUX)" "1"
- dd if=/dev/zero bs=1k count=2 > /install/cdrom/boot/isolinux/boot.catalog
- cp $(DIR_SRC)/config/syslinux/boot.png /install/cdrom/boot/isolinux/boot.png
- cp /usr/share/syslinux/isolinux.bin /install/cdrom/boot/isolinux/isolinux.bin
- cp /usr/share/hwdata/pci.ids /install/cdrom/boot/isolinux/pci.ids
- cp -vf /usr/share/syslinux/*.c32 /install/cdrom/boot/isolinux/
+ dd if=/dev/zero bs=1k count=2 > $(DIR_TMP)/cdrom/boot/isolinux/boot.catalog
+ cp $(DIR_SRC)/config/syslinux/boot.png $(DIR_TMP)/cdrom/boot/isolinux/boot.png
+ cp /usr/share/syslinux/isolinux.bin $(DIR_TMP)/cdrom/boot/isolinux/isolinux.bin
+ cp /usr/share/hwdata/pci.ids $(DIR_TMP)/cdrom/boot/isolinux/pci.ids
+ cp -vf /usr/share/syslinux/*.c32 $(DIR_TMP)/cdrom/boot/isolinux/
sed -e "s/VERSION/$(VERSION) - Core $(CORE)/g" \
$(DIR_SRC)/config/syslinux/syslinux.cfg \
- > /install/cdrom/boot/isolinux/isolinux.cfg
+ > $(DIR_TMP)/cdrom/boot/isolinux/isolinux.cfg
endif
ifeq "$(HAS_MEMTEST)" "1"
# Install memtest
- cp /usr/lib/memtest86+/memtest.bin /install/cdrom/boot/isolinux/memtest
+ cp /usr/lib/memtest86+/memtest.bin $(DIR_TMP)/cdrom/boot/isolinux/memtest
endif
ifeq "$(HAS_IPXE)" "1"
- cp /usr/share/ipfire-netboot/ipxe.lkrn /install/cdrom/boot/isolinux/netboot
+ cp /usr/share/ipfire-netboot/ipxe.lkrn $(DIR_TMP)/cdrom/boot/isolinux/netboot
endif
ifeq "$(EFI)" "1"
@@ -213,43 +216,43 @@ ifeq "$(EFI)" "1"
$(DIR_SRC)/config/cdrom/grub-efi.cfg > /tmp/grub-efi.cfg
# Build a GRUB EFI image
- mkdir -pv /install/cdrom/EFI/BOOT
+ mkdir -pv $(DIR_TMP)/cdrom/EFI/BOOT
grub-mkimage \
--format=$(GRUB_ARCH)-efi \
- --output=/install/cdrom/EFI/BOOT/boot$(EFI_ARCH).efi \
- --config=/tmp/grub-efi.cfg \
+ --output=$(DIR_TMP)/cdrom/EFI/BOOT/boot$(EFI_ARCH).efi \
+ --config=$(DIR_TMP)/grub-efi.cfg \
--compression=xz \
--prefix=/EFI/BOOT \
$$(for mod in $(GRUB_EFI_MODULES); do [ -f "/usr/lib/grub/$(GRUB_ARCH)-efi/$${mod}.mod" ] && echo "$${mod}"; done)
# Install GRUB configuration
- mkdir -pv /install/cdrom/EFI/BOOT
+ mkdir -pv $(DIR_TMP)/cdrom/EFI/BOOT
sed -e "s/NAME/$(NAME)/g" -e "s/VERSION/$(VERSION)/g" -e "s/ARCH/$(BUILD_ARCH)/g" \
- < $(DIR_SRC)/config/cdrom/grub.cfg > /install/cdrom/EFI/BOOT/grub.cfg
+ < $(DIR_SRC)/config/cdrom/grub.cfg > $(DIR_TMP)/cdrom/EFI/BOOT/grub.cfg
# Create the EFI Eltorito image
- dd if=/dev/zero of=/install/cdrom/boot/isolinux/efiboot.img bs=1k count=1440
- mkdosfs -F 12 -n "IPFIRE_EFI" /install/cdrom/boot/isolinux/efiboot.img
+ dd if=/dev/zero of=$(DIR_TMP)/cdrom/boot/isolinux/efiboot.img bs=1k count=1440
+ mkdosfs -F 12 -n "IPFIRE_EFI" $(DIR_TMP)/cdrom/boot/isolinux/efiboot.img
# Mount the EFI image
- mkdir -pv /install/efiboot.img
- mount -o loop /install/cdrom/boot/isolinux/efiboot.img /install/efiboot.img
+ mkdir -pv $(DIR_TMP)/efiboot.img
+ mount -o loop $(DIR_TMP)/cdrom/boot/isolinux/efiboot.img $(DIR_TMP)/efiboot.img
# Copy the bootloader into the image
- mkdir -pv /install/efiboot.img/EFI/BOOT
- cp -a /install/cdrom/EFI/BOOT/boot$(EFI_ARCH).efi \
- /install/efiboot.img/EFI/BOOT/boot$(EFI_ARCH).efi
+ mkdir -p $(DIR_TMP)/efiboot.img/EFI/BOOT
+ cp -a $(DIR_TMP)/cdrom/EFI/BOOT/boot$(EFI_ARCH).efi \
+ $(DIR_TMP)/efiboot.img/EFI/BOOT/boot$(EFI_ARCH).efi
# Cleanup
- umount /install/efiboot.img
- rm -rf /install/efiboot.img /tmp/grub-efi.cfg
+ umount $(DIR_TMP)/efiboot.img
+ rm -rf $(DIR_TMP)/efiboot.img $(DIR_TMP)/grub-efi.cfg
endif
- cd /install/cdrom && find -type f ! -name md5sum.txt | grep -v "./boot" | \
+ cd $(DIR_TMP)/cdrom && find -type f ! -name md5sum.txt | grep -v "./boot" | \
xargs md5sum > md5sum.txt
mkdir -p /install/images
- cd /install/cdrom && mkisofs $(ISO_ARGS) \
+ cd $(DIR_TMP)/cdrom && mkisofs $(ISO_ARGS) \
-o /install/images/$(SNAME)-$(VERSION).$(BUILD_ARCH)-full-core$(CORE).iso .
ifeq "$(HAS_ISOLINUX)" "1"
diff --git a/lfs/flash-images b/lfs/flash-images
index e4af896f1..422acac22 100644
--- a/lfs/flash-images
+++ b/lfs/flash-images
@@ -55,8 +55,8 @@ md5 :
###############################################################################
# Installation Details
###############################################################################
-MNThdd := /install/harddisk
-IMG := /install/images/image.img
+MNThdd := $(DIR_TMP)/harddisk
+IMG := $(DIR_TMP)/image.img
# All sizes in blocks
ifeq "$(BUILD_PLATFORM)" "arm"
@@ -152,9 +152,9 @@ endif
# Install IPFire
ifneq "$(BUILD_PLATFORM)" "arm"
- tar -x --xz -C $(MNThdd)/ -f /install/cdrom/distro.img
+ tar -x --xz -C $(MNThdd)/ -f $(DIR_TMP)/cdrom/distro.img
else
- tar -x -C $(MNThdd)/ -f /install/cdrom/distro.img
+ tar -x -C $(MNThdd)/ -f $(DIR_TMP)/cdrom/distro.img
endif
-touch $(MNThdd)/lib/modules/$(KVER)-ipfire/modules.dep
mkdir $(MNThdd)/proc
--
2.20.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 4/4] make.sh: Move ccache's temp directory into /tmp
2020-03-10 13:26 Reducing I/O of the build process Michael Tremer
` (2 preceding siblings ...)
2020-03-10 13:26 ` [PATCH 3/4] cdrom+flash-image: Move all temporary files to /tmp Michael Tremer
@ 2020-03-10 13:26 ` Michael Tremer
3 siblings, 0 replies; 6+ messages in thread
From: Michael Tremer @ 2020-03-10 13:26 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 1106 bytes --]
Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org>
---
make.sh | 3 +++
1 file changed, 3 insertions(+)
diff --git a/make.sh b/make.sh
index 188355a0d..9c88592da 100755
--- a/make.sh
+++ b/make.sh
@@ -504,6 +504,7 @@ prepareenv() {
# Run LFS static binary creation scripts one by one
export CCACHE_DIR=$BASEDIR/ccache
+ export CCACHE_TEMPDIR="/tmp"
export CCACHE_COMPRESS=1
export CCACHE_COMPILERCHECK="string:toolchain-${TOOLCHAINVER} ${BUILD_ARCH}"
@@ -575,6 +576,7 @@ enterchroot() {
BUILD_ARCH="${BUILD_ARCH}" \
BUILD_PLATFORM="${BUILD_PLATFORM}" \
CCACHE_DIR=/usr/src/ccache \
+ CCACHE_TEMPDIR="${CCACHE_TEMPDIR}" \
CCACHE_COMPRESS="${CCACHE_COMPRESS}" \
CCACHE_COMPILERCHECK="${CCACHE_COMPILERCHECK}" \
GOCACHE="/usr/src/ccache/go" \
@@ -658,6 +660,7 @@ lfsmake1() {
cd $BASEDIR/lfs && env -i \
PATH="${TOOLS_DIR}/ccache/bin:${TOOLS_DIR}/bin:$PATH" \
CCACHE_DIR="${CCACHE_DIR}" \
+ CCACHE_TEMPDIR="${CCACHE_TEMPDIR}" \
CCACHE_COMPRESS="${CCACHE_COMPRESS}" \
CCACHE_COMPILERCHECK="${CCACHE_COMPILERCHECK}" \
CFLAGS="${CFLAGS}" \
--
2.20.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH] make.sh: Umount /tmp when it is a ramdisk
2020-03-10 13:26 ` [PATCH 1/4] make.sh: Make /tmp a ramdisk if ramdisks are enabled Michael Tremer
@ 2020-03-10 13:28 ` Michael Tremer
0 siblings, 0 replies; 6+ messages in thread
From: Michael Tremer @ 2020-03-10 13:28 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 453 bytes --]
Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org>
---
make.sh | 1 +
1 file changed, 1 insertion(+)
diff --git a/make.sh b/make.sh
index 9c88592da..756b27f7a 100755
--- a/make.sh
+++ b/make.sh
@@ -285,6 +285,7 @@ stdumount() {
umount $BASEDIR/build/usr/src/log 2>/dev/null;
umount $BASEDIR/build/usr/src/src 2>/dev/null;
umount $BASEDIR/build/usr/src 2>/dev/null;
+ umount $BASEDIR/build/tmp 2>/dev/null;
}
now() {
--
2.20.1
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-03-10 13:28 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-03-10 13:26 Reducing I/O of the build process Michael Tremer
2020-03-10 13:26 ` [PATCH 1/4] make.sh: Make /tmp a ramdisk if ramdisks are enabled Michael Tremer
2020-03-10 13:28 ` [PATCH] make.sh: Umount /tmp when it is a ramdisk Michael Tremer
2020-03-10 13:26 ` [PATCH 2/4] cdrom: Do not write the temporary tarball to disk Michael Tremer
2020-03-10 13:26 ` [PATCH 3/4] cdrom+flash-image: Move all temporary files to /tmp Michael Tremer
2020-03-10 13:26 ` [PATCH 4/4] make.sh: Move ccache's temp directory into /tmp Michael Tremer
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox