From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4Zf5KC5plTz334L for ; Fri, 18 Apr 2025 07:01:35 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mail01.haj.ipfire.org", Issuer "R10" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4Zf5K81v1sz2xng for ; Fri, 18 Apr 2025 07:01:32 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4Zf5K722JBz2SF for ; Fri, 18 Apr 2025 07:01:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1744959691; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=+RAIuX/vgODDnVlYdIEAasujIwsPPqVeXGjrfMeIEeI=; b=HMOljWmhWA642ORJNdVYdTCWXujoDNVV4xIrsYWfdU2BRHBvXxCp/8QMym+mOC5L8O5MUY yg7ZGOs1UPyeROPixuinV8JYDzXbYKxrPS9NmTfM/eDz/fWfyu7QoBcSY2rKeHAdWfkQIR j4lQi/3jGRUqAeq+bWeq5ucE7Im8F6oWQjQHVatpPoi/LucAG4+iat8pNwQRN4tMdxwpdD jNMZPUAYee2d3nV0LaLrPnnPVEHB00PaWsCX2/MNXV6IF4oJFyiYQFVcZtIoTGbkxNRMiS se9dqSSOB2i/WdNUhmZtliWyd96oIBGEK64n/bCkNp2UmEqU2hpLw0QaT5xNXQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1744959691; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=+RAIuX/vgODDnVlYdIEAasujIwsPPqVeXGjrfMeIEeI=; b=rneTaZmGJ27IlYRaab5qpECT2AHBTBnfRadtqlBCaS5iOtMlrySm+knfkUvk+XW1vNaXrT JkOxMyLTRkwV9OBA== From: Michael Tremer Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Precedence: list List-Id: List-Subscribe: , List-Unsubscribe: , List-Post: List-Help: Sender: Mail-Followup-To: Mime-Version: 1.0 Subject: This Week In Pakfire: Build Environments & Snapshots Message-Id: <8307B50B-09F0-421C-822C-677585B3D7E0@ipfire.org> Date: Fri, 18 Apr 2025 08:00:02 +0100 To: "IPFire: Development-List" Hello all, Last week we learned about source packages. We also know about the jail. = But why all of this? Let me tell you about build environments=E2=80=A6 To build a package we need more than just the sources. We need a = compiler, a linker, lots of libraries, header files and many things = more. How do we get all those things that we need? We already have our = jail functionality, but how do get all the programs into it? Whenever Pakfire builds a package, it has to create a new build = environment. In the beginning that is just a directory somewhere in the = file system. Then, it will check what source package to build and = install it. The source package optionally has dependencies to all the = tools that are needed, but there is a default set of build tools that = will always be there because they are very common. The common build = tools are for example coreutils (cp, mv, =E2=80=A6), awk, a C compiler = and linker, glibc headers and many things more. Sometimes a program = wants to link against a third library. That can be pulled in by adding a = dependency to the source package and Pakfire will select all possible = tools and the best versions of those packages and install them into the = build environment. When the build is starting, all those tools are = available immediately. As we have learned before, the jail makes sure that the build process = cannot harm the host system. So once installed, the build environment is = living in its own little bubble. There is no internet access, no way to = install any more packages that have not been defined before. That way, = we can guarantee that we knew from the start which packages are needed = and which have been used. Later on, we might use this data to reproduce = the same build on a different host to ensure that we get the same = result. If a build fails, Pakfire will automatically open a shell for debugging. = In there, we will re-enable internet access and the option to install = further packages, for example a debugger, so that you can fix the = problem and run the build again. All in the very same environment. Once = the build is done, the environment will just be thrown away. That way, you can have as many build environments as you want on your = system. All you need is some disk space and probably computer power to = run multiple builds at the same time. They will be entire isolated from = each other and the outside world. Pretty cool isn=E2=80=99t it? It get=E2=80=99s better though=E2=80=A6 As mentioned before, there will always be some common packages = installed. Why would we want to extract the same packages again and = again and again? We don=E2=80=99t, because it takes a couple of seconds = to do this. Instead, we create a snapshot just once which will contain = all those packages, or let=E2=80=99s rather put it as: a very basic = build environments. When running a build we will use that snapshot and = install only the custom dependencies of the source package into the = snapshot. Usually that is only a few packages and runs within the = fraction of a second. That allows us to launch a build within seconds: = package the source package, mount the snapshot, install the source = package and the dependencies and run ./configure. Unless there is a = large source tarball to extract, we will reach the ./configure step = almost instantaneously. It is even so fast that a simple package like = beep is being completely built, packaged and the build environment is = being cleaned up again in about 2 seconds. This makes Pakfire really = fast and saves a lot of needless I/O on the build machines. I hope this was an interesting read. See you all next week. -Michael=