This Week In Pakfire: Build Environments & Snapshots

This Week In Pakfire: Build Environments & Snapshots

[Michael Tremer]

Hello all,

Last week we learned about source packages. We also know about the jail. But why all of this? Let me tell you about build environments…

To build a package we need more than just the sources. We need a compiler, a linker, lots of libraries, header files and many things more. How do we get all those things that we need? We already have our jail functionality, but how do get all the programs into it?

Whenever Pakfire builds a package, it has to create a new build environment. In the beginning that is just a directory somewhere in the file system. Then, it will check what source package to build and install it. The source package optionally has dependencies to all the tools that are needed, but there is a default set of build tools that will always be there because they are very common. The common build tools are for example coreutils (cp, mv, …), awk, a C compiler and linker, glibc headers and many things more. Sometimes a program wants to link against a third library. That can be pulled in by adding a dependency to the source package and Pakfire will select all possible tools and the best versions of those packages and install them into the build environment. When the build is starting, all those tools are available immediately.

As we have learned before, the jail makes sure that the build process cannot harm the host system. So once installed, the build environment is living in its own little bubble. There is no internet access, no way to install any more packages that have not been defined before. That way, we can guarantee that we knew from the start which packages are needed and which have been used. Later on, we might use this data to reproduce the same build on a different host to ensure that we get the same result.

If a build fails, Pakfire will automatically open a shell for debugging. In there, we will re-enable internet access and the option to install further packages, for example a debugger, so that you can fix the problem and run the build again. All in the very same environment. Once the build is done, the environment will just be thrown away.

That way, you can have as many build environments as you want on your system. All you need is some disk space and probably computer power to run multiple builds at the same time. They will be entire isolated from each other and the outside world.

Pretty cool isn’t it? It get’s better though…

As mentioned before, there will always be some common packages installed. Why would we want to extract the same packages again and again and again? We don’t, because it takes a couple of seconds to do this. Instead, we create a snapshot just once which will contain all those packages, or let’s rather put it as: a very basic build environments. When running a build we will use that snapshot and install only the custom dependencies of the source package into the snapshot. Usually that is only a few packages and runs within the fraction of a second. That allows us to launch a build within seconds: package the source package, mount the snapshot, install the source package and the dependencies and run ./configure. Unless there is a large source tarball to extract, we will reach the ./configure step almost instantaneously. It is even so fast that a simple package like beep is being completely built, packaged and the build environment is being cleaned up again in about 2 seconds. This makes Pakfire really fast and saves a lot of needless I/O on the build machines.

I hope this was an interesting read. See you all next week.

-Michael