This Week In Pakfire: Log Streaming

This Week In Pakfire: Log Streaming

[Michael Tremer]

Recently we learned about the jail that is running inside Pakfire. But how does communication work between the host and the secure environment? How is it possible to watch the build run on the build service? I will tell you how…

## Privilege Separation

The jail is creating another process that is contained inside the jail. It cannot break out of that. But in order to see what is going on, we will need some way to communicate with it. In Pakfire we went all the way to create a new Pseudo Terminal (https://en.wikipedia.org/wiki/Pseudoterminal) which basically looks like a classic old physical terminal inside the jail and outside is a pipe that we can use to communicate with. The reason why we don’t use a simple pipe is so that the process inside the jail cannot execute and commands that would reconfigure the terminal. That would be possibly by simply echoing some control characters which Pakfire would then forward. And that we don’t want.

So, once the process is launched, we can communicate with it and we will set up an event loop that will listen to any events that we receive. This usually is data being ready to be read, but we could also run an interactive jail where we have to forward data in the other direction, too. In addition to that, there are signals for when the child process has finished or even died, when the terminal is being resized and so on…

When running a build on the local terminal, we print the output of the build to the users terminal. Simples. But how do we get it into the build service?

## Streaming Logs in Realtime over the Internet

This all starts with a buffer where we store the last couple of messages. This is implemented here:

  https://git.ipfire.org/?p=pakfire.git;a=blob;f=src/pakfire/log_buffer.c;h=3ef66368f3d5a9e4cdc9c4716bcffa444743252d;hb=HEAD

If a client requests the log, a message is being sent to the daemon that is running somewhere else which requests the log to be streamed. Then, additional hooks are registered with the event loop to make sure that whenever the connection is free, we send some data. This is actually very quick and has pretty much no latency. A tail on the command line is exactly the same as viewing the log in the browser.

But how does it even get to the browser? For that we are using WebSocket connections which are being created whenever a user wants to see the log streaming. The control connection to the builders is also using WebSocket so that we can host them in multiple locations depending on cost, build power and availability.

I really like this feature, because of two reasons:

The first is that I would like to know what is happening. Sometimes a build takes a little bit extra time, but then we don’t know why. Just checking the logs although the build is running on the other side of the planet is really cool and answers that question immediately.

The second reason is that studying the build logs is important. There are a lot of warning flying around and although the build might have completed successfully, it might not be what we expected. Checking the logs while it is running is helping to use any time waiting for a build.