Hello,
On Tue, 2018-04-10 at 19:15 +0200, Peter Müller wrote:
Hello,
a few days ago, I had a discussion with Michael about privacy and security for IPFire updates (we mainly focused on IPFire 3.x, but some points might be applicable to 2.x, too).
It is good that we have a discussion about this in the public, too. We unfortunately have too many discussions in private which isn't too good.
But please, other people, contribute!
In order to get some more opinions about this, I would like to mention the main points here and ask for comments. Forgive me if you receive this mail twice; I'll post it on both development and mirror list.
You can CC the same email to multiple lists at the same time. I did this now, so people subscribed to both will only receive one.
We should also include the Pakfire list.
(a) Security Whenever you are updating an application or an entire operating systems, security is the most important aspect: An attacker must not be able to manipulate update packages, or fake information about the current patchlevel, or something similar.
In the past, we saw several incidents here - perhaps the most famous one was "Flame" (also known as "Flamer" or "Skywiper"), a sophisticated malware which was detected in Q1/2012 and spread via Windows Updates with a valid signature - and it turned out that strong cryptography is a very good way to be more robust here.
In IPFire 2.x, we recently switched from SHA1 to SHA512 for the Pakfire signatures, and plan to change the signing key (which is currently a 1024-bit-DSA one) with Core Update 121 to a more secure one.
So far, so good.
Keys in Pakfire 3 are usually 4k in size, RSA and we sign and hash with SHA512 only.
That is practically as best as we can go right now.
Speaking about IPFire 3.x, we plan to download updates over HTTPS only (see below why). However, there are still a few questions left:
We probably need to encourage some more people to move their mirrors to HTTPS too before we can go large. But I would prefer to have HTTPS only and fewer mirrors than having more mirrors with HTTP only.
However, this is only for privacy and not for security.
(i) Should we sign the mirror list? In 3.x, using a custom mirror (i.e. in a company network) will be possible. If not specified, we use the public mirror infrastructure; a list of all servers and paths will be published as it already is today.
The list is a bit more complex than that, but essentially serves the same purpose:
https://pakfire.ipfire.org/distro/ipfire3/repo/stable/mirrorlist?arch=x86_64
This is what it looks like right now.
It is not required at all to sign this list for the integrity of the entire package management system. The packages have a signature and it does not matter if the package was downloaded from a source that was not trustworthy since the signature is validated and either matches or it does not.
However, for the privacy argument, I can understand that there are some arguments for signing it so that no man in the middle can add other mirrors and gather information from any downloading clients.
The mirror list however is being downloaded over HTTPS and therefore we have transport security. TLS can be man-in-the-middle-ed of course.
Generally I would like to allow for users to download a package from a source that we do not know of verify. Debian is making a far stronger point towards that that they even ban HTTPS. They want bigger organizations to use proxy servers that cache data and they want to give them the opportunity to redirect them back to any self-hosted mirrors. That I completely regard out of scope for us since we don't create anywhere near the traffic that Debian creates (because both: size and number of users of our distribution). I would also like to emphasize that we consider security first and then bandwidth use.
I consider the likelihood that an attacker is inserting malicious mirrors in here very small. A signature on the list would also only show that we have seen the same list that a client has downloaded.
When we add a mirror to our list, we do not conduct any form of audit, so that it is even possible that some of the mirrors are compromised or configured in a fashion that we would not prefer. That - by design - is not a problem for the security of Pakfire. But it is possible that people just swap the files on the servers. That is an attack vector that we cannot remove unless we host all mirrors ourselves and never make any mistakes. Not going to happen.
Pakfire 2 has the mirror list being distributed over the mirrors. Therefore it *is* signed.
Pakfire 3 has a different approach. A central service is creating that list on demand and tries to *optimise* it for each client. That means putting mirrors that are closer or have a bigger pipe to the top of the list. Not sure how good our algorithm is right now, but we can change it on the server-side at any time and changes on the list will propagate quicker than with Pakfire 2.
Pakfire 2 also only has one key that is used to sign everything. I do not intend to go down that path why that is a bad idea, but Pakfire 3 is not doing this any more. In fact, packages can have multiple signatures.
That leads me to the question with what key the list should be signed. We would need to sign maybe up to one-hundred lists per second since we generate them live. We could now simplify the proximity algorithm so that each country only gets one list or something similar and then deliver that list from cache.
I do not think that the main key of the repository is a good idea. Potentially we should have an extra key just for the mirror lists on the server.
We would also need to let the signature expire so that mirrors that are found out to be compromised are *actually* removed. At the moment the client keeps using the mirror list until it can download a new one. What would happen if the download is not possible but the signature on a previous list has expired?
We would also make the entire package management system very prone to clock issues. If you are five minutes off, or an hour, the list could have expired and you cannot download any packages any more or you would always fall back to the main mirror.
In my opinion, we should sign that list, too, to prevent an attacker from inserting his/her mirror silently. On the other hand, packages are sill signed, so a manipulation here would not be possible (and we do not have to trust our mirrors), but an attacker might still gather some metadata.
So to bring this to a conclusion what I want to say here is, that I do not have a problem with it being signed. I just have a problem with all the new problems being created. If you can give me answers to the questions above and we can come up with an approach that improves security and privacy and also does not make bootstrapping a new system a pain in the rear end, then I am up for it.
But it will by design be a weak signature. We could probably not put the key into a HSM, etc.
[The mirror list can be viewed at https://mirrors.ipfire.org/, if anyone is interested.]
Pakfire 3 has its mirrors here: https://pakfire.ipfire.org/mirrors
(ii) Should we introduce signers? A package built for IPFire 3.x will be signed at the builder using a custom key for each machine. Since malicious activity might took place during the build, the key might became compromised.
Some Linux distributions are using dedicated signers, which are only signing data but never unpack or execute them. That way, we could also move the signing keys to a HSM (example: https://www.nitrokey.com/) and run the server at a secure location (not in a public data centre).
I am in favour of this.
This is just very hard for us to do. Can we bring the entire build service back to work again and then add this?
It is not very straight forward and since we won't have builders and the signers in the same DC, we would need to have a way to either transfer the package securely or do some remote signing. Both doesn't sound like a good idea.
(b) Privacy Fetching updates typically leaks a lot of information (such as your current patch level, or systems architecture, or IP address). By using HTTPS only, we avoid information leaks to eavesdroppers, which I consider a security benefit, too.
However, a mirror operator still has access to those information. Perhaps the IP address is the most critical one, since it allows tracing a system back to a city/country, or even to an organisation.
We have been hosting a ClamAV mirror once and it was very interesting to see this.
Also, many mirrors seem to open up the usage statistics through webalizer. So this will indeed be a public record.
Because of that, I do consider mirrors to be somewhat critical, and would like to see the list singed in 3.x, too.
As stated above, I do not think that this gets rid of the problem that you are describing here.
(i) Should we introduce mirror servers in the Tor network? One way to solve this problem is to download updates via a proxy, or an anonymisation network. In most cases, Tor fits the bill.
For best privacy, some mirror servers could be operated as so-called "hidden services", so traffic won't even leave the Tor network and pass some exit nodes. (Debian runs several services that way, including package mirrors: https://onion.debian.org/ .)
Since Tor is considered bad traffic in some corporate network (or even states), this technique should be disabled by default.
What are your opinions here?
I have never hosted a hidden service on Tor. I do not see a problem with that. It might only be possible that a very tiny of people are going to use this and therefore it is a lot of work to do this with only a few people benefiting from it.
What does it need so that Pakfire would be able to connect to the Tor network? How would this look like from a user's perspective? Where is this being configured? How to we send mirror lists or repository information?
(ii) Reducing update connections to anybody else Some resources (GeoIP database, IDS rulesets, proxy blacklists) are currently not fetched via the IPFire mirrors, causing some of the problems mentioned above.
For example, to fetch the GeoIP database, all systems sooner or later connect to "geolite.maxmind.com", so we can assume they see a lot of IP addresses IPFire systems are located behind. :-\ Michael and I are currently working on a replacement for this, called "libloc", but that is a different topic.
This is a huge problem for me. We cannot rely on any third parties any more. I guess the reports in the media over the last days and weeks have proven that there is too much of a conflict of interest. There are no free services from an organization that is trying to make billions of dollars.
Since it is very hard to get consent from the IPFire users on every of those, we should just get everything from one entity only.
Pushing all these resources into packages (if they are free, of course) and deliver them over our own mirrors would reduce some traffic to third party servers here. For libloc, we plan to do so.
If by package, you are NOT referring to a package in the sense of a pakfire package, then I agree.
Should we do this for other resources such as rulesets and blacklists, too?
Ideally yes, but realistically we cannot reinvent everything ourselves. I am personally involved into too many of these side-projects that there is only little time for the main thing. So I would rather consider that we work together with the blacklist people, or just leave it for now. I guess that is a thing for the blacklists because they are opt-in. People have to pick one and it is obvious that something is being downloaded. However, it is not obvious what the dangers are. The geo IP database however is not opt-in. And it isn't opt-out either.
Looking forward to read your comments.
Sorry this took a little while.
Thanks, and best regards, Peter Müller
-Michael