Thanks for the input. Let's go through it bit by bit...
On Wed, 2017-10-25 at 22:12 +0200, Peter Müller wrote:
Hello Michael,
Hello altogether,
to bring some movement into this I went ahead and installed Let's Encrypt certificates pretty much everywhere (with exception of the internal services like LDAP).
There you have it. I did it. I bit the bullet.
Thank you very much. :-) I really appreciate this.
It literally took me days since we have so many subdomains and I issued one certificate per domain which makes it easier to renew and revoke them if needed. They also throttled me to only issue a small number of certificates per week.
But in summary these things have changed (I would like to get some feedback from you since you all have probably a little bit more experience with a few things than I have):
- LE deployed for all web services with the exception of boot.ipfire.org and
fireinfo.
iPXE doesn't really support HTTPS that well and I shipped a new version of it that at least downloads some certificates (although it cannot really validate them - it can only check than a CA exists I think).
Hmmm, that is not very good - of course, everything is better than plaintext, but without validating the certificates...
How widely is iPXE used?
Loads of people use it. Probably every VM runs it :)
I enabled this: http://ipxe.org/cfg/crosscert
https://cgit.ipfire.org/oddments/ipfire-netboot.git/commit/?id=871fc3184ad5e...
I just found out that is has cross-signed all CAs in that folder with their own CA that is baked into the image, so they can verify if the CA they downloaded is actually trusted or not. It's not perfect, but I think it is good enough.
It is possible to bake in a CA into the image, but I didn't want to set it to LE and nothing else and there is not enough space in the image to import the usual CA bundle. I think the final image cannot be larger than 1M and the current IPFire CA bundle that we ship is about 700k.
Same with fireinfo. I have no idea if the profiles can be sent properly over HTTPS without touching the client first.
I guess we need to implement that in the clients first. At the moment I am quite busy reworking some broken patches and other stuff, but will have a look at that occasionally.
That would be cool. It uses the standard python urllib2 module which supports SSL. I am just not sure if it would follow the redirect properly. Needs some testing.
https://cgit.ipfire.org/oddments/fireinfo.git/tree/src/sendprofile
Using HTTPS on fireinfo an mirrors might be a good idea since a (even passive) attacker can see a lot of information by simply looking at the network traffic (Which version is the firewall running? Which architecture and network zones are in use? And so on.)
Our default mirror only supports HTTPS from now on and will redirect. However, the load-balancer will only redirect to HTTP which will then redirect to HTTPS. That's not ideal.
I need to add this here: https://cgit.ipfire.org/ipfire.org.git/tree/webapp/backend/mirrors.py#n418
It's very easy just to flag in the database which mirrors support HTTPS. But there will always be a chance that a client is redirected to a mirror that only supports HTTP.
I already did this for the Pakfire Build Service: https://cgit.ipfire.org/pbs.git/commit/?id=3163c789758423fad61335995b206e5af...
Just as a side note: Validating this traffic against DANE records would be nice. :-)
Oh I forgot to say that we have TLSA records for everything in here. Probably nobody validates this, but hey :)
[root@mail01 ~]# dig TLSA _443._tcp.www.ipfire.org
; <<>> DiG 9.11.1-P3-RedHat-9.11.1-2.P3.fc26 <<>> TLSA _443._tcp.www.ipfire.org ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41901 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;_443._tcp.www.ipfire.org. IN TLSA
;; ANSWER SECTION: _443._tcp.www.ipfire.org. 21599 IN CNAME _letsencrypt.certs.ipfire.org. _letsencrypt.certs.ipfire.org. 21599 IN TLSA 2 1 1 60B87575447DCBA2A36B7D11AC09FB24A9DB406FEE12D2CC90180517 616E8A18
;; Query time: 99 msec ;; SERVER: 172.28.1.1#53(172.28.1.1) ;; WHEN: Thu Oct 26 12:11:06 CEST 2017 ;; MSG SIZE rcvd: 133
- LE deployed on mail01.i.ipfire.org (Postfix + Dovecot)
Yep, seems to work.
However, a client certificate in Postfix is still missing:
Received: from mail01.ipfire.org (mail01.ipfire.org [81.3.27.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) --> (Client did not present a certificate) <-- [etc.]
In your main.cf, it can be enabled by using:
smtp_tls_cert_file = /Path/to/your/certificate smtp_tls_key_file = /Path/to/your/private/key
You might want not only to query client certificates, but also validate them. To do so, use:
smtpd_tls_CAfile = /Path/to/certificate/block
This requires all trusted intermediate certificates to be put together in one file, as far as I can remember.
(See http://www.postfix.org/postconf.5.html#smtpd_tls_CAfile for details)
I didn't have that set. Set it to this now: smtpd_tls_CAfile = /etc/pki/tls/certs/ca-bundle.crt
I did not test the mail server's cipherlist, but I hope it is secure for both SMTP server and client. :-)
Probably not :)
I think we both should walk through the configuration together...
LE deployed on im01.i.ipfire.org which runs Prosody, our XMPP server
I enabled HSTS for all web services since all of them are forcing the
browser to use HTTPS.
How about the IPFire mirrors? If one of them (i.e. "mirror1.ipfire.org") supports HTTPS, are the systems following the 30x redirects? In the future maybe hardcoding the protocol might be a good idea: If a mirror supports HTTPS, we use it there.
See above. And yes, clients follow a large number of redirects. Firefox follows 20.
But that is only a minor thing.
In general, HSTS is very nice to have here. :-)
- We only allow TLS 1.2 for web.
It seems like the key size is 2048 bits. Although this is not critical, I would recommend to move to 4096 bits at the next key rollover.
Well, I didn't check. Because security wasn't really what I had in mind here. Our old keys were 4096 bits long. I didn't even know this could be changed.
Let's Encrypt announced to support ECDSA in early 2018, so at that time we could also use dual-stack RSA and ECDSA, as we are doing with IPFire in Core Update 115.
They promised a lot. Let's see :)
Since everything is on SSL now, I also enabled HTTP/2.0.
I also enabled OCSP stapling.
Excellent.
The nginx configuration file looks as follows:
# Enable TLS 1.2 only ssl_protocols TLSv1.2;
# Enable protection against BEAST attacks ssl_prefer_server_ciphers on; ssl_ciphers "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM- SHA384:ECDHE- ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128- GCM- SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA- AES256- SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256";
Apparently there is an issue here: SSL Labs does not display the Chacha/Poly cipher. Perhaps nginx is linked against OpenSSL 1.0.x ?
The web server is running openssl 1.0.2k. Unfortunately it is still on Fedora 24 which I need to update as soon as I can.
I am not sure if we should prioritise this cipher suite over AES since it is safe too, but seems to faster on devices without AES-NI and a small CPU, such as Android smartphones. Decision is up to you.
I got this from Mozilla, so I thought they considered this.
# Enable session caching ssl_session_cache shared:SSL:50m; ssl_session_timeout 1d;
# Enable Certificate Stapling ssl_stapling on; ssl_stapling_verify on; resolver 172.28.1.1;
# HSTS (15768000 seconds = 6 months) add_header Strict-Transport-Security max-age=15768000;
This is inspired by the Mozilla recommendations.
So please guys, have a test. I am quite sure that although this is quite a "modern" configuration all recent web browsers should work fine. If you are on IE6, you are fucked. Tough luck.
Yes, I think supporting legacy clients is not a very good idea. Of course, there are some scenarios where you must do so (big company website, or anything else), but for this, I consider it safe.
Food for thoughts: Is there anything against updating OpenSSL to 1.1.x in IPFire? That way, we could also use Chacha/Poly there and disable TLS 1.0, but I am not sure how many clients will be broken by this.
I don't think there is nothing that is stopping us any more. For quite some time nothing built against OpenSSL 1.1 since they changed a lot in the API. Of course we will still need to ship 1.0.1 because loads of things link against it.
Before we also shipped OpenSSL 0.9.8 for quite a while until it was discontinued:
https://cgit.ipfire.org/ipfire-2.x.git/commit/lfs/openssl-compat?id=45ff420e... ad522dc6d0e53837a6e1951407fc
About disabling TLS 1.0. I am all for it, but it needs some good testing. One common scenario is that people use Postfix as a mail relay to deliver emails to an internal Exchange server. Those often require RC4 and MD5. I think that is Exchange 2003. This is bad and ugly but this is what people are using in 2017. And we need to be able to enable these ciphers optionally when they are required.
However, we should have strong defaults.
But if you find anything that should work, please let me know. You can also run some scanners against it and see what happens. I did a few and they show A+ rating which is what we are looking for.
Anything that I have forgotten to enable which could benefit security?
Not that I am aware of at the moment (besides the stuff mentioned above). Great job!
Cool!
-Michael
Best regards, Peter Müller
Looking for your input :)
Best, -Michael
On Mon, 2017-09-25 at 17:48 +0200, Peter Müller wrote:
Hello Michael,
Hi,
On Fri, 2017-09-15 at 22:26 +0200, Peter Müller wrote:
Hello Michael, hello Matthias,
a few thoughts on this from me:
(1) @Michael: Thank you for having a look at this. In my opinion, enabling HTTPS on all *.ipfire.org sites by default would be a huge security benefit.
Of course, this needs time and testing, so no pressure. :-)
Especially on download sites HTTPS would be nice in case someone verifies the downloaded ISO - when transmitted in plaintext, both image and checksum might be modified by a MITM.
Further, there is still SHA1 in use on http://download.ipfire.org/, but that is another issue.
Indeed it is. We have ticket for that: https://bugzilla.ipfire.org/show_bug.cgi?id=11345
Thanks for the ticket link.
(2) Speaking about the CAs: I do not know what opinion to have here.
On the one hand, using your own project CA is good since nobody except the visitors has to rely on some external certificate signing company.
Agreed. I do not trust any of those public CAs - at all. They have a commercial interest and that is it.
Publishing the TLSA-records tells everyone that this CA - which is untrusted by all current browsers - is legitimate on *.ipfire.org.
We do this.
I forgot that. Unfortunately, at the moment, "only" mail servers benefit from this (see below).
On the other hand, there are two points against it: (a) Nobody actually uses TLSA/DANE for validating web site certificates (this is different when it comes to SMTP). So, nearly every user will see a certificate warning.
Indeed not a single browser is verifying it. There is plugins that can show an icon and that's it.
We also do this for our mail server which does not have a publicly signed certificate.
As far as I am concerned, there are two different approaches to TLS:
(a) If you are running a mail server, TLS is mostly used in opportunistic mode, and certificates are not that important. The idea behind this is to prevent plaintext transport of e-mails, that's why so many big MX still support weak ciphers such as 3DES. (Personally, I am not a fan of opportunistic TLS since it only protects against passive attackers and one has to use ancient algorithms. At least RC4-MD5 finally disappeared...)
So, a self-signed certificate for a MX is no problem - at least, not a big one. Only a few MX actually ask for a client certificate, and even less systems are configured to present any.
(b) Web browsers have a completely different view on this: They first check the certificate against the DNS name, and if they match, encryption can be started. On the other hand, in case they see an untrusted certificate for whatever reason, everything becomes bad very quickly.
Integrity is much more important here in order to protect the users against active (MITM) attackers.
Because of this, I would prefer certs signed by well-known CAs for public web sites.
Personally, i suppose DANE will never be implemented in web browsers. There are too many collisions with the systems DNS service, and network stacks, and the network's firewall rules, and so on. Sad, but that is what we (do not) have.
I am not convinced that this has technical reasons. It just puts some big businesses will lose a machine that prints them money.
Partly I think. DANE depends on DNSSEC, and I remember some user complaints that their ISPs filter DNS queries to external name servers without using it on their own systems.
Nobody there is interested in the security. They are interested in the money.
Without being very deep into this, I assume you are right.
So, enabling HTTPS on all or at least some very important IPFire sites will cause more user complaints.
Indeed. That is why we don't enforce it.
Further, and that is a _very_ big problem in my eyes, it actually decreases transport encryption security: Every time a user accesses *.ipfire.org, he/she/it has to bypass a certificate warning - at least in FF, there is no easy way to store the exception permanently.
In case a MITM injects his own certificate to break TLS, the user will see a certificate warning which is nearly identical to those shown usually for IPFire's CA. And the user will mostly bypass this - very well-known - warning, an the attacker won.
To prevent this, you MUST check the certificates checksum against the value you know is legitimate. Every time.
Nobody will do so.
Agreed.
(b) As I mentioned above, I completely understand your point having an own CA. However, I do not think this is so important for public web services:
I emphasise here on "web services". I am not even sure if Let's Encrypt can issue certificates for our internal LDAP server or our mail server, etc.
This depends on the actual network (especially DNS) configuration. If the server does not have a public IP, but is configured in the DNS, LE can validate it without a direct connection:
- https://community.letsencrypt.org/t/cert-for-intranet/6337/4
- https://community.letsencrypt.org/t/on-the-state-of-the-dns-01-challenge
/480 5
As far as I am concerned, there is no "trust" in case of CA companies. Some of them showed a really unprofessional behaviour (DigiNotar, WoSign, Symantec, ...) and are/were either kicked off business by their customers or the web browser developers.
So why would we assume that Let's Encrypt is doing a better job? Just because they are sponsored by some of the "good guys"?
You only "trust" a CA to validate if a certain domain really belongs to the person who claims so.
In case there is a certificate in use signed by a foreign CA, what should happen? Nobody except the server admin has the private key, and there are many techniques to prevent modern browsers accepting any certificate for your domain, such as:
So let's split all of this:
- DANE/TLSA (specifies which certificate [chain] is valid)
We can just install that for the Let's Encrypt CA. It doesn't make sense to have TLSA records for each cert because they will be replaced very very quickly.
I remember a trick here: If you do not rotate the private key every time LE's signature expires, the public one also stays the same - and the TLSA record does not need to be changed.
Can't find the link right now...
- CAA (DNS record, rather new, specifies which CAs can issue
certificates for this site, might be supported by common browsers some day)
I need to implement that in our DNS server software, but that is not a big thing. Will do that soon.
Thanks.
- HSTS (preventing a MITM from downgrading to plaintext)
Who wants to take care of this?
Actually, HSTS is quite easy to deploy since it only tells web browsers to enforce TLS on a certain site. So, if there are no dependencies (unencrypted external resources, ad servers, ...), you can just set it up and forget it. :-)
- HPKP (as DANE, but supported by modern browsers, TOFU
[Trust On First Use] = first connection must be clean)
Same.
Indeed, HPKP is more complicated, it is like DANE without DNS. Further, some security experts (such as Ivan Ristic) argue that it might become dangerous in case of misconfiguration.
And who would like to create tickets on BZ for all of this?
Except from going out of business, I do not see any risk for a web site owner here. That is, in case all or some of the mentioned methods are implemented.
LOL, we are not a business. We are an Open Source project.
"Going out of business" was relating to CAs, not to IPFire. But yes, it would be very sad if this project disappears.
Let's Encrypt (LE) seems to be different from other CAs since they use standardised methods and act quite transparently. Further, there are some "major players" behind it (Mozilla, EFF, ...) which have a good reputation. This - hopefully - means that there will be no security breaches like in case of DigiNotar.
I don't share your hope, but I do not want to fight this. I have been trialing Let's Encrypt a little bit on some web shops and other things and it does the job. To be honest I do not see the point in spending the money on a different vendor of certificates. So take that as: They are not worse than those.
I don't want to fight either; but I am afraid I did not fully understand your point concerning the trust/distrust of public CAs (except for obvious reasons like security breaches).
To come to an end, using LE as a CA for *.ipfire.org will not harm the security of IPFire significantly. It reduces browser alerts and a possible security threat (see above), making it easier to deploy HTTPS.
Let's do it then.
All right. :-)
I need some support here, because I am running out of time.
Which web server are you running? At the moment, I am only using Apache and could provide some support here.
Best regards, Peter Müller
Needless to say, DANE/HSTS/CAA are obligatory.
Best regards, Peter Müller
-Michael