Hi,
On Thu, 2017-10-26 at 21:24 +0200, Peter Müller wrote:
Hello Michael,
Thanks for the input. Let's go through it bit by bit...
On Wed, 2017-10-25 at 22:12 +0200, Peter Müller wrote:
Hello Michael,
Hello altogether,
to bring some movement into this I went ahead and installed Let's Encrypt certificates pretty much everywhere (with exception of the internal services like LDAP).
There you have it. I did it. I bit the bullet.
Thank you very much. :-) I really appreciate this.
It literally took me days since we have so many subdomains and I issued one certificate per domain which makes it easier to renew and revoke them if needed. They also throttled me to only issue a small number of certificates per week.
But in summary these things have changed (I would like to get some feedback from you since you all have probably a little bit more experience with a few things than I have):
- LE deployed for all web services with the exception of boot.ipfire.org
and fireinfo.
iPXE doesn't really support HTTPS that well and I shipped a new version of it that at least downloads some certificates (although it cannot really validate them - it can only check than a CA exists I think).
Hmmm, that is not very good - of course, everything is better than plaintext, but without validating the certificates...
How widely is iPXE used?
Loads of people use it. Probably every VM runs it :)
I enabled this: http://ipxe.org/cfg/crosscert
https://cgit.ipfire.org/oddments/ipfire-netboot.git/commit/?id=871fc3184ad5e ce1633833f169e771629306ed94
I just found out that is has cross-signed all CAs in that folder with their own CA that is baked into the image, so they can verify if the CA they downloaded is actually trusted or not. It's not perfect, but I think it is good enough.
It is possible to bake in a CA into the image, but I didn't want to set it to LE and nothing else and there is not enough space in the image to import the usual CA bundle. I think the final image cannot be larger than 1M and the current IPFire CA bundle that we ship is about 700k.
That is another item on my list: To check the certificate bundle and throw out outdated and suspicious - such as BlueCoat, CINNIC, etc. - ones. But this requires some spare time, which I lack at the moment. :-)
We get it from Mozilla but it hasn't been updated in quite some time now. We probably need to do a better job with it and if you would put yourself forward as a maintainer that would be amazing.
However, going through this is probably not going to let you sleep well. There is scary stuff in it, but I do not see that it makes sense to filter this out manually. The chance to miss something is too high.
But if you have good ideas, let's talk about them...
Same with fireinfo. I have no idea if the profiles can be sent properly over HTTPS without touching the client first.
I guess we need to implement that in the clients first. At the moment I am quite busy reworking some broken patches and other stuff, but will have a look at that occasionally.
That would be cool. It uses the standard python urllib2 module which supports SSL. I am just not sure if it would follow the redirect properly. Needs some testing.
https://cgit.ipfire.org/oddments/fireinfo.git/tree/src/sendprofile
Using HTTPS on fireinfo an mirrors might be a good idea since a (even passive) attacker can see a lot of information by simply looking at the network traffic (Which version is the firewall running? Which architecture and network zones are in use? And so on.)
Our default mirror only supports HTTPS from now on and will redirect. However, the load-balancer will only redirect to HTTP which will then redirect to HTTPS. That's not ideal.
I need to add this here: https://cgit.ipfire.org/ipfire.org.git/tree/webapp/backend/mirrors.py#n418
It's very easy just to flag in the database which mirrors support HTTPS. But there will always be a chance that a client is redirected to a mirror that only supports HTTP.
Maybe the mirror hosters change this if we notify them. (Side question: What does the '*' at https://mirrors.ipfire.org/ mean? It seems to be somewhat random...)
It means that you will be redirected to one of those mirrors. They are "close" to you according to your GeoIP location.
Probably this needs some updating since it is quite slow. The Pakfire Build Service uses a better and quite simple approach which seems to work well.
We could also ship a pakfire mirror list with information about the HTTPS state of a mirror - and prevent the firewalls being redirected to plaintext if a mirror supports HTTPS.
Yes, if you dare to touch the old Pakfire :)
I already did this for the Pakfire Build Service: https://cgit.ipfire.org/pbs.git/commit/?id=3163c789758423fad61335995b206e5 af3a25f4f
Just as a side note: Validating this traffic against DANE records would be nice. :-)
Oh I forgot to say that we have TLSA records for everything in here. Probably nobody validates this, but hey :)
Well, there is a browser plugin for FF, which crashes surprisingly seldom on my machine. :-)
I used to use that, but just the icon doesn't really help much. And for stability and performance of my system I uninstalled this some time ago.
[root@mail01 ~]# dig TLSA _443._tcp.www.ipfire.org
; <<>> DiG 9.11.1-P3-RedHat-9.11.1-2.P3.fc26 <<>> TLSA _443._tcp.www.ipfire. org ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41901 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;_443._tcp.www.ipfire.org. IN TLSA
;; ANSWER SECTION: _443._tcp.www.ipfire.org. 21599 IN CNAME _letsencrypt.certs.ipfire.org. _letsencrypt.certs.ipfire.org. 21599 IN TLSA 2 1 1 60B87575447DCBA2A36B7D11AC09FB24A9DB406FEE12D2CC90180517 616E8A18
;; Query time: 99 msec ;; SERVER: 172.28.1.1#53(172.28.1.1) ;; WHEN: Thu Oct 26 12:11:06 CEST 2017 ;; MSG SIZE rcvd: 133
- LE deployed on mail01.i.ipfire.org (Postfix + Dovecot)
Yep, seems to work.
However, a client certificate in Postfix is still missing:
Received: from mail01.ipfire.org (mail01.ipfire.org [81.3.27.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) --> (Client did not present a certificate) <-- [etc.]
In your main.cf, it can be enabled by using:
smtp_tls_cert_file = /Path/to/your/certificate smtp_tls_key_file = /Path/to/your/private/key
You might want not only to query client certificates, but also validate them. To do so, use:
smtpd_tls_CAfile = /Path/to/certificate/block
This requires all trusted intermediate certificates to be put together in one file, as far as I can remember.
(See http://www.postfix.org/postconf.5.html#smtpd_tls_CAfile for details)
I didn't have that set. Set it to this now: smtpd_tls_CAfile = /etc/pki/tls/certs/ca-bundle.crt
Okay, so this mail should have a "(Verified OK)" somewhere at a Received- header...
It does when you send an email from gmail.com to your IPFire email address for example. I think this looks correct now.
I did not test the mail server's cipherlist, but I hope it is secure for both SMTP server and client. :-)
Probably not :)
I think we both should walk through the configuration together...
Okay.
Cool.
LE deployed on im01.i.ipfire.org which runs Prosody, our XMPP server
I enabled HSTS for all web services since all of them are forcing the
browser to use HTTPS.
How about the IPFire mirrors? If one of them (i.e. "mirror1.ipfire.org") supports HTTPS, are the systems following the 30x redirects? In the future maybe hardcoding the protocol might be a good idea: If a mirror supports HTTPS, we use it there.
See above. And yes, clients follow a large number of redirects. Firefox follows 20.
But that is only a minor thing.
In general, HSTS is very nice to have here. :-)
- We only allow TLS 1.2 for web.
It seems like the key size is 2048 bits. Although this is not critical, I would recommend to move to 4096 bits at the next key rollover.
Well, I didn't check. Because security wasn't really what I had in mind here. Our old keys were 4096 bits long. I didn't even know this could be changed.
Let's Encrypt announced to support ECDSA in early 2018, so at that time we could also use dual-stack RSA and ECDSA, as we are doing with IPFire in Core Update 115.
They promised a lot. Let's see :)
Since everything is on SSL now, I also enabled HTTP/2.0.
I also enabled OCSP stapling.
Excellent.
The nginx configuration file looks as follows:
# Enable TLS 1.2 only ssl_protocols TLSv1.2;
# Enable protection against BEAST attacks ssl_prefer_server_ciphers on; ssl_ciphers "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM- SHA384:ECDHE- ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA- AES128- GCM- SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE- RSA- AES256- SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256";
Apparently there is an issue here: SSL Labs does not display the Chacha/Poly cipher. Perhaps nginx is linked against OpenSSL 1.0.x ?
The web server is running openssl 1.0.2k. Unfortunately it is still on Fedora 24 which I need to update as soon as I can.
It seems like you have done so meanwhile, since SSLLabs now shows Chacha/Poly, too.
Yes, I did. Also manually verified this. We are on OpenSSL 1.1.x now.
I am not sure if we should prioritise this cipher suite over AES since it is safe too, but seems to faster on devices without AES-NI and a small CPU, such as Android smartphones. Decision is up to you.
I got this from Mozilla, so I thought they considered this.
Yes, as far as I can remember, there was a discussion about this on some Mozilla mailing list.
# Enable session caching ssl_session_cache shared:SSL:50m; ssl_session_timeout 1d;
# Enable Certificate Stapling ssl_stapling on; ssl_stapling_verify on; resolver 172.28.1.1;
# HSTS (15768000 seconds = 6 months) add_header Strict-Transport-Security max-age=15768000;
This is inspired by the Mozilla recommendations.
So please guys, have a test. I am quite sure that although this is quite a "modern" configuration all recent web browsers should work fine. If you are on IE6, you are fucked. Tough luck.
Yes, I think supporting legacy clients is not a very good idea. Of course, there are some scenarios where you must do so (big company website, or anything else), but for this, I consider it safe.
Food for thoughts: Is there anything against updating OpenSSL to 1.1.x in IPFire? That way, we could also use Chacha/Poly there and disable TLS 1.0, but I am not sure how many clients will be broken by this.
I don't think there is nothing that is stopping us any more. For quite some time nothing built against OpenSSL 1.1 since they changed a lot in the API. Of course we will still need to ship 1.0.1 because loads of things link against it.
Before we also shipped OpenSSL 0.9.8 for quite a while until it was discontinued:
https://cgit.ipfire.org/ipfire-2.x.git/commit/lfs/openssl-compat?id=45ff420e c7e3 ad522dc6d0e53837a6e1951407fc
This is my opinion, too. Of couse, shipping to version of one program is ugly, but I consider this safe here if OpenSSL 1.1.x breaks some things entirely.
I would prefer to have the entire system to build against the new one, but sometime people compile their own software and run it on IPFire. To not break that, we usually ship the old libraries for quite a while.
We removed openssl 0.9.8 because it wasn't patched any more.
About disabling TLS 1.0. I am all for it, but it needs some good testing. One common scenario is that people use Postfix as a mail relay to deliver emails to an internal Exchange server. Those often require RC4 and MD5. I think that is Exchange 2003. This is bad and ugly but this is what people are using in 2017. And we need to be able to enable these ciphers optionally when they are required.
As far as I am concerned, you have to manually build OpenSSL 1.1.x with RC4 support, by default it does not support it anymore.
My honest opinion here is that this is how it should be.
Anyone who relies on it needs to stop using it. Right now. It's not a *nice to have*. It's wrong and bad and ugly.
But unfortunately I get the phone calls where people complain about this not being supported any more. So there has to be some sort of trade-off.
I probably exaggerated this example a little bit too much, but there is certainly some patches in the system where we had to downgrade security a little for better compatibility. This is always difficult, but things like RC4 are not a question for me. I know what I would decide.
TLS with mailservers is always somewhat special since its goal is completely different from TLS used at web services. Mail servers want to prevent plain text transfer (protection against a passive attacker), and here, a weak encryption is better than none.
Maybe, but the question then is: Does RC4 still qualify as "encryption" where a passive attack is so simple to perform? It is a way of scrambling the data but that is about it.
Of course, I would be grateful if these old Exchange instances disappear...
Yes, and we have to work towards that by announcing that we are going to discontinue support for certain things then.
-Michael
However, we should have strong defaults.
But if you find anything that should work, please let me know. You can also run some scanners against it and see what happens. I did a few and they show A+ rating which is what we are looking for.
Anything that I have forgotten to enable which could benefit security?
Not that I am aware of at the moment (besides the stuff mentioned above). Great job!
Cool!
-Michael
Best regards, Peter Müller
Looking for your input :)
Best, -Michael
On Mon, 2017-09-25 at 17:48 +0200, Peter Müller wrote:
Hello Michael,
Hi,
On Fri, 2017-09-15 at 22:26 +0200, Peter Müller wrote: > Hello Michael, hello Matthias, > > a few thoughts on this from me: > > (1) @Michael: Thank you for having a look at this. In > my opinion, enabling HTTPS on all *.ipfire.org sites by > default would be a huge security benefit. > > Of course, this needs time and testing, so no pressure. :-) > > Especially on download sites HTTPS would be nice in > case someone verifies the downloaded ISO - when transmitted > in plaintext, both image and checksum might be modified > by a MITM. > > Further, there is still SHA1 in use on http://download.ipfire.org/ > , > but that is another issue.
Indeed it is. We have ticket for that: https://bugzilla.ipfire.org/show_bug.cgi?id=11345
Thanks for the ticket link.
> (2) Speaking about the CAs: I do not know what opinion > to have here. > > On the one hand, using your own project CA is good since > nobody except the visitors has to rely on some external > certificate signing company.
Agreed. I do not trust any of those public CAs - at all. They have a commercial interest and that is it.
> Publishing the TLSA-records tells everyone that this CA - > which is untrusted by all current browsers - is legitimate > on *.ipfire.org.
We do this.
I forgot that. Unfortunately, at the moment, "only" mail servers benefit from this (see below).
> On the other hand, there are two points against it: > (a) Nobody actually uses TLSA/DANE for validating web site > certificates (this is different when it comes to SMTP). So, > nearly every user will see a certificate warning.
Indeed not a single browser is verifying it. There is plugins that can show an icon and that's it.
We also do this for our mail server which does not have a publicly signed certificate.
As far as I am concerned, there are two different approaches to TLS:
(a) If you are running a mail server, TLS is mostly used in opportunistic mode, and certificates are not that important. The idea behind this is to prevent plaintext transport of e-mails, that's why so many big MX still support weak ciphers such as 3DES. (Personally, I am not a fan of opportunistic TLS since it only protects against passive attackers and one has to use ancient algorithms. At least RC4-MD5 finally disappeared...)
So, a self-signed certificate for a MX is no problem - at least, not a big one. Only a few MX actually ask for a client certificate, and even less systems are configured to present any.
(b) Web browsers have a completely different view on this: They first check the certificate against the DNS name, and if they match, encryption can be started. On the other hand, in case they see an untrusted certificate for whatever reason, everything becomes bad very quickly.
Integrity is much more important here in order to protect the users against active (MITM) attackers.
Because of this, I would prefer certs signed by well-known CAs for public web sites.
> Personally, i suppose DANE will never be implemented in > web browsers. There are too many collisions with the systems > DNS service, and network stacks, and the network's firewall > rules, and so on. Sad, but that is what we (do not) have.
I am not convinced that this has technical reasons. It just puts some big businesses will lose a machine that prints them money.
Partly I think. DANE depends on DNSSEC, and I remember some user complaints that their ISPs filter DNS queries to external name servers without using it on their own systems.
Nobody there is interested in the security. They are interested in the money.
Without being very deep into this, I assume you are right.
> So, enabling HTTPS on all or at least some very important > IPFire sites will cause more user complaints.
Indeed. That is why we don't enforce it.
> Further, and that is a _very_ big problem in my eyes, it > actually decreases transport encryption security: Every time > a user accesses *.ipfire.org, he/she/it has to bypass a > certificate warning - at least in FF, there is no easy way > to store the exception permanently. > > In case a MITM injects his own certificate to break TLS, > the user will see a certificate warning which is nearly > identical to those shown usually for IPFire's CA. And the > user will mostly bypass this - very well-known - warning, > an the attacker won. > > To prevent this, you MUST check the certificates checksum > against the value you know is legitimate. Every time. > > Nobody will do so.
Agreed.
> (b) As I mentioned above, I completely understand your point > having an own CA. However, I do not think this is so important > for public web services:
I emphasise here on "web services". I am not even sure if Let's Encrypt can issue certificates for our internal LDAP server or our mail server, etc.
This depends on the actual network (especially DNS) configuration. If the server does not have a public IP, but is configured in the DNS, LE can validate it without a direct connection:
- https://community.letsencrypt.org/t/cert-for-intranet/6337/4
- https://community.letsencrypt.org/t/on-the-state-of-the-dns-01-chall
enge /480 5
> As far as I am concerned, there is no "trust" in case of CA > companies. Some of them showed a really unprofessional behaviour > (DigiNotar, WoSign, Symantec, ...) and are/were either kicked > off business by their customers or the web browser > developers.
So why would we assume that Let's Encrypt is doing a better job? Just because they are sponsored by some of the "good guys"?
> You only "trust" a CA to validate if a certain domain really > belongs to the person who claims so. > > In case there is a certificate in use signed by a foreign CA, > what should happen? Nobody except the server admin has the > private key, and there are many techniques to prevent modern > browsers accepting any certificate for your domain, such as:
So let's split all of this:
> - DANE/TLSA (specifies which certificate [chain] is valid)
We can just install that for the Let's Encrypt CA. It doesn't make sense to have TLSA records for each cert because they will be replaced very very quickly.
I remember a trick here: If you do not rotate the private key every time LE's signature expires, the public one also stays the same - and the TLSA record does not need to be changed.
Can't find the link right now...
> - CAA (DNS record, rather new, specifies which CAs can issue > certificates for this site, might be supported by common browsers > some day)
I need to implement that in our DNS server software, but that is not a big thing. Will do that soon.
Thanks.
> - HSTS (preventing a MITM from downgrading to plaintext)
Who wants to take care of this?
Actually, HSTS is quite easy to deploy since it only tells web browsers to enforce TLS on a certain site. So, if there are no dependencies (unencrypted external resources, ad servers, ...), you can just set it up and forget it. :-)
> - HPKP (as DANE, but supported by modern browsers, TOFU > [Trust On First Use] = first connection must be clean)
Same.
Indeed, HPKP is more complicated, it is like DANE without DNS. Further, some security experts (such as Ivan Ristic) argue that it might become dangerous in case of misconfiguration.
And who would like to create tickets on BZ for all of this?
> Except from going out of business, I do not see any risk for > a web site owner here. That is, in case all or some of the > mentioned methods are implemented.
LOL, we are not a business. We are an Open Source project.
"Going out of business" was relating to CAs, not to IPFire. But yes, it would be very sad if this project disappears.
> Let's Encrypt (LE) seems to be different from other CAs since > they use standardised methods and act quite transparently. > Further, there are some "major players" behind it (Mozilla, > EFF, ...) which have a good reputation. This - hopefully - > means that there will be no security breaches like in case > of DigiNotar.
I don't share your hope, but I do not want to fight this. I have been trialing Let's Encrypt a little bit on some web shops and other things and it does the job. To be honest I do not see the point in spending the money on a different vendor of certificates. So take that as: They are not worse than those.
I don't want to fight either; but I am afraid I did not fully understand your point concerning the trust/distrust of public CAs (except for obvious reasons like security breaches).
> To come to an end, using LE as a CA for *.ipfire.org will > not harm the security of IPFire significantly. It reduces > browser alerts and a possible security threat (see above), > making it easier to deploy HTTPS.
Let's do it then.
All right. :-)
I need some support here, because I am running out of time.
Which web server are you running? At the moment, I am only using Apache and could provide some support here.
Best regards, Peter Müller
> Needless to say, DANE/HSTS/CAA are obligatory. > > Best regards, > Peter Müller
-Michael