Re: CA and HTTPS usage on *.ipfire.org

["Peter Müller" <peter.mueller@link38.eu>]

[-- Attachment #1: Type: text/plain, Size: 25067 bytes --]

Hello Michael,

> Thanks for the input. Let's go through it bit by bit...
> 
> On Wed, 2017-10-25 at 22:12 +0200, Peter Müller wrote:
> > Hello Michael,
> >   
> > > Hello altogether,
> > > 
> > > to bring some movement into this I went ahead and installed Let's Encrypt
> > > certificates pretty much everywhere (with exception of the internal services
> > > like LDAP).
> > > 
> > > There you have it. I did it. I bit the bullet.  
> > 
> > Thank you very much. :-) I really appreciate this.  
> > > 
> > > It literally took me days since we have so many subdomains and I issued one
> > > certificate per domain which makes it easier to renew and revoke them if
> > > needed.
> > > They also throttled me to only issue a small number of certificates per
> > > week.
> > > 
> > > But in summary these things have changed (I would like to get some feedback
> > > from
> > > you since you all have probably a little bit more experience with a few
> > > things
> > > than I have):
> > > 
> > > * LE deployed for all web services with the exception of boot.ipfire.org and
> > > fireinfo.
> > > 
> > > iPXE doesn't really support HTTPS that well and I shipped a new version of
> > > it
> > > that at least downloads some certificates (although it cannot really
> > > validate
> > > them - it can only check than a CA exists I think).  
> > 
> > Hmmm, that is not very good - of course, everything is better than plaintext,
> > but without validating the certificates...
> > 
> > How widely is iPXE used?  
> 
> Loads of people use it. Probably every VM runs it :)
> 
> I enabled this: http://ipxe.org/cfg/crosscert
> 
> https://cgit.ipfire.org/oddments/ipfire-netboot.git/commit/?id=871fc3184ad5ece1633833f169e771629306ed94
> 
> I just found out that is has cross-signed all CAs in that folder with their own
> CA that is baked into the image, so they can verify if the CA they downloaded is
> actually trusted or not. It's not perfect, but I think it is good enough.
> 
> It is possible to bake in a CA into the image, but I didn't want to set it to LE
> and nothing else and there is not enough space in the image to import the usual
> CA bundle. I think the final image cannot be larger than 1M and the current
> IPFire CA bundle that we ship is about 700k.
That is another item on my list: To check the certificate bundle and throw out
outdated and suspicious - such as BlueCoat, CINNIC, etc. - ones. But this requires
some spare time, which I lack at the moment. :-)
> 
> http://ipxe.org/crypto
> 
> > > 
> > > Same with fireinfo. I have no idea if the profiles can be sent properly over
> > > HTTPS without touching the client first.  
> > 
> > I guess we need to implement that in the clients first. At the moment I am
> > quite
> > busy reworking some broken patches and other stuff, but will have a look at
> > that
> > occasionally.  
> 
> That would be cool. It uses the standard python urllib2 module which supports
> SSL. I am just not sure if it would follow the redirect properly. Needs some
> testing.
> 
> https://cgit.ipfire.org/oddments/fireinfo.git/tree/src/sendprofile
> 
> > Using HTTPS on fireinfo an mirrors might be a good idea since a (even passive)
> > attacker can see a lot of information by simply looking at the network traffic
> > (Which version is the firewall running? Which architecture and network zones
> > are in use? And so on.)  
> 
> Our default mirror only supports HTTPS from now on and will redirect. However,
> the load-balancer will only redirect to HTTP which will then redirect to HTTPS.
> That's not ideal.
> 
> I need to add this here:
>   https://cgit.ipfire.org/ipfire.org.git/tree/webapp/backend/mirrors.py#n418
> 
> It's very easy just to flag in the database which mirrors support HTTPS. But
> there will always be a chance that a client is redirected to a mirror that only
> supports HTTP.
Maybe the mirror hosters change this if we notify them. (Side question: What does
the '*' at https://mirrors.ipfire.org/ mean? It seems to be somewhat random...)

We could also ship a pakfire mirror list with information about the HTTPS state
of a mirror - and prevent the firewalls being redirected to plaintext if a mirror
supports HTTPS.
> 
> I already did this for the Pakfire Build Service:
>   https://cgit.ipfire.org/pbs.git/commit/?id=3163c789758423fad61335995b206e5af3a25f4f
> 
> > Just as a side note: Validating this traffic against DANE records would be
> > nice. :-)  
> 
> Oh I forgot to say that we have TLSA records for everything in here. Probably
> nobody validates this, but hey :)
Well, there is a browser plugin for FF, which crashes surprisingly seldom on my
machine. :-)
> 
> [root(a)mail01 ~]# dig TLSA _443._tcp.www.ipfire.org
> 
> ; <<>> DiG 9.11.1-P3-RedHat-9.11.1-2.P3.fc26 <<>> TLSA _443._tcp.www.ipfire.org
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41901
> ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
> 
> ;; OPT PSEUDOSECTION:
> ; EDNS: version: 0, flags:; udp: 4096
> ;; QUESTION SECTION:
> ;_443._tcp.www.ipfire.org.      IN      TLSA
> 
> ;; ANSWER SECTION:
> _443._tcp.www.ipfire.org. 21599 IN      CNAME   _letsencrypt.certs.ipfire.org.
> _letsencrypt.certs.ipfire.org. 21599 IN TLSA    2 1 1 60B87575447DCBA2A36B7D11AC09FB24A9DB406FEE12D2CC90180517 616E8A18
> 
> ;; Query time: 99 msec
> ;; SERVER: 172.28.1.1#53(172.28.1.1)
> ;; WHEN: Thu Oct 26 12:11:06 CEST 2017
> ;; MSG SIZE  rcvd: 133
> 
> > > 
> > > * LE deployed on mail01.i.ipfire.org (Postfix + Dovecot)  
> > 
> > Yep, seems to work.
> > 
> > However, a client certificate in Postfix is still missing:
> > 
> > Received: from mail01.ipfire.org (mail01.ipfire.org [81.3.27.42])
> > 	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))  
> > -->	(Client did not present a certificate) <--  
> > 	[etc.]
> > 
> > In your main.cf, it can be enabled by using:
> > 
> > smtp_tls_cert_file = /Path/to/your/certificate
> > smtp_tls_key_file = /Path/to/your/private/key
> > 
> > You might want not only to query client certificates, but also validate
> > them. To do so, use:
> > 
> > smtpd_tls_CAfile = /Path/to/certificate/block
> > 
> > This requires all trusted intermediate certificates to be put together
> > in one file, as far as I can remember.
> > 
> > (See http://www.postfix.org/postconf.5.html#smtpd_tls_CAfile for details)  
> 
> I didn't have that set. Set it to this now:
>   smtpd_tls_CAfile = /etc/pki/tls/certs/ca-bundle.crt
Okay, so this mail should have a "(Verified OK)" somewhere at a Received-header...
> 
> > I did not test the mail server's cipherlist, but I hope it is secure for
> > both SMTP server and client. :-)  
> 
> Probably not :)
> 
> I think we both should walk through the configuration together...
Okay.
> 
> > > * LE deployed on im01.i.ipfire.org which runs Prosody, our XMPP server
> > > 
> > > * I enabled HSTS for all web services since all of them are forcing the
> > > browser
> > > to use HTTPS.  
> > 
> > How about the IPFire mirrors? If one of them (i.e. "mirror1.ipfire.org")
> > supports
> > HTTPS, are the systems following the 30x redirects? In the future maybe
> > hardcoding
> > the protocol might be a good idea: If a mirror supports HTTPS, we use it
> > there.  
> 
> See above. And yes, clients follow a large number of redirects.
> Firefox follows 20.
> 
> > But that is only a minor thing.
> > 
> > In general, HSTS is very nice to have here. :-)  
> > > 
> > > * We only allow TLS 1.2 for web.  
> > 
> > It seems like the key size is 2048 bits. Although this is not critical, I
> > would
> > recommend to move to 4096 bits at the next key rollover.  
> 
> Well, I didn't check. Because security wasn't really what I had in mind here.
> Our old keys were 4096 bits long. I didn't even know this could be changed.
> 
> > Let's Encrypt announced to support ECDSA in early 2018, so at that time we
> > could
> > also use dual-stack RSA and ECDSA, as we are doing with IPFire in Core Update
> > 115.  
> 
> They promised a lot. Let's see :)
> 
> > > * Since everything is on SSL now, I also enabled HTTP/2.0.
> > > 
> > > * I also enabled OCSP stapling.  
> > 
> > Excellent.  
> > > 
> > > The nginx configuration file looks as follows:
> > > 
> > >   # Enable TLS 1.2 only
> > >   ssl_protocols TLSv1.2;
> > > 
> > >   # Enable protection against BEAST attacks
> > >   ssl_prefer_server_ciphers on;
> > >   ssl_ciphers "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-
> > > SHA384:ECDHE-
> > >   ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-
> > > GCM-
> > >   SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-
> > > AES256-
> > >   SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256";  
> > 
> > Apparently there is an issue here: SSL Labs does not display the Chacha/Poly
> > cipher.
> > Perhaps nginx is linked against OpenSSL 1.0.x ?  
> 
> The web server is running openssl 1.0.2k. Unfortunately it is still on Fedora 24
> which I need to update as soon as I can.
It seems like you have done so meanwhile, since SSLLabs now shows Chacha/Poly, too.
> 
> > I am not sure if we should prioritise this cipher suite over AES since it is
> > safe
> > too, but seems to faster on devices without AES-NI and a small CPU, such as
> > Android smartphones. Decision is up to you.  
> 
> I got this from Mozilla, so I thought they considered this.
Yes, as far as I can remember, there was a discussion about this on some Mozilla
mailing list.
> 
> > >   # Enable session caching
> > >   ssl_session_cache shared:SSL:50m;
> > >   ssl_session_timeout 1d;
> > > 
> > >   # Enable Certificate Stapling
> > >   ssl_stapling on;
> > >   ssl_stapling_verify on;
> > >   resolver 172.28.1.1;
> > > 
> > >   # HSTS (15768000 seconds = 6 months)
> > >   add_header Strict-Transport-Security max-age=15768000;
> > > 
> > > This is inspired by the Mozilla recommendations.
> > > 
> > > So please guys, have a test. I am quite sure that although this is quite a
> > > "modern" configuration all recent web browsers should work fine. If you are
> > > on
> > > IE6, you are fucked. Tough luck.  
> > 
> > Yes, I think supporting legacy clients is not a very good idea. Of course,
> > there
> > are some scenarios where you must do so (big company website, or anything
> > else),
> > but for this, I consider it safe.
> > 
> > Food for thoughts: Is there anything against updating OpenSSL to 1.1.x in
> > IPFire?
> > That way, we could also use Chacha/Poly there and disable TLS 1.0, but I am
> > not
> > sure how many clients will be broken by this.  
> 
> I don't think there is nothing that is stopping us any more. For quite some time
> nothing built against OpenSSL 1.1 since they changed a lot in the API. Of course
> we will still need to ship 1.0.1 because loads of things link against it.
> 
> Before we also shipped OpenSSL 0.9.8 for quite a while until it was
> discontinued:
> 
> https://cgit.ipfire.org/ipfire-2.x.git/commit/lfs/openssl-compat?id=45ff420ec7e3
> ad522dc6d0e53837a6e1951407fc
This is my opinion, too. Of couse, shipping to version of one program is ugly,
but I consider this safe here if OpenSSL 1.1.x breaks some things entirely.
> 
> About disabling TLS 1.0. I am all for it, but it needs some good testing. One
> common scenario is that people use Postfix as a mail relay to deliver emails to
> an internal Exchange server. Those often require RC4 and MD5. I think that is
> Exchange 2003. This is bad and ugly but this is what people are using in 2017.
> And we need to be able to enable these ciphers optionally when they are
> required.
As far as I am concerned, you have to manually build OpenSSL 1.1.x with RC4 support,
by default it does not support it anymore.

TLS with mailservers is always somewhat special since its goal is completely
different from TLS used at web services. Mail servers want to prevent plain text
transfer (protection against a passive attacker), and here, a weak encryption is
better than none.

Of course, I would be grateful if these old Exchange instances disappear...
> 
> However, we should have strong defaults.
> 
> > > But if you find anything that should work, please let me know. You can also
> > > run
> > > some scanners against it and see what happens. I did a few and they show A+
> > > rating which is what we are looking for.
> > > 
> > > Anything that I have forgotten to enable which could benefit security?  
> > 
> > Not that I am aware of at the moment (besides the stuff mentioned above).
> > Great job!  
> 
> Cool!
> 
> -Michael
> 
> > 
> > Best regards,
> > Peter Müller  
> > > 
> > > Looking for your input :)
> > > 
> > > Best,
> > > -Michael
> > > 
> > > On Mon, 2017-09-25 at 17:48 +0200, Peter Müller wrote:  
> > > > Hello Michael,
> > > >     
> > > > > Hi,
> > > > > 
> > > > > On Fri, 2017-09-15 at 22:26 +0200, Peter Müller wrote:    
> > > > > > Hello Michael, hello Matthias,
> > > > > > 
> > > > > > a few thoughts on this from me:
> > > > > > 
> > > > > > (1) @Michael: Thank you for having a look at this. In
> > > > > > my opinion, enabling HTTPS on all *.ipfire.org sites by
> > > > > > default would be a huge security benefit.
> > > > > > 
> > > > > > Of course, this needs time and testing, so no pressure. :-)
> > > > > > 
> > > > > > Especially on download sites HTTPS would be nice in
> > > > > > case someone verifies the downloaded ISO - when transmitted
> > > > > > in plaintext, both image and checksum might be modified
> > > > > > by a MITM.
> > > > > > 
> > > > > > Further, there is still SHA1 in use on http://download.ipfire.org/,
> > > > > > but that is another issue.      
> > > > > 
> > > > > Indeed it is. We have ticket for that:
> > > > >   https://bugzilla.ipfire.org/show_bug.cgi?id=11345    
> > > > 
> > > > Thanks for the ticket link.    
> > > > >     
> > > > > > (2) Speaking about the CAs: I do not know what opinion
> > > > > > to have here.
> > > > > > 
> > > > > > On the one hand, using your own project CA is good since
> > > > > > nobody except the visitors has to rely on some external
> > > > > > certificate signing company.      
> > > > > 
> > > > > Agreed. I do not trust any of those public CAs - at all. They have a
> > > > > commercial interest and that is it.
> > > > >     
> > > > > > Publishing the TLSA-records tells everyone that this CA -
> > > > > > which is untrusted by all current browsers - is legitimate
> > > > > > on *.ipfire.org.      
> > > > > 
> > > > > We do this.
> > > > > 
> > > > > http://planet.ipfire.org/post/ipfire-org-is-signed-now    
> > > > 
> > > > I forgot that. Unfortunately, at the moment, "only" mail servers
> > > > benefit from this (see below).    
> > > > >     
> > > > > > On the other hand, there are two points against it:
> > > > > > (a) Nobody actually uses TLSA/DANE for validating web site
> > > > > > certificates (this is different when it comes to SMTP). So,
> > > > > > nearly every user will see a certificate warning.      
> > > > > 
> > > > > Indeed not a single browser is verifying it. There is plugins that can
> > > > > show an icon and that's it.
> > > > > 
> > > > > We also do this for our mail server which does not have a publicly
> > > > > signed certificate.    
> > > > 
> > > > As far as I am concerned, there are two different approaches to TLS:
> > > > 
> > > > (a) If you are running a mail server, TLS is mostly used in opportunistic
> > > > mode, and certificates are not that important. The idea behind this
> > > > is to prevent plaintext transport of e-mails, that's why so many big
> > > > MX still support weak ciphers such as 3DES. (Personally, I am not a
> > > > fan of opportunistic TLS since it only protects against passive attackers
> > > > and one has to use ancient algorithms. At least RC4-MD5 finally
> > > > disappeared...)
> > > > 
> > > > So, a self-signed certificate for a MX is no problem - at least, not
> > > > a big one. Only a few MX actually ask for a client certificate, and even
> > > > less systems are configured to present any.
> > > > 
> > > > (b) Web browsers have a completely different view on this: They first
> > > > check the certificate against the DNS name, and if they match, encryption
> > > > can be started. On the other hand, in case they see an untrusted
> > > > certificate
> > > > for whatever reason, everything becomes bad very quickly.
> > > > 
> > > > Integrity is much more important here in order to protect the users
> > > > against active (MITM) attackers.
> > > > 
> > > > Because of this, I would prefer certs signed by well-known CAs for public
> > > > web sites.
> > > >     
> > > > >     
> > > > > > Personally, i suppose DANE will never be implemented in
> > > > > > web browsers. There are too many collisions with the systems
> > > > > > DNS service, and network stacks, and the network's firewall
> > > > > > rules, and so on. Sad, but that is what we (do not) have.      
> > > > > 
> > > > > I am not convinced that this has technical reasons. It just puts some
> > > > > big businesses will lose a machine that prints them money.    
> > > > 
> > > > Partly I think. DANE depends on DNSSEC, and I remember some user
> > > > complaints
> > > > that their ISPs filter DNS queries to external name servers without
> > > > using it on their own systems.    
> > > > > 
> > > > > Nobody there is interested in the security. They are interested in the
> > > > > money.    
> > > > 
> > > > Without being very deep into this, I assume you are right.    
> > > > >     
> > > > > > So, enabling HTTPS on all or at least some very important
> > > > > > IPFire sites will cause more user complaints.      
> > > > > 
> > > > > Indeed. That is why we don't enforce it.
> > > > >     
> > > > > > Further, and that is a _very_ big problem in my eyes, it
> > > > > > actually decreases transport encryption security: Every time
> > > > > > a user accesses *.ipfire.org, he/she/it has to bypass a
> > > > > > certificate warning - at least in FF, there is no easy way
> > > > > > to store the exception permanently.
> > > > > > 
> > > > > > In case a MITM injects his own certificate to break TLS,
> > > > > > the user will see a certificate warning which is nearly
> > > > > > identical to those shown usually for IPFire's CA. And the
> > > > > > user will mostly bypass this - very well-known - warning,
> > > > > > an the attacker won.
> > > > > > 
> > > > > > To prevent this, you MUST check the certificates checksum
> > > > > > against the value you know is legitimate. Every time.
> > > > > > 
> > > > > > Nobody will do so.      
> > > > > 
> > > > > Agreed.
> > > > >     
> > > > > > (b) As I mentioned above, I completely understand your point
> > > > > > having an own CA. However, I do not think this is so important
> > > > > > for public web services:      
> > > > > 
> > > > > I emphasise here on "web services". I am not even sure if Let's Encrypt
> > > > > can issue certificates for our internal LDAP server or our mail server,
> > > > > etc.    
> > > > 
> > > > This depends on the actual network (especially DNS) configuration.
> > > > If the server does not have a public IP, but is configured in the DNS,
> > > > LE can validate it without a direct connection:
> > > > - https://community.letsencrypt.org/t/cert-for-intranet/6337/4 
> > > > - https://community.letsencrypt.org/t/on-the-state-of-the-dns-01-challenge
> > > > /480
> > > > 5    
> > > > >     
> > > > > > As far as I am concerned, there is no "trust" in case of CA
> > > > > > companies. Some of them showed a really unprofessional behaviour
> > > > > > (DigiNotar, WoSign, Symantec, ...) and are/were either kicked
> > > > > > off business by their customers or the web browser developers.      
> > > > > 
> > > > > So why would we assume that Let's Encrypt is doing a better job? Just
> > > > > because they are sponsored by some of the "good guys"?
> > > > >     
> > > > > > You only "trust" a CA to validate if a certain domain really
> > > > > > belongs to the person who claims so.
> > > > > > 
> > > > > > In case there is a certificate in use signed by a foreign CA,
> > > > > > what should happen? Nobody except the server admin has the
> > > > > > private key, and there are many techniques to prevent modern
> > > > > > browsers accepting any certificate for your domain, such as:      
> > > > > 
> > > > > So let's split all of this:
> > > > >     
> > > > > > - DANE/TLSA (specifies which certificate [chain] is valid)      
> > > > > 
> > > > > We can just install that for the Let's Encrypt CA. It doesn't make
> > > > > sense to have TLSA records for each cert because they will be replaced
> > > > > very very quickly.    
> > > > 
> > > > I remember a trick here: If you do not rotate the private key every
> > > > time LE's signature expires, the public one also stays the same -
> > > > and the TLSA record does not need to be changed.
> > > > 
> > > > Can't find the link right now...    
> > > > >     
> > > > > > - CAA (DNS record, rather new, specifies which CAs can issue
> > > > > > certificates for this site, might be supported by common browsers
> > > > > > some day)      
> > > > > 
> > > > > I need to implement that in our DNS server software, but that is not a
> > > > > big thing. Will do that soon.    
> > > > 
> > > > Thanks.    
> > > > >     
> > > > > > - HSTS (preventing a MITM from downgrading to plaintext)      
> > > > > 
> > > > > Who wants to take care of this?    
> > > > 
> > > > Actually, HSTS is quite easy to deploy since it only tells web browsers
> > > > to enforce TLS on a certain site. So, if there are no dependencies
> > > > (unencrypted external resources, ad servers, ...), you can just set it
> > > > up and forget it. :-)    
> > > > >     
> > > > > > - HPKP (as DANE, but supported by modern browsers, TOFU
> > > > > > [Trust On First Use] = first connection must be clean)      
> > > > > 
> > > > > Same.    
> > > > 
> > > > Indeed, HPKP is more complicated, it is like DANE without DNS.
> > > > Further, some security experts (such as Ivan Ristic) argue that
> > > > it might become dangerous in case of misconfiguration.    
> > > > > 
> > > > > And who would like to create tickets on BZ for all of this?
> > > > >     
> > > > > > Except from going out of business, I do not see any risk for
> > > > > > a web site owner here. That is, in case all or some of the
> > > > > > mentioned methods are implemented.      
> > > > > 
> > > > > LOL, we are not a business. We are an Open Source project.    
> > > > 
> > > > "Going out of business" was relating to CAs, not to IPFire. But
> > > > yes, it would be very sad if this project disappears.    
> > > > >     
> > > > > > Let's Encrypt (LE) seems to be different from other CAs since
> > > > > > they use standardised methods and act quite transparently.
> > > > > > Further, there are some "major players" behind it (Mozilla,
> > > > > > EFF, ...) which have a good reputation. This - hopefully -
> > > > > > means that there will be no security breaches like in case
> > > > > > of DigiNotar.      
> > > > > 
> > > > > I don't share your hope, but I do not want to fight this. I have been
> > > > > trialing Let's Encrypt a little bit on some web shops and other things
> > > > > and it does the job. To be honest I do not see the point in spending
> > > > > the money on a different vendor of certificates. So take that as: They
> > > > > are not worse than those.    
> > > > 
> > > > I don't want to fight either; but I am afraid I did not fully
> > > > understand your point concerning the trust/distrust of public CAs
> > > > (except for obvious reasons like security breaches).    
> > > > >     
> > > > > > To come to an end, using LE as a CA for *.ipfire.org will
> > > > > > not harm the security of IPFire significantly. It reduces
> > > > > > browser alerts and a possible security threat (see above),
> > > > > > making it easier to deploy HTTPS.      
> > > > > 
> > > > > Let's do it then.    
> > > > 
> > > > All right. :-)    
> > > > > 
> > > > > I need some support here, because I am running out of time.    
> > > > 
> > > > Which web server are you running? At the moment, I am only using
> > > > Apache and could provide some support here.
> > > > 
> > > > Best regards,
> > > > Peter Müller    
> > > > >     
> > > > > > Needless to say, DANE/HSTS/CAA are obligatory.
> > > > > > 
> > > > > > Best regards,
> > > > > > Peter Müller      
> > > > > 
> > > > > -Michael    
> > > > 
> > > >     
> > 
> >