Re: Upgrading to OpenSSL 1.1.0

[Horace Michael <horace.michael@gmx.com>]

[-- Attachment #1: Type: text/plain, Size: 18203 bytes --]

Hi,
Two inputs on validation based CRL and performance observed durin tests.

On Tue, 16 Jan 2018 14:56:11 +0200, Development wrote:
> Hi,
> 
> On Tue, 2018-01-16 at 12:36 +0100, ummeegge wrote:
> > Hi Michael,
> > and thanks for the Q&A which is important for me to clear things up, try to
> > make it as short as possible...
> > 
> > Am 15.01.2018 um 12:58 schrieb Michael Tremer:
> > 
> > > wow, long email…
> > 
> > sorry for that ;-) ...
> > 
> > > > I have had no problems building OpenVPN-2.4.4 -->
> > > > https://swupdate.openvpn.org/community/releases/openvpn-2.4.4.tar.xz
> > > > without
> > > > any further modifications of the LFS (except the md5 and version number),
> > > > there was also no need for the LZ4 lib ? What problems appears for you ?
> > > 
> > > None, I just haven't tried to build it because you are the maintainer of
> > > OpenVPN
> > > :)
> > 
> > Oh OK, didn´t know that :D .
> > 
> > > > To my current knowledge MySQL (possibly only an updated one) can also use
> > > > LZ4
> > > > may there are more candidates in IPFire which can participate from this
> > > > compression lib...
> > > 
> > > Not sure if people are using MySQL too much on IPFire. Hopefully they don't
> > > do
> > > that for any performance-critical applications.
> > > 
> > > I guess the only other useful application for LZ4 would be backups.
> > 
> > Do you have an idea where to place lz4 in make.sh ? Have placed it currently
> > before MySQL.
> 
> Put it where the others are as well. Very early in the IPFire stage.
> 
> > > > Another point in here is IPFire uses currently the level 3 of --script-
> > > > security for the RW " 3 -- Allow passwords to be passed to scripts via
> > > > environmental variables (potentially unsafe)" which is important for some
> > > > kinds of authentication methods but which has also been marked as unsafe
> > > > and
> > > > have also be again pointed out while the QuarkLabs and Cryptography
> > > > Engineering LCC audit -->
> > > > https://community.openvpn.net/openvpn/wiki/QuarkslabAndCryptographyEnginee
> > > > rAud
> > > > its --> under "OVPN-08-1: OpenVPN configuration risks: --script-security
> > > > 3"
> > > > you can find there also OpenVPNs statements causing this topic.
> > > 
> > > AFAIK we don't use any of those authentication methods. Or are we?
> > 
> > Possibly all people which uses OpenVPN as a client on IPFire are rely to this
> > auth method cause their VPN Provider sets usually "auth-user-pass-verify"
> > directives. Since this option is not officially supported, i think their on
> > their own...
> 
> Well, we don't support this at the moment. And I think we should leave it like
> this for now since we  want to release OpenSSL as early as possible.
> 
> We can add this later if we need to.
> 
> > 
> > > > The ON/OFF switch should be there to deactivate the automatic cipher
> > > > negotiation not the HWRNG. As far as i remember some crypto accelerators
> > > > can
> > > > only be used with certain types of encryption e.g. -->  https://doc.pfsens
> > > > e.or
> > > > g/index.php/Are_cryptographic_accelerators_supported . So in my
> > > > understanding
> > > > if OpenVPN negotiates for example AES-256-GCM with the client cause the
> > > > appropriate OpenSSL lib do supports it some potential existing
> > > > accelerators
> > > > like e.g. the "AMD Geode LX Security Block" won´t be used. If i get that
> > > > right
> > > > this might be one case where the automatic cipher negotiation should be
> > > > deactivated ? Another one might be setups with vast amounts of clients
> > > > where
> > > > possible less computing power with smaller key lengths is wanted.
> > > > That´s why i disabled the cipher negotiation per default and if wanted one
> > > > click via checkbox can be used to enable it for all clients whereby the
> > > > CCD
> > > > option can be used to deactivate the cipher negotiation for specific
> > > > clients. 
> > > > Not sure if this is the best practice and it might be great if someone can
> > > > give me feedback for this since this has not been debated since now !!!
> > > 
> > > OpenSSL choses to activate these things automatically when they are
> > > available.
> > > There is no need to do anything. Even apache and things like that use them
> > > because they use OpenSSL.
> > > 
> > > However, that is not an HWRNG. That is just a random number generator like
> > > RDRAND on Intel processors.
> > > 
> > > AESNI is not called a HWRNG.
> > 
> > Thanks for clarification on that, wasn´t sure causing this topic !
> > 
> > > 
> > > I do not see a reason why people would want to disable this. They work and
> > > if
> > > someone wants to disable them (because they are malfunctioning), that should
> > > be
> > > possible system-wide.
> > > 
> > > We should not have too many switches. It makes the web UI complicated.
> > 
> > So should i leave the the switch for '--ncp-disable' option out of the WUI ? 
> > In that case we will reduce the ciphers on IPFire significant since OpenVPN
> > will then use only AES-256-CBC or AES-256GCM if the clients are >=2.4.0 (which
> > mostly are meanwhile i think) !
> > 
> > What are you thinking about to leave the cipher-negotiation checkbox out of
> > the global section (in that case is all like before) but i think we should
> > deliver at least one possibility in CCD section to disable the cipher-
> > negotiation which would then looks like 
> > in here --> https://people.ipfire.org/~ummeegge/screenshoots/OpenVPN-
> > 2.4_beta2/Disable-NCP-CCD.png and is only located in the "Advanced client
> > options" in the CCD section ?
> 
> I am not sure if this negotiation makes any sense for us. It comes a bit too
> late. The reason is as follows:
> 
> If someone has deployed OpenVPN in the past, they had to choose the cipher.
> Let's say that was AES-256-CBC. It absolutely makes no sense to add any of the
> weaker ciphers like AES-128-CBC. Neither for security, nor performance or
> compatibility. Every client must have supported AES-256-CBC in the first place.
> 
> This makes only sense in the other direction where I would want to use AES-256-
> GCM instead of CBC. So I could still support my clients that are on OpenVPN 2.3
> and only support AES-256-CBC. But when I choose GCM, I certainly don't want any
> client to keep using CBC.
> 
> AES-256-CBC will probably have to be set with the "cipher" option. Not sure if
> OpenVPN always prefers that one then.
> 
> My point is: This is going to be complicated in the UI and giving the user a
> decision that they are not always competent enough to make.
> 
> So I would say we should add the new ciphers to the list and disable the
> negotiation for now. We won't have any disadvantages, yet.
> 
> And after we have shipped this, we can add something to the advanced page where
> people can choose additional ciphers similar to the IPsec settings. So people
> who want to can edit this, but generally we don't give the users too many
> options.
> 
> Does that sound a good idea?
> 
> > Since the choice of the cipher is an important setting it might be great in my
> > opinion the give the users at least a possibility (in the advanced settings
> > section) to configure this for each client if wanted ?
> 
> Yes, see above.
> 
> But I would prefer to defer this so that we can ship OpenSSL 1.1.0 really quick
> now and add that with the next Core Update then. This will also allow us to ship
> smaller changes that shouldn't break things too easily and we can track bugs
> easier.
> 
> > Above all, there are currently only 3 algorithms left on IPFire which are not
> > affected by the Sweet32 problem (AES, Camellia and Seed which are 128 bit
> > block ciphers).
> 
> Please mark everything as weak what is considered as such. Please add some
> reference in the commit message - even if that is just Wikipedia.
> 
> > > > but the server.conf serves --ncp-ciphers but also --cipher and --auth -->
> > > > 
> > > > #OpenVPN Server conf
> > > > 
> > > > daemon openvpnserver
> > > > writepid /var/run/openvpn.pid
> > > > #DAN prepare OpenVPN for listening on blue and orange
> > > > ;local ue.*.org
> > > > dev tun
> > > > proto udp
> > > > port 1194
> > > > script-security 3
> > > > ifconfig-pool-persist /var/ipfire/ovpn/ovpn-leases.db 3600
> > > > client-config-dir /var/ipfire/ovpn/ccd
> > > > tls-server
> > > > ca /var/ipfire/ovpn/ca/cacert.pem
> > > > cert /var/ipfire/ovpn/certs/servercert.pem
> > > > key /var/ipfire/ovpn/certs/serverkey.pem
> > > > dh /var/ipfire/ovpn/ca/dh1024.pem
> > > > server 10.2.17.0 255.255.255.0
> > > > tun-mtu 1400
> > > > mssfix 0
> > > > route 10.1.4.0 255.255.255.252
> > > > keepalive 10 60
> > > > status-version 1
> > > > status /var/run/ovpnserver.log 30
> > > > ncp-ciphers AES-256-GCM:AES-256-CBC:CAMELLIA-256-CBC
> > > > cipher AES-256-CBC
> > > > auth SHA512
> > > > tls-auth /var/ipfire/ovpn/certs/ta.key
> > > > max-clients 100
> > > > tls-verify /usr/lib/openvpn/verify
> > > > crl-verify /var/ipfire/ovpn/crls/cacrl.pem
> > > > user nobody
> > > > group nobody
> > > > persist-key
> > > > persist-tun
> > > > verb 3
> > > > 
> > > > so i think this should be no problem.
> > > 
> > > So we need to add the ncp-ciphers. If we didn't we would have some sort of
> > > functionality just like OpenVPN 2.3. Which is fine with me for the moment,
> > > too.
> > 
> > Yes we would need "--ncp-ciphers ALGs" for this new feature. Generally i think
> > this could be a security/performance gain for a lot´s of people out there
> > especially if they are using the old and deprecated ciphers like Blowfish (old
> > default) or the whole 3DES collection. 
> 
> Actually since you raise performance: I have only considered this for
> roadwarrior networks, yet.
>
> The ncp-ciphers options makes even less sense with net-to-net connections,
> because people control both sides of the tunnel. And then you would just want to
> pick one cipher and use exactly that. You don't want to leave it up to chance
> which cipher is used. This would also mitigate any downgrading attacks.
> 
> > > > Shouldn´t i test OpenVPN with the new OpenSSL before pushing this, if the
> > > > iso
> > > > is working i can make an installation in my testing environment to break
> > > > down
> > > > the most important steps ?
> > > 
> > > I thought that was tested already. However, the OpenSSL is branch is
> > > somewhat
> > > unstable and won't be released with the next core update. So if you send it
> > > now,
> > > it wouldn't break anyone's system.
> > 
> > All tests has been done with the old OpenSSL lib. One concern for me with the
> > new lib is if the old ovpn.cnf (OpenVPNs OpenSSL conf) is OK or if there needs
> > to be something done too.
> > But i´am calmed a little down if this changes are not released with the next
> > Core update.
> 
> Just test :) We will see what breaks.
> 
> OpenVPN has broken many many things in the past that we didn't expect because
> they care some shit about backwards-compatibility.
> 
> > > > I forgot also one important point to mention which we discovered really
> > > > late.
> > > > OpenVPN-2.4.x refactors the CRL handling --> https://github.com/OpenVPN/op
> > > > envp
> > > > n/commit/160504a2955c4478cd2c0323452929e07016a336 which is a problem cause
> > > > the
> > > > CRL will now be checked in the "validBefore" and "validAfter" fields.
> > > > Since
> > > > "default_crl_days" are set to 30 days we can run into a problem if the CRL
> > > > has
> > > > been expired. Error message looks like this:
> > > > 
> > > > Nov 22 22:56:26 OpenVPN-2-4 openvpnserver[16368]: a.b.c.d:46227 VERIFY
> > > > ERROR:
> > > > depth=0, error=CRL has expired: C=US, O=Tatoine, CN=LGG3
> > > > Nov 22 22:56:26 OpenVPN-2-4 openvpnserver[16368]: a.b.c.d:46227 OpenSSL:
> > > > error:14089086:SSL routines:ssl3_get_client_certificate:certificate verify
> > > > failed
> > > > Nov 22 22:56:26 OpenVPN-2-4 openvpnserver[16368]: a.b.c.d:46227 TLS_ERROR:
> > > > BIO
> > > > read tls_read_plaintext error
> > > > Nov 22 22:56:26 OpenVPN-2-4 openvpnserver[16368]: a.b.c.d:46227 TLS Error:
> > > > TLS
> > > > object -> incoming plaintext read error
> > > > Nov 22 22:56:26 OpenVPN-2-4 openvpnserver[16368]: a.b.c.d:46227 TLS Error:
> > > > TLS
> > > > handshake failed
> > > > Nov 22 22:56:26 OpenVPN-2-4 openvpnserver[16368]: a.b.c.d:46227 Fatal TLS
> > > > error (check_tls_errors_co), restarting
> > > > Nov 22 22:56:26 OpenVPN-2-4 openvpnserver[16368]: a.b.c.d:46227
> > > > SIGUSR1[soft,tls-error] received, client-instance restarting
> > > 
> > > The CRL has a valid before and after date? In that case we just need to
> > > regenerate it.
> > > 
> > > If the CA or some client certificates have expired, this is what we would
> > > expect, isn't it?
> > 
> > Exactly.
> 
> Again, an issue where backwards-compatibility was clearly not important.
> 
> > > 
> > > > A more detailed discussion in the IPFire forum is here -->
> > > > https://forum.ipfire.org/viewtopic.php?f=50&t=18067&start=30#p112529
> > > > located,
> > > > the Debian discussion --> https://bugs.debian.org/cgi-bin/bugreport.cgi?bu
> > > > g=84
> > > > 9909 and OpenVPN --> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=849
> > > > 909
> > > > . An idea there was to set the "nextUpdate" value in a far future as the
> > > > CA expiry date is, we decided in the forum to leave this idea behind and
> > > > started to develop a first idea for an CRL updater, please check this in
> > > > here --> https://forum.ipfire.org/viewtopic.php?f=50&t=18067&sid=e623c9c72
> > > > 6b7d0f5fb14b9659b4a6625&start=45#p112737 . In that case we would need a
> > > > script which checks periodically when a CRL update needs to be done. 
> > > 
> > > Not a fan of cron jobs for things like this, but I guess we have no other
> > > choice. This looks good.
> > 
> > Am also not really happy with this but have found so far no better solutions.
> > Should i place the script to /etc/fcron.daily/ ? Would make sense i think.
> 
> Yes.
>

Above topic link also contains example of a full Certificate Autorithy management using standard OpenSSL functionalities - I generated ROOT CA, Intermediary CA and server cerificates for apache. Apache init script uses functions to test all CA elements - ROOT and its CRL, Intermediary CA and its CRL, server certificate.

I tested it in several conditions and up to now the CA related tasks work very fast - at each an every service start.  
> > > 
> > > > If you have better ideas for this or another script it might be really
> > > > helpful.
> > > 
> > > No not really. We have to make sure that the CRL is up to date even when the
> > > system is not being rebooted in a while. So it has to be checked regularly
> > > which
> > > requires a cron job.
> > 
> > This is exactly the point. If clients are deleted via ovpnmain.cgi the CRL
> > will automatically be renewed via ovpnmain.cgi but if the systems runs without
> > some specific interactions, the CRL expires and all connections are over and
> > done until the CRL has been renewed.
> 
> Yes.
> 
> > 
> > > 
> > > Setting the expiry date to infinity would slightly ruin the idea of a CRL.
> > 
> > Yes this would make no sense at all.
> > 
> > > > > Peter has proposed a patch recently with improved crypto, please work
> > > > > together
> > > > > with him.
> > > > 
> > > > We can do this, no problem. @Peter would you write me to ummeegge at
> > > > ipfire.org ?
> > > 
> > > It is here https://patchwork.ipfire.org/patch/1594/
> > 
> > Thanks for pointing that out. 
> > I did also changes regarding the "vpn weak" warnings but this changes includes
> > more then the DH-parameter.
> > As above mentioned 6 from 12 the currently available on IPFires OpenVPN are
> > deprecated cause there can be attacked via Sweet32. OpenVPN implemented
> > "reneg-bytes 64000000" with v. 2.3.13 which forces frequent rekeying after 64
> > MB data transfer, this can lead to a lot more rekeying cycles then the one
> > hour cycle which is the regular behavior.
> 
> 64MB? That is once a second over a Gigabit link. Or roughly every 5 to 6 seconds
> over a 100M link.
> 
> Is performance not an issue here?
>

N2N tests (using OpenVPN 2.4 vs standard IPFire   OpenVPN) lead to 30% speed decrease using same network and same test file - a big, heavily compressed backup file moved between same file servers over N2N.

I did had lzo activated - and lzo got an upgrade on 2.4, but after reading above change is more likely that speed decrease is due to rekeying cycles.


> > All that can somehow be neglected with the cipher-negotiaion and newer clients
> > since there will then be used anyways AES-256 which is a 128 bit block cipher.
> > 
> > The SHA512 default setting changes from Peter are for N2N defaults which
> > should be OK in my opinion since every new connection needs to be created for
> > both sides (TLS-server and TLS-client), this should´t crash anything.
> 
> Okay. So please let's get that patch merged then.
> 
> > One more important thing. If we update to OpenVPN-2.4.x there is the need that
> > the changes will be written to the server.conf (save button) otherwise the
> > server do not start (causing the script-security entry), after i manually
> > updated the files i needed to stop the server, hit the save button and only
> > after then the server can be started again.
> > openvpnctrl do provides only stop|start but no save option do you know solve
> > this via command line parameter while the update ?
> 
> We can stop all OpenVPN instances on the command line.
> 
> For RW this is: openvpnctrl -k
> 
> And for all N2N connections that is: openvpnctrl -kn2n
> 
> > 
> > 
> > Again thanks for your feedback and help.
> > 
> > Greetings,
> > 
> > Erik
> > 
> > > 
> > 
> 
> -Michael
> 
> > 

Hope it helps,
-- 
Horace