Re: Question regarding IPsec N2N throughput with and without IPS

[Michael Tremer <michael.tremer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 6010 bytes --]

Hi,

> On 24 Jan 2020, at 14:32, Peter Müller <peter.mueller(a)ipfire.org> wrote:
> 
> Hello Michael,
> 
>> Hi,
>> 
>>> On 22 Jan 2020, at 16:24, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>> 
>>> Hello Stefan, hello list (CC'ed),
>>> 
>>> are you aware of any IPS bottlenecks regarding IPsec N2N throughput?
>> 
>> No, not at all.
>> 
>> There is actually not much the IPS can do with the ESP packets. I would assume it just passes them through.
> I am not sure about this as the decapsulated clear text packages show up on
> ppp0/red0, too - perhaps these pass Suricata twice?

Yes, after they are being decrypted, they are passing through suricata again.

You can try to add a RETURN rule that matches the private IP address ranges so that you can skip this step and find out if this is causing any problems.

>>> I finally (!) managed to get an IPsec connection between OpenBSD 6.6 (OpenIKED)
>>> and IPFire 2.23 Core Update 139 working.
>> 
>> Yay \o/.
> Well, actually not. PSK works, but certificate based authentication does not
> due to bug #12276 (OpenSSL does not use subjectAltNames from CSRs and is unable
> to be safely forced to do so - you will have to supply an additional configuration
> file to make CSR signing with subjectAltNames work which is a _very_ ugly thing
> to do if no CSR configuration is available on the signing machine).
> 
> Furthermore, Dead Peer Detection is tricky as OpenIKED does not support it
> - the OpenBSD people use ifstated instead, but restarting single IPsec connections
> is likely impossible at the moment -, so we can only rely on Strongswan here. :-/

Oh OpenBSD. I have no idea why you are doing this to yourself.

> Side note: OpenIKED does not support AES-GCM for ESP (see configuration below).

That’s bad because it is fast and secure.

>> 
>> Please do not forget to add the relevant documentation to the IPFire wiki.
> I will do so as soon this thing works stable and all corresponding IPFire bugs
> have been fixed. :-)

Great!

>> 
>>> During throughput tests, where
>>> I downloaded a 1 GByte test file from the machine via the IPsec tunnel,
>>> a rather large throughput difference with and without IPS enabled on RED
>>> has come to my attention:
>>> 
>>> With IPS enabled on RED, the download starts at ~ 2.5 MByte/sec. and
>>> continually decreases to ~ 580 kByte/sec. - 800 kByte/sec., which is
>>> even lower than OpenVPN performance. Without IPS enabled on RED, throughput
>>> is 4.0 MByte/sek. on average - running the IPS on other interfaces does
>>> not change this behaviour, neither does enabling monitoring mode.
>> 
>> How is CPU load?
> From the collectd graphs I can recall, load average (1 minute) was about 1.1 .

That is high. Please try disabling the Spectre/Meltdown mitigations just for the fun of it. It looks like something is blocking the processor from doing any actual work.

>> Did you see any retransmissions?
> I am currently unable to test this with IPsec but transferring the same test file
> via SCP (which results approximately in the same throughput) shows some duplicate
> ACKs, but much less than those we have seen on the similar bug investigated last
> year.

Ideally there should not be any.

>> 
>>> This suspiciously sounds like the issue we have had with Suricata last
>>> year - as far as I am concerned that was fully fixed. Are you aware of
>>> any other similar issue that could cause this massive throughput loss?
>> 
>> No. You can try Suricata 5 which has been posted to the list today.
> I would like to do so - is there a nightly/pre-testing ISO available?

Not yet. Next is still on core140.

I am hoping for next week. Arne?

>> 
>>> Anyway, thank you in advance for any help and hints. :-)
>>> 
>>> Kernel and CPU information of the OpenBSD machine:
>>>> openbsd# uname -a
>>>> OpenBSD openbsd 6.6 GENERIC.MP#4 amd64
>>>> openbsd# sysctl hw.model hw.machine hw.ncpu
>>>> hw.model=Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
>>>> hw.machine=amd64
>>>> hw.ncpu=2
>>> 
>>> Content of /etc/iked.conf on the OpenBSD machine:
>>>> set fragmentation
>>>> 
>>>> ikev2 "[REDACTED]" active esp \
>>>>  from 10.xxx.xxx.2/24 to 10.xxx.xxx.0/24 \
>>>>  local [REDACTED] peer [REDACTED] \
>>>>  ikesa auth hmac-sha2-512 enc aes-256 prf hmac-sha2-512 group curve25519 \
>>>>  childsa enc aes-256-gcm group curve25519 \
>>>>  srcid [REDACTED] dstid [REDACTED] \
>>>>  ikelifetime 3h \
>>>>  lifetime 1h
>>> 
>>> Kernel and CPU information of the IPFire machine:
>>>> [root(a)maverick ~]# uname -a
>>>> Linux maverick 4.14.154-ipfire #1 SMP Fri Nov 15 07:27:41 GMT 2019 x86_64 Intel(R) Celeron(R) CPU N3150 @ 1.60GHz GenuineIntel GNU/Linux
>> 
>> This is a really small Atom processor AFAIK. Could we struggle with Meltdown/Spectre mitigations here? Just to rule it out, can you boot the kernel with them disabled?
> Hm, I do not think we need to worry about them too much as the N3150 is not vulnerable
> to some CPU vulnerabilities:
>> [root(a)maverick ~]# grep . /sys/devices/system/cpu/vulnerabilities/*
>> /sys/devices/system/cpu/vulnerabilities/itlb_multihit:Not affected
>> /sys/devices/system/cpu/vulnerabilities/l1tf:Not affected
>> /sys/devices/system/cpu/vulnerabilities/mds:Mitigation: Clear CPU buffers; SMT disabled
>> /sys/devices/system/cpu/vulnerabilities/meltdown:Mitigation: PTI
>> /sys/devices/system/cpu/vulnerabilities/spec_store_bypass:Not affected
>> /sys/devices/system/cpu/vulnerabilities/spectre_v1:Mitigation: usercopy/swapgs barriers and __user pointer sanitization
>> /sys/devices/system/cpu/vulnerabilities/spectre_v2:Mitigation: Full generic retpoline, IBPB: conditional, IBRS_FW, STIBP: disabled, RSB filling
>> /sys/devices/system/cpu/vulnerabilities/tsx_async_abort:Not affected

PTI might be it. But that should not only affect IPsec traffic then, but all the rest as well.

> Thanks, and best regards,
> Peter Müller

-Michael