Hello,
On 16 Feb 2021, at 18:50, Adolf Belka (ipfire-dev) adolf.belka@ipfire.org wrote:
Hi Michael,
Daniel asked if I was running suricata and I was.
That email must have got lost.
But of course this explains it.
Removing that made everything much better. Now with -P 1 to -P 1000 I was getting ~950Mb/s. With -P 10000 I got only ~550Mb/s.
On 16/02/2021 17:16, Michael Tremer wrote:
Hello Adolf, This is very surprising to me. I am almost shocked. Maybe any of my assumptions are wrong, but if this is the actual throughput of this piece of hardware, I find this not enough.
On 16 Feb 2021, at 12:44, Adolf Belka (ipfire-dev) adolf.belka@ipfire.org wrote:
Hi All,
Following are the fireperf results I obtained:-
server: IPFire 2.25 - Core Update 153; Intel Celeron CPU J1900 @ 1.99GHz x4; I211 Gigabit Network Connection
You have a small processor here with a rather high clock rate. Four cores at 2 GHz is quite something. However, it is a Celeron processor and that means it is a bit more stripped down than others. Usually it is caches and pipeline throughput. It might be that, that bites you really bad here. You have a better than average NIC. The Intel network controllers are not bad, although the i2xx series is not fully active. Could you please send the output of “cat /proc/interrupts” so that we can see how many queues they have?
-bash-5.0$ cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 0: 40 0 0 0 IO-APIC 2-edge timer 1: 3 0 0 0 IO-APIC 1-edge i8042 4: 430 0 0 0 IO-APIC 4-edge ttyS0 8: 55 0 0 0 IO-APIC 8-fasteoi rtc0 9: 0 0 0 0 IO-APIC 9-fasteoi acpi 12: 4 0 0 0 IO-APIC 12-edge i8042 18: 0 0 0 0 IO-APIC 18-fasteoi i801_smbus 91: 2959648 0 0 0 PCI-MSI 311296-edge ahci[0000:00:13.0] 92: 292544 0 0 0 PCI-MSI 327680-edge xhci_hcd 93: 1 0 0 0 PCI-MSI 2097152-edge orange0 94: 332181 0 0 0 PCI-MSI 2097153-edge orange0-rx-0 95: 94258 0 0 0 PCI-MSI 2097154-edge orange0-rx-1 96: 328866 0 0 0 PCI-MSI 2097155-edge orange0-tx-0 97: 169838 0 0 0 PCI-MSI 2097156-edge orange0-tx-1 98: 1 0 0 0 PCI-MSI 3670016-edge red0 99: 9795304 0 0 0 PCI-MSI 3670017-edge red0-rx-0 100: 94258 0 0 0 PCI-MSI 3670018-edge red0-rx-1 101: 9574443 0 0 0 PCI-MSI 3670019-edge red0-tx-0 102: 1067926 0 0 0 PCI-MSI 3670020-edge red0-tx-1 103: 1 0 0 0 PCI-MSI 4194304-edge green0 104: 15302199 0 0 0 PCI-MSI 4194305-edge green0-rx-0 105: 94259 0 0 0 PCI-MSI 4194306-edge green0-rx-1 106: 13422909 0 0 0 PCI-MSI 4194307-edge green0-tx-0 107: 4977558 0 0 0 PCI-MSI 4194308-edge green0-tx-1 108: 1 0 0 0 PCI-MSI 4718592-edge blue0 109: 97391 0 0 0 PCI-MSI 4718593-edge blue0-rx-0 110: 94259 0 0 0 PCI-MSI 4718594-edge blue0-rx-1 111: 94259 0 0 0 PCI-MSI 4718595-edge blue0-tx-0 112: 137222 0 0 0 PCI-MSI 4718596-edge blue0-tx-1 NMI: 638 468 287 294 Non-maskable interrupts LOC: 18102811 13305397 18242209 25513971 Local timer interrupts SPU: 0 0 0 0 Spurious interrupts PMI: 638 468 287 294 Performance monitoring interrupts IWI: 9208 21 2 26 IRQ work interrupts RTR: 0 0 0 0 APIC ICR read retries RES: 563980 301713 552880 579914 Rescheduling interrupts CAL: 184217 137668 310137 256984 Function call interrupts TLB: 170395 122144 132849 103440 TLB shootdowns TRM: 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 Threshold APIC interrupts DFR: 0 0 0 0 Deferred Error APIC interrupts MCE: 0 0 0 0 Machine check exceptions MCP: 605 605 605 605 Machine check polls HYP: 0 0 0 0 Hypervisor callback interrupts ERR: 1 MIS: 0 PIN: 0 0 0 0 Posted-interrupt notification event NPI: 0 0 0 0 Nested posted-interrupt event PIW: 0 0 0 0 Posted-interrupt wakeup event
client: Arch Linux; Intel Core i5-8400 CPU @ 2.80GHz 6 core; 1GBit nic
Server: fireperf -s -P 10000 -p 63000:630010
Client: fireperf -c <IP address> -P 1 -x -p 63000:63010 -> 100 - 3000 cps strongly fluctuating. After a couple of minutes the client cps went down to 0 and stayed there. I had to stop fireperf and restart the terminal to get it working again.
fireperf -c <IP address> -P 10 -x -p 63000:63010 ->250 - 500 cps fluctuating
fireperf -c <IP address> -P 100 -x -p 63000:63010 -> 220 - 1000 cps fluctuating
fireperf -c <IP address> -P 1000 -x -p 63000:63010 -> 1200 - 2500 cps fluctuating
fireperf -c <IP address> -P 10000 -x -p 63000:63010 -> 0 - 7000 cps hugely fluctuating
From the beginning you have quite a large fluctuation here. Some is normal, but this is a lot. It seems that the system is overloaded from the very beginning. I have not done experiments with lots of different hardware (used the same usually), but Daniel has, and we normally have the systems being very idle with only one connection at a time. There isn’t too much to do for the CPU except waiting.
In all cases the cpu utilisation was quite low on both IPFire and the Arch Linux desktop.
Not surprising on the desktop side, because there wasn’t a lot stuff to do.
I then repeated the above tests removing the -x option so I could see the data bandwidth.
fireperf -c <IP address> -P 1 -p 63000:63010 -> 225Mb/s - 1 core at 100%, rest around 30% to 40%
This is the most surprising part. The IPFire Mini Appliance for example only has 1 GHz of clock and it doesn’t have any problems with transmissing a whole gigabit a second of data. This system has double the clock speed and the same NIC (or at least very similar to it).
fireperf -c <IP address> -P 10 -p 63000:63010 -> 185Mb/s - similar as above
fireperf -c <IP address> -P 100 -p 63000:63010 -> 210Mb/s - similar to above
The bandwidth should have increased here. That means we know that the bottleneck is not the network, but something else. The one core that is maxed out is to some good extend the fireperf process generating packets. The rest is overhead of the OS, network stack and NIC driver. Which feels way too high for me.
fireperf -c <IP address> -P 1000 -p 63000:63010 -> 370 - 450Mb/s - 2 cores at 100%, rest at 30% to 40%
fireperf -c <IP address> -P 10000 -p 63000:63010 -> 400Mb/s - 1Gb/s - 2 cores at 100%, rest at 40% to 50%
You seem to have more than one receive queue as it looks like. Did you actually achieve the 10k connections?
I recently got my Glass Fibre Gigabit connection connected. The supplier hooked his laptop directly to the media converter and got around 950Mb/s
Could you test “speedtest-cli” and see what that reports?
After turning off suricata I ran speedtest-cli on IPFire and got ~850Mb/s I ran speedtest-cli on my Arch Desktop and got 840Mb/s I ran speedtest++ on my Arch Desktop and got ~930Mb/s I ran my ISP's speedtest and got ~970Mb/s
Yeah, so it seems that the ISP speed test is doing something “different”. I am not sure if we can trust them. I definitely wouldn’t trust my ISP.
~840 MBit/s is still about 100 MBit/s away from the maximum which is 940 MBit/s taking away any overhead from Ethernet and IP.
970 MBit/s is technically not possible for an IP connection. It might work out if you include Ethernet headers into the maths.
How did the CPU load change? It suggests that the hardware still didn’t run at full capacity because of two cores being idle.
So all showing similar-ish values and around where they should be. So on my hardware suricata is making a very big difference. I now have to decide if that is worth it or not for my situation.
Very good question. Probably not best to be discussed here, but in general I would like to have this discussion.
An IPS is absolutely worth it, and everyone should have it. Unfortunately it doesn’t run on small hardware and therefore we need to be very careful what we recommend and what we compare with each other.
Discussions on the forum always ended up with people buying the cheapest stuff that they could get - that is of course a rational thing to do. However, I am simply running the IPS wherever I go. So that means that a raspberry pi is not useful for more than a megabit a second. However, it is difficult to predict IPS throughput because it depends on so many factors. It would be nice if we could find a reproducible way to benchmark this - at least somewhat accurate.
-Michael
Using the same speed test as he used but going through my IPFire hardware I get around 225Mb/s.
Although my hardware has four Intel I211 Gigabit NIC's, I have suspected that their performance is limited by the processor.
It sounds like it. Lets see what more information we can gather and hopefully find it. Can you run powertop along the bechmark and see what that says? -Michael
Regards,
Adolf.