From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Adolf Belka (ipfire-dev)" To: development@lists.ipfire.org Subject: Re: fireperf results Date: Tue, 16 Feb 2021 19:50:00 +0100 Message-ID: In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============6598414540009836750==" List-Id: --===============6598414540009836750== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi Michael, Daniel asked if I was running suricata and I was. Removing that made everythi= ng much better. Now with -P 1 to -P 1000 I was getting ~950Mb/s. With -P 1000= 0 I got only ~550Mb/s. On 16/02/2021 17:16, Michael Tremer wrote: > Hello Adolf, >=20 > This is very surprising to me. I am almost shocked. >=20 > Maybe any of my assumptions are wrong, but if this is the actual throughput= of this piece of hardware, I find this not enough. >=20 >> On 16 Feb 2021, at 12:44, Adolf Belka (ipfire-dev) wrote: >> >> Hi All, >> >> Following are the fireperf results I obtained:- >> >> server: IPFire 2.25 - Core Update 153; Intel Celeron CPU J1900 @ 1.99GHz x= 4; I211 Gigabit Network Connection >=20 > You have a small processor here with a rather high clock rate. Four cores a= t 2 GHz is quite something. >=20 > However, it is a Celeron processor and that means it is a bit more stripped= down than others. Usually it is caches and pipeline throughput. It might be = that, that bites you really bad here. >=20 > You have a better than average NIC. The Intel network controllers are not b= ad, although the i2xx series is not fully active. >=20 > Could you please send the output of =E2=80=9Ccat /proc/interrupts=E2=80=9D = so that we can see how many queues they have? -bash-5.0$ cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 0: 40 0 0 0 IO-APIC 2-edge t= imer 1: 3 0 0 0 IO-APIC 1-edge i= 8042 4: 430 0 0 0 IO-APIC 4-edge t= tyS0 8: 55 0 0 0 IO-APIC 8-fasteoi r= tc0 9: 0 0 0 0 IO-APIC 9-fasteoi a= cpi 12: 4 0 0 0 IO-APIC 12-edge i= 8042 18: 0 0 0 0 IO-APIC 18-fasteoi i= 801_smbus 91: 2959648 0 0 0 PCI-MSI 311296-edge = ahci[0000:00:13.0] 92: 292544 0 0 0 PCI-MSI 327680-edge = xhci_hcd 93: 1 0 0 0 PCI-MSI 2097152-edge = orange0 94: 332181 0 0 0 PCI-MSI 2097153-edge = orange0-rx-0 95: 94258 0 0 0 PCI-MSI 2097154-edge = orange0-rx-1 96: 328866 0 0 0 PCI-MSI 2097155-edge = orange0-tx-0 97: 169838 0 0 0 PCI-MSI 2097156-edge = orange0-tx-1 98: 1 0 0 0 PCI-MSI 3670016-edge = red0 99: 9795304 0 0 0 PCI-MSI 3670017-edge = red0-rx-0 100: 94258 0 0 0 PCI-MSI 3670018-edge = red0-rx-1 101: 9574443 0 0 0 PCI-MSI 3670019-edge = red0-tx-0 102: 1067926 0 0 0 PCI-MSI 3670020-edge = red0-tx-1 103: 1 0 0 0 PCI-MSI 4194304-edge = green0 104: 15302199 0 0 0 PCI-MSI 4194305-edge = green0-rx-0 105: 94259 0 0 0 PCI-MSI 4194306-edge = green0-rx-1 106: 13422909 0 0 0 PCI-MSI 4194307-edge = green0-tx-0 107: 4977558 0 0 0 PCI-MSI 4194308-edge = green0-tx-1 108: 1 0 0 0 PCI-MSI 4718592-edge = blue0 109: 97391 0 0 0 PCI-MSI 4718593-edge = blue0-rx-0 110: 94259 0 0 0 PCI-MSI 4718594-edge = blue0-rx-1 111: 94259 0 0 0 PCI-MSI 4718595-edge = blue0-tx-0 112: 137222 0 0 0 PCI-MSI 4718596-edge = blue0-tx-1 NMI: 638 468 287 294 Non-maskable interrupts LOC: 18102811 13305397 18242209 25513971 Local timer interrupts SPU: 0 0 0 0 Spurious interrupts PMI: 638 468 287 294 Performance monitoring i= nterrupts IWI: 9208 21 2 26 IRQ work interrupts RTR: 0 0 0 0 APIC ICR read retries RES: 563980 301713 552880 579914 Rescheduling interrupts CAL: 184217 137668 310137 256984 Function call interrupts TLB: 170395 122144 132849 103440 TLB shootdowns TRM: 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 Threshold APIC interrupts DFR: 0 0 0 0 Deferred Error APIC inte= rrupts MCE: 0 0 0 0 Machine check exceptions MCP: 605 605 605 605 Machine check polls HYP: 0 0 0 0 Hypervisor callback inte= rrupts ERR: 1 MIS: 0 PIN: 0 0 0 0 Posted-interrupt notific= ation event NPI: 0 0 0 0 Nested posted-interrupt = event PIW: 0 0 0 0 Posted-interrupt wakeup = event >=20 >> client: Arch Linux; Intel Core i5-8400 CPU @ 2.80GHz 6 core; 1GBit nic >> >> Server: >> fireperf -s -P 10000 -p 63000:630010 >> >> >> Client: >> fireperf -c -P 1 -x -p 63000:63010 -> 100 - 3000 cps strongly= fluctuating. After a couple of minutes the client cps went down to 0 and sta= yed there. I had to stop fireperf and restart the terminal to get it working = again. >> >> fireperf -c -P 10 -x -p 63000:63010 ->250 - 500 cps fluctuati= ng >> >> fireperf -c -P 100 -x -p 63000:63010 -> 220 - 1000 cps fluctu= ating >> >> fireperf -c -P 1000 -x -p 63000:63010 -> 1200 - 2500 cps fluc= tuating >> >> fireperf -c -P 10000 -x -p 63000:63010 -> 0 - 7000 cps hugely= fluctuating >=20 > From the beginning you have quite a large fluctuation here. Some is normal= , but this is a lot. It seems that the system is overloaded from the very beg= inning. >=20 > I have not done experiments with lots of different hardware (used the same = usually), but Daniel has, and we normally have the systems being very idle wi= th only one connection at a time. There isn=E2=80=99t too much to do for the = CPU except waiting. >=20 >> In all cases the cpu utilisation was quite low on both IPFire and the Arch= Linux desktop. >=20 > Not surprising on the desktop side, because there wasn=E2=80=99t a lot stuf= f to do. >=20 >> I then repeated the above tests removing the -x option so I could see the = data bandwidth. >> >> >> fireperf -c -P 1 -p 63000:63010 -> 225Mb/s - 1 core at 100%, = rest around 30% to 40% >=20 > This is the most surprising part. >=20 > The IPFire Mini Appliance for example only has 1 GHz of clock and it doesn= =E2=80=99t have any problems with transmissing a whole gigabit a second of da= ta. This system has double the clock speed and the same NIC (or at least very= similar to it). >=20 >> fireperf -c -P 10 -p 63000:63010 -> 185Mb/s - similar as above >> >> fireperf -c -P 100 -p 63000:63010 -> 210Mb/s - similar to abo= ve >=20 > The bandwidth should have increased here. That means we know that the bottl= eneck is not the network, but something else. >=20 > The one core that is maxed out is to some good extend the fireperf process = generating packets. The rest is overhead of the OS, network stack and NIC dri= ver. Which feels way too high for me. >=20 >> fireperf -c -P 1000 -p 63000:63010 -> 370 - 450Mb/s - 2 cores= at 100%, rest at 30% to 40% >> >> fireperf -c -P 10000 -p 63000:63010 -> 400Mb/s - 1Gb/s - 2 co= res at 100%, rest at 40% to 50% >=20 > You seem to have more than one receive queue as it looks like. >=20 > Did you actually achieve the 10k connections? >=20 >> I recently got my Glass Fibre Gigabit connection connected. The supplier h= ooked his laptop directly to the media converter and got around 950Mb/s >=20 > Could you test =E2=80=9Cspeedtest-cli=E2=80=9D and see what that reports? After turning off suricata I ran speedtest-cli on IPFire and got ~850Mb/s I ran speedtest-cli on my Arch Desktop and got 840Mb/s I ran speedtest++ on my Arch Desktop and got ~930Mb/s I ran my ISP's speedtest and got ~970Mb/s So all showing similar-ish values and around where they should be. So on my h= ardware suricata is making a very big difference. I now have to decide if tha= t is worth it or not for my situation. >=20 >> Using the same speed test as he used but going through my IPFire hardware = I get around 225Mb/s. >> >> Although my hardware has four Intel I211 Gigabit NIC's, I have suspected t= hat their performance is limited by the processor. >=20 > It sounds like it. Lets see what more information we can gather and hopeful= ly find it. >=20 > Can you run powertop along the bechmark and see what that says? >=20 > -Michael >=20 >> Regards, >> >> Adolf. >> >> >=20 --===============6598414540009836750==--