From mboxrd@z Thu Jan 1 00:00:00 1970 From: Tom Rymes To: development@lists.ipfire.org Subject: Re: Introducing fireperf Date: Thu, 04 Feb 2021 13:06:48 -0500 Message-ID: In-Reply-To: <082ACED3-5396-479B-AB43-20BEC1A3D317@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1547061644298594550==" List-Id: --===============1547061644298594550== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable No worries, Michael, you didn=E2=80=99t come across the wrong way. It is an i= nteresting subject. Tim > On Feb 4, 2021, at 11:48 AM, Michael Tremer w= rote: >=20 > =EF=BB=BFHi, >=20 > Sorry if I came across a wrong way. >=20 > I just didn=E2=80=99t know what to say apart from that it is of course poss= ible. According to German law, it is illegal to create and distribute somethi= ng colloquially called =E2=80=9Chacker tools=E2=80=9D. I do not think that th= is falls under it in that sense that a frying pan is not classified as a murd= er weapon although it could be used as one. >=20 > Potentially I would see this as a good tool to find out when a load-balance= r collapses because of the sheer amount of new connections being opened. Apac= he Benchmark could do the same - although I do not know how efficient it is w= ith the whole HTTP protocol layer. >=20 > -Michael >=20 >> On 4 Feb 2021, at 16:36, Tom Rymes wrote: >>=20 >> Just curious. I know that there=E2=80=99s an inherent tension in this area= , and I wanted to prompt discussion, nothing more. >>=20 >> Tom >>=20 >>>> On Feb 4, 2021, at 11:17 AM, Michael Tremer wrote: >>>=20 >>> Hello, >>>=20 >>> Well, it is a network testing tool. They all can be used for some evil st= uff, but so can be ping(8). >>>=20 >>> I was thinking about this during development, but I am not sure what coul= d be done to prevent this. >>>=20 >>> -Michael >>>=20 >>>> On 4 Feb 2021, at 16:16, Tom Rymes wrote: >>>>=20 >>>> Michael, >>>>=20 >>>> Any concerns that this could be used for evil? >>>>=20 >>>> Tom >>>>=20 >>>>> On Feb 4, 2021, at 10:57 AM, Michael Tremer wrote: >>>>>=20 >>>>> Hello all y=E2=80=99all, >>>>>=20 >>>>> I would like to introduce a small side-project I have been working on c= alled fireperf. >>>>>=20 >>>>> It is a networking benchmarking tool which I have written to debug some= issues with IPFire on AWS and I thought this would be very useful for others= , too. >>>>>=20 >>>>>=20 >>>>> A bit of a backstory >>>>>=20 >>>>> Everyone probably knows iperf and its newer brother iperf3 and has used= them. They can do bandwidth tests and usually should be quite good at then. = Unfortunately they are reaching their limits quite early. In my environment I= had at least a 5 GBit/s connection between my two machines and I wanted to c= reate lots and lots of connections to stress-test the connection tracking. >>>>>=20 >>>>> This was unfortunately not possible with either of them, because iperf = is starting a thread per connection and iperf3 limits its connections to 128 = per process. In both cases this simply did not scale because my goal was to c= reate connections in the 6 figures or more range. I simply ran out of memory. >>>>>=20 >>>>> Another issue that both sometimes have - and which I did not validate s= pecifically in this case - is that they cannot generate enough traffic to sat= urate a link. However, I need to be able to simply trust that this is possibl= e as long as I have the CPU resources available. >>>>>=20 >>>>> Therefore a new tool was needed. >>>>>=20 >>>>> When I started writing fireperf, I did not intend to make it fit for th= roughput tests, but since it was such a low-hanging fruit in the development = process I added it, too. The original goal was simply to open a number of con= nections (at least hundreds of thousands) and keep them open or let me know w= hen this is no longer possible. >>>>>=20 >>>>> Since I knew I was working on IPFire, I started to take advantage of Li= nux=E2=80=99 modern APIs and try to delegate as much work as possible to the = kernel. Especially since the whole Meltdown/Spectre debacle, sending data bet= ween the kernel and userland is slow, and the less work I have to do in the u= ser land, the more time I can spend on other things. >>>>>=20 >>>>> Therefore I use epoll() to let the kernel tell me when a socket is read= y to accept data and when something has happened and the connection broke dow= n. I am using getrandom() to get random data to send and I use timerfd to reg= ularly notify me when to print some statistics. Therefore this application is= not very easily portable (it wasn=E2=80=99t an original design goal), but I = am sure that there are some alternatives available if someone were to port th= is to another OS. >>>>>=20 >>>>> iperf3 - the most efficient one I knew - used up all of my 8GB of memor= y on my test system when started multiple times to create about 10k connectio= ns. Fireperf uses a few hundreds of kilobytes with tens of thousands of open = connections. In fact, it does not keep any state about the connections and th= erefore uses the same amount of memory no matter how many connections are ope= n. The kernel will use some memory though, but I could not measure how much. >>>>>=20 >>>>> Without saturating my processor I can saturate any network link that I = could test up to 10 GBit/s. CPU usage normally is less than 10% and fireperf = knows a mode (=E2=80=94k) in which it won=E2=80=99t send any data, but only k= eep the connections open and regularly let the kernel (again, because I am a = lazy developer) send some keep alive packets. That way, it uses next to no CP= U resources while still generating a lot of stress for the network. >>>>>=20 >>>>> So here it is, my new tool. I hope someone finds this useful. >>>>>=20 >>>>> It is nice and tiny and everything comes in one binary file which only = depends on the C standard library. >>>>>=20 >>>>> Sources are available on our Git server as usual: >>>>>=20 >>>>> https://git.ipfire.org/?p=3Dfireperf.git;a=3Dsummary >>>>>=20 >>>>> I tagged release number 0.1.0 and I will push a patch into next very so= on. There are also Debian packages available if you want to give fireperf a t= ry on Debian: >>>>>=20 >>>>> deb https://packages.ipfire.org/fireperf buster/ >>>>> deb-src https://packages.ipfire.org/fireperf buster/ >>>>>=20 >>>>> Replace buster with bullseye or sid if you are on those and do not forg= et to import the key: >>>>>=20 >>>>> curl https://packages.ipfire.org/79842AA7CDBA7AE3-pub.asc | apt-key add= - >>>>>=20 >>>>> Documentation in form of a man-page is available here: >>>>>=20 >>>>> https://man-pages.ipfire.org/fireperf/fireperf.html >>>>>=20 >>>>> It would be great, if fireperf would become a great tool to benchmark I= PFire. We definitely could do better in this department and hopefully gain be= tter insights on any regressions in performance, or if certain hardware is be= tter than other. I suppose throughput is not everything and fireperf should b= e able to help us measure other factors, too. >>>>>=20 >>>>> -Michael >>>>=20 >>>=20 >>=20 >=20 --===============1547061644298594550==--