From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: [PATCH] New package: IPTraffic 0.8.2 Date: Tue, 26 Jan 2021 11:38:45 +0000 Message-ID: In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2196226060787535313==" List-Id: --===============2196226060787535313== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello Bernhard, > On 25 Jan 2021, at 21:50, Bernhard Bitsch wrote: >=20 > Hi, >=20 > to come back to the facts, some annotations about the addon. I first decide= d to post them in the community thread about the addon, but in the situation = now I think it is better here.=20 Yes, let=E2=80=99s please keep all development work here, because not many of= us monitor the community portal all of the time, and it is easier to miss so= mething on there, too. > I've installed Frank's tar file from the community and studied the source. >=20 > - Doing the traffic measurement through iptables rules may be effitive but = ineffidcient. Did not investigate the overhead introduced into the system. Ma= ybe it isn't so much ( my tiny ALIX didn't get inresponive ), but it is just = a feeling for misuse of iptables. Yes, that is a misuse of iptables and won=E2=80=99t perform at all. In the past few years, we have been spending a lot of time on making sure tha= t the kernel takes advantage of multiple processors and multiple queues in th= e network adapters - one of the things the Lightning Wire Labs appliances do = really well. That way, we can have a smaller piece of hardware, but we can ta= ke advantage of all its features and make it really fast. Suricata was also a big challenge, but it now uses all available processor co= res. The kernel used to only evaluate iptables rules on the first core, which was = a big bottleneck and since modern hardware usually has more rather than faste= r cores, it would not have increased network throughput. The counters are mai= nly the problem here, because they require synchronisation between all proces= sor cores. Adding more rules like this would negatively impact network throughput - and = probably even more importantly, the CUSTOM* chains have been used which make = every single packet hit every single rule. One of the things that makes Linux= a fast firewall is the connection tracking and that we only need to look at = very few rules for existing connections. This is changed here when there is a= rule for every single monitored host on the network. > - Using the CUSTOM... chains isn't a good idea. As Michael mentioned these = should 'reserved' for user's purposes. Further, the rules are added without l= ooking into the existing contents. Also no good idea. > - The rules are generated 'on the fly'. Nice, tricky process, but not reall= y verifiable. That is how we generate firewall rules. There is nothing wrong with this. Our= other scripts have a debug option, so that the generated rules will be writt= en to a file to have a look at them before throwing them into the kernel. > - Graphs are generated at each update. This is the good old IPCop manner. T= his process seems to be very efficient, but it is not! The graphs are display= ed quicker in the WUI, but on the cost of a great load for generating them pe= riodically. I think 99,5% of the genarated images aren't looked at by the use= r. This is the reason, why IPFire generates such graphics on demand. Many people are running IPFire on flash storage and we try to keep write oper= ations to a minimum. It is of course not possible to save us from storing the= data in the RRD database, but the image does not need to be generated. > - I do not know whether using a SQLite data base for the data is really con= vinient. An invocation on the shell ( with argument '-d' ) shows a very slow = processing. The main program /usr/local/bin/iptraffic is called every minute. Oh wow. Why would there be a SQLite database when there are RRD databases? > To my opinion these observations could have been a start point for a discus= sion. But after Frank is leaving us ( why? ), I don't beleive there is somebo= dy now for basic developmnent and maintance of this feature. > Remains the question, do we need such a tool for measuring IP traffic for a= certain ( all ) client(s) in the local network(s). We do not urgently need it, but I would like to have that feature at some poi= nt. I would like it to monitor all traffic though without adding any hosts first,= because then we can simply do more with the data. This will be slightly tric= ky with many hosts, but it isn=E2=80=99t impossible when designed well. There= is pmacct which seems to be a good basis to start from without reinventing t= he wheel and it looks like it performs much better than iptables rules. But I= have never actually built it. -Michael > Best, > Bernhard >=20 > Thx, Michael, for bringing back the discussion from privae to public. >=20 >> Gesendet: Montag, 25. Januar 2021 um 22:01 Uhr >> Von: "Michael Tremer" >> An: "Frank Mainz" >> Cc: "development" >> Betreff: Re: [PATCH] New package: IPTraffic 0.8.2 >>=20 >> Hello Frank, >>=20 >> We have a policy to not take public conversations private: >>=20 >> https://wiki.ipfire.org/devel/contact >>=20 >> It is under "Never make a public conversation private=E2=80=9D and it is t= here because sometimes mail user agents do this because of bad UI, and someti= mes there are other reasons. >>=20 >> -Michael >>=20 >>> On 25 Jan 2021, at 20:58, Frank Mainz wrote: >>>=20 >>> Michael, >>> I had not answered without reason only privately. >>> Throw away the AddOn. Forget it. >>>=20 >>> unfortunately you can't unsubscribe with a simple "unsubscibe". >>>=20 >>> What is a spam filter for. >>>=20 >>> Am Montag, den 25.01.2021, 20:48 +0000 schrieb Michael Tremer: >>>> Hello Frank, >>>>=20 >>>>> On 25 Jan 2021, at 20:22, Frank Mainz >>>>> wrote: >>>>>=20 >>>>> Am Montag, den 25.01.2021, 20:05 +0000 schrieb Michael Tremer: >>>>>> Hi Frank, >>>>>>=20 >>>>>> Thank you for your email, but it didn=E2=80=99t answer any of my >>>>>> questions. >>>>>>=20 >>>>>> I wasn=E2=80=99t aware of the wiki pages, but I had a look at it. >>>>>>=20 >>>>>> The questions from my previous email remain though. I do not >>>>>> think a >>>>>> single one was answered. >>>>>=20 >>>>> That is not my problem. >>>>=20 >>>> Okay, I am sorry, but I would have to vote to reject this patch then. >>>>=20 >>>> It is very important that we review all code that is going into the >>>> distribution and that we have a reason that it is there. Code that is >>>> redundant, not needed and unmaintained is a potential security >>>> problem. I am sure this is very easy to understand. >>>>=20 >>>> Also, you can read here what it takes to get your patches into the >>>> distribution: >>>>=20 >>>> https://wiki.ipfire.org/devel/submit-patches >>>>=20 >>>> This is not there for our own entertainment. It is a tested and tuned >>>> process which guarantees that IPFire is a product of very high >>>> quality. >>>>=20 >>>>>>> On 25 Jan 2021, at 19:51, Frank Mainz >>>>>>> wrote: >>>>>>>=20 >>>>>>> Hello Michael, >>>>>>>=20 >>>>>>> look in the WIKI and you know what is displayed. Otherwise, the >>>>>>> traffic >>>>>>> of individual IP addresses is logged with the help of entries >>>>>>> in >>>>>>> the >>>>>>> CUSTOM chains and graphically prepared. >>>>>>=20 >>>>>> The CUSTOM* chains can=E2=80=99t be touched by any programs that come >>>>>> with >>>>>> the distributions. They exist for users to add any custom rules >>>>>> and >>>>>> they are under full control by the user only. >>>>>=20 >>>>> Then it does fit. >>>>=20 >>>> No, it doesn=E2=80=99t, because you are not the user of all those firewa= lls. >>>>=20 >>>> Every chain has a defined purpose, and you are using it for something >>>> it isn=E2=80=99t designed for and it will break setups. >>>>=20 >>>>>>> https://wiki.ipfire.org/addons/iptraffic-playground >>>>>>>=20 >>>>>>>=20 >>>>>>> Greetings Frank >>>>>>>=20 >>>>>>> PS: Fullquote removed >>>>>>=20 >>>>>> Why? I don=E2=80=99t recommend doing this because it removes context f= rom >>>>>> the >>>>>> conversation. >>>>>>=20 >>>>>> Best, >>>>>> -Michael >>>>>=20 >>>>> Then be careful next time not to quote >100 lines of source code. >>>>> How >>>>> this is supposed to preserve context is beyond me. >>>>=20 >>>> Because we write comments into the code. And we normally try to >>>> answer other people=E2=80=99s questions. >>>>=20 >>>> Do not forget that many people are reading this list and they do not >>>> devote a lot of time to get themselves involved in everything. >>>> Therefore things have to be understood at a quick glance. >>>>=20 >>>>> But do what you want. Exactly because of such reactions I didn't >>>>> want >>>>> to port the AddOn for IPFire. >>>>=20 >>>> I have absolutely no idea why you are reacting like this. The process >>>> is clear and I am here to make IPFire as good as I can possibly do. >>>>=20 >>>> I have simply asked what your code is and what it does. You seem to >>>> not want to answer those questions and why is unfortunately beyond >>>> me. >>>>=20 >>>>> Have a nice life. I am out. >>>>=20 >>>> Under those circumstances I must say that I find it very >>>> disappointing that you have wasted my time and Matthias=E2=80=99 time, t= oo. I >>>> have re-read my email and I do not see why this might have caused any >>>> offence. >>>>=20 >>>> I assume that nobody else wants to push this forward any more and >>>> maintain this code? >>>>=20 >>>> -Michael >>>=20 >>=20 >>=20 --===============2196226060787535313==--