Re: [PATCH] New package: IPTraffic 0.8.2

[Michael Tremer <michael.tremer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 9279 bytes --]

Hello Bernhard,

> On 25 Jan 2021, at 21:50, Bernhard Bitsch <Bernhard.Bitsch(a)gmx.de> wrote:
> 
> Hi,
> 
> to come back to the facts, some annotations about the addon. I first decided to post them in the community thread about the addon, but in the situation now I think it is better here. 

Yes, let’s please keep all development work here, because not many of us monitor the community portal all of the time, and it is easier to miss something on there, too.

> I've installed Frank's tar file from the community and studied the source.
> 
> - Doing the traffic measurement through iptables rules may be effitive but ineffidcient. Did not investigate the overhead introduced into the system. Maybe it isn't so much ( my tiny ALIX didn't get inresponive ), but it is just a feeling for misuse of iptables.

Yes, that is a misuse of iptables and won’t perform at all.

In the past few years, we have been spending a lot of time on making sure that the kernel takes advantage of multiple processors and multiple queues in the network adapters - one of the things the Lightning Wire Labs appliances do really well. That way, we can have a smaller piece of hardware, but we can take advantage of all its features and make it really fast.

Suricata was also a big challenge, but it now uses all available processor cores.

The kernel used to only evaluate iptables rules on the first core, which was a big bottleneck and since modern hardware usually has more rather than faster cores, it would not have increased network throughput. The counters are mainly the problem here, because they require synchronisation between all processor cores.

Adding more rules like this would negatively impact network throughput - and probably even more importantly, the CUSTOM* chains have been used which make every single packet hit every single rule. One of the things that makes Linux a fast firewall is the connection tracking and that we only need to look at very few rules for existing connections. This is changed here when there is a rule for every single monitored host on the network.

> - Using the CUSTOM... chains isn't a good idea. As Michael mentioned these should 'reserved' for user's purposes. Further, the rules are added without looking into the existing contents. Also no good idea.
> - The rules are generated 'on the fly'. Nice, tricky process, but not really verifiable.

That is how we generate firewall rules. There is nothing wrong with this. Our other scripts have a debug option, so that the generated rules will be written to a file to have a look at them before throwing them into the kernel.

> - Graphs are generated at each update. This is the good old IPCop manner. This process seems to be very efficient, but it is not! The graphs are displayed quicker in the WUI, but on the cost of a great load for generating them periodically. I think 99,5% of the genarated images aren't looked at by the user. This is the reason, why IPFire generates such graphics on demand.

Many people are running IPFire on flash storage and we try to keep write operations to a minimum. It is of course not possible to save us from storing the data in the RRD database, but the image does not need to be generated.

> - I do not know whether using a SQLite data base for the data is really convinient. An invocation on the shell ( with argument '-d' ) shows a very slow processing. The main program /usr/local/bin/iptraffic is called every minute.

Oh wow. Why would there be a SQLite database when there are RRD databases?

> To my opinion these observations could have been a start point for a discussion. But after Frank is leaving us ( why? ), I don't beleive there is somebody now for basic developmnent and maintance of this feature.
> Remains the question, do we need such a tool for measuring IP traffic for a certain ( all ) client(s) in the local network(s).

We do not urgently need it, but I would like to have that feature at some point.

I would like it to monitor all traffic though without adding any hosts first, because then we can simply do more with the data. This will be slightly tricky with many hosts, but it isn’t impossible when designed well. There is pmacct which seems to be a good basis to start from without reinventing the wheel and it looks like it performs much better than iptables rules. But I have never actually built it.

-Michael

> Best,
> Bernhard
> 
> Thx, Michael, for bringing back the discussion from privae to public.
> 
>> Gesendet: Montag, 25. Januar 2021 um 22:01 Uhr
>> Von: "Michael Tremer" <michael.tremer(a)ipfire.org>
>> An: "Frank Mainz" <frank(a)cybermainzel.de>
>> Cc: "development" <development(a)lists.ipfire.org>
>> Betreff: Re: [PATCH] New package: IPTraffic 0.8.2
>> 
>> Hello Frank,
>> 
>> We have a policy to not take public conversations private:
>> 
>>  https://wiki.ipfire.org/devel/contact
>> 
>> It is under "Never make a public conversation private” and it is there because sometimes mail user agents do this because of bad UI, and sometimes there are other reasons.
>> 
>> -Michael
>> 
>>> On 25 Jan 2021, at 20:58, Frank Mainz <frank(a)cybermainzel.de> wrote:
>>> 
>>> Michael,
>>> I had not answered without reason only privately.
>>> Throw away the AddOn. Forget it.
>>> 
>>> unfortunately you can't unsubscribe with a simple "unsubscibe".
>>> 
>>> What is a spam filter for.
>>> 
>>> Am Montag, den 25.01.2021, 20:48 +0000 schrieb Michael Tremer:
>>>> Hello Frank,
>>>> 
>>>>> On 25 Jan 2021, at 20:22, Frank Mainz <frank(a)cybermainzel.de>
>>>>> wrote:
>>>>> 
>>>>> Am Montag, den 25.01.2021, 20:05 +0000 schrieb Michael Tremer:
>>>>>> Hi Frank,
>>>>>> 
>>>>>> Thank you for your email, but it didn’t answer any of my
>>>>>> questions.
>>>>>> 
>>>>>> I wasn’t aware of the wiki pages, but I had a look at it.
>>>>>> 
>>>>>> The questions from my previous email remain though. I do not
>>>>>> think a
>>>>>> single one was answered.
>>>>> 
>>>>> That is not my problem.
>>>> 
>>>> Okay, I am sorry, but I would have to vote to reject this patch then.
>>>> 
>>>> It is very important that we review all code that is going into the
>>>> distribution and that we have a reason that it is there. Code that is
>>>> redundant, not needed and unmaintained is a potential security
>>>> problem. I am sure this is very easy to understand.
>>>> 
>>>> Also, you can read here what it takes to get your patches into the
>>>> distribution:
>>>> 
>>>> https://wiki.ipfire.org/devel/submit-patches
>>>> 
>>>> This is not there for our own entertainment. It is a tested and tuned
>>>> process which guarantees that IPFire is a product of very high
>>>> quality.
>>>> 
>>>>>>> On 25 Jan 2021, at 19:51, Frank Mainz <frank(a)cybermainzel.de>
>>>>>>> wrote:
>>>>>>> 
>>>>>>> Hello Michael,
>>>>>>> 
>>>>>>> look in the WIKI and you know what is displayed. Otherwise, the
>>>>>>> traffic
>>>>>>> of individual IP addresses is logged with the help of entries
>>>>>>> in
>>>>>>> the
>>>>>>> CUSTOM chains and graphically prepared.
>>>>>> 
>>>>>> The CUSTOM* chains can’t be touched by any programs that come
>>>>>> with
>>>>>> the distributions. They exist for users to add any custom rules
>>>>>> and
>>>>>> they are under full control by the user only.
>>>>> 
>>>>> Then it does fit.
>>>> 
>>>> No, it doesn’t, because you are not the user of all those firewalls.
>>>> 
>>>> Every chain has a defined purpose, and you are using it for something
>>>> it isn’t designed for and it will break setups.
>>>> 
>>>>>>> https://wiki.ipfire.org/addons/iptraffic-playground
>>>>>>> 
>>>>>>> 
>>>>>>> Greetings Frank
>>>>>>> 
>>>>>>> PS: Fullquote removed
>>>>>> 
>>>>>> Why? I don’t recommend doing this because it removes context from
>>>>>> the
>>>>>> conversation.
>>>>>> 
>>>>>> Best,
>>>>>> -Michael
>>>>> 
>>>>> Then be careful next time not to quote >100 lines of source code.
>>>>> How
>>>>> this is supposed to preserve context is beyond me.
>>>> 
>>>> Because we write comments into the code. And we normally try to
>>>> answer other people’s questions.
>>>> 
>>>> Do not forget that many people are reading this list and they do not
>>>> devote a lot of time to get themselves involved in everything.
>>>> Therefore things have to be understood at a quick glance.
>>>> 
>>>>> But do what you want. Exactly because of such reactions I didn't
>>>>> want
>>>>> to port the AddOn for IPFire.
>>>> 
>>>> I have absolutely no idea why you are reacting like this. The process
>>>> is clear and I am here to make IPFire as good as I can possibly do.
>>>> 
>>>> I have simply asked what your code is and what it does. You seem to
>>>> not want to answer those questions and why is unfortunately beyond
>>>> me.
>>>> 
>>>>> Have a nice life. I am out.
>>>> 
>>>> Under those circumstances I must say that I find it very
>>>> disappointing that you have wasted my time and Matthias’ time, too. I
>>>> have re-read my email and I do not see why this might have caused any
>>>> offence.
>>>> 
>>>> I assume that nobody else wants to push this forward any more and
>>>> maintain this code?
>>>> 
>>>> -Michael
>>> 
>> 
>>