Hello Frank,
On 25 Jan 2021, at 20:22, Frank Mainz frank@cybermainzel.de wrote:
Am Montag, den 25.01.2021, 20:05 +0000 schrieb Michael Tremer:
Hi Frank,
Thank you for your email, but it didn’t answer any of my questions.
I wasn’t aware of the wiki pages, but I had a look at it.
The questions from my previous email remain though. I do not think a single one was answered.
That is not my problem.
Okay, I am sorry, but I would have to vote to reject this patch then.
It is very important that we review all code that is going into the distribution and that we have a reason that it is there. Code that is redundant, not needed and unmaintained is a potential security problem. I am sure this is very easy to understand.
Also, you can read here what it takes to get your patches into the distribution:
https://wiki.ipfire.org/devel/submit-patches
This is not there for our own entertainment. It is a tested and tuned process which guarantees that IPFire is a product of very high quality.
On 25 Jan 2021, at 19:51, Frank Mainz frank@cybermainzel.de wrote:
Hello Michael,
look in the WIKI and you know what is displayed. Otherwise, the traffic of individual IP addresses is logged with the help of entries in the CUSTOM chains and graphically prepared.
The CUSTOM* chains can’t be touched by any programs that come with the distributions. They exist for users to add any custom rules and they are under full control by the user only.
Then it does fit.
No, it doesn’t, because you are not the user of all those firewalls.
Every chain has a defined purpose, and you are using it for something it isn’t designed for and it will break setups.
https://wiki.ipfire.org/addons/iptraffic-playground
Greetings Frank
PS: Fullquote removed
Why? I don’t recommend doing this because it removes context from the conversation.
Best, -Michael
Then be careful next time not to quote >100 lines of source code. How this is supposed to preserve context is beyond me.
Because we write comments into the code. And we normally try to answer other people’s questions.
Do not forget that many people are reading this list and they do not devote a lot of time to get themselves involved in everything. Therefore things have to be understood at a quick glance.
But do what you want. Exactly because of such reactions I didn't want to port the AddOn for IPFire.
I have absolutely no idea why you are reacting like this. The process is clear and I am here to make IPFire as good as I can possibly do.
I have simply asked what your code is and what it does. You seem to not want to answer those questions and why is unfortunately beyond me.
Have a nice life. I am out.
Under those circumstances I must say that I find it very disappointing that you have wasted my time and Matthias’ time, too. I have re-read my email and I do not see why this might have caused any offence.
I assume that nobody else wants to push this forward any more and maintain this code?
-Michael
Michael, I had not answered without reason only privately. Throw away the AddOn. Forget it.
unfortunately you can't unsubscribe with a simple "unsubscibe".
What is a spam filter for.
Am Montag, den 25.01.2021, 20:48 +0000 schrieb Michael Tremer:
Hello Frank,
On 25 Jan 2021, at 20:22, Frank Mainz frank@cybermainzel.de wrote:
Am Montag, den 25.01.2021, 20:05 +0000 schrieb Michael Tremer:
Hi Frank,
Thank you for your email, but it didn’t answer any of my questions.
I wasn’t aware of the wiki pages, but I had a look at it.
The questions from my previous email remain though. I do not think a single one was answered.
That is not my problem.
Okay, I am sorry, but I would have to vote to reject this patch then.
It is very important that we review all code that is going into the distribution and that we have a reason that it is there. Code that is redundant, not needed and unmaintained is a potential security problem. I am sure this is very easy to understand.
Also, you can read here what it takes to get your patches into the distribution:
https://wiki.ipfire.org/devel/submit-patches
This is not there for our own entertainment. It is a tested and tuned process which guarantees that IPFire is a product of very high quality.
On 25 Jan 2021, at 19:51, Frank Mainz frank@cybermainzel.de wrote:
Hello Michael,
look in the WIKI and you know what is displayed. Otherwise, the traffic of individual IP addresses is logged with the help of entries in the CUSTOM chains and graphically prepared.
The CUSTOM* chains can’t be touched by any programs that come with the distributions. They exist for users to add any custom rules and they are under full control by the user only.
Then it does fit.
No, it doesn’t, because you are not the user of all those firewalls.
Every chain has a defined purpose, and you are using it for something it isn’t designed for and it will break setups.
https://wiki.ipfire.org/addons/iptraffic-playground
Greetings Frank
PS: Fullquote removed
Why? I don’t recommend doing this because it removes context from the conversation.
Best, -Michael
Then be careful next time not to quote >100 lines of source code. How this is supposed to preserve context is beyond me.
Because we write comments into the code. And we normally try to answer other people’s questions.
Do not forget that many people are reading this list and they do not devote a lot of time to get themselves involved in everything. Therefore things have to be understood at a quick glance.
But do what you want. Exactly because of such reactions I didn't want to port the AddOn for IPFire.
I have absolutely no idea why you are reacting like this. The process is clear and I am here to make IPFire as good as I can possibly do.
I have simply asked what your code is and what it does. You seem to not want to answer those questions and why is unfortunately beyond me.
Have a nice life. I am out.
Under those circumstances I must say that I find it very disappointing that you have wasted my time and Matthias’ time, too. I have re-read my email and I do not see why this might have caused any offence.
I assume that nobody else wants to push this forward any more and maintain this code?
-Michael
Hello Frank,
We have a policy to not take public conversations private:
https://wiki.ipfire.org/devel/contact
It is under "Never make a public conversation private” and it is there because sometimes mail user agents do this because of bad UI, and sometimes there are other reasons.
-Michael
On 25 Jan 2021, at 20:58, Frank Mainz frank@cybermainzel.de wrote:
Michael, I had not answered without reason only privately. Throw away the AddOn. Forget it.
unfortunately you can't unsubscribe with a simple "unsubscibe".
What is a spam filter for.
Am Montag, den 25.01.2021, 20:48 +0000 schrieb Michael Tremer:
Hello Frank,
On 25 Jan 2021, at 20:22, Frank Mainz frank@cybermainzel.de wrote:
Am Montag, den 25.01.2021, 20:05 +0000 schrieb Michael Tremer:
Hi Frank,
Thank you for your email, but it didn’t answer any of my questions.
I wasn’t aware of the wiki pages, but I had a look at it.
The questions from my previous email remain though. I do not think a single one was answered.
That is not my problem.
Okay, I am sorry, but I would have to vote to reject this patch then.
It is very important that we review all code that is going into the distribution and that we have a reason that it is there. Code that is redundant, not needed and unmaintained is a potential security problem. I am sure this is very easy to understand.
Also, you can read here what it takes to get your patches into the distribution:
https://wiki.ipfire.org/devel/submit-patches
This is not there for our own entertainment. It is a tested and tuned process which guarantees that IPFire is a product of very high quality.
On 25 Jan 2021, at 19:51, Frank Mainz frank@cybermainzel.de wrote:
Hello Michael,
look in the WIKI and you know what is displayed. Otherwise, the traffic of individual IP addresses is logged with the help of entries in the CUSTOM chains and graphically prepared.
The CUSTOM* chains can’t be touched by any programs that come with the distributions. They exist for users to add any custom rules and they are under full control by the user only.
Then it does fit.
No, it doesn’t, because you are not the user of all those firewalls.
Every chain has a defined purpose, and you are using it for something it isn’t designed for and it will break setups.
https://wiki.ipfire.org/addons/iptraffic-playground
Greetings Frank
PS: Fullquote removed
Why? I don’t recommend doing this because it removes context from the conversation.
Best, -Michael
Then be careful next time not to quote >100 lines of source code. How this is supposed to preserve context is beyond me.
Because we write comments into the code. And we normally try to answer other people’s questions.
Do not forget that many people are reading this list and they do not devote a lot of time to get themselves involved in everything. Therefore things have to be understood at a quick glance.
But do what you want. Exactly because of such reactions I didn't want to port the AddOn for IPFire.
I have absolutely no idea why you are reacting like this. The process is clear and I am here to make IPFire as good as I can possibly do.
I have simply asked what your code is and what it does. You seem to not want to answer those questions and why is unfortunately beyond me.
Have a nice life. I am out.
Under those circumstances I must say that I find it very disappointing that you have wasted my time and Matthias’ time, too. I have re-read my email and I do not see why this might have caused any offence.
I assume that nobody else wants to push this forward any more and maintain this code?
-Michael
Hi,
to come back to the facts, some annotations about the addon. I first decided to post them in the community thread about the addon, but in the situation now I think it is better here. I've installed Frank's tar file from the community and studied the source.
- Doing the traffic measurement through iptables rules may be effitive but ineffidcient. Did not investigate the overhead introduced into the system. Maybe it isn't so much ( my tiny ALIX didn't get inresponive ), but it is just a feeling for misuse of iptables. - Using the CUSTOM... chains isn't a good idea. As Michael mentioned these should 'reserved' for user's purposes. Further, the rules are added without looking into the existing contents. Also no good idea. - The rules are generated 'on the fly'. Nice, tricky process, but not really verifiable. - Graphs are generated at each update. This is the good old IPCop manner. This process seems to be very efficient, but it is not! The graphs are displayed quicker in the WUI, but on the cost of a great load for generating them periodically. I think 99,5% of the genarated images aren't looked at by the user. This is the reason, why IPFire generates such graphics on demand. - I do not know whether using a SQLite data base for the data is really convinient. An invocation on the shell ( with argument '-d' ) shows a very slow processing. The main program /usr/local/bin/iptraffic is called every minute.
To my opinion these observations could have been a start point for a discussion. But after Frank is leaving us ( why? ), I don't beleive there is somebody now for basic developmnent and maintance of this feature. Remains the question, do we need such a tool for measuring IP traffic for a certain ( all ) client(s) in the local network(s).
Best, Bernhard
Thx, Michael, for bringing back the discussion from privae to public.
Gesendet: Montag, 25. Januar 2021 um 22:01 Uhr Von: "Michael Tremer" michael.tremer@ipfire.org An: "Frank Mainz" frank@cybermainzel.de Cc: "development" development@lists.ipfire.org Betreff: Re: [PATCH] New package: IPTraffic 0.8.2
Hello Frank,
We have a policy to not take public conversations private:
https://wiki.ipfire.org/devel/contact
It is under "Never make a public conversation private” and it is there because sometimes mail user agents do this because of bad UI, and sometimes there are other reasons.
-Michael
On 25 Jan 2021, at 20:58, Frank Mainz frank@cybermainzel.de wrote:
Michael, I had not answered without reason only privately. Throw away the AddOn. Forget it.
unfortunately you can't unsubscribe with a simple "unsubscibe".
What is a spam filter for.
Am Montag, den 25.01.2021, 20:48 +0000 schrieb Michael Tremer:
Hello Frank,
On 25 Jan 2021, at 20:22, Frank Mainz frank@cybermainzel.de wrote:
Am Montag, den 25.01.2021, 20:05 +0000 schrieb Michael Tremer:
Hi Frank,
Thank you for your email, but it didn’t answer any of my questions.
I wasn’t aware of the wiki pages, but I had a look at it.
The questions from my previous email remain though. I do not think a single one was answered.
That is not my problem.
Okay, I am sorry, but I would have to vote to reject this patch then.
It is very important that we review all code that is going into the distribution and that we have a reason that it is there. Code that is redundant, not needed and unmaintained is a potential security problem. I am sure this is very easy to understand.
Also, you can read here what it takes to get your patches into the distribution:
https://wiki.ipfire.org/devel/submit-patches
This is not there for our own entertainment. It is a tested and tuned process which guarantees that IPFire is a product of very high quality.
On 25 Jan 2021, at 19:51, Frank Mainz frank@cybermainzel.de wrote:
Hello Michael,
look in the WIKI and you know what is displayed. Otherwise, the traffic of individual IP addresses is logged with the help of entries in the CUSTOM chains and graphically prepared.
The CUSTOM* chains can’t be touched by any programs that come with the distributions. They exist for users to add any custom rules and they are under full control by the user only.
Then it does fit.
No, it doesn’t, because you are not the user of all those firewalls.
Every chain has a defined purpose, and you are using it for something it isn’t designed for and it will break setups.
https://wiki.ipfire.org/addons/iptraffic-playground
Greetings Frank
PS: Fullquote removed
Why? I don’t recommend doing this because it removes context from the conversation.
Best, -Michael
Then be careful next time not to quote >100 lines of source code. How this is supposed to preserve context is beyond me.
Because we write comments into the code. And we normally try to answer other people’s questions.
Do not forget that many people are reading this list and they do not devote a lot of time to get themselves involved in everything. Therefore things have to be understood at a quick glance.
But do what you want. Exactly because of such reactions I didn't want to port the AddOn for IPFire.
I have absolutely no idea why you are reacting like this. The process is clear and I am here to make IPFire as good as I can possibly do.
I have simply asked what your code is and what it does. You seem to not want to answer those questions and why is unfortunately beyond me.
Have a nice life. I am out.
Under those circumstances I must say that I find it very disappointing that you have wasted my time and Matthias’ time, too. I have re-read my email and I do not see why this might have caused any offence.
I assume that nobody else wants to push this forward any more and maintain this code?
-Michael
Hello Bernhard,
On 25 Jan 2021, at 21:50, Bernhard Bitsch Bernhard.Bitsch@gmx.de wrote:
Hi,
to come back to the facts, some annotations about the addon. I first decided to post them in the community thread about the addon, but in the situation now I think it is better here.
Yes, let’s please keep all development work here, because not many of us monitor the community portal all of the time, and it is easier to miss something on there, too.
I've installed Frank's tar file from the community and studied the source.
- Doing the traffic measurement through iptables rules may be effitive but ineffidcient. Did not investigate the overhead introduced into the system. Maybe it isn't so much ( my tiny ALIX didn't get inresponive ), but it is just a feeling for misuse of iptables.
Yes, that is a misuse of iptables and won’t perform at all.
In the past few years, we have been spending a lot of time on making sure that the kernel takes advantage of multiple processors and multiple queues in the network adapters - one of the things the Lightning Wire Labs appliances do really well. That way, we can have a smaller piece of hardware, but we can take advantage of all its features and make it really fast.
Suricata was also a big challenge, but it now uses all available processor cores.
The kernel used to only evaluate iptables rules on the first core, which was a big bottleneck and since modern hardware usually has more rather than faster cores, it would not have increased network throughput. The counters are mainly the problem here, because they require synchronisation between all processor cores.
Adding more rules like this would negatively impact network throughput - and probably even more importantly, the CUSTOM* chains have been used which make every single packet hit every single rule. One of the things that makes Linux a fast firewall is the connection tracking and that we only need to look at very few rules for existing connections. This is changed here when there is a rule for every single monitored host on the network.
- Using the CUSTOM... chains isn't a good idea. As Michael mentioned these should 'reserved' for user's purposes. Further, the rules are added without looking into the existing contents. Also no good idea.
- The rules are generated 'on the fly'. Nice, tricky process, but not really verifiable.
That is how we generate firewall rules. There is nothing wrong with this. Our other scripts have a debug option, so that the generated rules will be written to a file to have a look at them before throwing them into the kernel.
- Graphs are generated at each update. This is the good old IPCop manner. This process seems to be very efficient, but it is not! The graphs are displayed quicker in the WUI, but on the cost of a great load for generating them periodically. I think 99,5% of the genarated images aren't looked at by the user. This is the reason, why IPFire generates such graphics on demand.
Many people are running IPFire on flash storage and we try to keep write operations to a minimum. It is of course not possible to save us from storing the data in the RRD database, but the image does not need to be generated.
- I do not know whether using a SQLite data base for the data is really convinient. An invocation on the shell ( with argument '-d' ) shows a very slow processing. The main program /usr/local/bin/iptraffic is called every minute.
Oh wow. Why would there be a SQLite database when there are RRD databases?
To my opinion these observations could have been a start point for a discussion. But after Frank is leaving us ( why? ), I don't beleive there is somebody now for basic developmnent and maintance of this feature. Remains the question, do we need such a tool for measuring IP traffic for a certain ( all ) client(s) in the local network(s).
We do not urgently need it, but I would like to have that feature at some point.
I would like it to monitor all traffic though without adding any hosts first, because then we can simply do more with the data. This will be slightly tricky with many hosts, but it isn’t impossible when designed well. There is pmacct which seems to be a good basis to start from without reinventing the wheel and it looks like it performs much better than iptables rules. But I have never actually built it.
-Michael
Best, Bernhard
Thx, Michael, for bringing back the discussion from privae to public.
Gesendet: Montag, 25. Januar 2021 um 22:01 Uhr Von: "Michael Tremer" michael.tremer@ipfire.org An: "Frank Mainz" frank@cybermainzel.de Cc: "development" development@lists.ipfire.org Betreff: Re: [PATCH] New package: IPTraffic 0.8.2
Hello Frank,
We have a policy to not take public conversations private:
https://wiki.ipfire.org/devel/contact
It is under "Never make a public conversation private” and it is there because sometimes mail user agents do this because of bad UI, and sometimes there are other reasons.
-Michael
On 25 Jan 2021, at 20:58, Frank Mainz frank@cybermainzel.de wrote:
Michael, I had not answered without reason only privately. Throw away the AddOn. Forget it.
unfortunately you can't unsubscribe with a simple "unsubscibe".
What is a spam filter for.
Am Montag, den 25.01.2021, 20:48 +0000 schrieb Michael Tremer:
Hello Frank,
On 25 Jan 2021, at 20:22, Frank Mainz frank@cybermainzel.de wrote:
Am Montag, den 25.01.2021, 20:05 +0000 schrieb Michael Tremer:
Hi Frank,
Thank you for your email, but it didn’t answer any of my questions.
I wasn’t aware of the wiki pages, but I had a look at it.
The questions from my previous email remain though. I do not think a single one was answered.
That is not my problem.
Okay, I am sorry, but I would have to vote to reject this patch then.
It is very important that we review all code that is going into the distribution and that we have a reason that it is there. Code that is redundant, not needed and unmaintained is a potential security problem. I am sure this is very easy to understand.
Also, you can read here what it takes to get your patches into the distribution:
https://wiki.ipfire.org/devel/submit-patches
This is not there for our own entertainment. It is a tested and tuned process which guarantees that IPFire is a product of very high quality.
> On 25 Jan 2021, at 19:51, Frank Mainz frank@cybermainzel.de > wrote: > > Hello Michael, > > look in the WIKI and you know what is displayed. Otherwise, the > traffic > of individual IP addresses is logged with the help of entries > in > the > CUSTOM chains and graphically prepared.
The CUSTOM* chains can’t be touched by any programs that come with the distributions. They exist for users to add any custom rules and they are under full control by the user only.
Then it does fit.
No, it doesn’t, because you are not the user of all those firewalls.
Every chain has a defined purpose, and you are using it for something it isn’t designed for and it will break setups.
> https://wiki.ipfire.org/addons/iptraffic-playground > > > Greetings Frank > > PS: Fullquote removed
Why? I don’t recommend doing this because it removes context from the conversation.
Best, -Michael
Then be careful next time not to quote >100 lines of source code. How this is supposed to preserve context is beyond me.
Because we write comments into the code. And we normally try to answer other people’s questions.
Do not forget that many people are reading this list and they do not devote a lot of time to get themselves involved in everything. Therefore things have to be understood at a quick glance.
But do what you want. Exactly because of such reactions I didn't want to port the AddOn for IPFire.
I have absolutely no idea why you are reacting like this. The process is clear and I am here to make IPFire as good as I can possibly do.
I have simply asked what your code is and what it does. You seem to not want to answer those questions and why is unfortunately beyond me.
Have a nice life. I am out.
Under those circumstances I must say that I find it very disappointing that you have wasted my time and Matthias’ time, too. I have re-read my email and I do not see why this might have caused any offence.
I assume that nobody else wants to push this forward any more and maintain this code?
-Michael
Started a new thread for pmacct - hope that was OK. This is continued from: https://lists.ipfire.org/pipermail/development/2021-January/009174.html https://lists.ipfire.org/pipermail/development/2021-January/009174.html
Remains the question, do we need such a tool for measuring IP traffic for a certain ( all ) client(s) in the local network(s).
We do not urgently need it, but I would like to have that feature at some point.
I would like it to monitor all traffic though without adding any hosts first, because then we can simply do more with the data. This will be slightly tricky with many hosts, but it isn’t impossible when designed well. There is pmacct which seems to be a good basis to start from without reinventing the wheel and it looks like it performs much better than iptables rules. But I have never actually built it.
Michael, I’ve been using pmacct on & off since Oct 2019. There is information here:
https://community.ipfire.org/t/pmacct-a-lightweight-passive-network-monitori... https://community.ipfire.org/t/pmacct-a-lightweight-passive-network-monitoring-tool/47
My Use Case: I’ve been over-running our data caps for a year and I wanted to track down the main problem(s). I had a good idea of the problem devices but I wanted to be sure. AND I am a big fan of "You can't fix what you can't track". So that means these type of metrics.
Hopefully the above thread will give you an idea about pmacct.
Regards, Jon
-
Bernhard Bitsch -
Frank Mainz -
Jon Murphy -
Michael Tremer