As a contributor to the SANS ISC, I email my IPTABLES logs hourly to the Dshield database, however the default IPFire IPTABLES logs are incomplete since packets dropped by Locationblock are not logged to syslog.
As a workaround I edit the locationblock function of rules.pl to add a IPTABLES rule to log the dropped packets. This edit does not of course survive any Core-Updates that upgrade rules.pl and the edit needs to be re-applied.
In addition, unless the Locationblock packets are logged to syslog the data displayed in the Firewall logs sections are incomplete as the packets dropped by Location Block are not displayed.
I understand that the original intension of the Location Block feature was to reduce the amount of log messages on installations running on extremely cheap flash storage. I suspect that these installations are now almost zero especially since IPFire no longer supports 32 bit versions.
The obvious solution to this to make Location Block logging selectable as a firewall option and my quick look at making a patches for this would seem to be fairly simple task to rules.pl and and optionsfw.cgi.
I would be happy to provide the necessary patches should this be acceptable.
Hello Rob,
Good to hear from you again.
On 9 Nov 2024, at 14:00, Rob Brewer ipfire-devel@grantura.co.uk wrote:
As a contributor to the SANS ISC, I email my IPTABLES logs hourly to the Dshield database, however the default IPFire IPTABLES logs are incomplete since packets dropped by Locationblock are not logged to syslog.
This is correct.
As a workaround I edit the locationblock function of rules.pl to add a IPTABLES rule to log the dropped packets. This edit does not of course survive any Core-Updates that upgrade rules.pl and the edit needs to be re-applied.
In addition, unless the Locationblock packets are logged to syslog the data displayed in the Firewall logs sections are incomplete as the packets dropped by Location Block are not displayed.
I understand that the original intension of the Location Block feature was to reduce the amount of log messages on installations running on extremely cheap flash storage. I suspect that these installations are now almost zero especially since IPFire no longer supports 32 bit versions.
The obvious solution to this to make Location Block logging selectable as a firewall option and my quick look at making a patches for this would seem to be fairly simple task to rules.pl and and optionsfw.cgi.
I would be happy to provide the necessary patches should this be acceptable.
I think a couple of things have changed since we introduced this feature and we have also learned a couple of lessons here:
* Yes it is true that this was introduced to keep the logs smaller, but not necessarily to avoid using storage, mainly to keep them readable by a human. I was always a bit doubtful about this and believe that apart from this, there is no more value in a general location blocking feature. The firewall will drop those packets anyway.
* People seem to misunderstand this feature: The block as many countries as they can click on and think that this makes things “more secure”, which of course it doesn’t.
* What I personally like as a feature is to apply location filters to individual rules. That allows to apply rate limits in a few places and easily select some larger groups of potentially less interesting traffic.
* Back when this was introduced there was also a lot less use of botnets as we know them now. Pretty much all recent DoS attacks I remember have been coming from all over the world because systems where compromised wherever they were. This renders the filter absolutely useless.
* Infrastructure is becoming more centralised. Some people attempt to block the US and pretty much nothing works any more. Well…
Therefore I am happy to rethink this feature - personally I would even remove it entirely, but I don’t like to handle the stress about this now.
I think this should log now. I don’t think there is any legitimate reason to disable firewall logging at all. This is the paper trail that you want when debugging anything or doing forensics on an attack. Certainly bad flash is not a valid reason for me.
I would be happy to hear some more opinions on this from the list.
-Michael
On Tue, 19 Nov 2024 17:10:24 +0000, Michael Tremer wrote:
Hi Michael, thank for your response, I was beginning to think there was little interest in this topic.
See below...
Hello Rob,
Good to hear from you again.
On 9 Nov 2024, at 14:00, Rob Brewer ipfire-devel@grantura.co.uk wrote:
As a contributor to the SANS ISC, I email my IPTABLES logs hourly to the Dshield database, however the default IPFire IPTABLES logs are incomplete since packets dropped by Locationblock are not logged to syslog.
This is correct.
As a workaround I edit the locationblock function of rules.pl to add a IPTABLES rule to log the dropped packets. This edit does not of course survive any Core-Updates that upgrade rules.pl and the edit needs to be re-applied.
In addition, unless the Locationblock packets are logged to syslog the data displayed in the Firewall logs sections are incomplete as the packets dropped by Location Block are not displayed.
I understand that the original intension of the Location Block feature was to reduce the amount of log messages on installations running on extremely cheap flash storage. I suspect that these installations are now almost zero especially since IPFire no longer supports 32 bit versions.
The obvious solution to this to make Location Block logging selectable as a firewall option and my quick look at making a patches for this would seem to be fairly simple task to rules.pl and and optionsfw.cgi.
I would be happy to provide the necessary patches should this be acceptable.
I think a couple of things have changed since we introduced this feature and we have also learned a couple of lessons here:
- Yes it is true that this was introduced to keep the logs smaller, but
not necessarily to avoid using storage, mainly to keep them readable by a human. I was always a bit doubtful about this and believe that apart from this, there is no more value in a general location blocking feature. The firewall will drop those packets anyway.
IPFire does a very good job in protecting the internal network from unwanted probes from the WAN, and I agree that for most users, location block does not enhance this protection.
As I run a mail server, I use locationblock as part of my filtering to prevent bad actors from accessing my mail server and I think would be useful for anyone who are running servers on their networks.
I don't like blocking traffic, but I have witnessed over the last few years a substantial rise in abuse from certain countries from where I would never expect to receive emails and locationblock works very well for me in filtering out those unwanted connections.
I currently have 21 countries blocked including XD Hostile networks. All of these blocked countries have been seen to support abusive networks.
It would be possible to generate individual rules for these networks which in some cases I do with Banish, but it is sometimes easier just to drop the whole country.
- People seem to misunderstand this feature: The block as many countries
as they can click on and think that this makes things “more secure”, which of course it doesn’t.
I think this is education and I suspect that very few of these people are running servers on their system and need location block.
- What I personally like as a feature is to apply location filters to
individual rules. That allows to apply rate limits in a few places and easily select some larger groups of potentially less interesting traffic.
I agree and is similar to what I am doing for ports 25 and 587 although with location block.
- Back when this was introduced there was also a lot less use of botnets
as we know them now. Pretty much all recent DoS attacks I remember have been coming from all over the world because systems where compromised wherever they were. This renders the filter absolutely useless.
For this, I am currently testing a dynamic block list on my system which adds an entry to an ipset when an ip address exceeds a threshold in the previous 24 hours. Currently there are about 7000 entries in this set which have been detected over the previous month. Not for everyone because it needs a Whitelist to prevent some 'push' technologies such as Facebook and to protect from bulk inputs such as mailing lists. I also blacklist a large number of ip addresses which are associated with port scanners.
This would work well as a botnet filter but it needs an understanding of the blocking process, so it would not be suitable for everyone.
- Infrastructure is becoming more centralised. Some people attempt to
block the US and pretty much nothing works any more. Well…
:)
Therefore I am happy to rethink this feature - personally I would even remove it entirely, but I don’t like to handle the stress about this now.
No I think you should keep it. Locationblock is very useful, but needs to be used in a considered way. I could live without it but its availability makes my life easier. Also being updated on a weekly basis makes keeping track of abusers easier.
I think this should log now. I don’t think there is any legitimate reason to disable firewall logging at all. This is the paper trail that you want when debugging anything or doing forensics on an attack. Certainly bad flash is not a valid reason for me.
I have started generating some patches to make 'locationblock logging' as a selectable feature and could post them here in the next day or so if of any use.
In fact I think that IPFire's logging environment, which was inherited from IPCop needs a re-visit.
I was pleased to see that a while back there was a discussion of replacing sylogd with rsyslogd which I think would be a very useful upgrade and enable kernel messages, like on most current linux systems, to be logged to alternative log files instead of everything ending up in /var/log/ messages.
The replacement with rsyslgd could also inhibit the "last message repeated x times" in syslog and which are ignored by IPFire's Log menu and causes erroneous results in the 'Count' values.
I would be happy to hear some more opinions on this from the list.
-Michael
I hope these comments are useful and would be pleased to help where I can.
Rob
Hi Rob & Michael,
On 20/11/2024 14:36, Rob Brewer wrote:
On Tue, 19 Nov 2024 17:10:24 +0000, Michael Tremer wrote:
Hi Michael, thank for your response, I was beginning to think there was little interest in this topic.
I think people have either been away on vacation or very busy hence the slower response time.
See below...
Hello Rob,
Good to hear from you again.
On 9 Nov 2024, at 14:00, Rob Brewer ipfire-devel@grantura.co.uk wrote:
As a contributor to the SANS ISC, I email my IPTABLES logs hourly to the Dshield database, however the default IPFire IPTABLES logs are incomplete since packets dropped by Locationblock are not logged to syslog.
This is correct.
As a workaround I edit the locationblock function of rules.pl to add a IPTABLES rule to log the dropped packets. This edit does not of course survive any Core-Updates that upgrade rules.pl and the edit needs to be re-applied.
In addition, unless the Locationblock packets are logged to syslog the data displayed in the Firewall logs sections are incomplete as the packets dropped by Location Block are not displayed.
I understand that the original intension of the Location Block feature was to reduce the amount of log messages on installations running on extremely cheap flash storage. I suspect that these installations are now almost zero especially since IPFire no longer supports 32 bit versions.
The obvious solution to this to make Location Block logging selectable as a firewall option and my quick look at making a patches for this would seem to be fairly simple task to rules.pl and and optionsfw.cgi.
I would be happy to provide the necessary patches should this be acceptable.
I think a couple of things have changed since we introduced this feature and we have also learned a couple of lessons here:
- Yes it is true that this was introduced to keep the logs smaller, but
not necessarily to avoid using storage, mainly to keep them readable by a human. I was always a bit doubtful about this and believe that apart from this, there is no more value in a general location blocking feature. The firewall will drop those packets anyway.
IPFire does a very good job in protecting the internal network from unwanted probes from the WAN, and I agree that for most users, location block does not enhance this protection.
As I run a mail server, I use locationblock as part of my filtering to prevent bad actors from accessing my mail server and I think would be useful for anyone who are running servers on their networks.
I don't like blocking traffic, but I have witnessed over the last few years a substantial rise in abuse from certain countries from where I would never expect to receive emails and locationblock works very well for me in filtering out those unwanted connections.
I currently have 21 countries blocked including XD Hostile networks. All of these blocked countries have been seen to support abusive networks.
It would be possible to generate individual rules for these networks which in some cases I do with Banish, but it is sometimes easier just to drop the whole country.
- People seem to misunderstand this feature: The block as many countries
as they can click on and think that this makes things “more secure”, which of course it doesn’t.
I think this is education and I suspect that very few of these people are running servers on their system and need location block.
- What I personally like as a feature is to apply location filters to
individual rules. That allows to apply rate limits in a few places and easily select some larger groups of potentially less interesting traffic.
I agree and is similar to what I am doing for ports 25 and 587 although with location block.
- Back when this was introduced there was also a lot less use of botnets
as we know them now. Pretty much all recent DoS attacks I remember have been coming from all over the world because systems where compromised wherever they were. This renders the filter absolutely useless.
For this, I am currently testing a dynamic block list on my system which adds an entry to an ipset when an ip address exceeds a threshold in the previous 24 hours. Currently there are about 7000 entries in this set which have been detected over the previous month. Not for everyone because it needs a Whitelist to prevent some 'push' technologies such as Facebook and to protect from bulk inputs such as mailing lists. I also blacklist a large number of ip addresses which are associated with port scanners.
This would work well as a botnet filter but it needs an understanding of the blocking process, so it would not be suitable for everyone.
- Infrastructure is becoming more centralised. Some people attempt to
block the US and pretty much nothing works any more. Well…
:)
Therefore I am happy to rethink this feature - personally I would even remove it entirely, but I don’t like to handle the stress about this now.
No I think you should keep it. Locationblock is very useful, but needs to be used in a considered way. I could live without it but its availability makes my life easier. Also being updated on a weekly basis makes keeping track of abusers easier.
I think this should log now. I don’t think there is any legitimate reason to disable firewall logging at all. This is the paper trail that you want when debugging anything or doing forensics on an attack. Certainly bad flash is not a valid reason for me.
I have started generating some patches to make 'locationblock logging' as a selectable feature and could post them here in the next day or so if of any use.
In fact I think that IPFire's logging environment, which was inherited from IPCop needs a re-visit.
I was pleased to see that a while back there was a discussion of replacing sylogd with rsyslogd which I think would be a very useful upgrade and enable kernel messages, like on most current linux systems, to be logged to alternative log files instead of everything ending up in /var/log/ messages.
The replacement with rsyslgd could also inhibit the "last message repeated x times" in syslog and which are ignored by IPFire's Log menu and causes erroneous results in the 'Count' values.
I would be happy to hear some more opinions on this from the list.
I had added this topic onto the agenda for the next video conf call but if it gets resolved on the mailing list that is also fine and I will then, at that point, remove it from the agenda.
I don't use Location Blocking on my production system at all. Virtually everything is blocked as standard. I just have port 80 open for Lets Encrypt and as they use multiple servers from multiple ASN's to validate you have to not use Location Blocking otherwise you can end up blocking the update process. They will not provide a list of the IP's of their servers.
Therefore from my point of view I have no issue with logging being able to be turned on for Location Blocking as I don't use it anyway.
Regards,
Adolf.
-Michael
I hope these comments are useful and would be pleased to help where I can.
Rob
Hello,
On 20 Nov 2024, at 13:36, Rob Brewer ipfire-devel@grantura.co.uk wrote:
On Tue, 19 Nov 2024 17:10:24 +0000, Michael Tremer wrote:
Hi Michael, thank for your response, I was beginning to think there was little interest in this topic.
I was traveling and so have a little bit of a back log as usual :)
I can’t speak for the others, though.
See below...
Hello Rob,
Good to hear from you again.
On 9 Nov 2024, at 14:00, Rob Brewer ipfire-devel@grantura.co.uk wrote:
As a contributor to the SANS ISC, I email my IPTABLES logs hourly to the Dshield database, however the default IPFire IPTABLES logs are incomplete since packets dropped by Locationblock are not logged to syslog.
This is correct.
As a workaround I edit the locationblock function of rules.pl to add a IPTABLES rule to log the dropped packets. This edit does not of course survive any Core-Updates that upgrade rules.pl and the edit needs to be re-applied.
In addition, unless the Locationblock packets are logged to syslog the data displayed in the Firewall logs sections are incomplete as the packets dropped by Location Block are not displayed.
I understand that the original intension of the Location Block feature was to reduce the amount of log messages on installations running on extremely cheap flash storage. I suspect that these installations are now almost zero especially since IPFire no longer supports 32 bit versions.
The obvious solution to this to make Location Block logging selectable as a firewall option and my quick look at making a patches for this would seem to be fairly simple task to rules.pl and and optionsfw.cgi.
I would be happy to provide the necessary patches should this be acceptable.
I think a couple of things have changed since we introduced this feature and we have also learned a couple of lessons here:
- Yes it is true that this was introduced to keep the logs smaller, but
not necessarily to avoid using storage, mainly to keep them readable by a human. I was always a bit doubtful about this and believe that apart from this, there is no more value in a general location blocking feature. The firewall will drop those packets anyway.
IPFire does a very good job in protecting the internal network from unwanted probes from the WAN, and I agree that for most users, location block does not enhance this protection.
As I run a mail server, I use locationblock as part of my filtering to prevent bad actors from accessing my mail server and I think would be useful for anyone who are running servers on their networks.
I don't like blocking traffic, but I have witnessed over the last few years a substantial rise in abuse from certain countries from where I would never expect to receive emails and locationblock works very well for me in filtering out those unwanted connections.
I currently have 21 countries blocked including XD Hostile networks. All of these blocked countries have been seen to support abusive networks.
It would be possible to generate individual rules for these networks which in some cases I do with Banish, but it is sometimes easier just to drop the whole country.
Easy yes. I am just not sure what we are actually getting out of this.
I don’t like to become political on this list and so I won’t, but I know that some people hold the following beliefs:
* There are “bad” countries out there. In the western world those are Russia, China, and more that fall into this kind of “category”. Nothing is going to stop them to open up an account on AWS and suddenly they have access to resources anywhere in the world and so actually, if those players were up to anything “bad”, they can come out of any IP space they want. There is also Tor which allows this.
* Looking at actual data, the bullet proof ISPs that annoy us are not that far away. They are hosted in Europe and very prominently so in the Netherlands - a “good” country.
This is just an example to explain that we all have our idea of the world in our head, but the Internet doesn’t know these borders. Geopolitics is complicated and I don’t think that this is a tool that helps to fight this kind of problem. I hope this is clear what I am trying to say.
I personally trust in Spamhaus which is feeding data into our “Hostile Networks” blocker and they are looking at the actual traffic that they see from those subnets that they are blocking. They have a clear set of rules and they don’t look just at the country and say “ah yes, the bad guys again”.
I have stated this recently in the RPZ discussion that I feel that we are lacking good sources for this kind of data. For blocking anything. Loads of people believe “more is more” and to be fair… once you get to blocking 90% of the internet you probably have blocked 90% of the noise. However this won’t do anything for targeted attacks. I personally would rather have more quality here.
- People seem to misunderstand this feature: The block as many countries
as they can click on and think that this makes things “more secure”, which of course it doesn’t.
I think this is education and I suspect that very few of these people are running servers on their system and need location block.
I am personally not here to educate people. I am happy to help out and give my opinion, but I suppose considering that people think as I have outlined above, who am I to educate them?
It is just very regretful to me when people are ab-using a feature just to “feel more safe”.
- What I personally like as a feature is to apply location filters to
individual rules. That allows to apply rate limits in a few places and easily select some larger groups of potentially less interesting traffic.
I agree and is similar to what I am doing for ports 25 and 587 although with location block.
- Back when this was introduced there was also a lot less use of botnets
as we know them now. Pretty much all recent DoS attacks I remember have been coming from all over the world because systems where compromised wherever they were. This renders the filter absolutely useless.
For this, I am currently testing a dynamic block list on my system which adds an entry to an ipset when an ip address exceeds a threshold in the previous 24 hours. Currently there are about 7000 entries in this set which have been detected over the previous month. Not for everyone because it needs a Whitelist to prevent some 'push' technologies such as Facebook and to protect from bulk inputs such as mailing lists. I also blacklist a large number of ip addresses which are associated with port scanners.
This would work well as a botnet filter but it needs an understanding of the blocking process, so it would not be suitable for everyone.
We have talked about this on the last team call and came to the conclusion that the whole set of blocking features is a total mess. There are several places where things can be blocked, they all heavily overlap. Not to be going on about Spamhaus again, but the DROP list is in the “Hostile Networks” feature, there is a special country code for it, pretty much every IPS ruleset contains it, the IP blocklist feature supports it. Now there are people throwing this all into DNS as well. I believe this is all getting a little bit out of hand.
I personally would prefer *one* place and we really make this work well.
Some people find the IPS has too much overhead - I am beginning to disagree.
Some people add all sorts of blacklists that are clearly for experimental/research use and suddenly complain about things not working anymore. It is hard to figure out what is actually blocking traffic when there are about five places to look for. The result often is to disable the IPS, then something else, and after a while all of this is turned off and the firewall is configured to a bare minimum. We don’t want people to run it like that.
- Infrastructure is becoming more centralised. Some people attempt to
block the US and pretty much nothing works any more. Well…
:)
Therefore I am happy to rethink this feature - personally I would even remove it entirely, but I don’t like to handle the stress about this now.
No I think you should keep it. Locationblock is very useful, but needs to be used in a considered way. I could live without it but its availability makes my life easier. Also being updated on a weekly basis makes keeping track of abusers easier.
We issue a new location database once a day, but the IPFire systems update it only once a week. We deemed that as sufficient as there are not that many changes from one version to another one.
I think this should log now. I don’t think there is any legitimate reason to disable firewall logging at all. This is the paper trail that you want when debugging anything or doing forensics on an attack. Certainly bad flash is not a valid reason for me.
I have started generating some patches to make 'locationblock logging' as a selectable feature and could post them here in the next day or so if of any use.
What would this basically do? Log it as a simple rule for all, or do you include the selected country? I think the latter would make more sense.
What is the rationale to turn this off? Why don’t we turn this on for everybody all of the time?
In fact I think that IPFire's logging environment, which was inherited from IPCop needs a re-visit.
Yes. But this is probably not a thing I would be looking at in IPFire 2.
I am also happy to think a little bit more outside of the box and see what we would fundamentally change in the future, but not implement in IPFire 2.
I was pleased to see that a while back there was a discussion of replacing sylogd with rsyslogd which I think would be a very useful upgrade and enable kernel messages, like on most current linux systems, to be logged to alternative log files instead of everything ending up in /var/log/ messages.
The currentl syslog server could do that. The problem is more that we don’t have good tools to view the logs on the web UI. So there is simply one place where most things go into and we will then filter it out when users use the web UI. This is tremendously slow.
The replacement with rsyslgd could also inhibit the "last message repeated x times" in syslog and which are ignored by IPFire's Log menu and causes erroneous results in the 'Count' values.
This might be a setting that we can turn off in the current implementation.
We could also think about moving all firewall hits into /var/log/firewall.log or something. That should free up enough in /var/log/messages to make debugging easier.
I would be happy to hear some more opinions on this from the list.
-Michael
I hope these comments are useful and would be pleased to help where I can.
Of course :)
-Michael
Rob
Hi Michael and Adolf,
I'll combine my reply to both of you.
On Wed, 20 Nov 2024 15:21:28 +0000, Michael Tremer wrote:
Hello,
On 20 Nov 2024, at 13:36, Rob Brewer ipfire-devel@grantura.co.uk wrote:
On Tue, 19 Nov 2024 17:10:24 +0000, Michael Tremer wrote:
Hi Michael, thank for your response, I was beginning to think there was little interest in this topic.
I was traveling and so have a little bit of a back log as usual :)
I can’t speak for the others, though.
Not a problem, I should have said that I previously raised this a topic on the forum and Abolf suggested that I raised it on the mailing list as well.
See below...
Hello Rob,
Good to hear from you again.
On 9 Nov 2024, at 14:00, Rob Brewer ipfire-devel@grantura.co.uk wrote:
As a contributor to the SANS ISC, I email my IPTABLES logs hourly to the Dshield database, however the default IPFire IPTABLES logs are incomplete since packets dropped by Locationblock are not logged to syslog.
This is correct.
As a workaround I edit the locationblock function of rules.pl to add a IPTABLES rule to log the dropped packets. This edit does not of course survive any Core-Updates that upgrade rules.pl and the edit needs to be re-applied.
In addition, unless the Locationblock packets are logged to syslog the data displayed in the Firewall logs sections are incomplete as the packets dropped by Location Block are not displayed.
I understand that the original intension of the Location Block feature was to reduce the amount of log messages on installations running on extremely cheap flash storage. I suspect that these installations are now almost zero especially since IPFire no longer supports 32 bit versions.
The obvious solution to this to make Location Block logging selectable as a firewall option and my quick look at making a patches for this would seem to be fairly simple task to rules.pl and and optionsfw.cgi.
I would be happy to provide the necessary patches should this be acceptable.
I think a couple of things have changed since we introduced this feature and we have also learned a couple of lessons here:
- Yes it is true that this was introduced to keep the logs smaller,
but not necessarily to avoid using storage, mainly to keep them readable by a human. I was always a bit doubtful about this and believe that apart from this, there is no more value in a general location blocking feature. The firewall will drop those packets anyway.
IPFire does a very good job in protecting the internal network from unwanted probes from the WAN, and I agree that for most users, location block does not enhance this protection.
As I run a mail server, I use locationblock as part of my filtering to prevent bad actors from accessing my mail server and I think would be useful for anyone who are running servers on their networks.
I don't like blocking traffic, but I have witnessed over the last few years a substantial rise in abuse from certain countries from where I would never expect to receive emails and locationblock works very well for me in filtering out those unwanted connections.
I currently have 21 countries blocked including XD Hostile networks. All of these blocked countries have been seen to support abusive networks.
It would be possible to generate individual rules for these networks which in some cases I do with Banish, but it is sometimes easier just to drop the whole country.
Easy yes. I am just not sure what we are actually getting out of this.
I don’t like to become political on this list and so I won’t, but I know that some people hold the following beliefs:
- There are “bad” countries out there. In the western world those are
Russia, China, and more that fall into this kind of “category”. Nothing is going to stop them to open up an account on AWS and suddenly they have access to resources anywhere in the world and so actually, if those players were up to anything “bad”, they can come out of any IP space they want. There is also Tor which allows this.
I have been submitting my iptables logs to dshield for over 20 years and in the early days I would be sending a lot less than 1000 a day (this was with IPCop of course).
Over the last few years the number of firewall iptables logs have escalated until it had reached dshield's daily limit of 100,000 /day and I started looking at ways to limit the logging of some of the 'bad actors' firewall probes.
- Looking at actual data, the bullet proof ISPs that annoy us are not
that far away. They are hosted in Europe and very prominently so in the Netherlands - a “good” country.
I use Banish for this as it can block by ASN and can target bullet proof ISPs not only in NL but also in the USA and elsewhere. Using the 'location function' the blocked ASN's are updated when the database is updated.
This is just an example to explain that we all have our idea of the world in our head, but the Internet doesn’t know these borders. Geopolitics is complicated and I don’t think that this is a tool that helps to fight this kind of problem. I hope this is clear what I am trying to say.
Yes my approach is to use multiple tools and Locationblock is just one of the tools I am using. Maybe my approach is over the top though.
I personally trust in Spamhaus which is feeding data into our “Hostile Networks” blocker and they are looking at the actual traffic that they see from those subnets that they are blocking. They have a clear set of rules and they don’t look just at the country and say “ah yes, the bad guys again”.
For emails I also drop packets if listed in a number of DNS blackhole lists using Auto-Dnsbl which I adapted for IPFire. See:
https://people.ipfire.org/~helix/auto-dnsbl/README for info.
I have been using this for about a year and can confirm that it is very effective in removing spammers.
I have stated this recently in the RPZ discussion that I feel that we are lacking good sources for this kind of data. For blocking anything. Loads of people believe “more is more” and to be fair… once you get to blocking 90% of the internet you probably have blocked 90% of the noise. However this won’t do anything for targeted attacks. I personally would rather have more quality here.
I only use a few blocklists with ipblocklist and rely mainly on my locally derived data from failed smtp connections or from data from dshield's daily report.
- People seem to misunderstand this feature: The block as many
countries as they can click on and think that this makes things “more secure”, which of course it doesn’t.
I think this is education and I suspect that very few of these people are running servers on their system and need location block.
I am personally not here to educate people. I am happy to help out and give my opinion, but I suppose considering that people think as I have outlined above, who am I to educate them?
Sorry I didn't express my reply well, I agree that IPFire users should be responsible for their own actions.
It is just very regretful to me when people are ab-using a feature just to “feel more safe”.
- What I personally like as a feature is to apply location filters to
individual rules. That allows to apply rate limits in a few places and easily select some larger groups of potentially less interesting traffic.
I agree and is similar to what I am doing for ports 25 and 587 although with location block.
- Back when this was introduced there was also a lot less use of
botnets as we know them now. Pretty much all recent DoS attacks I remember have been coming from all over the world because systems where compromised wherever they were. This renders the filter absolutely useless.
For this, I am currently testing a dynamic block list on my system which adds an entry to an ipset when an ip address exceeds a threshold in the previous 24 hours. Currently there are about 7000 entries in this set which have been detected over the previous month. Not for everyone because it needs a Whitelist to prevent some 'push' technologies such as Facebook and to protect from bulk inputs such as mailing lists. I also blacklist a large number of ip addresses which are associated with port scanners.
This would work well as a botnet filter but it needs an understanding of the blocking process, so it would not be suitable for everyone.
We have talked about this on the last team call and came to the conclusion that the whole set of blocking features is a total mess. There are several places where things can be blocked, they all heavily overlap. Not to be going on about Spamhaus again, but the DROP list is in the “Hostile Networks” feature, there is a special country code for it, pretty much every IPS ruleset contains it, the IP blocklist feature supports it. Now there are people throwing this all into DNS as well. I believe this is all getting a little bit out of hand.
I don't use the IPS mainly due to bad experiences when trying to use Snort with IPCop. I find it difficult to set up and select the appropriate Rulsets for my requirements. I suspect the IPS would be very effective if I could get it set up correctly.
I personally would prefer *one* place and we really make this work well.
Some people find the IPS has too much overhead - I am beginning to disagree.
Some people add all sorts of blacklists that are clearly for experimental/research use and suddenly complain about things not working anymore. It is hard to figure out what is actually blocking traffic when there are about five places to look for. The result often is to disable the IPS, then something else, and after a while all of this is turned off and the firewall is configured to a bare minimum. We don’t want people to run it like that.
- Infrastructure is becoming more centralised. Some people attempt to
block the US and pretty much nothing works any more. Well…
:)
Therefore I am happy to rethink this feature - personally I would even remove it entirely, but I don’t like to handle the stress about this now.
No I think you should keep it. Locationblock is very useful, but needs to be used in a considered way. I could live without it but its availability makes my life easier. Also being updated on a weekly basis makes keeping track of abusers easier.
We issue a new location database once a day, but the IPFire systems update it only once a week. We deemed that as sufficient as there are not that many changes from one version to another one.
I think this should log now. I don’t think there is any legitimate reason to disable firewall logging at all. This is the paper trail that you want when debugging anything or doing forensics on an attack. Certainly bad flash is not a valid reason for me.
I have started generating some patches to make 'locationblock logging' as a selectable feature and could post them here in the next day or so if of any use.
What would this basically do? Log it as a simple rule for all, or do you include the selected country? I think the latter would make more sense.
OK - I have modified location-block.cgi to add an additional checkbox at the top of the page titled "Log dropped packets", very similar to ipblocklist.cgi. This adds a entry to the /var/ipfire/firewall/ locationblock settings file ON or OFF.
In /usr/lib/firewall/rules.pl I add an additional Iptables LOCATIONBLOCK log rule with the prefix 'LOCBLOCK-$location '. This is selectable from the Log dropped packets entry in the locationblock settings file.
This generates kernel messages such as:
LOCBLOCK-CN IN=ppp0 OUT=ppp0 MAC= SRC=114.139.42.71 DST=81.187.175.103 LEN=60 TOS=0x00 PREC=0x00 TTL=47 ID=34664 DF PROTO=TCP SPT=46837 DPT=23 WINDOW=29040 RES=0x00 SYN URGP=0
As you can see the prefix contains the country code of the dropped packets.
What is the rationale to turn this off? Why don’t we turn this on for everybody all of the time?
I would be happy with that I was suggesting that we use selectable logging to be compatible with earlier IPFire versions.
In fact I think that IPFire's logging environment, which was inherited from IPCop needs a re-visit.
Yes. But this is probably not a thing I would be looking at in IPFire 2.
I am also happy to think a little bit more outside of the box and see what we would fundamentally change in the future, but not implement in IPFire 2.
I was pleased to see that a while back there was a discussion of replacing sylogd with rsyslogd which I think would be a very useful upgrade and enable kernel messages, like on most current linux systems, to be logged to alternative log files instead of everything ending up in /var/log/ messages.
The currentl syslog server could do that. The problem is more that we don’t have good tools to view the logs on the web UI. So there is simply one place where most things go into and we will then filter it out when users use the web UI. This is tremendously slow.
I did look at this a while back and believe that with syslog all kernel messages must go to the same log file and cannot be filtered off as can be done with rsyslog. I use this facility in rsyslog.d on my server to filter off SYN Flood logs to a separate log file with ":msg,contains". I don't believe that syslog has this option.
The replacement with rsyslgd could also inhibit the "last message repeated x times" in syslog and which are ignored by IPFire's Log menu and causes erroneous results in the 'Count' values.
This might be a setting that we can turn off in the current implementation.
I understand that this is hard coded into syslogd and can't be changed. In rsyslog it defaults to "off".
https://www.rsyslog.com/doc/configuration/action/ rsconf1_repeatedmsgreduction.html
We could also think about moving all firewall hits into /var/log/firewall.log or something. That should free up enough in /var/log/messages to make debugging easier.
That would be a sensible option, currently I filter out firewall hits from the messages.log with a perl regex.
I would be happy to hear some more opinions on this from the list.
-Michael
I hope these comments are useful and would be pleased to help where I can.
Of course :)
-Michael
Rob
On Wed, 20 Nov 2024 23:08:39 -0000 (UTC), Rob Brewer wrote:
I've tested these patches on Core-Update 189 and can enable and disable locationblock logging. I can provide the patches in a different format if preferred.
Rob
OK - I have modified location-block.cgi to add an additional checkbox at the top of the page titled "Log dropped packets", very similar to ipblocklist.cgi. This adds a entry to the /var/ipfire/firewall/ locationblock settings file ON or OFF.
/srv/web/ipfire/cgi-bin/location-block.cgi ================================================================= 62a63,69
# Check if we want to disable logging. if (exists $cgiparams{'LOGGING_ENABLED'}) { $settings{'LOGGING_ENABLED'} = "on"; } else { $settings{'LOGGING_ENABLED'} = "off"; }
93a101
my $logging;
97a106,111
if ($settings{'LOGGING_ENABLED'} eq "on") { $logging = "checked='checked'"; }
108a123,125
<td width='50%'
class='base'>$Lang::tr{'ipblocklist log'}
<td><input type='checkbox'
name='LOGGING_ENABLED' $logging></td>
</tr>
==========================================================================
In /usr/lib/firewall/rules.pl I add an additional Iptables LOCATIONBLOCK log rule with the prefix 'LOCBLOCK-$location '. This is selectable from the Log dropped packets entry in the locationblock settings file.
/usr/lib/firewall/rules.pl ===================================================================== 76c76,77 < "LOCATIONBLOCK_ENABLED" => "off" ---
"LOCATIONBLOCK_ENABLED" => "off", "LOGGING_ENABLED" => "off",
719a721,725
# add loggging if enabled from location-block.cgi if ($locationsettings{'LOGGING_ENABLED'} eq "on") { run("$IPTABLES -A LOCATIONBLOCK -m set --match-set
$location src -j LOG --log-prefix 'LOCBLOCK-$location '");
}
=========================================================================
On Wed, 20 Nov 2024 23:08:39 -0000 (UTC), Rob Brewer wrote:
Sorry I got caught out by my editor's word wrap. I've now turned it off so hopefully this will be OK.
Rob
OK - I have modified location-block.cgi to add an additional checkbox at the top of the page titled "Log dropped packets", very similar to ipblocklist.cgi. This adds a entry to the /var/ipfire/firewall/ locationblock settings file ON or OFF.
/srv/web/ipfire/cgi-bin/location-block.cgi ========================================================= 62a63,69
# Check if we want to disable logging. if (exists $cgiparams{'LOGGING_ENABLED'}) { $settings{'LOGGING_ENABLED'} = "on"; } else { $settings{'LOGGING_ENABLED'} = "off"; }
93a101
my $logging;
97a106,111
if ($settings{'LOGGING_ENABLED'} eq "on") { $logging = "checked='checked'"; }
108a123,125
<td width='50%' class='base'>$Lang::tr{'ipblocklist log'} <td><input type='checkbox' name='LOGGING_ENABLED' $logging></td> </tr>
====================================================================
In /usr/lib/firewall/rules.pl I add an additional Iptables LOCATIONBLOCK log rule with the prefix 'LOCBLOCK-$location '. This is selectable from the Log dropped packets entry in the locationblock settings file.
/usr/lib/firewall/rules.pl ===================================================================== 76c76,77 < "LOCATIONBLOCK_ENABLED" => "off" ---
"LOCATIONBLOCK_ENABLED" => "off", "LOGGING_ENABLED" => "off",
719a721,725
# add loggging if enabled from location-block.cgi if ($locationsettings{'LOGGING_ENABLED'} eq "on") { run("$IPTABLES -A LOCATIONBLOCK -m set --match-set $location src -j LOG --log-prefix 'LOCBLOCK-$location '"); }
======================================================================= End of encapsulated message
Hello,
On 21 Nov 2024, at 00:08, Rob Brewer ipfire-devel@grantura.co.uk wrote:
Hi Michael and Adolf,
I'll combine my reply to both of you.
On Wed, 20 Nov 2024 15:21:28 +0000, Michael Tremer wrote:
Hello,
On 20 Nov 2024, at 13:36, Rob Brewer ipfire-devel@grantura.co.uk wrote:
On Tue, 19 Nov 2024 17:10:24 +0000, Michael Tremer wrote:
Hi Michael, thank for your response, I was beginning to think there was little interest in this topic.
I was traveling and so have a little bit of a back log as usual :)
I can’t speak for the others, though.
Not a problem, I should have said that I previously raised this a topic on the forum and Abolf suggested that I raised it on the mailing list as well.
Yes, I really prefer this here, because the forum is very overwhelming sometimes. There is too many threads, too many people… Mailing lists are great for development.
See below...
Hello Rob,
Good to hear from you again.
On 9 Nov 2024, at 14:00, Rob Brewer ipfire-devel@grantura.co.uk wrote:
As a contributor to the SANS ISC, I email my IPTABLES logs hourly to the Dshield database, however the default IPFire IPTABLES logs are incomplete since packets dropped by Locationblock are not logged to syslog.
This is correct.
As a workaround I edit the locationblock function of rules.pl to add a IPTABLES rule to log the dropped packets. This edit does not of course survive any Core-Updates that upgrade rules.pl and the edit needs to be re-applied.
In addition, unless the Locationblock packets are logged to syslog the data displayed in the Firewall logs sections are incomplete as the packets dropped by Location Block are not displayed.
I understand that the original intension of the Location Block feature was to reduce the amount of log messages on installations running on extremely cheap flash storage. I suspect that these installations are now almost zero especially since IPFire no longer supports 32 bit versions.
The obvious solution to this to make Location Block logging selectable as a firewall option and my quick look at making a patches for this would seem to be fairly simple task to rules.pl and and optionsfw.cgi.
I would be happy to provide the necessary patches should this be acceptable.
I think a couple of things have changed since we introduced this feature and we have also learned a couple of lessons here:
- Yes it is true that this was introduced to keep the logs smaller,
but not necessarily to avoid using storage, mainly to keep them readable by a human. I was always a bit doubtful about this and believe that apart from this, there is no more value in a general location blocking feature. The firewall will drop those packets anyway.
IPFire does a very good job in protecting the internal network from unwanted probes from the WAN, and I agree that for most users, location block does not enhance this protection.
As I run a mail server, I use locationblock as part of my filtering to prevent bad actors from accessing my mail server and I think would be useful for anyone who are running servers on their networks.
I don't like blocking traffic, but I have witnessed over the last few years a substantial rise in abuse from certain countries from where I would never expect to receive emails and locationblock works very well for me in filtering out those unwanted connections.
I currently have 21 countries blocked including XD Hostile networks. All of these blocked countries have been seen to support abusive networks.
It would be possible to generate individual rules for these networks which in some cases I do with Banish, but it is sometimes easier just to drop the whole country.
Easy yes. I am just not sure what we are actually getting out of this.
I don’t like to become political on this list and so I won’t, but I know that some people hold the following beliefs:
- There are “bad” countries out there. In the western world those are
Russia, China, and more that fall into this kind of “category”. Nothing is going to stop them to open up an account on AWS and suddenly they have access to resources anywhere in the world and so actually, if those players were up to anything “bad”, they can come out of any IP space they want. There is also Tor which allows this.
I have been submitting my iptables logs to dshield for over 20 years and in the early days I would be sending a lot less than 1000 a day (this was with IPCop of course).
Over the last few years the number of firewall iptables logs have escalated until it had reached dshield's daily limit of 100,000 /day and I started looking at ways to limit the logging of some of the 'bad actors' firewall probes.
What do you use to send them the logs? The Python script or one of those alternatives?
- Looking at actual data, the bullet proof ISPs that annoy us are not
that far away. They are hosted in Europe and very prominently so in the Netherlands - a “good” country.
I use Banish for this as it can block by ASN and can target bullet proof ISPs not only in NL but also in the USA and elsewhere. Using the 'location function' the blocked ASN's are updated when the database is updated.
This is just an example to explain that we all have our idea of the world in our head, but the Internet doesn’t know these borders. Geopolitics is complicated and I don’t think that this is a tool that helps to fight this kind of problem. I hope this is clear what I am trying to say.
Yes my approach is to use multiple tools and Locationblock is just one of the tools I am using. Maybe my approach is over the top though.
Probably not massively, but maybe slightly.
Wouldn’t it be nicer to have everything in one place instead of having multiple tools that work together in concert?
I personally trust in Spamhaus which is feeding data into our “Hostile Networks” blocker and they are looking at the actual traffic that they see from those subnets that they are blocking. They have a clear set of rules and they don’t look just at the country and say “ah yes, the bad guys again”.
For emails I also drop packets if listed in a number of DNS blackhole lists using Auto-Dnsbl which I adapted for IPFire. See:
https://people.ipfire.org/~helix/auto-dnsbl/README for info.
I have been using this for about a year and can confirm that it is very effective in removing spammers.
I have stated this recently in the RPZ discussion that I feel that we are lacking good sources for this kind of data. For blocking anything. Loads of people believe “more is more” and to be fair… once you get to blocking 90% of the internet you probably have blocked 90% of the noise. However this won’t do anything for targeted attacks. I personally would rather have more quality here.
I only use a few blocklists with ipblocklist and rely mainly on my locally derived data from failed smtp connections or from data from dshield's daily report.
- People seem to misunderstand this feature: The block as many
countries as they can click on and think that this makes things “more secure”, which of course it doesn’t.
I think this is education and I suspect that very few of these people are running servers on their system and need location block.
I am personally not here to educate people. I am happy to help out and give my opinion, but I suppose considering that people think as I have outlined above, who am I to educate them?
Sorry I didn't express my reply well, I agree that IPFire users should be responsible for their own actions.
Naturally. But we should also not give them sharp knives when we don’t expect them to use them properly.
It is just very regretful to me when people are ab-using a feature just to “feel more safe”.
- What I personally like as a feature is to apply location filters to
individual rules. That allows to apply rate limits in a few places and easily select some larger groups of potentially less interesting traffic.
I agree and is similar to what I am doing for ports 25 and 587 although with location block.
- Back when this was introduced there was also a lot less use of
botnets as we know them now. Pretty much all recent DoS attacks I remember have been coming from all over the world because systems where compromised wherever they were. This renders the filter absolutely useless.
For this, I am currently testing a dynamic block list on my system which adds an entry to an ipset when an ip address exceeds a threshold in the previous 24 hours. Currently there are about 7000 entries in this set which have been detected over the previous month. Not for everyone because it needs a Whitelist to prevent some 'push' technologies such as Facebook and to protect from bulk inputs such as mailing lists. I also blacklist a large number of ip addresses which are associated with port scanners.
This would work well as a botnet filter but it needs an understanding of the blocking process, so it would not be suitable for everyone.
We have talked about this on the last team call and came to the conclusion that the whole set of blocking features is a total mess. There are several places where things can be blocked, they all heavily overlap. Not to be going on about Spamhaus again, but the DROP list is in the “Hostile Networks” feature, there is a special country code for it, pretty much every IPS ruleset contains it, the IP blocklist feature supports it. Now there are people throwing this all into DNS as well. I believe this is all getting a little bit out of hand.
I don't use the IPS mainly due to bad experiences when trying to use Snort with IPCop. I find it difficult to set up and select the appropriate Rulsets for my requirements. I suspect the IPS would be very effective if I could get it set up correctly.
This has massively improved in recent times. And it is FAST!
I personally would prefer *one* place and we really make this work well.
Some people find the IPS has too much overhead - I am beginning to disagree.
Some people add all sorts of blacklists that are clearly for experimental/research use and suddenly complain about things not working anymore. It is hard to figure out what is actually blocking traffic when there are about five places to look for. The result often is to disable the IPS, then something else, and after a while all of this is turned off and the firewall is configured to a bare minimum. We don’t want people to run it like that.
- Infrastructure is becoming more centralised. Some people attempt to
block the US and pretty much nothing works any more. Well…
:)
Therefore I am happy to rethink this feature - personally I would even remove it entirely, but I don’t like to handle the stress about this now.
No I think you should keep it. Locationblock is very useful, but needs to be used in a considered way. I could live without it but its availability makes my life easier. Also being updated on a weekly basis makes keeping track of abusers easier.
We issue a new location database once a day, but the IPFire systems update it only once a week. We deemed that as sufficient as there are not that many changes from one version to another one.
I think this should log now. I don’t think there is any legitimate reason to disable firewall logging at all. This is the paper trail that you want when debugging anything or doing forensics on an attack. Certainly bad flash is not a valid reason for me.
I have started generating some patches to make 'locationblock logging' as a selectable feature and could post them here in the next day or so if of any use.
What would this basically do? Log it as a simple rule for all, or do you include the selected country? I think the latter would make more sense.
OK - I have modified location-block.cgi to add an additional checkbox at the top of the page titled "Log dropped packets", very similar to ipblocklist.cgi. This adds a entry to the /var/ipfire/firewall/ locationblock settings file ON or OFF.
In /usr/lib/firewall/rules.pl I add an additional Iptables LOCATIONBLOCK log rule with the prefix 'LOCBLOCK-$location '. This is selectable from the Log dropped packets entry in the locationblock settings file.
This generates kernel messages such as:
LOCBLOCK-CN IN=ppp0 OUT=ppp0 MAC= SRC=114.139.42.71 DST=81.187.175.103 LEN=60 TOS=0x00 PREC=0x00 TTL=47 ID=34664 DF PROTO=TCP SPT=46837 DPT=23 WINDOW=29040 RES=0x00 SYN URGP=0
As you can see the prefix contains the country code of the dropped packets.
Cool. So I think coding-wise this is pretty straight forward.
The more crucial question for me is below…
What is the rationale to turn this off? Why don’t we turn this on for everybody all of the time?
I would be happy with that I was suggesting that we use selectable logging to be compatible with earlier IPFire versions.
What do we have to be compatible with? Just IPFire’s behaviour from last year?
I would be happy to break that if we have a good reason, because the world sometimes moves on. What we have released 10 years ago probably doesn’t have good defaults any more. Sometimes we have to revisit those and sometimes we even have to be pushy with some changes if we want some long-standing users to adopt them. IPFire users usually have no reason to do a reinstall and we don’t want to leave the people behind who have been running IPFire perfectly well for the last 10 years or even longer. So at least I would like to discuss what we push a little harder.
In fact I think that IPFire's logging environment, which was inherited from IPCop needs a re-visit.
Yes. But this is probably not a thing I would be looking at in IPFire 2.
I am also happy to think a little bit more outside of the box and see what we would fundamentally change in the future, but not implement in IPFire 2.
I was pleased to see that a while back there was a discussion of replacing sylogd with rsyslogd which I think would be a very useful upgrade and enable kernel messages, like on most current linux systems, to be logged to alternative log files instead of everything ending up in /var/log/ messages.
The currentl syslog server could do that. The problem is more that we don’t have good tools to view the logs on the web UI. So there is simply one place where most things go into and we will then filter it out when users use the web UI. This is tremendously slow.
I did look at this a while back and believe that with syslog all kernel messages must go to the same log file and cannot be filtered off as can be done with rsyslog. I use this facility in rsyslog.d on my server to filter off SYN Flood logs to a separate log file with ":msg,contains". I don't believe that syslog has this option.
Different messages could be sent to different “facilities” and then we can filter them into individual files.
I would not want to split firewall hits into different files though. I just wouldn’t mind if firewall hits were no longer in /var/log/messages.
The replacement with rsyslgd could also inhibit the "last message repeated x times" in syslog and which are ignored by IPFire's Log menu and causes erroneous results in the 'Count' values.
This might be a setting that we can turn off in the current implementation.
I understand that this is hard coded into syslogd and can't be changed. In rsyslog it defaults to "off".
https://www.rsyslog.com/doc/configuration/action/ rsconf1_repeatedmsgreduction.html
What impact does this have for you? I never noticed any problem by summarising the logs.
We could also think about moving all firewall hits into /var/log/firewall.log or something. That should free up enough in /var/log/messages to make debugging easier.
That would be a sensible option, currently I filter out firewall hits from the messages.log with a perl regex.
I would be happy to hear some more opinions on this from the list.
-Michael
I hope these comments are useful and would be pleased to help where I can.
Of course :)
-Michael
Rob
-Michael
On Sat, 23 Nov 2024 10:26:07 +0100, Michael Tremer wrote:
Hello,
On 21 Nov 2024, at 00:08, Rob Brewer ipfire-devel@grantura.co.uk
wrote:
Hi Michael and Adolf,
I'll combine my reply to both of you.
On Wed, 20 Nov 2024 15:21:28 +0000, Michael Tremer wrote:
Hello,
On 20 Nov 2024, at 13:36, Rob Brewer ipfire-devel@grantura.co.uk wrote:
On Tue, 19 Nov 2024 17:10:24 +0000, Michael Tremer wrote:
Hi Michael, thank for your response, I was beginning to think there
was
little interest in this topic.
I was traveling and so have a little bit of a back log as usual :)
I can’t speak for the others, though.
Not a problem, I should have said that I previously raised this a
topic on
the forum and Adolf suggested that I raised it on the mailing list as well.
Yes, I really prefer this here, because the forum is very overwhelming
sometimes. There is too many threads, too many people… Mailing lists are great for development.
That's fine, one query though you don't seem to snip older parts of the post, is this your preference?
See below...
Hello Rob,
Good to hear from you again.
On 9 Nov 2024, at 14:00, Rob Brewer ipfire-devel@grantura.co.uk wrote:
As a contributor to the SANS ISC, I email my IPTABLES logs hourly to the Dshield database, however the default IPFire IPTABLES logs
are
incomplete since packets dropped by Locationblock are not logged to syslog.
This is correct.
As a workaround I edit the locationblock function of rules.pl to
add
a IPTABLES rule to log the dropped packets. This edit does not of course survive any Core-Updates that upgrade rules.pl and the edit needs to be re-applied.
In addition, unless the Locationblock packets are logged to syslog the data displayed in the Firewall logs sections are incomplete as the packets dropped by Location Block are not displayed.
I understand that the original intension of the Location Block feature was to reduce the amount of log messages on installations running on extremely cheap flash storage. I suspect that these installations are now almost zero especially since IPFire no longer supports 32 bit versions.
The obvious solution to this to make Location Block logging selectable as a firewall option and my quick look at making a
patches
for this would seem to be fairly simple task to rules.pl and and optionsfw.cgi.
I would be happy to provide the necessary patches should this be acceptable.
I think a couple of things have changed since we introduced this feature and we have also learned a couple of lessons here:
- Yes it is true that this was introduced to keep the logs smaller,
but not necessarily to avoid using storage, mainly to keep them readable by a human. I was always a bit doubtful about this and believe that apart from this, there is no more value in a general location blocking feature. The firewall will drop those packets anyway.
IPFire does a very good job in protecting the internal network from unwanted probes from the WAN, and I agree that for most users,
location
block does not enhance this protection.
As I run a mail server, I use locationblock as part of my filtering
to
prevent bad actors from accessing my mail server and I think would be useful for anyone who are running servers on their networks.
I don't like blocking traffic, but I have witnessed over the last few years a substantial rise in abuse from certain countries from where I would never expect to receive emails and locationblock works very
well
for me in filtering out those unwanted connections.
I currently have 21 countries blocked including XD Hostile networks. All of these blocked countries have been seen to support abusive networks.
It would be possible to generate individual rules for these networks which in some cases I do with Banish, but it is sometimes easier just to drop the whole country.
Easy yes. I am just not sure what we are actually getting out of this.
I don’t like to become political on this list and so I won’t, but I
know
that some people hold the following beliefs:
- There are “bad” countries out there. In the western world those are
Russia, China, and more that fall into this kind of “category”.
Nothing
is going to stop them to open up an account on AWS and suddenly they have access to resources anywhere in the world and so actually, if
those
players were up to anything “bad”, they can come out of any IP space they want. There is also Tor which allows this.
I have been submitting my iptables logs to dshield for over 20 years
and
in the early days I would be sending a lot less than 1000 a day (this
was
with IPCop of course).
Over the last few years the number of firewall iptables logs have escalated until it had reached dshield's daily limit of 100,000 /day
and
I started looking at ways to limit the logging of some of the 'bad
actors'
firewall probes.
What do you use to send them the logs? The Python script or one of
those alternatives?
I use a modified Perl script (iptables.pl) which is about 20 years old was originally written for Smoothwall and Ipcop1. This was part of the Logsend add-on and I modified for IPCop2 about 15 years ago.
Now modified for IPFire and using DMA email instead of the original sendEmail. I've uploaded my version to people.ipfire.org/~helix/dshield.
I had intended to write an install/uninstall script but is in the todo list. Do you think it may be of interest to anyone if I do?
- Looking at actual data, the bullet proof ISPs that annoy us are not
that far away. They are hosted in Europe and very prominently so in
the
Netherlands - a “good” country.
I use Banish for this as it can block by ASN and can target bullet
proof
ISPs not only in NL but also in the USA and elsewhere. Using the
'location
function' the blocked ASN's are updated when the database is updated.
This is just an example to explain that we all have our idea of the world in our head, but the Internet doesn’t know these borders. Geopolitics is complicated and I don’t think that this is a tool that helps to fight this kind of problem. I hope this is clear what I am trying to say.
Yes my approach is to use multiple tools and Locationblock is just one
of
the tools I am using. Maybe my approach is over the top though.
Probably not massively, but maybe slightly.
Wouldn’t it be nicer to have everything in one place instead of having
multiple tools that work together in concert?
I think that would be a lot of work, would you consider that for IPFire2 or 3?
I personally trust in Spamhaus which is feeding data into our “Hostile Networks” blocker and they are looking at the actual traffic that they see from those subnets that they are blocking. They have a clear set
of
rules and they don’t look just at the country and say “ah yes, the bad guys again”.
For emails I also drop packets if listed in a number of DNS blackhole lists using Auto-Dnsbl which I adapted for IPFire. See:
https://people.ipfire.org/~helix/auto-dnsbl/README for info.
I have been using this for about a year and can confirm that it is
very
effective in removing spammers.
I have stated this recently in the RPZ discussion that I feel that we are lacking good sources for this kind of data. For blocking anything. Loads of people believe “more is more” and to be fair… once you get to blocking 90% of the internet you probably have blocked 90% of the
noise.
However this won’t do anything for targeted attacks. I personally
would
rather have more quality here.
I only use a few blocklists with ipblocklist and rely mainly on my
locally
derived data from failed smtp connections or from data from dshield's daily report.
- People seem to misunderstand this feature: The block as many
countries as they can click on and think that this makes things
“more
secure”, which of course it doesn’t.
I think this is education and I suspect that very few of these people are running servers on their system and need location block.
I am personally not here to educate people. I am happy to help out and give my opinion, but I suppose considering that people think as I have outlined above, who am I to educate them?
Sorry I didn't express my reply well, I agree that IPFire users should
be
responsible for their own actions.
Naturally. But we should also not give them sharp knives when we don’t
expect them to use them properly.
It is just very regretful to me when people are ab-using a feature
just
to “feel more safe”.
- What I personally like as a feature is to apply location filters
to
individual rules. That allows to apply rate limits in a few places
and
easily select some larger groups of potentially less interesting traffic.
I agree and is similar to what I am doing for ports 25 and 587
although
with location block.
- Back when this was introduced there was also a lot less use of
botnets as we know them now. Pretty much all recent DoS attacks I remember have been coming from all over the world because systems where compromised wherever they were. This renders the filter absolutely useless.
For this, I am currently testing a dynamic block list on my system which adds an entry to an ipset when an ip address exceeds a
threshold
in the previous 24 hours. Currently there are about 7000 entries in this set which have been detected over the previous month. Not for everyone because it needs a Whitelist to prevent some 'push' technologies such as Facebook and to protect from bulk inputs such as mailing lists. I also blacklist a large number of ip addresses which are associated with port scanners.
This would work well as a botnet filter but it needs an understanding of the blocking process, so it would not be suitable for everyone.
We have talked about this on the last team call and came to the conclusion that the whole set of blocking features is a total mess. There are several places where things can be blocked, they all heavily overlap. Not to be going on about Spamhaus again, but the DROP list is in the “Hostile Networks” feature, there is a special country code for it, pretty much every IPS ruleset contains it, the IP blocklist
feature
supports it. Now there are people throwing this all into DNS as well.
I
believe this is all getting a little bit out of hand.
I don't use the IPS mainly due to bad experiences when trying to use
Snort
with IPCop. I find it difficult to set up and select the appropriate Rulsets for my requirements. I suspect the IPS would be very effective
if
I could get it set up correctly.
This has massively improved in recent times. And it is FAST!
I'll have another look, but my current set-up is working well, so I'm reluctant to change.
I personally would prefer *one* place and we really make this work
well.
Some people find the IPS has too much overhead - I am beginning to disagree.
Some people add all sorts of blacklists that are clearly for experimental/research use and suddenly complain about things not
working
anymore. It is hard to figure out what is actually blocking traffic
when
there are about five places to look for. The result often is to
disable
the IPS, then something else, and after a while all of this is turned off and the firewall is configured to a bare minimum. We don’t want people to run it like that.
- Infrastructure is becoming more centralised. Some people attempt
to
block the US and pretty much nothing works any more. Well…
:)
Therefore I am happy to rethink this feature - personally I would
even
remove it entirely, but I don’t like to handle the stress about this now.
No I think you should keep it. Locationblock is very useful, but
needs
to be used in a considered way. I could live without it but its availability makes my life easier. Also being updated on a weekly
basis
makes keeping track of abusers easier.
We issue a new location database once a day, but the IPFire systems update it only once a week. We deemed that as sufficient as there are not that many changes from one version to another one.
I think this should log now. I don’t think there is any legitimate reason to disable firewall logging at all. This is the paper trail that you want when debugging anything or doing forensics on an
attack.
Certainly bad flash is not a valid reason for me.
I have started generating some patches to make 'locationblock
logging'
as a selectable feature and could post them here in the next day or
so
if of any use.
What would this basically do? Log it as a simple rule for all, or do
you
include the selected country? I think the latter would make more
sense.
OK - I have modified location-block.cgi to add an additional checkbox
at
the top of the page titled "Log dropped packets", very similar to ipblocklist.cgi. This adds a entry to the /var/ipfire/firewall/ locationblock settings file ON or OFF.
In /usr/lib/firewall/rules.pl I add an additional Iptables
LOCATIONBLOCK
log rule with the prefix 'LOCBLOCK-$location '. This is selectable
from
the Log dropped packets entry in the locationblock settings file.
This generates kernel messages such as:
LOCBLOCK-CN IN=ppp0 OUT=ppp0 MAC= SRC=114.139.42.71 DST=81.187.175.103 LEN=60 TOS=0x00 PREC=0x00 TTL=47 ID=34664 DF PROTO=TCP SPT=46837
DPT=23
WINDOW=29040 RES=0x00 SYN URGP=0
As you can see the prefix contains the country code of the dropped packets.
Cool. So I think coding-wise this is pretty straight forward.
The more crucial question for me is below…
Do have a look at the patches I posted and let me have any comments. If it's not needed and we can add locationblock logging without selection then that's what I have been doing since LB was first implemented.
What is the rationale to turn this off? Why don’t we turn this on for everybody all of the time?
I would be happy with that I was suggesting that we use selectable
logging
to be compatible with earlier IPFire versions.
What do we have to be compatible with? Just IPFire’s behaviour from
last year?
I would be happy to break that if we have a good reason, because the
world sometimes moves on. What we have released 10 years ago probably doesn’t have good defaults any more. Sometimes we have to revisit those and sometimes we even have to be pushy with some changes if we want some long-standing users to adopt them. IPFire users usually have no reason to do a reinstall and we don’t want to leave the people behind who have been running IPFire perfectly well for the last 10 years or even longer. So at least I would like to discuss what we push a little harder.
As Adolf pointed out in the Forum,the WIKI stated the rational for not logging was to protect flash memory from excessive use. I don't think that is appropriate now.
In fact I think that IPFire's logging environment, which was
inherited
from IPCop needs a re-visit.
Yes. But this is probably not a thing I would be looking at in IPFire
2.
I am also happy to think a little bit more outside of the box and see what we would fundamentally change in the future, but not implement in IPFire 2.
I was pleased to see that a while back there was a discussion of replacing sylogd with rsyslogd which I think would be a very useful upgrade and enable kernel messages, like on most current linux
systems,
to be logged to alternative log files instead of everything ending up in /var/log/ messages.
The currentl syslog server could do that. The problem is more that we don’t have good tools to view the logs on the web UI. So there is
simply
one place where most things go into and we will then filter it out
when
users use the web UI. This is tremendously slow.
I did look at this a while back and believe that with syslog all
kernel
messages must go to the same log file and cannot be filtered off as
can be
done with rsyslog. I use this facility in rsyslog.d on my server to
filter
off SYN Flood logs to a separate log file with ":msg,contains". I
don't
believe that syslog has this option.
Different messages could be sent to different “facilities” and then we
can filter them into individual files.
I would not want to split firewall hits into different files though. I
just wouldn’t mind if firewall hits were no longer in /var/log/messages.
You need to be careful with the LOG* menus which would need to look in 2 or more places for historical logs and so the cgi's would need to be modified.
I would however welcome kernel message filtering since it would allow my use of a separate log directory for my auto-dnsbl program to monitor for non-dropped incoming SMTP connections without appearing in 'messages' log file.
The replacement with rsyslgd could also inhibit the "last message repeated x times" in syslog and which are ignored by IPFire's Log
menu
and causes erroneous results in the 'Count' values.
This might be a setting that we can turn off in the current implementation.
I understand that this is hard coded into syslogd and can't be
changed. In
rsyslog it defaults to "off".
https://www.rsyslog.com/doc/configuration/action/ rsconf1_repeatedmsgreduction.html
What impact does this have for you? I never noticed any problem by
summarising the logs.
I came across this and it illustrates the point well:
https://dcid.me/notes/syslog-last-message-repeated-x-times-rant
and makes the point that kernel messages can be delayed by several seconds and could affect any DOS filtering that may be in use.
The dshield iptables.pl script tries to recover the lost repeated messages but looses the time-stamp.
When comparing the dshield logs with IPFire's logs there is invariably a discrepancy with the IPFire's count sometimes considerably less especially with multiple port-scans over a minute or so.
Any chance of looking again at rsyslog? IPFire is the only linux system I know of still using syslog.
I seem to remember that Adolf had completed all the necessary changes to introduce rsyslog a years or so ago but was not implemented,
We could also think about moving all firewall hits into /var/log/firewall.log or something. That should free up enough in /var/log/messages to make debugging easier.
That would be a sensible option, currently I filter out firewall hits
from
the messages.log with a perl regex.
I would be happy to hear some more opinions on this from the list.
-Michael
I hope these comments are useful and would be pleased to help where I can.
Of course :)
-Michael
Rob
-Michael
Rob
Hi Rob,
On 23/11/2024 15:36, Rob Brewer wrote:
On Sat, 23 Nov 2024 10:26:07 +0100, Michael Tremer wrote:
Hello,
On 21 Nov 2024, at 00:08, Rob Brewer ipfire-devel@grantura.co.uk
wrote:
Hi Michael and Adolf,
I'll combine my reply to both of you.
On Wed, 20 Nov 2024 15:21:28 +0000, Michael Tremer wrote:
Hello,
On 20 Nov 2024, at 13:36, Rob Brewer ipfire-devel@grantura.co.uk wrote:
On Tue, 19 Nov 2024 17:10:24 +0000, Michael Tremer wrote:
Hi Michael, thank for your response, I was beginning to think there
was
little interest in this topic.
I was traveling and so have a little bit of a back log as usual :)
I can’t speak for the others, though.
Not a problem, I should have said that I previously raised this a
topic on
the forum and Adolf suggested that I raised it on the mailing list as well.
Yes, I really prefer this here, because the forum is very overwhelming
sometimes. There is too many threads, too many people… Mailing lists are great for development.
That's fine, one query though you don't seem to snip older parts of the post, is this your preference?
See below...
Hello Rob,
Good to hear from you again.
> On 9 Nov 2024, at 14:00, Rob Brewer ipfire-devel@grantura.co.uk > wrote: > > As a contributor to the SANS ISC, I email my IPTABLES logs hourly > to the Dshield database, however the default IPFire IPTABLES logs
are
> incomplete since packets dropped by Locationblock are not logged to > syslog. This is correct.
> As a workaround I edit the locationblock function of rules.pl to
add
> a IPTABLES rule to log the dropped packets. This edit does not of > course survive any Core-Updates that upgrade rules.pl and the edit > needs to be re-applied. > > In addition, unless the Locationblock packets are logged to syslog > the data displayed in the Firewall logs sections are incomplete as > the packets dropped by Location Block are not displayed. > > I understand that the original intension of the Location Block > feature was to reduce the amount of log messages on installations > running on extremely cheap flash storage. I suspect that these > installations are now almost zero especially since IPFire no longer > supports 32 bit versions. > > The obvious solution to this to make Location Block logging > selectable as a firewall option and my quick look at making a
patches
> for this would seem to be fairly simple task to rules.pl and and > optionsfw.cgi. > > I would be happy to provide the necessary patches should this be > acceptable. I think a couple of things have changed since we introduced this feature and we have also learned a couple of lessons here:
- Yes it is true that this was introduced to keep the logs smaller,
but not necessarily to avoid using storage, mainly to keep them readable by a human. I was always a bit doubtful about this and believe that apart from this, there is no more value in a general location blocking feature. The firewall will drop those packets anyway.
IPFire does a very good job in protecting the internal network from unwanted probes from the WAN, and I agree that for most users,
location
block does not enhance this protection.
As I run a mail server, I use locationblock as part of my filtering
to
prevent bad actors from accessing my mail server and I think would be useful for anyone who are running servers on their networks.
I don't like blocking traffic, but I have witnessed over the last few years a substantial rise in abuse from certain countries from where I would never expect to receive emails and locationblock works very
well
for me in filtering out those unwanted connections.
I currently have 21 countries blocked including XD Hostile networks. All of these blocked countries have been seen to support abusive networks.
It would be possible to generate individual rules for these networks which in some cases I do with Banish, but it is sometimes easier just to drop the whole country.
Easy yes. I am just not sure what we are actually getting out of this.
I don’t like to become political on this list and so I won’t, but I
know
that some people hold the following beliefs:
- There are “bad” countries out there. In the western world those are
Russia, China, and more that fall into this kind of “category”.
Nothing
is going to stop them to open up an account on AWS and suddenly they have access to resources anywhere in the world and so actually, if
those
players were up to anything “bad”, they can come out of any IP space they want. There is also Tor which allows this.
I have been submitting my iptables logs to dshield for over 20 years
and
in the early days I would be sending a lot less than 1000 a day (this
was
with IPCop of course).
Over the last few years the number of firewall iptables logs have escalated until it had reached dshield's daily limit of 100,000 /day
and
I started looking at ways to limit the logging of some of the 'bad
actors'
firewall probes.
What do you use to send them the logs? The Python script or one of
those alternatives?
I use a modified Perl script (iptables.pl) which is about 20 years old was originally written for Smoothwall and Ipcop1. This was part of the Logsend add-on and I modified for IPCop2 about 15 years ago.
Now modified for IPFire and using DMA email instead of the original sendEmail. I've uploaded my version to people.ipfire.org/~helix/dshield.
I had intended to write an install/uninstall script but is in the todo list. Do you think it may be of interest to anyone if I do?
- Looking at actual data, the bullet proof ISPs that annoy us are not
that far away. They are hosted in Europe and very prominently so in
the
Netherlands - a “good” country.
I use Banish for this as it can block by ASN and can target bullet
proof
ISPs not only in NL but also in the USA and elsewhere. Using the
'location
function' the blocked ASN's are updated when the database is updated.
This is just an example to explain that we all have our idea of the world in our head, but the Internet doesn’t know these borders. Geopolitics is complicated and I don’t think that this is a tool that helps to fight this kind of problem. I hope this is clear what I am trying to say.
Yes my approach is to use multiple tools and Locationblock is just one
of
the tools I am using. Maybe my approach is over the top though.
Probably not massively, but maybe slightly.
Wouldn’t it be nicer to have everything in one place instead of having
multiple tools that work together in concert? I think that would be a lot of work, would you consider that for IPFire2 or 3?
I personally trust in Spamhaus which is feeding data into our “Hostile Networks” blocker and they are looking at the actual traffic that they see from those subnets that they are blocking. They have a clear set
of
rules and they don’t look just at the country and say “ah yes, the bad guys again”.
For emails I also drop packets if listed in a number of DNS blackhole lists using Auto-Dnsbl which I adapted for IPFire. See:
https://people.ipfire.org/~helix/auto-dnsbl/README for info.
I have been using this for about a year and can confirm that it is
very
effective in removing spammers.
I have stated this recently in the RPZ discussion that I feel that we are lacking good sources for this kind of data. For blocking anything. Loads of people believe “more is more” and to be fair… once you get to blocking 90% of the internet you probably have blocked 90% of the
noise.
However this won’t do anything for targeted attacks. I personally
would
rather have more quality here.
I only use a few blocklists with ipblocklist and rely mainly on my
locally
derived data from failed smtp connections or from data from dshield's daily report.
- People seem to misunderstand this feature: The block as many
countries as they can click on and think that this makes things
“more
secure”, which of course it doesn’t.
I think this is education and I suspect that very few of these people are running servers on their system and need location block.
I am personally not here to educate people. I am happy to help out and give my opinion, but I suppose considering that people think as I have outlined above, who am I to educate them?
Sorry I didn't express my reply well, I agree that IPFire users should
be
responsible for their own actions.
Naturally. But we should also not give them sharp knives when we don’t
expect them to use them properly.
It is just very regretful to me when people are ab-using a feature
just
to “feel more safe”.
- What I personally like as a feature is to apply location filters
to
individual rules. That allows to apply rate limits in a few places
and
easily select some larger groups of potentially less interesting traffic.
I agree and is similar to what I am doing for ports 25 and 587
although
with location block.
- Back when this was introduced there was also a lot less use of
botnets as we know them now. Pretty much all recent DoS attacks I remember have been coming from all over the world because systems where compromised wherever they were. This renders the filter absolutely useless.
For this, I am currently testing a dynamic block list on my system which adds an entry to an ipset when an ip address exceeds a
threshold
in the previous 24 hours. Currently there are about 7000 entries in this set which have been detected over the previous month. Not for everyone because it needs a Whitelist to prevent some 'push' technologies such as Facebook and to protect from bulk inputs such as mailing lists. I also blacklist a large number of ip addresses which are associated with port scanners.
This would work well as a botnet filter but it needs an understanding of the blocking process, so it would not be suitable for everyone.
We have talked about this on the last team call and came to the conclusion that the whole set of blocking features is a total mess. There are several places where things can be blocked, they all heavily overlap. Not to be going on about Spamhaus again, but the DROP list is in the “Hostile Networks” feature, there is a special country code for it, pretty much every IPS ruleset contains it, the IP blocklist
feature
supports it. Now there are people throwing this all into DNS as well.
I
believe this is all getting a little bit out of hand.
I don't use the IPS mainly due to bad experiences when trying to use
Snort
with IPCop. I find it difficult to set up and select the appropriate Rulsets for my requirements. I suspect the IPS would be very effective
if
I could get it set up correctly.
This has massively improved in recent times. And it is FAST!
I'll have another look, but my current set-up is working well, so I'm reluctant to change.
I personally would prefer *one* place and we really make this work
well.
Some people find the IPS has too much overhead - I am beginning to disagree.
Some people add all sorts of blacklists that are clearly for experimental/research use and suddenly complain about things not
working
anymore. It is hard to figure out what is actually blocking traffic
when
there are about five places to look for. The result often is to
disable
the IPS, then something else, and after a while all of this is turned off and the firewall is configured to a bare minimum. We don’t want people to run it like that.
- Infrastructure is becoming more centralised. Some people attempt
to
block the US and pretty much nothing works any more. Well…
:)
Therefore I am happy to rethink this feature - personally I would
even
remove it entirely, but I don’t like to handle the stress about this now.
No I think you should keep it. Locationblock is very useful, but
needs
to be used in a considered way. I could live without it but its availability makes my life easier. Also being updated on a weekly
basis
makes keeping track of abusers easier.
We issue a new location database once a day, but the IPFire systems update it only once a week. We deemed that as sufficient as there are not that many changes from one version to another one.
I think this should log now. I don’t think there is any legitimate reason to disable firewall logging at all. This is the paper trail that you want when debugging anything or doing forensics on an
attack.
Certainly bad flash is not a valid reason for me.
I have started generating some patches to make 'locationblock
logging'
as a selectable feature and could post them here in the next day or
so
if of any use.
What would this basically do? Log it as a simple rule for all, or do
you
include the selected country? I think the latter would make more
sense.
OK - I have modified location-block.cgi to add an additional checkbox
at
the top of the page titled "Log dropped packets", very similar to ipblocklist.cgi. This adds a entry to the /var/ipfire/firewall/ locationblock settings file ON or OFF.
In /usr/lib/firewall/rules.pl I add an additional Iptables
LOCATIONBLOCK
log rule with the prefix 'LOCBLOCK-$location '. This is selectable
from
the Log dropped packets entry in the locationblock settings file.
This generates kernel messages such as:
LOCBLOCK-CN IN=ppp0 OUT=ppp0 MAC= SRC=114.139.42.71 DST=81.187.175.103 LEN=60 TOS=0x00 PREC=0x00 TTL=47 ID=34664 DF PROTO=TCP SPT=46837
DPT=23
WINDOW=29040 RES=0x00 SYN URGP=0
As you can see the prefix contains the country code of the dropped packets.
Cool. So I think coding-wise this is pretty straight forward.
The more crucial question for me is below…
Do have a look at the patches I posted and let me have any comments. If it's not needed and we can add locationblock logging without selection then that's what I have been doing since LB was first implemented.
What is the rationale to turn this off? Why don’t we turn this on for everybody all of the time?
I would be happy with that I was suggesting that we use selectable
logging
to be compatible with earlier IPFire versions.
What do we have to be compatible with? Just IPFire’s behaviour from
last year?
I would be happy to break that if we have a good reason, because the
world sometimes moves on. What we have released 10 years ago probably doesn’t have good defaults any more. Sometimes we have to revisit those and sometimes we even have to be pushy with some changes if we want some long-standing users to adopt them. IPFire users usually have no reason to do a reinstall and we don’t want to leave the people behind who have been running IPFire perfectly well for the last 10 years or even longer. So at least I would like to discuss what we push a little harder. As Adolf pointed out in the Forum,the WIKI stated the rational for not logging was to protect flash memory from excessive use. I don't think that is appropriate now.
In fact I think that IPFire's logging environment, which was
inherited
from IPCop needs a re-visit.
Yes. But this is probably not a thing I would be looking at in IPFire
I am also happy to think a little bit more outside of the box and see what we would fundamentally change in the future, but not implement in IPFire 2.
I was pleased to see that a while back there was a discussion of replacing sylogd with rsyslogd which I think would be a very useful upgrade and enable kernel messages, like on most current linux
systems,
to be logged to alternative log files instead of everything ending up in /var/log/ messages.
The currentl syslog server could do that. The problem is more that we don’t have good tools to view the logs on the web UI. So there is
simply
one place where most things go into and we will then filter it out
when
users use the web UI. This is tremendously slow.
I did look at this a while back and believe that with syslog all
kernel
messages must go to the same log file and cannot be filtered off as
can be
done with rsyslog. I use this facility in rsyslog.d on my server to
filter
off SYN Flood logs to a separate log file with ":msg,contains". I
don't
believe that syslog has this option.
Different messages could be sent to different “facilities” and then we
can filter them into individual files.
I would not want to split firewall hits into different files though. I
just wouldn’t mind if firewall hits were no longer in /var/log/messages.
You need to be careful with the LOG* menus which would need to look in 2 or more places for historical logs and so the cgi's would need to be modified.
I would however welcome kernel message filtering since it would allow my use of a separate log directory for my auto-dnsbl program to monitor for non-dropped incoming SMTP connections without appearing in 'messages' log file.
The replacement with rsyslgd could also inhibit the "last message repeated x times" in syslog and which are ignored by IPFire's Log
menu
and causes erroneous results in the 'Count' values.
This might be a setting that we can turn off in the current implementation.
I understand that this is hard coded into syslogd and can't be
changed. In
rsyslog it defaults to "off".
https://www.rsyslog.com/doc/configuration/action/ rsconf1_repeatedmsgreduction.html
What impact does this have for you? I never noticed any problem by
summarising the logs.
I came across this and it illustrates the point well:
https://dcid.me/notes/syslog-last-message-repeated-x-times-rant
and makes the point that kernel messages can be delayed by several seconds and could affect any DOS filtering that may be in use.
The dshield iptables.pl script tries to recover the lost repeated messages but looses the time-stamp.
When comparing the dshield logs with IPFire's logs there is invariably a discrepancy with the IPFire's count sometimes considerably less especially with multiple port-scans over a minute or so.
Any chance of looking again at rsyslog? IPFire is the only linux system I know of still using syslog.
I seem to remember that Adolf had completed all the necessary changes to introduce rsyslog a years or so ago but was not implemented,
It wasn't me. It was Erik Kapfer. I have looked at it to have a go at it but there have been a lot of other things that are on my plate that I am still working through and of course sometimes things come along that have to be addressed and jump ahead in the queue.
Regards,
Adolf.
We could also think about moving all firewall hits into /var/log/firewall.log or something. That should free up enough in /var/log/messages to make debugging easier.
That would be a sensible option, currently I filter out firewall hits
from
the messages.log with a perl regex.
I would be happy to hear some more opinions on this from the list.
-Michael
I hope these comments are useful and would be pleased to help where I can.
Of course :)
-Michael
Rob
-Michael
Rob
Hello
On 23 Nov 2024, at 15:47, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Rob,
On 23/11/2024 15:36, Rob Brewer wrote:
On Sat, 23 Nov 2024 10:26:07 +0100, Michael Tremer wrote:
Hello,
On 21 Nov 2024, at 00:08, Rob Brewer ipfire-devel@grantura.co.uk
wrote:
Hi Michael and Adolf,
I'll combine my reply to both of you.
On Wed, 20 Nov 2024 15:21:28 +0000, Michael Tremer wrote:
Hello,
On 20 Nov 2024, at 13:36, Rob Brewer ipfire-devel@grantura.co.uk wrote:
On Tue, 19 Nov 2024 17:10:24 +0000, Michael Tremer wrote:
Hi Michael, thank for your response, I was beginning to think there
was
little interest in this topic.
I was traveling and so have a little bit of a back log as usual :)
I can’t speak for the others, though.
Not a problem, I should have said that I previously raised this a
topic on
the forum and Adolf suggested that I raised it on the mailing list as well.
Yes, I really prefer this here, because the forum is very overwhelming
sometimes. There is too many threads, too many people… Mailing lists are great for development.
That's fine, one query though you don't seem to snip older parts of the post, is this your preference?
See below...
> Hello Rob, > > Good to hear from you again. > >> On 9 Nov 2024, at 14:00, Rob Brewer ipfire-devel@grantura.co.uk >> wrote: >> >> As a contributor to the SANS ISC, I email my IPTABLES logs hourly >> to the Dshield database, however the default IPFire IPTABLES logs
are
>> incomplete since packets dropped by Locationblock are not logged to >> syslog. > This is correct. > >> As a workaround I edit the locationblock function of rules.pl to
add
>> a IPTABLES rule to log the dropped packets. This edit does not of >> course survive any Core-Updates that upgrade rules.pl and the edit >> needs to be re-applied. >> >> In addition, unless the Locationblock packets are logged to syslog >> the data displayed in the Firewall logs sections are incomplete as >> the packets dropped by Location Block are not displayed. >> >> I understand that the original intension of the Location Block >> feature was to reduce the amount of log messages on installations >> running on extremely cheap flash storage. I suspect that these >> installations are now almost zero especially since IPFire no longer >> supports 32 bit versions. >> >> The obvious solution to this to make Location Block logging >> selectable as a firewall option and my quick look at making a
patches
>> for this would seem to be fairly simple task to rules.pl and and >> optionsfw.cgi. >> >> I would be happy to provide the necessary patches should this be >> acceptable. > I think a couple of things have changed since we introduced this > feature and we have also learned a couple of lessons here: > > * Yes it is true that this was introduced to keep the logs smaller, > but not necessarily to avoid using storage, mainly to keep them > readable by a human. I was always a bit doubtful about this and > believe that apart from this, there is no more value in a general > location blocking feature. The firewall will drop those packets > anyway. > > IPFire does a very good job in protecting the internal network from unwanted probes from the WAN, and I agree that for most users,
location
block does not enhance this protection.
As I run a mail server, I use locationblock as part of my filtering
to
prevent bad actors from accessing my mail server and I think would be useful for anyone who are running servers on their networks.
I don't like blocking traffic, but I have witnessed over the last few years a substantial rise in abuse from certain countries from where I would never expect to receive emails and locationblock works very
well
for me in filtering out those unwanted connections.
I currently have 21 countries blocked including XD Hostile networks. All of these blocked countries have been seen to support abusive networks.
It would be possible to generate individual rules for these networks which in some cases I do with Banish, but it is sometimes easier just to drop the whole country.
Easy yes. I am just not sure what we are actually getting out of this.
I don’t like to become political on this list and so I won’t, but I
know
that some people hold the following beliefs:
- There are “bad” countries out there. In the western world those are
Russia, China, and more that fall into this kind of “category”.
Nothing
is going to stop them to open up an account on AWS and suddenly they have access to resources anywhere in the world and so actually, if
those
players were up to anything “bad”, they can come out of any IP space they want. There is also Tor which allows this.
I have been submitting my iptables logs to dshield for over 20 years
and
in the early days I would be sending a lot less than 1000 a day (this
was
with IPCop of course).
Over the last few years the number of firewall iptables logs have escalated until it had reached dshield's daily limit of 100,000 /day
and
I started looking at ways to limit the logging of some of the 'bad
actors'
firewall probes.
What do you use to send them the logs? The Python script or one of
those alternatives?
I use a modified Perl script (iptables.pl) which is about 20 years old was originally written for Smoothwall and Ipcop1. This was part of the Logsend add-on and I modified for IPCop2 about 15 years ago.
Now modified for IPFire and using DMA email instead of the original sendEmail. I've uploaded my version to people.ipfire.org/~helix/dshield.
I had intended to write an install/uninstall script but is in the todo list. Do you think it may be of interest to anyone if I do?
- Looking at actual data, the bullet proof ISPs that annoy us are not
that far away. They are hosted in Europe and very prominently so in
the
Netherlands - a “good” country.
I use Banish for this as it can block by ASN and can target bullet
proof
ISPs not only in NL but also in the USA and elsewhere. Using the
'location
function' the blocked ASN's are updated when the database is updated.
This is just an example to explain that we all have our idea of the world in our head, but the Internet doesn’t know these borders. Geopolitics is complicated and I don’t think that this is a tool that helps to fight this kind of problem. I hope this is clear what I am trying to say.
Yes my approach is to use multiple tools and Locationblock is just one
of
the tools I am using. Maybe my approach is over the top though.
Probably not massively, but maybe slightly.
Wouldn’t it be nicer to have everything in one place instead of having
multiple tools that work together in concert? I think that would be a lot of work, would you consider that for IPFire2 or 3?
I personally trust in Spamhaus which is feeding data into our “Hostile Networks” blocker and they are looking at the actual traffic that they see from those subnets that they are blocking. They have a clear set
of
rules and they don’t look just at the country and say “ah yes, the bad guys again”.
For emails I also drop packets if listed in a number of DNS blackhole lists using Auto-Dnsbl which I adapted for IPFire. See:
https://people.ipfire.org/~helix/auto-dnsbl/README for info.
I have been using this for about a year and can confirm that it is
very
effective in removing spammers.
I have stated this recently in the RPZ discussion that I feel that we are lacking good sources for this kind of data. For blocking anything. Loads of people believe “more is more” and to be fair… once you get to blocking 90% of the internet you probably have blocked 90% of the
noise.
However this won’t do anything for targeted attacks. I personally
would
rather have more quality here.
I only use a few blocklists with ipblocklist and rely mainly on my
locally
derived data from failed smtp connections or from data from dshield's daily report.
> * People seem to misunderstand this feature: The block as many > countries as they can click on and think that this makes things
“more
> secure”, > which of course it doesn’t. > > I think this is education and I suspect that very few of these people are running servers on their system and need location block.
I am personally not here to educate people. I am happy to help out and give my opinion, but I suppose considering that people think as I have outlined above, who am I to educate them?
Sorry I didn't express my reply well, I agree that IPFire users should
be
responsible for their own actions.
Naturally. But we should also not give them sharp knives when we don’t
expect them to use them properly.
It is just very regretful to me when people are ab-using a feature
just
to “feel more safe”.
> * What I personally like as a feature is to apply location filters
to
> individual rules. That allows to apply rate limits in a few places
and
> easily select some larger groups of potentially less interesting > traffic. I agree and is similar to what I am doing for ports 25 and 587
although
with location block.
> * Back when this was introduced there was also a lot less use of > botnets as we know them now. Pretty much all recent DoS attacks I > remember have been coming from all over the world because systems > where compromised wherever they were. This renders the filter > absolutely useless. For this, I am currently testing a dynamic block list on my system which adds an entry to an ipset when an ip address exceeds a
threshold
in the previous 24 hours. Currently there are about 7000 entries in this set which have been detected over the previous month. Not for everyone because it needs a Whitelist to prevent some 'push' technologies such as Facebook and to protect from bulk inputs such as mailing lists. I also blacklist a large number of ip addresses which are associated with port scanners.
This would work well as a botnet filter but it needs an understanding of the blocking process, so it would not be suitable for everyone.
We have talked about this on the last team call and came to the conclusion that the whole set of blocking features is a total mess. There are several places where things can be blocked, they all heavily overlap. Not to be going on about Spamhaus again, but the DROP list is in the “Hostile Networks” feature, there is a special country code for it, pretty much every IPS ruleset contains it, the IP blocklist
feature
supports it. Now there are people throwing this all into DNS as well.
I
believe this is all getting a little bit out of hand.
I don't use the IPS mainly due to bad experiences when trying to use
Snort
with IPCop. I find it difficult to set up and select the appropriate Rulsets for my requirements. I suspect the IPS would be very effective
if
I could get it set up correctly.
This has massively improved in recent times. And it is FAST!
I'll have another look, but my current set-up is working well, so I'm reluctant to change.
I personally would prefer *one* place and we really make this work
well.
Some people find the IPS has too much overhead - I am beginning to disagree.
Some people add all sorts of blacklists that are clearly for experimental/research use and suddenly complain about things not
working
anymore. It is hard to figure out what is actually blocking traffic
when
there are about five places to look for. The result often is to
disable
the IPS, then something else, and after a while all of this is turned off and the firewall is configured to a bare minimum. We don’t want people to run it like that.
> * Infrastructure is becoming more centralised. Some people attempt
to
> block the US and pretty much nothing works any more. Well… > > :)
> Therefore I am happy to rethink this feature - personally I would
even
> remove it entirely, but I don’t like to handle the stress about this > now. No I think you should keep it. Locationblock is very useful, but
needs
to be used in a considered way. I could live without it but its availability makes my life easier. Also being updated on a weekly
basis
makes keeping track of abusers easier.
We issue a new location database once a day, but the IPFire systems update it only once a week. We deemed that as sufficient as there are not that many changes from one version to another one.
> I think this should log now. I don’t think there is any legitimate > reason to disable firewall logging at all. This is the paper trail > that you want when debugging anything or doing forensics on an
attack.
> Certainly bad flash is not a valid reason for me. > > I have started generating some patches to make 'locationblock
logging'
as a selectable feature and could post them here in the next day or
so
if of any use.
What would this basically do? Log it as a simple rule for all, or do
you
include the selected country? I think the latter would make more
sense.
OK - I have modified location-block.cgi to add an additional checkbox
at
the top of the page titled "Log dropped packets", very similar to ipblocklist.cgi. This adds a entry to the /var/ipfire/firewall/ locationblock settings file ON or OFF.
In /usr/lib/firewall/rules.pl I add an additional Iptables
LOCATIONBLOCK
log rule with the prefix 'LOCBLOCK-$location '. This is selectable
from
the Log dropped packets entry in the locationblock settings file.
This generates kernel messages such as:
LOCBLOCK-CN IN=ppp0 OUT=ppp0 MAC= SRC=114.139.42.71 DST=81.187.175.103 LEN=60 TOS=0x00 PREC=0x00 TTL=47 ID=34664 DF PROTO=TCP SPT=46837
DPT=23
WINDOW=29040 RES=0x00 SYN URGP=0
As you can see the prefix contains the country code of the dropped packets.
Cool. So I think coding-wise this is pretty straight forward.
The more crucial question for me is below…
Do have a look at the patches I posted and let me have any comments. If it's not needed and we can add locationblock logging without selection then that's what I have been doing since LB was first implemented.
What is the rationale to turn this off? Why don’t we turn this on for everybody all of the time?
I would be happy with that I was suggesting that we use selectable
logging
to be compatible with earlier IPFire versions.
What do we have to be compatible with? Just IPFire’s behaviour from
last year?
I would be happy to break that if we have a good reason, because the
world sometimes moves on. What we have released 10 years ago probably doesn’t have good defaults any more. Sometimes we have to revisit those and sometimes we even have to be pushy with some changes if we want some long-standing users to adopt them. IPFire users usually have no reason to do a reinstall and we don’t want to leave the people behind who have been running IPFire perfectly well for the last 10 years or even longer. So at least I would like to discuss what we push a little harder. As Adolf pointed out in the Forum,the WIKI stated the rational for not logging was to protect flash memory from excessive use. I don't think that is appropriate now.
In fact I think that IPFire's logging environment, which was
inherited
from IPCop needs a re-visit.
Yes. But this is probably not a thing I would be looking at in IPFire
I am also happy to think a little bit more outside of the box and see what we would fundamentally change in the future, but not implement in IPFire 2.
I was pleased to see that a while back there was a discussion of replacing sylogd with rsyslogd which I think would be a very useful upgrade and enable kernel messages, like on most current linux
systems,
to be logged to alternative log files instead of everything ending up in /var/log/ messages.
The currentl syslog server could do that. The problem is more that we don’t have good tools to view the logs on the web UI. So there is
simply
one place where most things go into and we will then filter it out
when
users use the web UI. This is tremendously slow.
I did look at this a while back and believe that with syslog all
kernel
messages must go to the same log file and cannot be filtered off as
can be
done with rsyslog. I use this facility in rsyslog.d on my server to
filter
off SYN Flood logs to a separate log file with ":msg,contains". I
don't
believe that syslog has this option.
Different messages could be sent to different “facilities” and then we
can filter them into individual files.
I would not want to split firewall hits into different files though. I
just wouldn’t mind if firewall hits were no longer in /var/log/messages.
You need to be careful with the LOG* menus which would need to look in 2 or more places for historical logs and so the cgi's would need to be modified.
I would however welcome kernel message filtering since it would allow my use of a separate log directory for my auto-dnsbl program to monitor for non-dropped incoming SMTP connections without appearing in 'messages' log file.
The replacement with rsyslgd could also inhibit the "last message repeated x times" in syslog and which are ignored by IPFire's Log
menu
and causes erroneous results in the 'Count' values.
This might be a setting that we can turn off in the current implementation.
I understand that this is hard coded into syslogd and can't be
changed. In
rsyslog it defaults to "off".
https://www.rsyslog.com/doc/configuration/action/ rsconf1_repeatedmsgreduction.html
What impact does this have for you? I never noticed any problem by
summarising the logs.
I came across this and it illustrates the point well:
https://dcid.me/notes/syslog-last-message-repeated-x-times-rant
and makes the point that kernel messages can be delayed by several seconds and could affect any DOS filtering that may be in use.
The dshield iptables.pl script tries to recover the lost repeated messages but looses the time-stamp.
When comparing the dshield logs with IPFire's logs there is invariably a discrepancy with the IPFire's count sometimes considerably less especially with multiple port-scans over a minute or so.
Any chance of looking again at rsyslog? IPFire is the only linux system I know of still using syslog.
I seem to remember that Adolf had completed all the necessary changes to introduce rsyslog a years or so ago but was not implemented,
It wasn't me. It was Erik Kapfer. I have looked at it to have a go at it but there have been a lot of other things that are on my plate that I am still working through and of course sometimes things come along that have to be addressed and jump ahead in the queue.
Story of my life :)
Regards,
Adolf.
We could also think about moving all firewall hits into /var/log/firewall.log or something. That should free up enough in /var/log/messages to make debugging easier.
That would be a sensible option, currently I filter out firewall hits
from
the messages.log with a perl regex.
> I would be happy to hear some more opinions on this from the list. > > -Michael I hope these comments are useful and would be pleased to help where I can.
Of course :)
-Michael
Rob
-Michael
Rob
-
Adolf Belka -
Michael Tremer -
Rob Brewer