Hello,
I've not had a chance to look at the code in detail, but a few comments:
- The disable value in the sources file is intended to disable conflicting sets when necessary. It's partly in reaction to people who just enable everything without thinking about it, but it's also because for the blocklists it's relatively easy to identify which lists are subsets of others. A good example is the three Feodo lists which are are different versions of the same list, but with different probabilities of blocking legitimate traffic.
- I set the list to update every 15 minutes after discussion with Michael, as some of the lists update very quickly (every five minutes for some lists). I don't think there's any particular reason why a given update period is preferred.
- I had code to create either a hash:ip or had:net ipset. The reason for this is that aparently (if I remember the documentation correctly), looking for an IP address against a hash:net requires checking each of the possible containing networks (/32, /31, /30...). It thus takes significantly longer than checking against a hash:ip. As the majority of lists are only single addresses this seemed like a worthwhile optimisation.
- It's probably worth reviewing the list of blocklists. I would certainly remove Alienvault, which seems to be defunct. Charles Brown suggested the 3COREsec lists, which look interesting. This could be left to post-first release. As a note of caution, I found some interesting lists for blocking command and control servers which were licenced under terms that I thought would make it difficult to include in the IPFire distribution.
- Longer term it may be worth adding the capability for a sources.local, but again I think this is better left until post-first release. It may be worth adding a 'Do not edit this file' in the sources file since it could be overwritten by future updates.
- I think IPFire now has four different ways of using the Spamhaus DROP list (IPS, hostile networks, XD country code and IP Blocklist). When this is released it may be worth writing a blog post or a wiki page on the advantages and disadvantages of each (or even a more general one looking at all the different malware etc. facilities).
Tim
On 10/04/2022 19:47, Stefan Schantl wrote:
Hello Charles,
thanks for a first look and your feedback.
Now after you have put my attention to those "disable" value in the sources file, I have to admit I totally have overseen this.
When interpreting this field and the values, it seems has been designed to automatically disable conflicting/obsolete sets in case such a set is enabled.
So for this feature I have go back to the drawing board and put some attention to it.
In the meantime please go on with testing and report any other issues.
Best regards,
-Stefan
Tim, Stefan, I have installed the ipblocklist feature. It looks great. I’m curious about the disable attribute in the sources file. I have all the lists enabled, I would have thought enabling EMERGING_FWRULE would have the DSHIELD list automatically disabled. However, I am showing several hits on DSHIELD and I see 20 entries in ipset for DSHIELD. Is the disable attribute in sources there for informational purposes only? Thanks for your excellent work on this feature, Charles Brown
Hello Tim, hello *,
welcome back to the list and thanks for your detailed feedback.
Hello,
I've not had a chance to look at the code in detail, but a few comments:
- The disable value in the sources file is intended to disable
conflicting sets when necessary. It's partly in reaction to people who just enable everything without thinking about it, but it's also because for the blocklists it's relatively easy to identify which lists are subsets of others. A good example is the three Feodo lists which are are different versions of the same list, but with different probabilities of blocking legitimate traffic.
Thanks for clarify this. Theoretically it is no problem to enable and use all blocklists at the same time, they do not harm each other. But as you pointed this would be a massive waste of resources, because some of the lists are subsets of other lists.
When walking through the code of your main "ipblacklist" script, I didn't find any code to handle this. Did I overseen something or was that a planed feature for the future?
- I set the list to update every 15 minutes after discussion with
Michael, as some of the lists update very quickly (every five minutes for some lists). I don't think there's any particular reason why a given update period is preferred.
Currently I set the call of the update script to one a hour via cron job. This easily can be increased or decreased.
- I had code to create either a hash:ip or had:net ipset. The reason
for this is that aparently (if I remember the documentation correctly), looking for an IP address against a hash:net requires checking each of the possible containing networks (/32, /31, /30...). It thus takes significantly longer than checking against a hash:ip. As the majority of lists are only single addresses this seemed like a worthwhile optimisation.
I see - I only checked that single addresses also can be used in a net:hash and therefore simplified to generally use this kind of hash for the sets. I was not aware of the performance impact of this decision.
With this knowledge it absolutely make sense to bring back some code to handle this in a proper way.
- It's probably worth reviewing the list of blocklists. I would
certainly remove Alienvault, which seems to be defunct. Charles Brown suggested the 3COREsec lists, which look interesting. This could be left to post-first release. As a note of caution, I found some interesting lists for blocking command and control servers which were licenced under terms that I thought would make it difficult to include in the IPFire distribution.
That sounds great. I'm also in favor to move a single provider from the IDS to the blocklist, because it only serves a set of addresses in his rulesfile and also serves a bare list on their website.
- Longer term it may be worth adding the capability for a
sources.local, but again I think this is better left until post-first release. It may be worth adding a 'Do not edit this file' in the sources file since it could be overwritten by future updates.
- I think IPFire now has four different ways of using the Spamhaus
DROP list (IPS, hostile networks, XD country code and IP Blocklist). When this is released it may be worth writing a blog post or a wiki page on the advantages and disadvantages of each (or even a more general one looking at all the different malware etc. facilities).
I've the XD and Spamhaus running on my system and both are dropping packets. The huge amount will be match the XD but a few packets slip through and will be blocked by the more accurate list from ipblocklist.
- Stefan
Tim
On 10/04/2022 19:47, Stefan Schantl wrote:
Hello Charles,
thanks for a first look and your feedback.
Now after you have put my attention to those "disable" value in the sources file, I have to admit I totally have overseen this.
When interpreting this field and the values, it seems has been designed to automatically disable conflicting/obsolete sets in case such a set is enabled.
So for this feature I have go back to the drawing board and put some attention to it.
In the meantime please go on with testing and report any other issues.
Best regards,
-Stefan
Tim, Stefan, I have installed the ipblocklist feature. It looks great. I’m curious about the disable attribute in the sources file. I have all the lists enabled, I would have thought enabling EMERGING_FWRULE would have the DSHIELD list automatically disabled. However, I am showing several hits on DSHIELD and I see 20 entries in ipset for DSHIELD. Is the disable attribute in sources there for informational purposes only? Thanks for your excellent work on this feature, Charles Brown
Hi Stefan, Tim
On Tuesday 12 April 2022 05:12 Stefan Schantl wrote:
Hello Tim, hello *,
welcome back to the list and thanks for your detailed feedback.
Hello,
I've not had a chance to look at the code in detail, but a few comments:
- The disable value in the sources file is intended to disable
conflicting sets when necessary. It's partly in reaction to people who just enable everything without thinking about it, but it's also because for the blocklists it's relatively easy to identify which lists are subsets of others. A good example is the three Feodo lists which are are different versions of the same list, but with different probabilities of blocking legitimate traffic.
Thanks for clarify this. Theoretically it is no problem to enable and use all blocklists at the same time, they do not harm each other. But as you pointed this would be a massive waste of resources, because some of the lists are subsets of other lists.
When walking through the code of your main "ipblacklist" script, I didn't find any code to handle this. Did I overseen something or was that a planed feature for the future?
The disable code is working on Tim's original code and I have made good use of it when selecting my blacklists. I'll have a look and see if I can find the relevant bits.
- I set the list to update every 15 minutes after discussion with
Michael, as some of the lists update very quickly (every five minutes for some lists). I don't think there's any particular reason why a given update period is preferred.
The 15 minute update is very useful for list that update once an hour since if the update runs just after the list updates you have another hour to wait before the your blocklist is updated. A 15 minute update will update the blocklist and will pick up the new data in less than 15 minutes.
I suggest you keep the 15 minute update as there doesn't seem to be any overhead for not doing so.
Currently I set the call of the update script to one a hour via cron job. This easily can be increased or decreased.
- I had code to create either a hash:ip or had:net ipset. The reason
for this is that apparently (if I remember the documentation correctly), looking for an IP address against a hash:net requires checking each of the possible containing networks (/32, /31, /30...). It thus takes significantly longer than checking against a hash:ip. As the majority of lists are only single addresses this seemed like a worthwhile optimisation.
I see - I only checked that single addresses also can be used in a net:hash and therefore simplified to generally use this kind of hash for the sets. I was not aware of the performance impact of this decision.
With this knowledge it absolutely make sense to bring back some code to handle this in a proper way.
These are the lists I am using and certainly seem to be sufficient for my use:
[root@griffith6 ~]# ipset -t list Name: BANISH Type: hash:net Revision: 6 Header: family inet hashsize 256 maxelem 1024 Size in memory: 9152 References: 2 Number of entries: 191
Name: BLOCKLIST_DE Type: hash:ip Revision: 4 Header: family inet hashsize 32768 maxelem 65536 Size in memory: 794288 References: 2 Number of entries: 22985
Name: CIARMY Type: hash:ip Revision: 4 Header: family inet hashsize 16384 maxelem 32768 Size in memory: 471440 References: 2 Number of entries: 15000
Name: EMERGING_COMPROMISED Type: hash:ip Revision: 4 Header: family inet hashsize 1024 maxelem 2048 Size in memory: 28520 References: 2 Number of entries: 887
Name: EMERGING_FWRULE Type: hash:net Revision: 6 Header: family inet hashsize 2048 maxelem 4096 Size in memory: 72920 References: 2 Number of entries: 1600
Name: LOCAL_BLOCKLIST Type: hash:net Revision: 6 Header: family inet hashsize 4096 maxelem 8192 Size in memory: 145544 References: 2 Number of entries: 3283
Name: SHODAN Type: hash:ip Revision: 4 Header: family inet hashsize 64 maxelem 1024 Size in memory: 1784 References: 2 Number of entries: 44
Name: SPAMHAUS_EDROP Type: hash:net Revision: 6 Header: family inet hashsize 64 maxelem 1024 Size in memory: 2880 References: 2 Number of entries: 58
There are 4 hash:net and 4 hash:ip (2 of hash:nets are locally generated). I only have port 25 open so I have chosen lists which are most effective for smtp. I also use Location block and fail2ban (IPTABLES) as additional blocklists .
The most effective lists blocking port 25 varies on a day by day basis and also time of day so each of my selected lists complement each other.
For me the selected lists seems to be a good combination.
- It's probably worth reviewing the list of blocklists. I would
certainly remove Alienvault, which seems to be defunct. Charles Brown suggested the 3COREsec lists, which look interesting. This could be left to post-first release. As a note of caution, I found some interesting lists for blocking command and control servers which were licenced under terms that I thought would make it difficult to include in the IPFire distribution.
That sounds great. I'm also in favor to move a single provider from the IDS to the blocklist, because it only serves a set of addresses in his rulesfile and also serves a bare list on their website.
I think that would be a mistake since different sources seem to be optimized for different ports. Looking at my logs for port 23 the majority of blocks come from CIARMY and Location Block whereas SMTP blocks mostly come from BLOCKLIST_DE and my local blocklists.
As most users have different requirements they should be able to chose a blocklist that suits their environment.
- Longer term it may be worth adding the capability for a
sources.local, but again I think this is better left until post-first release. It may be worth adding a 'Do not edit this file' in the sources file since it could be overwritten by future updates.
As above I am running 2 local blocklists one on IPFire's apache server and the other on the web server of my SMTP server.
BANISH runs on IPFire and is useful to block addresses when you don't want to block whole countries with Location Block but find abusive domains such as spamming outfits that don't appear on other blocklists.
The one on my SMTP server catches the failed SMTP AUTH attempts from ip's that fall through the other blocklists.
- I think IPFire now has four different ways of using the Spamhaus
DROP list (IPS, hostile networks, XD country code and IP Blocklist). When this is released it may be worth writing a blog post or a wiki page on the advantages and disadvantages of each (or even a more general one looking at all the different malware etc. facilities).
I've the XD and Spamhaus running on my system and both are dropping packets. The huge amount will be match the XD but a few packets slip through and will be blocked by the more accurate list from ipblocklist.
I use SPAMHAUS_EDROP but it catches very few IP's. Often the daily total is Zero. I also run Zen on my mail server but since I have been using IPblacklist I rarely see any blocks from spamhaus in the mail server logs, so presumably attempted connections are rejected before they reach the server.
Rob
- Stefan
Tim
On 10/04/2022 19:47, Stefan Schantl wrote:
Hello Charles,
thanks for a first look and your feedback.
Now after you have put my attention to those "disable" value in the sources file, I have to admit I totally have overseen this.
When interpreting this field and the values, it seems has been designed to automatically disable conflicting/obsolete sets in case such a set is enabled.
So for this feature I have go back to the drawing board and put some attention to it.
In the meantime please go on with testing and report any other issues.
Best regards,
-Stefan
Tim, Stefan, I have installed the ipblocklist feature. It looks great. I’m curious about the disable attribute in the sources file. I have all the lists enabled, I would have thought enabling EMERGING_FWRULE would have the DSHIELD list automatically disabled. However, I am showing several hits on DSHIELD and I see 20 entries in ipset for DSHIELD. Is the disable attribute in sources there for informational purposes only? Thanks for your excellent work on this feature, Charles Brown
-
Rob Brewer -
Stefan Schantl -
Tim FitzGeorge