Hello Tim, hello *,
welcome back to the list and thanks for your detailed feedback.
Hello,
I've not had a chance to look at the code in detail, but a few comments:
- The disable value in the sources file is intended to disable
conflicting sets when necessary. It's partly in reaction to people who just enable everything without thinking about it, but it's also because for the blocklists it's relatively easy to identify which lists are subsets of others. A good example is the three Feodo lists which are are different versions of the same list, but with different probabilities of blocking legitimate traffic.
Thanks for clarify this. Theoretically it is no problem to enable and use all blocklists at the same time, they do not harm each other. But as you pointed this would be a massive waste of resources, because some of the lists are subsets of other lists.
When walking through the code of your main "ipblacklist" script, I didn't find any code to handle this. Did I overseen something or was that a planed feature for the future?
- I set the list to update every 15 minutes after discussion with
Michael, as some of the lists update very quickly (every five minutes for some lists). I don't think there's any particular reason why a given update period is preferred.
Currently I set the call of the update script to one a hour via cron job. This easily can be increased or decreased.
- I had code to create either a hash:ip or had:net ipset. The reason
for this is that aparently (if I remember the documentation correctly), looking for an IP address against a hash:net requires checking each of the possible containing networks (/32, /31, /30...). It thus takes significantly longer than checking against a hash:ip. As the majority of lists are only single addresses this seemed like a worthwhile optimisation.
I see - I only checked that single addresses also can be used in a net:hash and therefore simplified to generally use this kind of hash for the sets. I was not aware of the performance impact of this decision.
With this knowledge it absolutely make sense to bring back some code to handle this in a proper way.
- It's probably worth reviewing the list of blocklists. I would
certainly remove Alienvault, which seems to be defunct. Charles Brown suggested the 3COREsec lists, which look interesting. This could be left to post-first release. As a note of caution, I found some interesting lists for blocking command and control servers which were licenced under terms that I thought would make it difficult to include in the IPFire distribution.
That sounds great. I'm also in favor to move a single provider from the IDS to the blocklist, because it only serves a set of addresses in his rulesfile and also serves a bare list on their website.
- Longer term it may be worth adding the capability for a
sources.local, but again I think this is better left until post-first release. It may be worth adding a 'Do not edit this file' in the sources file since it could be overwritten by future updates.
- I think IPFire now has four different ways of using the Spamhaus
DROP list (IPS, hostile networks, XD country code and IP Blocklist). When this is released it may be worth writing a blog post or a wiki page on the advantages and disadvantages of each (or even a more general one looking at all the different malware etc. facilities).
I've the XD and Spamhaus running on my system and both are dropping packets. The huge amount will be match the XD but a few packets slip through and will be blocked by the more accurate list from ipblocklist.
- Stefan
Tim
On 10/04/2022 19:47, Stefan Schantl wrote:
Hello Charles,
thanks for a first look and your feedback.
Now after you have put my attention to those "disable" value in the sources file, I have to admit I totally have overseen this.
When interpreting this field and the values, it seems has been designed to automatically disable conflicting/obsolete sets in case such a set is enabled.
So for this feature I have go back to the drawing board and put some attention to it.
In the meantime please go on with testing and report any other issues.
Best regards,
-Stefan
Tim, Stefan, I have installed the ipblocklist feature. It looks great. I’m curious about the disable attribute in the sources file. I have all the lists enabled, I would have thought enabling EMERGING_FWRULE would have the DSHIELD list automatically disabled. However, I am showing several hits on DSHIELD and I see 20 entries in ipset for DSHIELD. Is the disable attribute in sources there for informational purposes only? Thanks for your excellent work on this feature, Charles Brown