Re: ipblocklist - Call for testers (disable attribute in sources)

[Rob Brewer <ipfire-devel@grantura.co.uk>]

[-- Attachment #1: Type: text/plain, Size: 8702 bytes --]

Hi Stefan, Tim

On Tuesday 12 April 2022 05:12 Stefan Schantl wrote:

> Hello Tim, hello *,
> 
> welcome back to the list and thanks for your detailed feedback.
>> Hello,
>> 
>> I've not had a chance to look at the code in detail, but a few
>> comments:
>> 
>> - The disable value in the sources file is intended to disable
>> conflicting sets when necessary.  It's partly in
>> reaction to people who just enable everything without thinking about
>> it, but it's also because for the blocklists
>> it's relatively easy to identify which lists are subsets of others.
>> A good example is the three Feodo lists which
>> are are different versions of the same list, but with different
>> probabilities of blocking legitimate traffic.
> 
> Thanks for clarify this. Theoretically it is no problem to enable and
> use all blocklists at the same time, they do not harm each other.
> But as you pointed this would be a massive waste of resources, because
> some of the lists are subsets of other lists.
> 
> When walking through the code of your main "ipblacklist" script, I
> didn't find any code to handle this. Did I overseen something or was
> that a planed feature for the future?
>> 

The disable code is working on Tim's original code and I have made good use 
of it when selecting my blacklists.  I'll have a look and see if I can find 
the relevant bits.


>> - I set the list to update every 15 minutes after discussion with
>> Michael, as some of the lists update very quickly
>> (every five minutes for some lists).  I don't think there's any
>> particular reason why a given update period is
>> preferred.
>
The 15 minute update is very useful for list that update once an hour since 
if the update runs just after the list updates you have another hour to wait 
before the your blocklist is updated. A 15 minute update will update the 
blocklist and will pick up the new data in less than 15 minutes.

I suggest you keep the 15 minute update as there doesn't seem to be any 
overhead for not doing so.
 
> Currently I set the call of the update script to one a hour via cron
> job. This easily can be increased or decreased.
>> 
>> - I had code to create either a hash:ip or had:net ipset.  The reason
>> for this is that apparently (if I remember
>> the documentation correctly), looking for an IP address against a
>> hash:net requires checking each of the possible
>> containing networks (/32, /31, /30...).  It thus takes significantly
>> longer than checking against a hash:ip.  As
>> the majority of lists are only single addresses this seemed like a
>> worthwhile optimisation.
> 
> I see - I only checked that single addresses also can be used in a
> net:hash and therefore simplified to generally use this kind of hash
> for the sets. I was not aware of the performance impact of this
> decision.
> 
> With this knowledge it absolutely make sense to bring back some code to
> handle this in a proper way.
>> 
These are the lists I am using and certainly seem to be sufficient for my 
use:

[root(a)griffith6 ~]# ipset -t list
Name: BANISH
Type: hash:net
Revision: 6
Header: family inet hashsize 256 maxelem 1024
Size in memory: 9152
References: 2
Number of entries: 191

Name: BLOCKLIST_DE
Type: hash:ip
Revision: 4
Header: family inet hashsize 32768 maxelem 65536
Size in memory: 794288
References: 2
Number of entries: 22985

Name: CIARMY
Type: hash:ip
Revision: 4
Header: family inet hashsize 16384 maxelem 32768
Size in memory: 471440
References: 2
Number of entries: 15000

Name: EMERGING_COMPROMISED
Type: hash:ip
Revision: 4
Header: family inet hashsize 1024 maxelem 2048
Size in memory: 28520
References: 2
Number of entries: 887

Name: EMERGING_FWRULE
Type: hash:net
Revision: 6
Header: family inet hashsize 2048 maxelem 4096
Size in memory: 72920
References: 2
Number of entries: 1600

Name: LOCAL_BLOCKLIST
Type: hash:net
Revision: 6
Header: family inet hashsize 4096 maxelem 8192
Size in memory: 145544
References: 2
Number of entries: 3283

Name: SHODAN
Type: hash:ip
Revision: 4
Header: family inet hashsize 64 maxelem 1024
Size in memory: 1784
References: 2
Number of entries: 44

Name: SPAMHAUS_EDROP
Type: hash:net
Revision: 6
Header: family inet hashsize 64 maxelem 1024
Size in memory: 2880
References: 2
Number of entries: 58

There are 4 hash:net and 4 hash:ip (2 of hash:nets are locally generated). I 
only have port 25 open so I have chosen lists which are most effective for 
smtp.  I also use Location block and fail2ban (IPTABLES) as additional 
blocklists .

The most effective lists blocking port 25 varies on a day by day basis and 
also time of day so each of my selected lists complement each other.

For me the selected lists seems to be a good combination. 


>> - It's probably worth reviewing the list of blocklists.  I would
>> certainly remove Alienvault, which seems to be
>> defunct.  Charles Brown suggested the 3COREsec lists, which look
>> interesting.  This could be left to post-first release.  As a note of
>> caution, I found some interesting lists for blocking command and
>> control servers which
>> were licenced under terms that I thought would make it difficult to
>> include in the IPFire distribution.
> 
> That sounds great. I'm also in favor to move a single provider from the
> IDS to the blocklist, because it only serves a set of addresses in his
> rulesfile and also serves a bare list on their website.
>> 

I think that would be a mistake since different sources seem to be optimized 
for different ports. Looking at my logs for port 23 the majority of blocks 
come from CIARMY and Location Block whereas SMTP blocks mostly come from 
BLOCKLIST_DE and my local blocklists.

As most users have different requirements they should be able to chose a 
blocklist that suits their environment.

>> - Longer term it may be worth adding the capability for a
>> sources.local, but again I think this is better left
>> until post-first release.  It may be worth adding a 'Do not edit this
>> file' in the sources file since it could
>> be overwritten by future updates.
>>

As above I am running 2 local blocklists one on IPFire's apache server and 
the other on the web server of my SMTP server. 

BANISH runs on IPFire and is useful to block addresses when you don't want 
to block whole countries with  Location Block but find abusive domains such 
as spamming outfits that don't appear on other blocklists. 

The one on my SMTP server catches the failed SMTP AUTH attempts from ip's 
that fall through the other blocklists.

>> - I think IPFire now has four different ways of using the Spamhaus
>> DROP list (IPS, hostile networks, XD country
>> code and IP Blocklist).  When this is released it may be worth
>> writing a blog post or a wiki page on the
>> advantages and disadvantages of each (or even a more general one
>> looking at all the different malware etc.
>> facilities).
> 
> I've the XD and Spamhaus running on my system and both are dropping
> packets. The huge amount will be match the XD but a few packets slip
> through and will be blocked by the more accurate list from ipblocklist.
> 
I use SPAMHAUS_EDROP but it catches very few IP's. Often the daily total is 
Zero. I also run Zen on my mail server but since I have been using 
IPblacklist I rarely see any blocks from spamhaus in the mail server logs, 
so presumably attempted connections are rejected before they reach the 
server.

Rob

> - Stefan
>> 
>> Tim
>> 


>> On 10/04/2022 19:47, Stefan Schantl wrote:
>> > Hello Charles,
>> > 
>> > thanks for a first look and your feedback.
>> > 
>> > Now after you have put my attention to those "disable" value in the
>> > sources file, I have to admit I totally have overseen this.
>> > 
>> > When interpreting this field and the values, it seems has been
>> > designed
>> > to automatically disable conflicting/obsolete sets in case such a
>> > set
>> > is enabled.
>> > 
>> > So for this feature I have go back to the drawing board and put
>> > some
>> > attention to it.
>> > 
>> > In the meantime please go on with testing and report any other
>> > issues.
>> > 
>> > Best regards,
>> > 
>> > -Stefan
>> > > Tim, Stefan,
>> > > I have installed the ipblocklist feature. It looks great.
>> > > I’m curious about the disable attribute in the sources file.
>> > > I have all the lists enabled, I would have thought enabling
>> > > EMERGING_FWRULE would have the DSHIELD list automatically
>> > > disabled.
>> > > However, I am showing several hits on DSHIELD and I see 20
>> > > entries in
>> > > ipset for DSHIELD. Is the disable attribute in sources there for
>> > > informational purposes only?
>> > > Thanks for your excellent work on this feature,
>> > > Charles Brown
>> > > 
>> > > 
>> > 
>> > 
>>