Re: ipblocklist - Call for testers (disable attribute in sources)

Re: ipblocklist - Call for testers (disable attribute in sources)

[Stefan Schantl]

[-- Attachment #1: Type: text/plain, Size: 1053 bytes --]

Hello Charles,

thanks for a first look and your feedback.

Now after you have put my attention to those "disable" value in the
sources file, I have to admit I totally have overseen this.

When interpreting this field and the values, it seems has been designed
to automatically disable conflicting/obsolete sets in case such a set
is enabled.

So for this feature I have go back to the drawing board and put some
attention to it.

In the meantime please go on with testing and report any other issues.

Best regards,

-Stefan 
>  Tim, Stefan,
> I have installed the ipblocklist feature. It looks great.
> I’m curious about the disable attribute in the sources file. 
>  I have all the lists enabled, I would have thought enabling
> EMERGING_FWRULE would have the DSHIELD list automatically disabled.
> However, I am showing several hits on DSHIELD and I see 20 entries in
> ipset for DSHIELD. Is the disable attribute in sources there for
> informational purposes only? 
> Thanks for your excellent work on this feature,
>  Charles Brown
> 
>  

Re: ipblocklist - Call for testers (disable attribute in sources)

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 700 bytes --]

Hi Stefan, hi Tim,

I hadn't the time to test yet - but found a small typo.

Patch is attached.

Great work!

Best,
Matthias

On 10.04.2022 20:21, Charles Brown wrote:
> Tim, Stefan,
> 
> I have installed the ipblocklist feature. It looks great.
> 
> I’m curious about the disable attribute in the sources file.
> 
> I have all the lists enabled, I would have thought enabling 
> EMERGING_FWRULE would have the DSHIELD list automatically disabled. 
> However, I am showing several hits on DSHIELD and I see 20 entries in 
> ipset for DSHIELD. Is the disable attribute in sources there for 
> informational purposes only?
> 
> Thanks for your excellent work on this feature,
> Charles Brown
> 
> 



[-- Attachment #2: fix_aggresive.patch --]
[-- Type: text/plain, Size: 1683 bytes --]

diff -U 3 a/var/ipfire/ipblocklist/sources b/var/ipfire/ipblocklist/sources
--- a/var/ipfire/ipblocklist/sources	Sun Apr  3 13:22:06 2022
+++ b/var/ipfire/ipblocklist/sources	Sun Apr 10 20:48:08 2022
@@ -42,7 +42,7 @@
                                     'parser'   => 'ip-or-net-list',
                                     'rate'     => '1h',
                                     'category' => 'composite',
-                                    'disable'  => ['FEODO_RECOMMENDED', 'FEODO_IP', 'FEODO_AGGRESIVE', 'SPAMHAUS_DROP', 'DSHIELD'] },
+                                    'disable'  => ['FEODO_RECOMMENDED', 'FEODO_IP', 'FEODO_AGGRESSIVE', 'SPAMHAUS_DROP', 'DSHIELD'] },
              'EMERGING_COMPROMISED' => { 'name' => 'Emerging Threats Compromised IPs',
                                     'url'      => 'https://rules.emergingthreats.net/blockrules/compromised-ips.txt',
                                     'info'     => 'https://doc.emergingthreats.net/bin/view/Main/CompromisedHost',
@@ -80,7 +80,7 @@
                                     'rate'     => '5m',
                                     'category' => 'c and c',
                                     'disable'  => 'FEODO_RECOMMENDED' },
-             'FEODO_AGGRESIVE' => { 'name'     => 'Feodo Trojan IP Blocklist (Aggresive)',
+             'FEODO_AGGRESSIVE' => { 'name'     => 'Feodo Trojan IP Blocklist (Aggressive)',
                                     'url'      => 'https://feodotracker.abuse.ch/downloads/ipblocklist_aggressive.txt',
                                     'info'     => 'https://feodotracker.abuse.ch/blocklist',
                                     'parser'   => 'ip-or-net-list',

Re: ipblocklist - Call for testers (disable attribute in sources)

[Rob Brewer]

[-- Attachment #1: Type: text/plain, Size: 1548 bytes --]

Hi Charles

On Sunday 10 April 2022 19:21 Charles Brown wrote:

> Tim, Stefan,
> 
> I have installed the ipblocklist feature. It looks great.
> 
> I’m curious about the disable attribute in the sources file.
> 
> I have all the lists enabled, I would have thought enabling
> EMERGING_FWRULE would have the DSHIELD list automatically disabled.
> However, I am showing several hits on DSHIELD and I see 20 entries in
> ipset for DSHIELD. Is the disable attribute in sources there for
> informational purposes only?
> 
> Thanks for your excellent work on this feature,
> Charles Brown

I have been running Tim's original ipbl?list for about 2 months now and find I 
only need a few Bl?cklists enabled. I am mainly interrest in protecting port 25 
and find the most effective list is BLOCKLIST_DE.
CIARMY is very good at catching port scanners.
I also run a locally sourced blocklist and Banish which are optimised for port 
25.

I don't think it is a good idea to enable all of the lists and conflicting lists 
should be disabled by the original Attributes feature which you have noticed.

This was from my logs yesterday:

Blacklist		Category	Packets Dropped In	Packets Dropped Out
			Count	Percentage	Count	Percentage

BANISH		Attacker	74	0%	7	100%
BLOCKLIST_DE	Attacker	3615	8%	0	0%
CIARMY		Reputation	35598	77%	0	0%
EMERGING_COMPROMISED	Attacker	248	1%	0	0%
EMERGING_FWRULE	Composite	6235	13%	0	0%
LOCAL_BLOCKLIST	Attacker	575	1%	0	0%
SHODAN	Scanner	0	0%	0	0%
SPAMHAUS_EDROP	Reputation	4	0%	0	0%

Rob

Re: ipblocklist - Call for testers (disable attribute in sources)

[Rob Brewer]

[-- Attachment #1: Type: text/plain, Size: 8702 bytes --]

Hi Stefan, Tim

On Tuesday 12 April 2022 05:12 Stefan Schantl wrote:

> Hello Tim, hello *,
> 
> welcome back to the list and thanks for your detailed feedback.
>> Hello,
>> 
>> I've not had a chance to look at the code in detail, but a few
>> comments:
>> 
>> - The disable value in the sources file is intended to disable
>> conflicting sets when necessary.  It's partly in
>> reaction to people who just enable everything without thinking about
>> it, but it's also because for the blocklists
>> it's relatively easy to identify which lists are subsets of others.
>> A good example is the three Feodo lists which
>> are are different versions of the same list, but with different
>> probabilities of blocking legitimate traffic.
> 
> Thanks for clarify this. Theoretically it is no problem to enable and
> use all blocklists at the same time, they do not harm each other.
> But as you pointed this would be a massive waste of resources, because
> some of the lists are subsets of other lists.
> 
> When walking through the code of your main "ipblacklist" script, I
> didn't find any code to handle this. Did I overseen something or was
> that a planed feature for the future?
>> 

The disable code is working on Tim's original code and I have made good use 
of it when selecting my blacklists.  I'll have a look and see if I can find 
the relevant bits.


>> - I set the list to update every 15 minutes after discussion with
>> Michael, as some of the lists update very quickly
>> (every five minutes for some lists).  I don't think there's any
>> particular reason why a given update period is
>> preferred.
>
The 15 minute update is very useful for list that update once an hour since 
if the update runs just after the list updates you have another hour to wait 
before the your blocklist is updated. A 15 minute update will update the 
blocklist and will pick up the new data in less than 15 minutes.

I suggest you keep the 15 minute update as there doesn't seem to be any 
overhead for not doing so.
 
> Currently I set the call of the update script to one a hour via cron
> job. This easily can be increased or decreased.
>> 
>> - I had code to create either a hash:ip or had:net ipset.  The reason
>> for this is that apparently (if I remember
>> the documentation correctly), looking for an IP address against a
>> hash:net requires checking each of the possible
>> containing networks (/32, /31, /30...).  It thus takes significantly
>> longer than checking against a hash:ip.  As
>> the majority of lists are only single addresses this seemed like a
>> worthwhile optimisation.
> 
> I see - I only checked that single addresses also can be used in a
> net:hash and therefore simplified to generally use this kind of hash
> for the sets. I was not aware of the performance impact of this
> decision.
> 
> With this knowledge it absolutely make sense to bring back some code to
> handle this in a proper way.
>> 
These are the lists I am using and certainly seem to be sufficient for my 
use:

[root(a)griffith6 ~]# ipset -t list
Name: BANISH
Type: hash:net
Revision: 6
Header: family inet hashsize 256 maxelem 1024
Size in memory: 9152
References: 2
Number of entries: 191

Name: BLOCKLIST_DE
Type: hash:ip
Revision: 4
Header: family inet hashsize 32768 maxelem 65536
Size in memory: 794288
References: 2
Number of entries: 22985

Name: CIARMY
Type: hash:ip
Revision: 4
Header: family inet hashsize 16384 maxelem 32768
Size in memory: 471440
References: 2
Number of entries: 15000

Name: EMERGING_COMPROMISED
Type: hash:ip
Revision: 4
Header: family inet hashsize 1024 maxelem 2048
Size in memory: 28520
References: 2
Number of entries: 887

Name: EMERGING_FWRULE
Type: hash:net
Revision: 6
Header: family inet hashsize 2048 maxelem 4096
Size in memory: 72920
References: 2
Number of entries: 1600

Name: LOCAL_BLOCKLIST
Type: hash:net
Revision: 6
Header: family inet hashsize 4096 maxelem 8192
Size in memory: 145544
References: 2
Number of entries: 3283

Name: SHODAN
Type: hash:ip
Revision: 4
Header: family inet hashsize 64 maxelem 1024
Size in memory: 1784
References: 2
Number of entries: 44

Name: SPAMHAUS_EDROP
Type: hash:net
Revision: 6
Header: family inet hashsize 64 maxelem 1024
Size in memory: 2880
References: 2
Number of entries: 58

There are 4 hash:net and 4 hash:ip (2 of hash:nets are locally generated). I 
only have port 25 open so I have chosen lists which are most effective for 
smtp.  I also use Location block and fail2ban (IPTABLES) as additional 
blocklists .

The most effective lists blocking port 25 varies on a day by day basis and 
also time of day so each of my selected lists complement each other.

For me the selected lists seems to be a good combination. 


>> - It's probably worth reviewing the list of blocklists.  I would
>> certainly remove Alienvault, which seems to be
>> defunct.  Charles Brown suggested the 3COREsec lists, which look
>> interesting.  This could be left to post-first release.  As a note of
>> caution, I found some interesting lists for blocking command and
>> control servers which
>> were licenced under terms that I thought would make it difficult to
>> include in the IPFire distribution.
> 
> That sounds great. I'm also in favor to move a single provider from the
> IDS to the blocklist, because it only serves a set of addresses in his
> rulesfile and also serves a bare list on their website.
>> 

I think that would be a mistake since different sources seem to be optimized 
for different ports. Looking at my logs for port 23 the majority of blocks 
come from CIARMY and Location Block whereas SMTP blocks mostly come from 
BLOCKLIST_DE and my local blocklists.

As most users have different requirements they should be able to chose a 
blocklist that suits their environment.

>> - Longer term it may be worth adding the capability for a
>> sources.local, but again I think this is better left
>> until post-first release.  It may be worth adding a 'Do not edit this
>> file' in the sources file since it could
>> be overwritten by future updates.
>>

As above I am running 2 local blocklists one on IPFire's apache server and 
the other on the web server of my SMTP server. 

BANISH runs on IPFire and is useful to block addresses when you don't want 
to block whole countries with  Location Block but find abusive domains such 
as spamming outfits that don't appear on other blocklists. 

The one on my SMTP server catches the failed SMTP AUTH attempts from ip's 
that fall through the other blocklists.

>> - I think IPFire now has four different ways of using the Spamhaus
>> DROP list (IPS, hostile networks, XD country
>> code and IP Blocklist).  When this is released it may be worth
>> writing a blog post or a wiki page on the
>> advantages and disadvantages of each (or even a more general one
>> looking at all the different malware etc.
>> facilities).
> 
> I've the XD and Spamhaus running on my system and both are dropping
> packets. The huge amount will be match the XD but a few packets slip
> through and will be blocked by the more accurate list from ipblocklist.
> 
I use SPAMHAUS_EDROP but it catches very few IP's. Often the daily total is 
Zero. I also run Zen on my mail server but since I have been using 
IPblacklist I rarely see any blocks from spamhaus in the mail server logs, 
so presumably attempted connections are rejected before they reach the 
server.

Rob

> - Stefan
>> 
>> Tim
>> 


>> On 10/04/2022 19:47, Stefan Schantl wrote:
>> > Hello Charles,
>> > 
>> > thanks for a first look and your feedback.
>> > 
>> > Now after you have put my attention to those "disable" value in the
>> > sources file, I have to admit I totally have overseen this.
>> > 
>> > When interpreting this field and the values, it seems has been
>> > designed
>> > to automatically disable conflicting/obsolete sets in case such a
>> > set
>> > is enabled.
>> > 
>> > So for this feature I have go back to the drawing board and put
>> > some
>> > attention to it.
>> > 
>> > In the meantime please go on with testing and report any other
>> > issues.
>> > 
>> > Best regards,
>> > 
>> > -Stefan
>> > > Tim, Stefan,
>> > > I have installed the ipblocklist feature. It looks great.
>> > > I’m curious about the disable attribute in the sources file.
>> > > I have all the lists enabled, I would have thought enabling
>> > > EMERGING_FWRULE would have the DSHIELD list automatically
>> > > disabled.
>> > > However, I am showing several hits on DSHIELD and I see 20
>> > > entries in
>> > > ipset for DSHIELD. Is the disable attribute in sources there for
>> > > informational purposes only?
>> > > Thanks for your excellent work on this feature,
>> > > Charles Brown
>> > > 
>> > > 
>> > 
>> > 
>>

Re: ipblocklist - Call for testers (disable attribute in sources)

[Stefan Schantl]

[-- Attachment #1: Type: text/plain, Size: 4996 bytes --]

Hello Tim, hello *,

welcome back to the list and thanks for your detailed feedback.
> Hello,
> 
> I've not had a chance to look at the code in detail, but a few
> comments:
> 
> - The disable value in the sources file is intended to disable
> conflicting sets when necessary.  It's partly in
> reaction to people who just enable everything without thinking about
> it, but it's also because for the blocklists
> it's relatively easy to identify which lists are subsets of others. 
> A good example is the three Feodo lists which
> are are different versions of the same list, but with different
> probabilities of blocking legitimate traffic.

Thanks for clarify this. Theoretically it is no problem to enable and
use all blocklists at the same time, they do not harm each other.
But as you pointed this would be a massive waste of resources, because
some of the lists are subsets of other lists.

When walking through the code of your main "ipblacklist" script, I
didn't find any code to handle this. Did I overseen something or was
that a planed feature for the future?  
> 
> - I set the list to update every 15 minutes after discussion with
> Michael, as some of the lists update very quickly
> (every five minutes for some lists).  I don't think there's any
> particular reason why a given update period is
> preferred.

Currently I set the call of the update script to one a hour via cron
job. This easily can be increased or decreased.
> 
> - I had code to create either a hash:ip or had:net ipset.  The reason
> for this is that aparently (if I remember
> the documentation correctly), looking for an IP address against a
> hash:net requires checking each of the possible
> containing networks (/32, /31, /30...).  It thus takes significantly
> longer than checking against a hash:ip.  As
> the majority of lists are only single addresses this seemed like a
> worthwhile optimisation.

I see - I only checked that single addresses also can be used in a
net:hash and therefore simplified to generally use this kind of hash
for the sets. I was not aware of the performance impact of this
decision.

With this knowledge it absolutely make sense to bring back some code to
handle this in a proper way. 
> 
> - It's probably worth reviewing the list of blocklists.  I would
> certainly remove Alienvault, which seems to be
> defunct.  Charles Brown suggested the 3COREsec lists, which look
> interesting.  This could be left to post-first release.  As a note of
> caution, I found some interesting lists for blocking command and
> control servers which
> were licenced under terms that I thought would make it difficult to
> include in the IPFire distribution.

That sounds great. I'm also in favor to move a single provider from the
IDS to the blocklist, because it only serves a set of addresses in his
rulesfile and also serves a bare list on their website. 
> 
> - Longer term it may be worth adding the capability for a
> sources.local, but again I think this is better left
> until post-first release.  It may be worth adding a 'Do not edit this
> file' in the sources file since it could
> be overwritten by future updates.
> 
> - I think IPFire now has four different ways of using the Spamhaus
> DROP list (IPS, hostile networks, XD country
> code and IP Blocklist).  When this is released it may be worth
> writing a blog post or a wiki page on the
> advantages and disadvantages of each (or even a more general one
> looking at all the different malware etc.
> facilities).

I've the XD and Spamhaus running on my system and both are dropping
packets. The huge amount will be match the XD but a few packets slip
through and will be blocked by the more accurate list from ipblocklist.

- Stefan
> 
> Tim
> 
> On 10/04/2022 19:47, Stefan Schantl wrote:
> > Hello Charles,
> > 
> > thanks for a first look and your feedback.
> > 
> > Now after you have put my attention to those "disable" value in the
> > sources file, I have to admit I totally have overseen this.
> > 
> > When interpreting this field and the values, it seems has been
> > designed
> > to automatically disable conflicting/obsolete sets in case such a
> > set
> > is enabled.
> > 
> > So for this feature I have go back to the drawing board and put
> > some
> > attention to it.
> > 
> > In the meantime please go on with testing and report any other
> > issues.
> > 
> > Best regards,
> > 
> > -Stefan 
> > >  Tim, Stefan,
> > > I have installed the ipblocklist feature. It looks great.
> > > I’m curious about the disable attribute in the sources file. 
> > >  I have all the lists enabled, I would have thought enabling
> > > EMERGING_FWRULE would have the DSHIELD list automatically
> > > disabled.
> > > However, I am showing several hits on DSHIELD and I see 20
> > > entries in
> > > ipset for DSHIELD. Is the disable attribute in sources there for
> > > informational purposes only? 
> > > Thanks for your excellent work on this feature,
> > >  Charles Brown
> > > 
> > >  
> > 
> > 
> 

Re: ipblocklist - Call for testers (disable attribute in sources)

[Tim FitzGeorge]

[-- Attachment #1: Type: text/plain, Size: 3449 bytes --]

Hello,

I've not had a chance to look at the code in detail, but a few comments:

- The disable value in the sources file is intended to disable conflicting sets when necessary.  It's partly in
reaction to people who just enable everything without thinking about it, but it's also because for the blocklists
it's relatively easy to identify which lists are subsets of others.  A good example is the three Feodo lists which
are are different versions of the same list, but with different probabilities of blocking legitimate traffic.

- I set the list to update every 15 minutes after discussion with Michael, as some of the lists update very quickly
(every five minutes for some lists).  I don't think there's any particular reason why a given update period is
preferred.

- I had code to create either a hash:ip or had:net ipset.  The reason for this is that aparently (if I remember
the documentation correctly), looking for an IP address against a hash:net requires checking each of the possible
containing networks (/32, /31, /30...).  It thus takes significantly longer than checking against a hash:ip.  As
the majority of lists are only single addresses this seemed like a worthwhile optimisation.

- It's probably worth reviewing the list of blocklists.  I would certainly remove Alienvault, which seems to be
defunct.  Charles Brown suggested the 3COREsec lists, which look interesting.  This could be left to post-first release.  As a note of caution, I found some interesting lists for blocking command and control servers which
were licenced under terms that I thought would make it difficult to include in the IPFire distribution.

- Longer term it may be worth adding the capability for a sources.local, but again I think this is better left
until post-first release.  It may be worth adding a 'Do not edit this file' in the sources file since it could
be overwritten by future updates.

- I think IPFire now has four different ways of using the Spamhaus DROP list (IPS, hostile networks, XD country
code and IP Blocklist).  When this is released it may be worth writing a blog post or a wiki page on the
advantages and disadvantages of each (or even a more general one looking at all the different malware etc.
facilities).

Tim

On 10/04/2022 19:47, Stefan Schantl wrote:
> Hello Charles,
>
> thanks for a first look and your feedback.
>
> Now after you have put my attention to those "disable" value in the
> sources file, I have to admit I totally have overseen this.
>
> When interpreting this field and the values, it seems has been designed
> to automatically disable conflicting/obsolete sets in case such a set
> is enabled.
>
> So for this feature I have go back to the drawing board and put some
> attention to it.
>
> In the meantime please go on with testing and report any other issues.
>
> Best regards,
>
> -Stefan 
>>  Tim, Stefan,
>> I have installed the ipblocklist feature. It looks great.
>> I’m curious about the disable attribute in the sources file. 
>>  I have all the lists enabled, I would have thought enabling
>> EMERGING_FWRULE would have the DSHIELD list automatically disabled.
>> However, I am showing several hits on DSHIELD and I see 20 entries in
>> ipset for DSHIELD. Is the disable attribute in sources there for
>> informational purposes only? 
>> Thanks for your excellent work on this feature,
>>  Charles Brown
>>
>>  
>
>