IP Address Blacklists

IP Address Blacklists

[Tim FitzGeorge]

[-- Attachment #1: Type: text/plain, Size: 977 bytes --]

Hi,

I've pushed the my changes to implement IP Address Blacklists to the
repository at git://git.ipfire.org/people/timf/ipfire-2.x.git on the
ipblacklist branch.

As a result of discussions with Michael, this has a number of changes
from my first patch series:

- Removed autoblacklist.
- Added WUI log pages.
- Removed status from settings WUI page.
- Simplified download.
- Modified sources file 'rate' to allow unit to be specified.
- Updated sources file 'disable' to allow list to be specified.
- Changed Dshield download URL to preferred address.
- Removed Abuse.ch blacklist (discontinued).
- Removed Talos Malicious blacklist (not appropriate).
- Added Feodo recommended blacklist.
- Added blocklist.de all blacklist.
- Updated ignored messages in logwatch.

There's also some additional code on the addresscheck branch which adds
a WUI page that can check why a URL or address is being blocked.  It's
not production ready, but may possibly be useful in testing.

Tim

Re: IP Address Blacklists

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 1316 bytes --]

Hi Tim,

Thanks for your email.

I am not entirely sure what I can do now :)

> On 15 Feb 2020, at 15:40, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
> 
> Hi,
> 
> I've pushed the my changes to implement IP Address Blacklists to the
> repository at git://git.ipfire.org/people/timf/ipfire-2.x.git on the
> ipblacklist branch.
> 
> As a result of discussions with Michael, this has a number of changes
> from my first patch series:
> 
> - Removed autoblacklist.

Cool.

> - Added WUI log pages.
> - Removed status from settings WUI page.
> - Simplified download.
> - Modified sources file 'rate' to allow unit to be specified.
> - Updated sources file 'disable' to allow list to be specified.
> - Changed Dshield download URL to preferred address.
> - Removed Abuse.ch blacklist (discontinued).
> - Removed Talos Malicious blacklist (not appropriate).

Why is this one not appropriate?

> - Added Feodo recommended blacklist.
> - Added blocklist.de all blacklist.
> - Updated ignored messages in logwatch.

Are you going to submit this as a patchset again or how should we proceed?

> There's also some additional code on the addresscheck branch which adds
> a WUI page that can check why a URL or address is being blocked.  It's
> not production ready, but may possibly be useful in testing.
> 
> Tim

-Michael

Re: IP Address Blacklists

[ummeegge]

[-- Attachment #1: Type: text/plain, Size: 1489 bytes --]

Hi all,

Am Samstag, den 15.02.2020, 15:40 +0000 schrieb Tim FitzGeorge:
> Hi,
> 
> I've pushed the my changes to implement IP Address Blacklists to the
> repository at git://git.ipfire.org/people/timf/ipfire-2.x.git on the
> ipblacklist branch.
> 
> As a result of discussions with Michael, this has a number of changes
> from my first patch series:
> 
> - Removed autoblacklist.
> - Added WUI log pages.
> - Removed status from settings WUI page.
> - Simplified download.
> - Modified sources file 'rate' to allow unit to be specified.
> - Updated sources file 'disable' to allow list to be specified.
> - Changed Dshield download URL to preferred address.
> - Removed Abuse.ch blacklist (discontinued).
> - Removed Talos Malicious blacklist (not appropriate).
> - Added Feodo recommended blacklist.
> - Added blocklist.de all blacklist.
> - Updated ignored messages in logwatch.
> 
> There's also some additional code on the addresscheck branch which
> adds
> a WUI page that can check why a URL or address is being
> blocked.  It's
> not production ready, but may possibly be useful in testing.
> 
> Tim

thanks for your hard work here which looks great. 
As far as i can see, there are no possiblities to add own lists. Might
it be an idea for such a possibility ? I use currently e.g. lists from
firehol --> http://iplists.firehol.org/ via script and IPSet.
Am currently not sure how difficult it is to give the user there some
individuality to choose it´s own list ?

Best,

Erik

Re: IP Address Blacklists

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 1910 bytes --]

Hi,

> On 18 Feb 2020, at 16:49, ummeegge <ummeegge(a)ipfire.org> wrote:
> 
> Hi all,
> 
> Am Samstag, den 15.02.2020, 15:40 +0000 schrieb Tim FitzGeorge:
>> Hi,
>> 
>> I've pushed the my changes to implement IP Address Blacklists to the
>> repository at git://git.ipfire.org/people/timf/ipfire-2.x.git on the
>> ipblacklist branch.
>> 
>> As a result of discussions with Michael, this has a number of changes
>> from my first patch series:
>> 
>> - Removed autoblacklist.
>> - Added WUI log pages.
>> - Removed status from settings WUI page.
>> - Simplified download.
>> - Modified sources file 'rate' to allow unit to be specified.
>> - Updated sources file 'disable' to allow list to be specified.
>> - Changed Dshield download URL to preferred address.
>> - Removed Abuse.ch blacklist (discontinued).
>> - Removed Talos Malicious blacklist (not appropriate).
>> - Added Feodo recommended blacklist.
>> - Added blocklist.de all blacklist.
>> - Updated ignored messages in logwatch.
>> 
>> There's also some additional code on the addresscheck branch which
>> adds
>> a WUI page that can check why a URL or address is being
>> blocked.  It's
>> not production ready, but may possibly be useful in testing.
>> 
>> Tim
> 
> thanks for your hard work here which looks great. 
> As far as i can see, there are no possiblities to add own lists. Might
> it be an idea for such a possibility ? I use currently e.g. lists from
> firehol --> http://iplists.firehol.org/ via script and IPSet.
> Am currently not sure how difficult it is to give the user there some
> individuality to choose it´s own list ?

We currently do not allow this for the IPS either.

And I am not really sure if we should. Why would we not add the lists for all users if we see any value in them.

What reasons are there to allow users to do their own thing?

> 
> Best,
> 
> Erik
> 

Re: IP Address Blacklists

[ummeegge]

[-- Attachment #1: Type: text/plain, Size: 2640 bytes --]

Hi Michael,

Am Mittwoch, den 19.02.2020, 11:52 +0000 schrieb Michael Tremer:
> Hi,
> 
> > On 18 Feb 2020, at 16:49, ummeegge <ummeegge(a)ipfire.org> wrote:
> > 
> > Hi all,
> > 
> > Am Samstag, den 15.02.2020, 15:40 +0000 schrieb Tim FitzGeorge:
> > > Hi,
> > > 
> > > I've pushed the my changes to implement IP Address Blacklists to
> > > the
> > > repository at git://git.ipfire.org/people/timf/ipfire-2.x.git on
> > > the
> > > ipblacklist branch.
> > > 
> > > As a result of discussions with Michael, this has a number of
> > > changes
> > > from my first patch series:
> > > 
> > > - Removed autoblacklist.
> > > - Added WUI log pages.
> > > - Removed status from settings WUI page.
> > > - Simplified download.
> > > - Modified sources file 'rate' to allow unit to be specified.
> > > - Updated sources file 'disable' to allow list to be specified.
> > > - Changed Dshield download URL to preferred address.
> > > - Removed Abuse.ch blacklist (discontinued).
> > > - Removed Talos Malicious blacklist (not appropriate).
> > > - Added Feodo recommended blacklist.
> > > - Added blocklist.de all blacklist.
> > > - Updated ignored messages in logwatch.
> > > 
> > > There's also some additional code on the addresscheck branch
> > > which
> > > adds
> > > a WUI page that can check why a URL or address is being
> > > blocked.  It's
> > > not production ready, but may possibly be useful in testing.
> > > 
> > > Tim
> > 
> > thanks for your hard work here which looks great. 
> > As far as i can see, there are no possiblities to add own lists.
> > Might
> > it be an idea for such a possibility ? I use currently e.g. lists
> > from
> > firehol --> http://iplists.firehol.org/ via script and IPSet.
> > Am currently not sure how difficult it is to give the user there
> > some
> > individuality to choose it´s own list ?
> 
> We currently do not allow this for the IPS either.
> 
> And I am not really sure if we should. Why would we not add the lists
> for all users if we see any value in them.
> 
> What reasons are there to allow users to do their own thing?
Use cases can be different e.g. i remeber a project in the old forum
which was about a company blocker (facebook, Windows, Apple) or in
general the whole telemetry stuff can also be unwanted and there are
some lists out there which can help to block also the "good" ones. If
there are own vast lists of unwanted IPs, IPSet which is working here,
is then the best way to do so, therefor my idea to bring in some
flexibility in this great project to prevent scripting around in
parallel for, let´s say, doing the same twice.

> 
> > 
> > Best,
> > 
> > Erik
> > 
> 
> 

Re: IP Address Blacklists

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 2799 bytes --]

Hi,

> On 19 Feb 2020, at 17:13, ummeegge <ummeegge(a)ipfire.org> wrote:
> 
> Hi Michael,
> 
> Am Mittwoch, den 19.02.2020, 11:52 +0000 schrieb Michael Tremer:
>> Hi,
>> 
>>> On 18 Feb 2020, at 16:49, ummeegge <ummeegge(a)ipfire.org> wrote:
>>> 
>>> Hi all,
>>> 
>>> Am Samstag, den 15.02.2020, 15:40 +0000 schrieb Tim FitzGeorge:
>>>> Hi,
>>>> 
>>>> I've pushed the my changes to implement IP Address Blacklists to
>>>> the
>>>> repository at git://git.ipfire.org/people/timf/ipfire-2.x.git on
>>>> the
>>>> ipblacklist branch.
>>>> 
>>>> As a result of discussions with Michael, this has a number of
>>>> changes
>>>> from my first patch series:
>>>> 
>>>> - Removed autoblacklist.
>>>> - Added WUI log pages.
>>>> - Removed status from settings WUI page.
>>>> - Simplified download.
>>>> - Modified sources file 'rate' to allow unit to be specified.
>>>> - Updated sources file 'disable' to allow list to be specified.
>>>> - Changed Dshield download URL to preferred address.
>>>> - Removed Abuse.ch blacklist (discontinued).
>>>> - Removed Talos Malicious blacklist (not appropriate).
>>>> - Added Feodo recommended blacklist.
>>>> - Added blocklist.de all blacklist.
>>>> - Updated ignored messages in logwatch.
>>>> 
>>>> There's also some additional code on the addresscheck branch
>>>> which
>>>> adds
>>>> a WUI page that can check why a URL or address is being
>>>> blocked.  It's
>>>> not production ready, but may possibly be useful in testing.
>>>> 
>>>> Tim
>>> 
>>> thanks for your hard work here which looks great. 
>>> As far as i can see, there are no possiblities to add own lists.
>>> Might
>>> it be an idea for such a possibility ? I use currently e.g. lists
>>> from
>>> firehol --> http://iplists.firehol.org/ via script and IPSet.
>>> Am currently not sure how difficult it is to give the user there
>>> some
>>> individuality to choose it´s own list ?
>> 
>> We currently do not allow this for the IPS either.
>> 
>> And I am not really sure if we should. Why would we not add the lists
>> for all users if we see any value in them.
>> 
>> What reasons are there to allow users to do their own thing?
> Use cases can be different e.g. i remeber a project in the old forum
> which was about a company blocker (facebook, Windows, Apple) or in
> general the whole telemetry stuff can also be unwanted and there are
> some lists out there which can help to block also the "good" ones. If
> there are own vast lists of unwanted IPs, IPSet which is working here,
> is then the best way to do so, therefor my idea to bring in some
> flexibility in this great project to prevent scripting around in
> parallel for, let´s say, doing the same twice.

Is it not better to block the whole AS of those companies in the firewall?

> 
>> 
>>> 
>>> Best,
>>> 
>>> Erik
>>> 
>> 
>> 
> 

Re: IP Address Blacklists

[ummeegge]

[-- Attachment #1: Type: text/plain, Size: 4445 bytes --]

Hi Michael,

Am Mittwoch, den 19.02.2020, 17:21 +0000 schrieb Michael Tremer:
> Hi,
> 
> > On 19 Feb 2020, at 17:13, ummeegge <ummeegge(a)ipfire.org> wrote:
> > 
> > Hi Michael,
> > 
> > Am Mittwoch, den 19.02.2020, 11:52 +0000 schrieb Michael Tremer:
> > > Hi,
> > > 
> > > > On 18 Feb 2020, at 16:49, ummeegge <ummeegge(a)ipfire.org> wrote:
> > > > 
> > > > Hi all,
> > > > 
> > > > Am Samstag, den 15.02.2020, 15:40 +0000 schrieb Tim FitzGeorge:
> > > > > Hi,
> > > > > 
> > > > > I've pushed the my changes to implement IP Address Blacklists
> > > > > to
> > > > > the
> > > > > repository at git://git.ipfire.org/people/timf/ipfire-2.x.git 
> > > > > on
> > > > > the
> > > > > ipblacklist branch.
> > > > > 
> > > > > As a result of discussions with Michael, this has a number of
> > > > > changes
> > > > > from my first patch series:
> > > > > 
> > > > > - Removed autoblacklist.
> > > > > - Added WUI log pages.
> > > > > - Removed status from settings WUI page.
> > > > > - Simplified download.
> > > > > - Modified sources file 'rate' to allow unit to be specified.
> > > > > - Updated sources file 'disable' to allow list to be
> > > > > specified.
> > > > > - Changed Dshield download URL to preferred address.
> > > > > - Removed Abuse.ch blacklist (discontinued).
> > > > > - Removed Talos Malicious blacklist (not appropriate).
> > > > > - Added Feodo recommended blacklist.
> > > > > - Added blocklist.de all blacklist.
> > > > > - Updated ignored messages in logwatch.
> > > > > 
> > > > > There's also some additional code on the addresscheck branch
> > > > > which
> > > > > adds
> > > > > a WUI page that can check why a URL or address is being
> > > > > blocked.  It's
> > > > > not production ready, but may possibly be useful in testing.
> > > > > 
> > > > > Tim
> > > > 
> > > > thanks for your hard work here which looks great. 
> > > > As far as i can see, there are no possiblities to add own
> > > > lists.
> > > > Might
> > > > it be an idea for such a possibility ? I use currently e.g.
> > > > lists
> > > > from
> > > > firehol --> http://iplists.firehol.org/ via script and IPSet.
> > > > Am currently not sure how difficult it is to give the user
> > > > there
> > > > some
> > > > individuality to choose it´s own list ?
> > > 
> > > We currently do not allow this for the IPS either.
> > > 
> > > And I am not really sure if we should. Why would we not add the
> > > lists
> > > for all users if we see any value in them.
> > > 
> > > What reasons are there to allow users to do their own thing?
> > 
> > Use cases can be different e.g. i remeber a project in the old
> > forum
> > which was about a company blocker (facebook, Windows, Apple) or in
> > general the whole telemetry stuff can also be unwanted and there
> > are
> > some lists out there which can help to block also the "good" ones.
> > If
> > there are own vast lists of unwanted IPs, IPSet which is working
> > here,
> > is then the best way to do so, therefor my idea to bring in some
> > flexibility in this great project to prevent scripting around in
> > parallel for, let´s say, doing the same twice.
> 
> Is it not better to block the whole AS of those companies in the
> firewall?
The performance gain with IPSet causing the hash table can be
significant --> 
https://workshop.netfilter.org/2013/wiki/images/a/ab/Jozsef_Kadlecsik_ipset-osd-public.pdf#page=10
<-- haven´t found some new performance tests but i thnk the results
today are related close to another. Have experienced it by my own (with
my ALIX back in the days) that the system was not usable after the try
to handle some thousands IPs/CIDRs via IPTables (therefore i pushed
IPSet at that time). Since this project delivers WUI access to IPSet,
which is really great, beneath the really practical handling of block
lists, some more advantages are may on the doorstep ? 

We would have also a comparable design with the Proxy/URL-Filter
whereby the user do also handle the challenge/possibility to
upload/download/integrate own lists.

Sorry for appearing may a little unhumble which i really do not want
and am also really not sure how much more work that costs but wanted to
bring on some ideas and am nonetheless thankful with the already made
process.

> 
> > 
> > > 
> > > > 
> > > > Best,
> > > > 
> > > > Erik
> > > > 
> > > 
> > > 
> 
> 

Thoughts regarding IP-based ad blockers (was: Re: IP Address Blacklists)

[Peter Müller]

[-- Attachment #1: Type: text/plain, Size: 1950 bytes --]

Hello *,

as a footnote on including blacklist feeds covering not inherently
malicious networks, I do not think it is wise to overdo things here.

In my point of view, the main intention of Tim's effort was to drop
connections from criminal or heavily abused IP addresses/networks
efficiently. Compared to what we have today, this is a huge improvement
of which nobody I think is in denial.

However, we must keep in mind dropping packets from and to certain
destinations can cause massive confusion and endless troubleshooting
if somebody sits behind such a firewall without being informed about
this.

>From my own experience, convincing (network) administration staff and/or
management folks to do take step can be quite challenging - it is hard
to imagine they will like a transparent advertisement blocker at IP level
better.

Further, this reminds me of projects like PiHole - which is basically
doing the same thing at DNS level - which I never liked at all. If
anyone intends to block or filter traffic to certain destinations, non-
transparent proxies are the way to do it: Everybody is aware of something
between client and server and does not take unlimited connectivity for
granted. Everybody can easily tell the difference between connection
failures due to network issues and policy reasons.

PiHole et al. exists because we unfortunately have to deal with devices
(a) lacking support for HTTP(S) proxies
(b) connecting to advertisement, tracking or even worse destinations.

Personally, I made good experience in strictly enforcing proxy support:
If a device lacks it, it will not get any internet connectivity. Period.

Thereof, I suggest not to include non-malicious blacklists in this feature
and attempt to ship a first operational version of it rather than arguing
for a long time about possible improvements or disadvantages. I certainly
have to swipe my own hallway first on this... :-)

Thanks, and best regards,
Peter Müller

Re: IP Address Blacklists

[Tim FitzGeorge]

[-- Attachment #1: Type: text/plain, Size: 5791 bytes --]

On 19/02/2020 18:43, ummeegge wrote:


Hi,

On 19/02/2020 18:43, ummeegge wrote:
> Hi Michael,
>
> Am Mittwoch, den 19.02.2020, 17:21 +0000 schrieb Michael Tremer:
>> Hi,
>>
>>> On 19 Feb 2020, at 17:13, ummeegge <ummeegge(a)ipfire.org> wrote:
>>>
>>> Hi Michael,
>>>
>>> Am Mittwoch, den 19.02.2020, 11:52 +0000 schrieb Michael Tremer:
>>>> Hi,
>>>>
>>>>> On 18 Feb 2020, at 16:49, ummeegge <ummeegge(a)ipfire.org> wrote:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> Am Samstag, den 15.02.2020, 15:40 +0000 schrieb Tim FitzGeorge:
>>>>>> Hi,
>>>>>>
>>>>>> I've pushed the my changes to implement IP Address Blacklists
>>>>>> to
>>>>>> the
>>>>>> repository at git://git.ipfire.org/people/timf/ipfire-2.x.git
>>>>>> on
>>>>>> the
>>>>>> ipblacklist branch.
>>>>>>
>>>>>> As a result of discussions with Michael, this has a number of
>>>>>> changes
>>>>>> from my first patch series:
>>>>>>
>>>>>> - Removed autoblacklist.
>>>>>> - Added WUI log pages.
>>>>>> - Removed status from settings WUI page.
>>>>>> - Simplified download.
>>>>>> - Modified sources file 'rate' to allow unit to be specified.
>>>>>> - Updated sources file 'disable' to allow list to be
>>>>>> specified.
>>>>>> - Changed Dshield download URL to preferred address.
>>>>>> - Removed Abuse.ch blacklist (discontinued).
>>>>>> - Removed Talos Malicious blacklist (not appropriate).
>>>>>> - Added Feodo recommended blacklist.
>>>>>> - Added blocklist.de all blacklist.
>>>>>> - Updated ignored messages in logwatch.
>>>>>>
>>>>>> There's also some additional code on the addresscheck branch
>>>>>> which
>>>>>> adds
>>>>>> a WUI page that can check why a URL or address is being
>>>>>> blocked.  It's
>>>>>> not production ready, but may possibly be useful in testing.
>>>>>>
>>>>>> Tim
>>>>>
>>>>> thanks for your hard work here which looks great.
>>>>> As far as i can see, there are no possiblities to add own
>>>>> lists.
>>>>> Might
>>>>> it be an idea for such a possibility ? I use currently e.g.
>>>>> lists
>>>>> from
>>>>> firehol --> http://iplists.firehol.org/ via script and IPSet.
>>>>> Am currently not sure how difficult it is to give the user
>>>>> there
>>>>> some
>>>>> individuality to choose it´s own list ?

There's currently no way for a user to add their own list, apart from
editing the text file specifying the list sources.  This would obviously
be overwritten by updates, but it wouldn't be difficult to allow for a
second locally defined file to be read as well and merged with the
distributed list.  The WUI could be updated to allow this list to be edited.

My preference is to get the initial version in IPFire and sort out any
points that people raise before adding additional features.
>>>>
>>>> We currently do not allow this for the IPS either.
The IPS can have local rules defined in a file.

>>>>
>>>> And I am not really sure if we should. Why would we not add the
>>>> lists
>>>> for all users if we see any value in them.
>>>>
>>>> What reasons are there to allow users to do their own thing?
I agree that we should add additional lists where they're useful, but
there are a lot of lists out there.  Look at http://iplists.firehol.org/
- there's a list of monitored lists on the left hand side, down the page
a bit.  There's just under 400 of them.  We don't want to put that many
into IPFire because it would be too confusing, but it's possible that
they could be useful to an 'advanced' user.

There are also lists like the ones from Bambenek Consulting
(http://osint.bambenekconsulting.com/feeds/) which look interesting
since they target individual malware C&C channels, but the licence
doesn't cover commercial use.  I don't think these could be included in
IPFire because of the licence (unless we extend the WUI to include the
licence in some way), but they could certainly be of use to individuals
or charities.

Tim

>>>
>>> Use cases can be different e.g. i remeber a project in the old
>>> forum
>>> which was about a company blocker (facebook, Windows, Apple) or in
>>> general the whole telemetry stuff can also be unwanted and there
>>> are
>>> some lists out there which can help to block also the "good" ones.
>>> If
>>> there are own vast lists of unwanted IPs, IPSet which is working
>>> here,
>>> is then the best way to do so, therefor my idea to bring in some
>>> flexibility in this great project to prevent scripting around in
>>> parallel for, let´s say, doing the same twice.
>>
>> Is it not better to block the whole AS of those companies in the
>> firewall?
> The performance gain with IPSet causing the hash table can be
> significant -->
>
https://workshop.netfilter.org/2013/wiki/images/a/ab/Jozsef_Kadlecsik_ipset-osd-public.pdf#page=10
> <-- haven´t found some new performance tests but i thnk the results
> today are related close to another. Have experienced it by my own (with
> my ALIX back in the days) that the system was not usable after the try
> to handle some thousands IPs/CIDRs via IPTables (therefore i pushed
> IPSet at that time). Since this project delivers WUI access to IPSet,
> which is really great, beneath the really practical handling of block
> lists, some more advantages are may on the doorstep ?
>
> We would have also a comparable design with the Proxy/URL-Filter
> whereby the user do also handle the challenge/possibility to
> upload/download/integrate own lists.
>
> Sorry for appearing may a little unhumble which i really do not want
> and am also really not sure how much more work that costs but wanted to
> bring on some ideas and am nonetheless thankful with the already made
> process.
>
>>
>>>
>>>>
>>>>>
>>>>> Best,
>>>>>
>>>>> Erik
>>>>>
>>>>
>>>>
>>
>>
>