Hi,
I've pushed the my changes to implement IP Address Blacklists to the repository at git://git.ipfire.org/people/timf/ipfire-2.x.git on the ipblacklist branch.
As a result of discussions with Michael, this has a number of changes from my first patch series:
- Removed autoblacklist. - Added WUI log pages. - Removed status from settings WUI page. - Simplified download. - Modified sources file 'rate' to allow unit to be specified. - Updated sources file 'disable' to allow list to be specified. - Changed Dshield download URL to preferred address. - Removed Abuse.ch blacklist (discontinued). - Removed Talos Malicious blacklist (not appropriate). - Added Feodo recommended blacklist. - Added blocklist.de all blacklist. - Updated ignored messages in logwatch.
There's also some additional code on the addresscheck branch which adds a WUI page that can check why a URL or address is being blocked. It's not production ready, but may possibly be useful in testing.
Tim
Hi Tim,
Thanks for your email.
I am not entirely sure what I can do now :)
On 15 Feb 2020, at 15:40, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Hi,
I've pushed the my changes to implement IP Address Blacklists to the repository at git://git.ipfire.org/people/timf/ipfire-2.x.git on the ipblacklist branch.
As a result of discussions with Michael, this has a number of changes from my first patch series:
- Removed autoblacklist.
Cool.
- Added WUI log pages.
- Removed status from settings WUI page.
- Simplified download.
- Modified sources file 'rate' to allow unit to be specified.
- Updated sources file 'disable' to allow list to be specified.
- Changed Dshield download URL to preferred address.
- Removed Abuse.ch blacklist (discontinued).
- Removed Talos Malicious blacklist (not appropriate).
Why is this one not appropriate?
- Added Feodo recommended blacklist.
- Added blocklist.de all blacklist.
- Updated ignored messages in logwatch.
Are you going to submit this as a patchset again or how should we proceed?
There's also some additional code on the addresscheck branch which adds a WUI page that can check why a URL or address is being blocked. It's not production ready, but may possibly be useful in testing.
Tim
-Michael
Hi all,
Am Samstag, den 15.02.2020, 15:40 +0000 schrieb Tim FitzGeorge:
Hi,
I've pushed the my changes to implement IP Address Blacklists to the repository at git://git.ipfire.org/people/timf/ipfire-2.x.git on the ipblacklist branch.
As a result of discussions with Michael, this has a number of changes from my first patch series:
- Removed autoblacklist.
- Added WUI log pages.
- Removed status from settings WUI page.
- Simplified download.
- Modified sources file 'rate' to allow unit to be specified.
- Updated sources file 'disable' to allow list to be specified.
- Changed Dshield download URL to preferred address.
- Removed Abuse.ch blacklist (discontinued).
- Removed Talos Malicious blacklist (not appropriate).
- Added Feodo recommended blacklist.
- Added blocklist.de all blacklist.
- Updated ignored messages in logwatch.
There's also some additional code on the addresscheck branch which adds a WUI page that can check why a URL or address is being blocked. It's not production ready, but may possibly be useful in testing.
Tim
thanks for your hard work here which looks great. As far as i can see, there are no possiblities to add own lists. Might it be an idea for such a possibility ? I use currently e.g. lists from firehol --> http://iplists.firehol.org/ via script and IPSet. Am currently not sure how difficult it is to give the user there some individuality to choose it´s own list ?
Best,
Erik
Hi,
On 18 Feb 2020, at 16:49, ummeegge ummeegge@ipfire.org wrote:
Hi all,
Am Samstag, den 15.02.2020, 15:40 +0000 schrieb Tim FitzGeorge:
Hi,
I've pushed the my changes to implement IP Address Blacklists to the repository at git://git.ipfire.org/people/timf/ipfire-2.x.git on the ipblacklist branch.
As a result of discussions with Michael, this has a number of changes from my first patch series:
- Removed autoblacklist.
- Added WUI log pages.
- Removed status from settings WUI page.
- Simplified download.
- Modified sources file 'rate' to allow unit to be specified.
- Updated sources file 'disable' to allow list to be specified.
- Changed Dshield download URL to preferred address.
- Removed Abuse.ch blacklist (discontinued).
- Removed Talos Malicious blacklist (not appropriate).
- Added Feodo recommended blacklist.
- Added blocklist.de all blacklist.
- Updated ignored messages in logwatch.
There's also some additional code on the addresscheck branch which adds a WUI page that can check why a URL or address is being blocked. It's not production ready, but may possibly be useful in testing.
Tim
thanks for your hard work here which looks great. As far as i can see, there are no possiblities to add own lists. Might it be an idea for such a possibility ? I use currently e.g. lists from firehol --> http://iplists.firehol.org/ via script and IPSet. Am currently not sure how difficult it is to give the user there some individuality to choose it´s own list ?
We currently do not allow this for the IPS either.
And I am not really sure if we should. Why would we not add the lists for all users if we see any value in them.
What reasons are there to allow users to do their own thing?
Best,
Erik
Hi Michael,
Am Mittwoch, den 19.02.2020, 11:52 +0000 schrieb Michael Tremer:
Hi,
On 18 Feb 2020, at 16:49, ummeegge ummeegge@ipfire.org wrote:
Hi all,
Am Samstag, den 15.02.2020, 15:40 +0000 schrieb Tim FitzGeorge:
Hi,
I've pushed the my changes to implement IP Address Blacklists to the repository at git://git.ipfire.org/people/timf/ipfire-2.x.git on the ipblacklist branch.
As a result of discussions with Michael, this has a number of changes from my first patch series:
- Removed autoblacklist.
- Added WUI log pages.
- Removed status from settings WUI page.
- Simplified download.
- Modified sources file 'rate' to allow unit to be specified.
- Updated sources file 'disable' to allow list to be specified.
- Changed Dshield download URL to preferred address.
- Removed Abuse.ch blacklist (discontinued).
- Removed Talos Malicious blacklist (not appropriate).
- Added Feodo recommended blacklist.
- Added blocklist.de all blacklist.
- Updated ignored messages in logwatch.
There's also some additional code on the addresscheck branch which adds a WUI page that can check why a URL or address is being blocked. It's not production ready, but may possibly be useful in testing.
Tim
thanks for your hard work here which looks great. As far as i can see, there are no possiblities to add own lists. Might it be an idea for such a possibility ? I use currently e.g. lists from firehol --> http://iplists.firehol.org/ via script and IPSet. Am currently not sure how difficult it is to give the user there some individuality to choose it´s own list ?
We currently do not allow this for the IPS either.
And I am not really sure if we should. Why would we not add the lists for all users if we see any value in them.
What reasons are there to allow users to do their own thing?
Use cases can be different e.g. i remeber a project in the old forum which was about a company blocker (facebook, Windows, Apple) or in general the whole telemetry stuff can also be unwanted and there are some lists out there which can help to block also the "good" ones. If there are own vast lists of unwanted IPs, IPSet which is working here, is then the best way to do so, therefor my idea to bring in some flexibility in this great project to prevent scripting around in parallel for, let´s say, doing the same twice.
Best,
Erik
Hi,
On 19 Feb 2020, at 17:13, ummeegge ummeegge@ipfire.org wrote:
Hi Michael,
Am Mittwoch, den 19.02.2020, 11:52 +0000 schrieb Michael Tremer:
Hi,
On 18 Feb 2020, at 16:49, ummeegge ummeegge@ipfire.org wrote:
Hi all,
Am Samstag, den 15.02.2020, 15:40 +0000 schrieb Tim FitzGeorge:
Hi,
I've pushed the my changes to implement IP Address Blacklists to the repository at git://git.ipfire.org/people/timf/ipfire-2.x.git on the ipblacklist branch.
As a result of discussions with Michael, this has a number of changes from my first patch series:
- Removed autoblacklist.
- Added WUI log pages.
- Removed status from settings WUI page.
- Simplified download.
- Modified sources file 'rate' to allow unit to be specified.
- Updated sources file 'disable' to allow list to be specified.
- Changed Dshield download URL to preferred address.
- Removed Abuse.ch blacklist (discontinued).
- Removed Talos Malicious blacklist (not appropriate).
- Added Feodo recommended blacklist.
- Added blocklist.de all blacklist.
- Updated ignored messages in logwatch.
There's also some additional code on the addresscheck branch which adds a WUI page that can check why a URL or address is being blocked. It's not production ready, but may possibly be useful in testing.
Tim
thanks for your hard work here which looks great. As far as i can see, there are no possiblities to add own lists. Might it be an idea for such a possibility ? I use currently e.g. lists from firehol --> http://iplists.firehol.org/ via script and IPSet. Am currently not sure how difficult it is to give the user there some individuality to choose it´s own list ?
We currently do not allow this for the IPS either.
And I am not really sure if we should. Why would we not add the lists for all users if we see any value in them.
What reasons are there to allow users to do their own thing?
Use cases can be different e.g. i remeber a project in the old forum which was about a company blocker (facebook, Windows, Apple) or in general the whole telemetry stuff can also be unwanted and there are some lists out there which can help to block also the "good" ones. If there are own vast lists of unwanted IPs, IPSet which is working here, is then the best way to do so, therefor my idea to bring in some flexibility in this great project to prevent scripting around in parallel for, let´s say, doing the same twice.
Is it not better to block the whole AS of those companies in the firewall?
Best,
Erik
Hi Michael,
Am Mittwoch, den 19.02.2020, 17:21 +0000 schrieb Michael Tremer:
Hi,
On 19 Feb 2020, at 17:13, ummeegge ummeegge@ipfire.org wrote:
Hi Michael,
Am Mittwoch, den 19.02.2020, 11:52 +0000 schrieb Michael Tremer:
Hi,
On 18 Feb 2020, at 16:49, ummeegge ummeegge@ipfire.org wrote:
Hi all,
Am Samstag, den 15.02.2020, 15:40 +0000 schrieb Tim FitzGeorge:
Hi,
I've pushed the my changes to implement IP Address Blacklists to the repository at git://git.ipfire.org/people/timf/ipfire-2.x.git on the ipblacklist branch.
As a result of discussions with Michael, this has a number of changes from my first patch series:
- Removed autoblacklist.
- Added WUI log pages.
- Removed status from settings WUI page.
- Simplified download.
- Modified sources file 'rate' to allow unit to be specified.
- Updated sources file 'disable' to allow list to be
specified.
- Changed Dshield download URL to preferred address.
- Removed Abuse.ch blacklist (discontinued).
- Removed Talos Malicious blacklist (not appropriate).
- Added Feodo recommended blacklist.
- Added blocklist.de all blacklist.
- Updated ignored messages in logwatch.
There's also some additional code on the addresscheck branch which adds a WUI page that can check why a URL or address is being blocked. It's not production ready, but may possibly be useful in testing.
Tim
thanks for your hard work here which looks great. As far as i can see, there are no possiblities to add own lists. Might it be an idea for such a possibility ? I use currently e.g. lists from firehol --> http://iplists.firehol.org/ via script and IPSet. Am currently not sure how difficult it is to give the user there some individuality to choose it´s own list ?
We currently do not allow this for the IPS either.
And I am not really sure if we should. Why would we not add the lists for all users if we see any value in them.
What reasons are there to allow users to do their own thing?
Use cases can be different e.g. i remeber a project in the old forum which was about a company blocker (facebook, Windows, Apple) or in general the whole telemetry stuff can also be unwanted and there are some lists out there which can help to block also the "good" ones. If there are own vast lists of unwanted IPs, IPSet which is working here, is then the best way to do so, therefor my idea to bring in some flexibility in this great project to prevent scripting around in parallel for, let´s say, doing the same twice.
Is it not better to block the whole AS of those companies in the firewall?
The performance gain with IPSet causing the hash table can be significant --> https://workshop.netfilter.org/2013/wiki/images/a/ab/Jozsef_Kadlecsik_ipset-... <-- haven´t found some new performance tests but i thnk the results today are related close to another. Have experienced it by my own (with my ALIX back in the days) that the system was not usable after the try to handle some thousands IPs/CIDRs via IPTables (therefore i pushed IPSet at that time). Since this project delivers WUI access to IPSet, which is really great, beneath the really practical handling of block lists, some more advantages are may on the doorstep ?
We would have also a comparable design with the Proxy/URL-Filter whereby the user do also handle the challenge/possibility to upload/download/integrate own lists.
Sorry for appearing may a little unhumble which i really do not want and am also really not sure how much more work that costs but wanted to bring on some ideas and am nonetheless thankful with the already made process.
Best,
Erik
Hello *,
as a footnote on including blacklist feeds covering not inherently malicious networks, I do not think it is wise to overdo things here.
In my point of view, the main intention of Tim's effort was to drop connections from criminal or heavily abused IP addresses/networks efficiently. Compared to what we have today, this is a huge improvement of which nobody I think is in denial.
However, we must keep in mind dropping packets from and to certain destinations can cause massive confusion and endless troubleshooting if somebody sits behind such a firewall without being informed about this.
From my own experience, convincing (network) administration staff and/or
management folks to do take step can be quite challenging - it is hard to imagine they will like a transparent advertisement blocker at IP level better.
Further, this reminds me of projects like PiHole - which is basically doing the same thing at DNS level - which I never liked at all. If anyone intends to block or filter traffic to certain destinations, non- transparent proxies are the way to do it: Everybody is aware of something between client and server and does not take unlimited connectivity for granted. Everybody can easily tell the difference between connection failures due to network issues and policy reasons.
PiHole et al. exists because we unfortunately have to deal with devices (a) lacking support for HTTP(S) proxies (b) connecting to advertisement, tracking or even worse destinations.
Personally, I made good experience in strictly enforcing proxy support: If a device lacks it, it will not get any internet connectivity. Period.
Thereof, I suggest not to include non-malicious blacklists in this feature and attempt to ship a first operational version of it rather than arguing for a long time about possible improvements or disadvantages. I certainly have to swipe my own hallway first on this... :-)
Thanks, and best regards, Peter Müller
On 19/02/2020 18:43, ummeegge wrote:
Hi,
On 19/02/2020 18:43, ummeegge wrote:
Hi Michael,
Am Mittwoch, den 19.02.2020, 17:21 +0000 schrieb Michael Tremer:
Hi,
On 19 Feb 2020, at 17:13, ummeegge ummeegge@ipfire.org wrote:
Hi Michael,
Am Mittwoch, den 19.02.2020, 11:52 +0000 schrieb Michael Tremer:
Hi,
On 18 Feb 2020, at 16:49, ummeegge ummeegge@ipfire.org wrote:
Hi all,
Am Samstag, den 15.02.2020, 15:40 +0000 schrieb Tim FitzGeorge:
Hi,
I've pushed the my changes to implement IP Address Blacklists to the repository at git://git.ipfire.org/people/timf/ipfire-2.x.git on the ipblacklist branch.
As a result of discussions with Michael, this has a number of changes from my first patch series:
- Removed autoblacklist.
- Added WUI log pages.
- Removed status from settings WUI page.
- Simplified download.
- Modified sources file 'rate' to allow unit to be specified.
- Updated sources file 'disable' to allow list to be
specified.
- Changed Dshield download URL to preferred address.
- Removed Abuse.ch blacklist (discontinued).
- Removed Talos Malicious blacklist (not appropriate).
- Added Feodo recommended blacklist.
- Added blocklist.de all blacklist.
- Updated ignored messages in logwatch.
There's also some additional code on the addresscheck branch which adds a WUI page that can check why a URL or address is being blocked. It's not production ready, but may possibly be useful in testing.
Tim
thanks for your hard work here which looks great. As far as i can see, there are no possiblities to add own lists. Might it be an idea for such a possibility ? I use currently e.g. lists from firehol --> http://iplists.firehol.org/ via script and IPSet. Am currently not sure how difficult it is to give the user there some individuality to choose it´s own list ?
There's currently no way for a user to add their own list, apart from editing the text file specifying the list sources. This would obviously be overwritten by updates, but it wouldn't be difficult to allow for a second locally defined file to be read as well and merged with the distributed list. The WUI could be updated to allow this list to be edited.
My preference is to get the initial version in IPFire and sort out any points that people raise before adding additional features.
We currently do not allow this for the IPS either.
The IPS can have local rules defined in a file.
And I am not really sure if we should. Why would we not add the lists for all users if we see any value in them.
What reasons are there to allow users to do their own thing?
I agree that we should add additional lists where they're useful, but there are a lot of lists out there. Look at http://iplists.firehol.org/ - there's a list of monitored lists on the left hand side, down the page a bit. There's just under 400 of them. We don't want to put that many into IPFire because it would be too confusing, but it's possible that they could be useful to an 'advanced' user.
There are also lists like the ones from Bambenek Consulting (http://osint.bambenekconsulting.com/feeds/) which look interesting since they target individual malware C&C channels, but the licence doesn't cover commercial use. I don't think these could be included in IPFire because of the licence (unless we extend the WUI to include the licence in some way), but they could certainly be of use to individuals or charities.
Tim
Use cases can be different e.g. i remeber a project in the old forum which was about a company blocker (facebook, Windows, Apple) or in general the whole telemetry stuff can also be unwanted and there are some lists out there which can help to block also the "good" ones. If there are own vast lists of unwanted IPs, IPSet which is working here, is then the best way to do so, therefor my idea to bring in some flexibility in this great project to prevent scripting around in parallel for, let´s say, doing the same twice.
Is it not better to block the whole AS of those companies in the firewall?
The performance gain with IPSet causing the hash table can be significant -->
https://workshop.netfilter.org/2013/wiki/images/a/ab/Jozsef_Kadlecsik_ipset-...
<-- haven´t found some new performance tests but i thnk the results today are related close to another. Have experienced it by my own (with my ALIX back in the days) that the system was not usable after the try to handle some thousands IPs/CIDRs via IPTables (therefore i pushed IPSet at that time). Since this project delivers WUI access to IPSet, which is really great, beneath the really practical handling of block lists, some more advantages are may on the doorstep ?
We would have also a comparable design with the Proxy/URL-Filter whereby the user do also handle the challenge/possibility to upload/download/integrate own lists.
Sorry for appearing may a little unhumble which i really do not want and am also really not sure how much more work that costs but wanted to bring on some ideas and am nonetheless thankful with the already made process.
Best,
Erik
-
Michael Tremer -
Peter Müller -
Tim FitzGeorge -
ummeegge