Hi,
On 16 Dec 2019, at 20:06, Tim FitzGeorge lists@tfitzgeorge.me.uk wrote:
Hi,
I've attached the current GUI screenshot.
Thanks for that.
I have a couple of suggestions/concerns about it:
a) I am not sure what the value is about the top box that is called “Status”. It is basically a summary of the iptables output, but will it help the user? A blacklist with more or fewer entries is not better or worse, blocking more packets isn’t better than blocking fewer. It is about the quality of the blocks.
b) I would prefer to move the settings box to the top. If you like you can show the size of the blacklists there and when they have been updated, too.
c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
d) I do not understand what the “update check rate” means. I suppose we should show the update interval of the lists in the table and just refresh them according to that.
e) Do we want to keep the automatic blacklists now? I do not think it will actually improve the security of IPFire.
I'll update the code based on our discussions, and submit an updated set of patches - I imagine there will have to be at least one more iteration.
Let’s wait until we have come to decisions :)
-Michael
Tim
(No additional comments below)
On 13/12/2019 23:11, Michael Tremer wrote:
Hi,
Again my apologies for my late reply. Busy busy weeks.
On 8 Dec 2019, at 20:50, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Hello,
It's my turn to apologise for being slow to respond - I've had a busy week, but I should have plenty of time over the next couple of weeks.
No worries. Turns out we all do :)
I've made most of the comments inline, however I think Michael had a question (which I can't find now) about what happens if someone enables all the lists. One thing which would perhaps make this less likely is that the WUI tags the available lists with whether they're safe or not, with a footnote that safe means that the list only blocks malicious traffic. This won't guarantee that a user won't still try to enable all the lists, but it should make them realise that they should think first.
We had a couple of features going slightly wrong or being “misunderstood” by some users. People still seem to panic when they see “local recursor”. To this day I do not know why.
We cannot make everything idiot-proof. And when some user if of that category, they probably should shutdown their IPFire box, educate themselves and then come back again. So I do not want to limit people, but make things as easy as possible.
If someone enables all the lists, good luck with passing packets :)
I have considered replacing this tag with a risk high/medium/low and maybe adding a category (invalid/application/scanner/C&C or something like that), but that may provide too much information and dissuade them from actually following the links to checkout what the list actually does.
Can we have a screenshot of the GUI right now? I didn’t run the code, yet.
We should document the lists like we do it with the rulesets of the IPS. People might ignore this, but that is on them.
Tim
On 05/12/2019 22:25, Michael Tremer wrote:
Hello,
On 4 Dec 2019, at 17:05, Peter Müller peter.mueller@ipfire.org wrote:
Hello Tim, hello Michael,
please see my responses inline...
>> >> We could periodically update the blacklists on our main mirror (and >> wait for the network to sync it), make sure it is signed and write >> a small downloader that fetches, validates and installs them. >> >> @All: Thoughts on this? > > I think there are a number of points here. > > Firstly, from the point of a third party using IPFire, is this really > solving the privacy disclosure problem? There's no way round disclosing > your public IP address to someone you're downloading from; all this does > is change who that information is being disclosed to. For the user > there's no way of knowing whether the source is more or less protective > of the user's privacy than the blacklist provider. Indeed it won't be > possible to know who the lists are being downloaded from until the > download starts.
There is a way: Tor. But that is a totally different story.
Well, I see a third option on this: Use the mirror infrastructure we already have. Every IPFire installation discloses its public IP address to one of these servers sooner or later, so we do not disclose additional data if the blacklists were fetched from these.
Needless to say, Tor (hidden services) would be better, but that is a different story indeed. :-)
The point is rather that a forget list can be sent instead of the “real” one.
I did not get this. Forget? Forged?? ???
Yes, I meant to write forged, but auto-correct didn’t let me.
> Secondly, latency; some of the lists are updated every 5 minutes. While > I've limited the maximum check rate to hourly, will the updates > propagate quickly enough. For reference on my main system the 24 > updates on the CIARMY list made 143 498 changes (additions or > deletions). I've seem it do over 200 000.
Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not document a recommended update interval anywhere, so we can only guess.
Personally, more static lists seem to be preferable for packet filtering. Highly dynamic ones such as CIArmy should be done via DNSBL queries or something similar
- do we really want to have that list here?
It is not really an option to implement a DNSBL into a packet filter, but I get your point.
One of the 'selling points' for an IP address blacklist is that it can respond quickly to new threats - or rather new attackers. While a new IDS/IPS rule needs time to analyse the threat, generate a rule and check it, it's easy to add an address to a list. So, I think the CIArmy list is potentially useful for protecting home systems etc. with budget hardware, but I would be very careful about using it for a protecting a general access website.
If they are very volatile, we should honour that and update them often, too.
It probably is more about false-positives being removed very quickly instead of adding threats very quickly. The average IPFire user is probably not under threats like these to need to react very quickly.
So I do not see much value in adding those lists and then updating once a day. Does it have to be every 5 min? No. I would suggest 15 which should be good enough for everyone.
Other lists should of course not be updated every 15 minutes when not needed.
Running every 15 minutes would allow us to retry downloading lists that are on an hourly schedule if the download failed.
How did you come up with the hour? Will it be retried more often if the download was not successful?
One hour is the most common interval indeed, but adding some random time might be useful in order to reduce load on the servers providing a blacklist.
Yes, definitely. Otherwise we will shoot down our mirrors.
When I implemented that section of code, specifying the minimum check period in hours seemed to provide a convenient way of allowing a check period covering a wide range, with an hour as the fastest and a week as the slowest. I didn't looked at the CIArmy list until much later. Most of the lists don't change nearly as much, but the CIArmy list is described as one that deliberately responds quickly.
From my production system, for yesterday:
The following block lists were updated: BLOCKLIST_DE: 24 Time(s) - 9341 change(s) BOGON_FULL: 1 Time(s) - 10 change(s) CIARMY: 24 Time(s) - 159134 change(s) DSHIELD: 7 Time(s) - 18 change(s) EMERGING_FWRULE: 1 Time(s) - 50 change(s) FEODO_AGGRESIVE: 24 Time(s) - 13 change(s) SHODAN: 1 Time(s) - 0 change(s) SPAMHAUS_DROP: 1 Time(s) - 0 change(s) TOR_EXIT: 24 Time(s) - 162 change(s)
Very interesting statistics.
and my test system:
The following block lists were updated: ALIENVAULT: 19 Time(s) - 5331 change(s) EMERGING_COMPROMISED: 1 Time(s) - 26 change(s) TALOS_MALICIOUS: 1 Time(s) - 36 change(s)
That covers most of the lists. From the WUI, since 1 Dec:
Blacklist Entries pkts bytes Last updated in in AUTOBLACKLIST 0 731 51144 Sun Dec 8 18:40:02 2019 BLOCKLIST_DE 28020 857 46735 Sun Dec 8 18:40:02 2019 BOGON_FULL 214 5255 189K Sun Dec 8 16:50:04 2019 CIARMY 15000 19774 976K Sun Dec 8 18:04:01 2019 DSHIELD 20 7992 321K Sun Dec 8 16:45:13 2019 EMERGING_FWRULE 1647 197 8383 Fri Dec 6 05:29:07 2019 FEODO_AGGRESIVE 7169 0 0 Sun Dec 8 18:50:07 2019 SHODAN 32 34 1530 Sun Dec 8 17:54:09 2019 SPAMHAUS_DROP 823 0 0 Fri Dec 6 18:22:35 2019 SPAMHAUS_EDROP 111 82 16433 Thu Dec 5 17:27:09 2019 TOR_EXIT 1055 0 0 Sun Dec 8 18:31:02 2019
This as well.
Those are more packets than I would have expected.
(I've left out the pkts/bytes out fields which were all 0)
Note that where possible I do a HEAD request first and then only download the list if the modification time has changed since the last check. For dynamically generated lists this isn't possible.
You won’t need a HEAD request for it. You can include it in the GET request.
Have a look at location downloader where I use that. The server will respond with 304 and not send any payload.
https://git.ipfire.org/?p=location/libloc.git;a=blob;f=src/python/location-d...
If the download isn't successful it just gives up and waits for the next attempt (apart from the usual retries in the library). I probably should to change that so that it only applies the per list minimum update period in this case (specified in the sources file) rather than the user specified value as well.
I think it is not the worst if an update fails. It might just happen every once in a while.
So I would suggest to just re-run the script more often and when the mtime of the file is older than the threshold, a download is attempted. You can use that timestamp for the GET request.
I already use a time offset on the downloads - when it's started from boot, backup restore or WUI enable, it checks to see if it's installed in the fcrontab, and if not adds itself at a randomly generated offset in the hour.
That should potentially go to red.up, or if we can settle on 15 minutes, I would consider that often enough to quickly update all lists after a reboot.
> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is > the largest at a few MB), there are going to be a lot of them. How will > this affect the willingness of people to mirror IPFire?
I do not consider this being a problem as we do not generate that much traffic to them. Of course, that depends on the update interval again.
That depends on your point of view.
I do not have a problem with this at all in my data center, but there are plenty of people with a volume-based LTE plan or simply a 128 kBit/s connection. It will take a longer time to download the lists for them. We need to mind that.
> >> >> Talking about the preference of packet filter and IPS, I prefer to >> use the latter as well as it gains more insight in what kind of malicious >> traffic tried to pass a firewall machine. On systems with low resources, >> this might be problematic and removing load from the IPS can be preferred >> (make this configurable?!), on others, people might want to have both >> results. >> > You're only going to get one result for a packet whichever way round the > IP blacklist and IPS are since whichever comes first will drop the > packet before it reaches the second (well it would be possible to put > the IP blacklist first and get it to log and mark packets which are then > dropped after the IPS, but I think that's getting a little complicated. > In addition I've seen the messages about the trouble marking was > causing in the QoS). > > I think it's a 50/50 choice as to which is more valuable first; it's > probably going to differ from packet to packet. For me the possibility > of reducing the IPS load means I prefer putting the IP blacklist first > > It should be fairly easy to add the choice of where to put the IP > blacklist. I think it'll have to be in the main firewall script, so > it'll require a firewall restart, but it's not something that'll be > changed often.
I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier.
Agreed.
I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads.
We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily.
So dropping packets from blacklisted IP addresses/networks before IPS is it, then.
I do not even think it makes sense to swap the order in the outgoing direction.
Me too.
What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this.
Based on my personal experience, firewall hits observed on a single machine exposed to the internet are interesting, but the overall situation across multiple machines is even more interesting. Very quickly, you'll end on something like a centralised logging server and custom statistical analysis here...
Probably a project for IPFire 4.0 :)
Or use one of the existing services, like the DSHIELD client https://dshield.org/howto.html (subject to privacy concerns again).
>> >> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default. >> I would love to see the bogon ruleset here, too (think about 8chan successor >> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause >> interference if RED does not have a public IP address assigned. > > I can add a field to the options file that controls whether a list is > enabled by default.
Thank you. :-)
To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat.
This is my only objection against this patchset. Now, what can we do about it? One possibility is to apply the patchset now and implement a custom download source thing later on, or do that before releasing Core Update 139 (or which version the patchset will be to) after we agreed on something.
I do not see this being merged for 139. But that is not important. We need to get it right first and then release it.
As far as I know, nobody has tested this, yet.
There are a number of people who have been running an earlier version which I shared on GitHub. There were a few early issues, but it seems to be OK now.
https://forum.ipfire.org/viewtopic.php?f=27&t=21845
This version wasn't integrated into IPFire, so (for example) it inserted itself into the INPUT IPTables chain rather than having it's chains created as part of the firewall start-up script.
I have huge concerns about the automatic blacklist. @Peter: What is your opinion on this?
While I implemented it, I'm aware of its potential to cause problems, which is why it has to be separately enabled. It's not caused me any issues at the default settings (blocks at over 10 packets per hour until 1 hour has passed without seeing packets from the address), but I've not used it on a site with publicly announced services. If I was going to use it on a web site I would want to, at the very minimum, drop the block period drastically.
I suppose this is entirely unusable on an IPFire box in a data center that hosts things. Let’s say our rack in Hanover.
You will have hits from some broken IP stacks and people might just end up on this without doing anything wrong.
You can denial-of-service easily as well and I suppose without aggregation of data from many many systems, it does not make sense to instantly block addresses. And even then you probably would block an entire subnet.
So I am not sure how I should feel about it. I do not think that it adds anything because the packets would have been blocked anyways.
On the other hand, it's good at responding quickly. Usually I see only 1-2% of blocks from the automatic list:
Reason Count % First Last CIARMY 2416 45 Dec 6 00:00 Dec 6 23:59 DSHIELD 1353 25 Dec 6 00:00 Dec 6 23:59 INPUT 1294 24 Dec 6 00:00 Dec 6 23:59 AUTOBLACKLIST 122 2 Dec 6 00:20 Dec 6 16:28 BLOCKLIST_DE 89 2 Dec 6 00:20 Dec 6 23:46
and sometimes none at all, but one one occasion it blocked over 8000 packets. Again I'm aware this is for a home system, which is rather different than from a Web server.
If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not seem to be problematic, as they are hosted on 3rd party servers, too).
One of them will be, sooner or later. And one is enough I suppose.
DShield (https://dshield.org/api/#threatfeeds) and firehol (http://iplists.firehol.org/) seem to host copies of most of the lists as well.
I do not really want to overthink this - we didn’t do this with clamav for example either. But it probably is a decision to be made by the user what they want to enable and we should not enable anything by default. So no data will be leaked as long as the user does not consent.
-Michael
Thanks, and best regards, Peter Müller
<Screenshot_2019-12-16_IP_Address_Blacklists.png>
On 12/16/2019 5:20 PM, Michael Tremer wrote:> Hi,
On 16 Dec 2019, at 20:06, Tim FitzGeorge lists@tfitzgeorge.me.uk
wrote:
Hi,
I've attached the current GUI screenshot.
Thanks for that.
I have a couple of suggestions/concerns about it:
[snip]
c) I would suggest to remove the “safe” column because that is a very
hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
[snip]
May I opine that the "Safe" information would be helpful to me in the WUI. Perhaps we can be more explicit, or better explain, such as is often done with RBLs in mail server settings, where lists are sometimes described in terms of their likelihood to cause false-positives.
It's all well and good in the documentation, but a quick "Safe|Moderate|Risky" listing in the WUI will prove handy, IMHO.
Just my $0.02 as more of a user than a developer,
Tom
Hi,
On 16 Dec 2019, at 23:05, Tom Rymes trymes@rymes.com wrote:
On 12/16/2019 5:20 PM, Michael Tremer wrote:> Hi,
On 16 Dec 2019, at 20:06, Tim FitzGeorge lists@tfitzgeorge.me.uk wrote:
Hi,
I've attached the current GUI screenshot.
Thanks for that.
I have a couple of suggestions/concerns about it:
[snip]
c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
[snip]
May I opine that the "Safe" information would be helpful to me in the WUI. Perhaps we can be more explicit, or better explain, such as is often done with RBLs in mail server settings, where lists are sometimes described in terms of their likelihood to cause false-positives.
It's all well and good in the documentation, but a quick "Safe|Moderate|Risky" listing in the WUI will prove handy, IMHO.
Just my $0.02 as more of a user than a developer,
I appreciate your input, but I still disagree with is that we take the decision if something is “risky” or not. There are too many things that need to be taken into account to make that decision and it probably varies for each user.
What I take from your comment though is that we should categorise the lists, and that is something we can do.
We can add a headline to the table and group the lists by “Blocking ambiguous packets”, “Blocking Malware”, etc.
That makes it easier for the user to decide which lists are interesting or even necessary depending on what they want to achieve.
How is that?
-Michael
Tom
Hi,
Having decided that we'll categorise the lists, the question is what categories to use. They need to be:
- Short (to fit on the screen) - Easily translatable - and above all, useful.
Looking at the lists the obvious categories are:
- Invalid Address (on the public internet) BOGON, BOGON_FULL
- Scanner (not by itself malicious) SHODAN
- Application (potentially unwanted) TOR_ALL, TOR_EXIT
- Malware C & C FEODO_RECOMMENDED, FEODO_IP, FEODO_AGGRESIVE
- Composite EMERGING_FWRULE
Less obvious are:
- Reputation ALIENVAULT, CIARMY, SPAMHAUS_DROP, SPAMHAUS_EDROP
- Attacks BLOCKLIST_DE, DSHIELD, EMERGING_COMPROMISED
I'm not sure that the distinction between these two is going to be helpful to most people (I'm not sure I understand it myself).
We could use:
- Top attackers DSHIELD, EMERGING_COMPROMISED, SPAMHAUS_DROP, SPAMHAUS_EDROP
- Other attackers ALIENVAULT, BLOCKLIST_DE, CIARMY
but that might be making a distinction that is better made by the user.
Any opinions?
Tim
On 18/12/2019 12:10, Michael Tremer wrote:
Hi,
On 16 Dec 2019, at 23:05, Tom Rymes trymes@rymes.com wrote:
On 12/16/2019 5:20 PM, Michael Tremer wrote:> Hi,
On 16 Dec 2019, at 20:06, Tim FitzGeorge lists@tfitzgeorge.me.uk wrote:
Hi,
I've attached the current GUI screenshot.
Thanks for that.
I have a couple of suggestions/concerns about it:
[snip]
c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
[snip]
May I opine that the "Safe" information would be helpful to me in the WUI. Perhaps we can be more explicit, or better explain, such as is often done with RBLs in mail server settings, where lists are sometimes described in terms of their likelihood to cause false-positives.
It's all well and good in the documentation, but a quick "Safe|Moderate|Risky" listing in the WUI will prove handy, IMHO.
Just my $0.02 as more of a user than a developer,
I appreciate your input, but I still disagree with is that we take the decision if something is “risky” or not. There are too many things that need to be taken into account to make that decision and it probably varies for each user.
What I take from your comment though is that we should categorise the lists, and that is something we can do.
We can add a headline to the table and group the lists by “Blocking ambiguous packets”, “Blocking Malware”, etc.
That makes it easier for the user to decide which lists are interesting or even necessary depending on what they want to achieve.
How is that?
-Michael
Tom
Hello Tim,
On 28 Dec 2019, at 21:17, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Hi,
Having decided that we'll categorise the lists, the question is what categories to use. They need to be:
- Short (to fit on the screen)
- Easily translatable
- and above all, useful.
Looking at the lists the obvious categories are:
Invalid Address (on the public internet) BOGON, BOGON_FULL
Scanner (not by itself malicious) SHODAN
Application (potentially unwanted) TOR_ALL, TOR_EXIT
Malware C & C FEODO_RECOMMENDED, FEODO_IP, FEODO_AGGRESIVE
Composite EMERGING_FWRULE
I like all these a lot.
Less obvious are:
Reputation ALIENVAULT, CIARMY, SPAMHAUS_DROP, SPAMHAUS_EDROP
Attacks BLOCKLIST_DE, DSHIELD, EMERGING_COMPROMISED
I even like those two, although I would potentially consider merging “Invalid Address” and Reputation. They are kind of the same to me. IP addresses I under no circumstances I want to talk to.
I also like the Attacks category, although the name is very generic. But I cannot come up with anything better. The only thing that might be worth considering is to merge it with Malware and just call it “Malicious”.
I'm not sure that the distinction between these two is going to be helpful to most people (I'm not sure I understand it myself).
We could use:
Top attackers DSHIELD, EMERGING_COMPROMISED, SPAMHAUS_DROP, SPAMHAUS_EDROP
Other attackers ALIENVAULT, BLOCKLIST_DE, CIARMY
but that might be making a distinction that is better made by the user.
Agreed. It is not obvious why some are top attackers and others are not.
So I would 100% prefer the first option from above.
Best, -Michael
Any opinions?
Tim
On 18/12/2019 12:10, Michael Tremer wrote:
Hi,
On 16 Dec 2019, at 23:05, Tom Rymes trymes@rymes.com wrote:
On 12/16/2019 5:20 PM, Michael Tremer wrote:> Hi,
On 16 Dec 2019, at 20:06, Tim FitzGeorge lists@tfitzgeorge.me.uk wrote:
Hi,
I've attached the current GUI screenshot.
Thanks for that.
I have a couple of suggestions/concerns about it:
[snip]
c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
[snip]
May I opine that the "Safe" information would be helpful to me in the WUI. Perhaps we can be more explicit, or better explain, such as is often done with RBLs in mail server settings, where lists are sometimes described in terms of their likelihood to cause false-positives.
It's all well and good in the documentation, but a quick "Safe|Moderate|Risky" listing in the WUI will prove handy, IMHO.
Just my $0.02 as more of a user than a developer,
I appreciate your input, but I still disagree with is that we take the decision if something is “risky” or not. There are too many things that need to be taken into account to make that decision and it probably varies for each user.
What I take from your comment though is that we should categorise the lists, and that is something we can do.
We can add a headline to the table and group the lists by “Blocking ambiguous packets”, “Blocking Malware”, etc.
That makes it easier for the user to decide which lists are interesting or even necessary depending on what they want to achieve.
How is that?
-Michael
Tom
Hello Michael,
On 06/01/2020 11:21, Michael Tremer wrote:
Hello Tim,
On 28 Dec 2019, at 21:17, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Hi,
Having decided that we'll categorise the lists, the question is what categories to use. They need to be:
- Short (to fit on the screen)
- Easily translatable
- and above all, useful.
Looking at the lists the obvious categories are:
Invalid Address (on the public internet) BOGON, BOGON_FULL
Scanner (not by itself malicious) SHODAN
Application (potentially unwanted) TOR_ALL, TOR_EXIT
Malware C & C FEODO_RECOMMENDED, FEODO_IP, FEODO_AGGRESIVE
Composite EMERGING_FWRULE
I like all these a lot.
Less obvious are:
Reputation ALIENVAULT, CIARMY, SPAMHAUS_DROP, SPAMHAUS_EDROP
Attacks BLOCKLIST_DE, DSHIELD, EMERGING_COMPROMISED
I even like those two, although I would potentially consider merging
“Invalid Address” and Reputation. They are kind of the same to me. IP addresses I under no circumstances I want to talk to.
I think I'd rather keep them separate, since conceptually they're rather different. Also, if your red interface has a private address, you definitely wouldn't want to enable these lists.
I also like the Attacks category, although the name is very generic.
But I cannot come up with anything better. The only thing that might be worth considering is to merge it with Malware and just call it “Malicious”.
I suspect that anything more specific that Attacks is going to only describe a single list. It's (unfortunately) a catch-all category for anything that doesn't fit better somewhere else.
I'd rather not merge it with Malware C&C since the behaviour of the two is rather different - under most circumstances lists in this category shouldn't block any packets, whereas the Attacks category is expected to block a lot of inbound traffic.
I'm not sure that the distinction between these two is going to be helpful to most people (I'm not sure I understand it myself).
We could use:
Top attackers DSHIELD, EMERGING_COMPROMISED, SPAMHAUS_DROP, SPAMHAUS_EDROP
Other attackers ALIENVAULT, BLOCKLIST_DE, CIARMY
but that might be making a distinction that is better made by the user.
Agreed. It is not obvious why some are top attackers and others are not.
So I would 100% prefer the first option from above.
Best, -Michael
Tim
Any opinions?
Tim
On 18/12/2019 12:10, Michael Tremer wrote:
Hi,
On 16 Dec 2019, at 23:05, Tom Rymes trymes@rymes.com wrote:
On 12/16/2019 5:20 PM, Michael Tremer wrote:> Hi,
On 16 Dec 2019, at 20:06, Tim FitzGeorge
lists@tfitzgeorge.me.uk wrote:
Hi,
I've attached the current GUI screenshot.
Thanks for that.
I have a couple of suggestions/concerns about it:
[snip]
c) I would suggest to remove the “safe” column because that is a
very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
[snip]
May I opine that the "Safe" information would be helpful to me in
the WUI. Perhaps we can be more explicit, or better explain, such as is often done with RBLs in mail server settings, where lists are sometimes described in terms of their likelihood to cause false-positives.
It's all well and good in the documentation, but a quick
"Safe|Moderate|Risky" listing in the WUI will prove handy, IMHO.
Just my $0.02 as more of a user than a developer,
I appreciate your input, but I still disagree with is that we take
the decision if something is “risky” or not. There are too many things that need to be taken into account to make that decision and it probably varies for each user.
What I take from your comment though is that we should categorise
the lists, and that is something we can do.
We can add a headline to the table and group the lists by “Blocking
ambiguous packets”, “Blocking Malware”, etc.
That makes it easier for the user to decide which lists are
interesting or even necessary depending on what they want to achieve.
How is that?
-Michael
Tom
On 22 Jan 2020, at 20:35, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Hello Michael,
On 06/01/2020 11:21, Michael Tremer wrote:
Hello Tim,
On 28 Dec 2019, at 21:17, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Hi,
Having decided that we'll categorise the lists, the question is what categories to use. They need to be:
- Short (to fit on the screen)
- Easily translatable
- and above all, useful.
Looking at the lists the obvious categories are:
Invalid Address (on the public internet) BOGON, BOGON_FULL
Scanner (not by itself malicious) SHODAN
Application (potentially unwanted) TOR_ALL, TOR_EXIT
Malware C & C FEODO_RECOMMENDED, FEODO_IP, FEODO_AGGRESIVE
Composite EMERGING_FWRULE
I like all these a lot.
Less obvious are:
Reputation ALIENVAULT, CIARMY, SPAMHAUS_DROP, SPAMHAUS_EDROP
Attacks BLOCKLIST_DE, DSHIELD, EMERGING_COMPROMISED
I even like those two, although I would potentially consider merging
“Invalid Address” and Reputation. They are kind of the same to me. IP addresses I under no circumstances I want to talk to.
I think I'd rather keep them separate, since conceptually they're rather different. Also, if your red interface has a private address, you definitely wouldn't want to enable these lists.
Yes, that makes sense.
I also like the Attacks category, although the name is very generic.
But I cannot come up with anything better. The only thing that might be worth considering is to merge it with Malware and just call it “Malicious”.
I suspect that anything more specific that Attacks is going to only describe a single list. It's (unfortunately) a catch-all category for anything that doesn't fit better somewhere else.
I'd rather not merge it with Malware C&C since the behaviour of the two is rather different - under most circumstances lists in this category shouldn't block any packets, whereas the Attacks category is expected to block a lot of inbound traffic.
I can follow your argument and I am all for it.
However I am not sure if this is obvious from the category names. Instead of wasting too much time thinking about them (because I am sure that we can slice this cake in many different ways) I am okay with what you proposed if everybody else is.
-Michael
I'm not sure that the distinction between these two is going to be helpful to most people (I'm not sure I understand it myself).
We could use:
Top attackers DSHIELD, EMERGING_COMPROMISED, SPAMHAUS_DROP, SPAMHAUS_EDROP
Other attackers ALIENVAULT, BLOCKLIST_DE, CIARMY
but that might be making a distinction that is better made by the user.
Agreed. It is not obvious why some are top attackers and others are not.
So I would 100% prefer the first option from above.
Best, -Michael
Tim
Any opinions?
Tim
On 18/12/2019 12:10, Michael Tremer wrote:
Hi,
On 16 Dec 2019, at 23:05, Tom Rymes trymes@rymes.com wrote:
On 12/16/2019 5:20 PM, Michael Tremer wrote:> Hi,
> On 16 Dec 2019, at 20:06, Tim FitzGeorge
lists@tfitzgeorge.me.uk wrote:
> > Hi, > > I've attached the current GUI screenshot.
Thanks for that.
I have a couple of suggestions/concerns about it:
[snip]
c) I would suggest to remove the “safe” column because that is a
very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
[snip]
May I opine that the "Safe" information would be helpful to me in
the WUI. Perhaps we can be more explicit, or better explain, such as is often done with RBLs in mail server settings, where lists are sometimes described in terms of their likelihood to cause false-positives.
It's all well and good in the documentation, but a quick
"Safe|Moderate|Risky" listing in the WUI will prove handy, IMHO.
Just my $0.02 as more of a user than a developer,
I appreciate your input, but I still disagree with is that we take
the decision if something is “risky” or not. There are too many things that need to be taken into account to make that decision and it probably varies for each user.
What I take from your comment though is that we should categorise
the lists, and that is something we can do.
We can add a headline to the table and group the lists by “Blocking
ambiguous packets”, “Blocking Malware”, etc.
That makes it easier for the user to decide which lists are
interesting or even necessary depending on what they want to achieve.
How is that?
-Michael
Tom
Hi,
I think we've agreed on the following:
- Checks to be made every 15 minutes, subject to per-list minimum rate. - Modify download to use If-Modified-Since rather than separate HEAD request. - Remove automatic blacklist.
other comments are below.
Tim
On 16/12/2019 22:20, Michael Tremer wrote:
Hi,
On 16 Dec 2019, at 20:06, Tim FitzGeorge lists@tfitzgeorge.me.uk wrote:
Hi,
I've attached the current GUI screenshot.
Thanks for that.
I have a couple of suggestions/concerns about it:
a) I am not sure what the value is about the top box that is called “Status”. It is basically a summary of the iptables output, but will it help the user? A blacklist with more or fewer entries is not better or worse, blocking more packets isn’t better than blocking fewer. It is about the quality of the blocks.
I agree that the size of the lists is not all that useful, but the information about the number of blocked packets can be an indication of compromise. It depends on the type of the list, but a list that targets C&C channels (e.g. the FEODO lists) should not show any input packets being blocked, unless one of the computers being protected is infected. Apart from the BOGON lists, none of the lists should show any output packets being blocked; again it's a potential indication of infection if otherwise.
b) I would prefer to move the settings box to the top. If you like you can show the size of the blacklists there and when they have been updated, too.
I put the status at the top since it's usually what I want to see when I go to the page. Under normal circumstances I don't change the settings, so I'd rather not have to scroll past them to get to the information I'm actually interested in. The update times are just useful to show that the lists are actually being updated. The size is interesting to me, but as you say it probably won't help the average user, especially if you don't know which lists block individual addresses and which block nets (BOGON, DSHIELD and EMERGING_COMPROMISED).
I originally had the status and settings in one list, but I decided that it was getting a bit messy and difficult to follow. I've attached a screenshot of a quickly hacked page. (Note that the status area only appears once the blacklists have been enabled.)
c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
I agree that it's it's an over simplistic summary of the lists. I note that Tom Rymes has asked to keep it, but I think its only value is if it suggests to users that things are more complicated than they appear and therefore prompts them to read the Wiki.
d) I do not understand what the “update check rate” means. I suppose we should show the update interval of the lists in the table and just refresh them according to that.
It's the minimum check period for updates. The system checks for updates at either the rate specific to the list or this rate, whichever is slower. This is a hangover from the early days, copied from the IPS updates. I suppose it could have a value for some people who may wish to lower the check rate, but I'm not sure that it has sufficient value to be worth keeping.
e) Do we want to keep the automatic blacklists now? I do not think it will actually improve the security of IPFire.
I'll remove them. I think they do have a use, in that they can detect someone doing a port scan and block them before they find an open port, but as you say there is also the possibility of a DOS.
I'll update the code based on our discussions, and submit an updated set of patches - I imagine there will have to be at least one more iteration.
Let’s wait until we have come to decisions :)
-Michael
Tim
(No additional comments below)
On 13/12/2019 23:11, Michael Tremer wrote:
Hi,
Again my apologies for my late reply. Busy busy weeks.
On 8 Dec 2019, at 20:50, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Hello,
It's my turn to apologise for being slow to respond - I've had a busy week, but I should have plenty of time over the next couple of weeks.
No worries. Turns out we all do :)
I've made most of the comments inline, however I think Michael had a question (which I can't find now) about what happens if someone enables all the lists. One thing which would perhaps make this less likely is that the WUI tags the available lists with whether they're safe or not, with a footnote that safe means that the list only blocks malicious traffic. This won't guarantee that a user won't still try to enable all the lists, but it should make them realise that they should think first.
We had a couple of features going slightly wrong or being “misunderstood” by some users. People still seem to panic when they see “local recursor”. To this day I do not know why.
We cannot make everything idiot-proof. And when some user if of that category, they probably should shutdown their IPFire box, educate themselves and then come back again. So I do not want to limit people, but make things as easy as possible.
If someone enables all the lists, good luck with passing packets :)
I have considered replacing this tag with a risk high/medium/low and maybe adding a category (invalid/application/scanner/C&C or something like that), but that may provide too much information and dissuade them from actually following the links to checkout what the list actually does.
Can we have a screenshot of the GUI right now? I didn’t run the code, yet.
We should document the lists like we do it with the rulesets of the IPS. People might ignore this, but that is on them.
Tim
On 05/12/2019 22:25, Michael Tremer wrote:
Hello,
On 4 Dec 2019, at 17:05, Peter Müller peter.mueller@ipfire.org wrote:
Hello Tim, hello Michael,
please see my responses inline... >>> >>> We could periodically update the blacklists on our main mirror (and >>> wait for the network to sync it), make sure it is signed and write >>> a small downloader that fetches, validates and installs them. >>> >>> @All: Thoughts on this? >> >> I think there are a number of points here. >> >> Firstly, from the point of a third party using IPFire, is this really >> solving the privacy disclosure problem? There's no way round disclosing >> your public IP address to someone you're downloading from; all this does >> is change who that information is being disclosed to. For the user >> there's no way of knowing whether the source is more or less protective >> of the user's privacy than the blacklist provider. Indeed it won't be >> possible to know who the lists are being downloaded from until the >> download starts. > > There is a way: Tor. But that is a totally different story. Well, I see a third option on this: Use the mirror infrastructure we already have. Every IPFire installation discloses its public IP address to one of these servers sooner or later, so we do not disclose additional data if the blacklists were fetched from these.
Needless to say, Tor (hidden services) would be better, but that is a different story indeed. :-) > > The point is rather that a forget list can be sent instead of the “real” one. I did not get this. Forget? Forged?? ???
Yes, I meant to write forged, but auto-correct didn’t let me.
>> Secondly, latency; some of the lists are updated every 5 minutes. While >> I've limited the maximum check rate to hourly, will the updates >> propagate quickly enough. For reference on my main system the 24 >> updates on the CIARMY list made 143 498 changes (additions or >> deletions). I've seem it do over 200 000. Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not document a recommended update interval anywhere, so we can only guess.
Personally, more static lists seem to be preferable for packet filtering. Highly dynamic ones such as CIArmy should be done via DNSBL queries or something similar
- do we really want to have that list here?
It is not really an option to implement a DNSBL into a packet filter, but I get your point.
One of the 'selling points' for an IP address blacklist is that it can respond quickly to new threats - or rather new attackers. While a new IDS/IPS rule needs time to analyse the threat, generate a rule and check it, it's easy to add an address to a list. So, I think the CIArmy list is potentially useful for protecting home systems etc. with budget hardware, but I would be very careful about using it for a protecting a general access website.
If they are very volatile, we should honour that and update them often, too.
It probably is more about false-positives being removed very quickly instead of adding threats very quickly. The average IPFire user is probably not under threats like these to need to react very quickly.
So I do not see much value in adding those lists and then updating once a day. Does it have to be every 5 min? No. I would suggest 15 which should be good enough for everyone.
Other lists should of course not be updated every 15 minutes when not needed.
Running every 15 minutes would allow us to retry downloading lists that are on an hourly schedule if the download failed.
> How did you come up with the hour? Will it be retried more often if the download was not successful? One hour is the most common interval indeed, but adding some random time might be useful in order to reduce load on the servers providing a blacklist.
Yes, definitely. Otherwise we will shoot down our mirrors.
When I implemented that section of code, specifying the minimum check period in hours seemed to provide a convenient way of allowing a check period covering a wide range, with an hour as the fastest and a week as the slowest. I didn't looked at the CIArmy list until much later. Most of the lists don't change nearly as much, but the CIArmy list is described as one that deliberately responds quickly.
From my production system, for yesterday:
The following block lists were updated: BLOCKLIST_DE: 24 Time(s) - 9341 change(s) BOGON_FULL: 1 Time(s) - 10 change(s) CIARMY: 24 Time(s) - 159134 change(s) DSHIELD: 7 Time(s) - 18 change(s) EMERGING_FWRULE: 1 Time(s) - 50 change(s) FEODO_AGGRESIVE: 24 Time(s) - 13 change(s) SHODAN: 1 Time(s) - 0 change(s) SPAMHAUS_DROP: 1 Time(s) - 0 change(s) TOR_EXIT: 24 Time(s) - 162 change(s)
Very interesting statistics.
and my test system:
The following block lists were updated: ALIENVAULT: 19 Time(s) - 5331 change(s) EMERGING_COMPROMISED: 1 Time(s) - 26 change(s) TALOS_MALICIOUS: 1 Time(s) - 36 change(s)
That covers most of the lists. From the WUI, since 1 Dec:
Blacklist Entries pkts bytes Last updated in in AUTOBLACKLIST 0 731 51144 Sun Dec 8 18:40:02 2019 BLOCKLIST_DE 28020 857 46735 Sun Dec 8 18:40:02 2019 BOGON_FULL 214 5255 189K Sun Dec 8 16:50:04 2019 CIARMY 15000 19774 976K Sun Dec 8 18:04:01 2019 DSHIELD 20 7992 321K Sun Dec 8 16:45:13 2019 EMERGING_FWRULE 1647 197 8383 Fri Dec 6 05:29:07 2019 FEODO_AGGRESIVE 7169 0 0 Sun Dec 8 18:50:07 2019 SHODAN 32 34 1530 Sun Dec 8 17:54:09 2019 SPAMHAUS_DROP 823 0 0 Fri Dec 6 18:22:35 2019 SPAMHAUS_EDROP 111 82 16433 Thu Dec 5 17:27:09 2019 TOR_EXIT 1055 0 0 Sun Dec 8 18:31:02 2019
This as well.
Those are more packets than I would have expected.
(I've left out the pkts/bytes out fields which were all 0)
Note that where possible I do a HEAD request first and then only download the list if the modification time has changed since the last check. For dynamically generated lists this isn't possible.
You won’t need a HEAD request for it. You can include it in the GET request.
Have a look at location downloader where I use that. The server will respond with 304 and not send any payload.
https://git.ipfire.org/?p=location/libloc.git;a=blob;f=src/python/location-d...
If the download isn't successful it just gives up and waits for the next attempt (apart from the usual retries in the library). I probably should to change that so that it only applies the per list minimum update period in this case (specified in the sources file) rather than the user specified value as well.
I think it is not the worst if an update fails. It might just happen every once in a while.
So I would suggest to just re-run the script more often and when the mtime of the file is older than the threshold, a download is attempted. You can use that timestamp for the GET request.
I already use a time offset on the downloads - when it's started from boot, backup restore or WUI enable, it checks to see if it's installed in the fcrontab, and if not adds itself at a randomly generated offset in the hour.
That should potentially go to red.up, or if we can settle on 15 minutes, I would consider that often enough to quickly update all lists after a reboot.
>> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is >> the largest at a few MB), there are going to be a lot of them. How will >> this affect the willingness of people to mirror IPFire? I do not consider this being a problem as we do not generate that much traffic to them. Of course, that depends on the update interval again.
That depends on your point of view.
I do not have a problem with this at all in my data center, but there are plenty of people with a volume-based LTE plan or simply a 128 kBit/s connection. It will take a longer time to download the lists for them. We need to mind that.
>> >>> >>> Talking about the preference of packet filter and IPS, I prefer to >>> use the latter as well as it gains more insight in what kind of malicious >>> traffic tried to pass a firewall machine. On systems with low resources, >>> this might be problematic and removing load from the IPS can be preferred >>> (make this configurable?!), on others, people might want to have both >>> results. >>> >> You're only going to get one result for a packet whichever way round the >> IP blacklist and IPS are since whichever comes first will drop the >> packet before it reaches the second (well it would be possible to put >> the IP blacklist first and get it to log and mark packets which are then >> dropped after the IPS, but I think that's getting a little complicated. >> In addition I've seen the messages about the trouble marking was >> causing in the QoS). >> >> I think it's a 50/50 choice as to which is more valuable first; it's >> probably going to differ from packet to packet. For me the possibility >> of reducing the IPS load means I prefer putting the IP blacklist first >> >> It should be fairly easy to add the choice of where to put the IP >> blacklist. I think it'll have to be in the main firewall script, so >> it'll require a firewall restart, but it's not something that'll be >> changed often. > > I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier. Agreed.
> > I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads. > > We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily. So dropping packets from blacklisted IP addresses/networks before IPS is it, then. > > I do not even think it makes sense to swap the order in the outgoing direction. Me too.
> > What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this. Based on my personal experience, firewall hits observed on a single machine exposed to the internet are interesting, but the overall situation across multiple machines is even more interesting. Very quickly, you'll end on something like a centralised logging server and custom statistical analysis here...
Probably a project for IPFire 4.0 :)
Or use one of the existing services, like the DSHIELD client https://dshield.org/howto.html (subject to privacy concerns again).
> >>> >>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default. >>> I would love to see the bogon ruleset here, too (think about 8chan successor >>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause >>> interference if RED does not have a public IP address assigned. >> >> I can add a field to the options file that controls whether a list is >> enabled by default. Thank you. :-) > > To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat. This is my only objection against this patchset. Now, what can we do about it? One possibility is to apply the patchset now and implement a custom download source thing later on, or do that before releasing Core Update 139 (or which version the patchset will be to) after we agreed on something.
I do not see this being merged for 139. But that is not important. We need to get it right first and then release it.
As far as I know, nobody has tested this, yet.
There are a number of people who have been running an earlier version which I shared on GitHub. There were a few early issues, but it seems to be OK now.
https://forum.ipfire.org/viewtopic.php?f=27&t=21845
This version wasn't integrated into IPFire, so (for example) it inserted itself into the INPUT IPTables chain rather than having it's chains created as part of the firewall start-up script.
I have huge concerns about the automatic blacklist. @Peter: What is your opinion on this?
While I implemented it, I'm aware of its potential to cause problems, which is why it has to be separately enabled. It's not caused me any issues at the default settings (blocks at over 10 packets per hour until 1 hour has passed without seeing packets from the address), but I've not used it on a site with publicly announced services. If I was going to use it on a web site I would want to, at the very minimum, drop the block period drastically.
I suppose this is entirely unusable on an IPFire box in a data center that hosts things. Let’s say our rack in Hanover.
You will have hits from some broken IP stacks and people might just end up on this without doing anything wrong.
You can denial-of-service easily as well and I suppose without aggregation of data from many many systems, it does not make sense to instantly block addresses. And even then you probably would block an entire subnet.
So I am not sure how I should feel about it. I do not think that it adds anything because the packets would have been blocked anyways.
On the other hand, it's good at responding quickly. Usually I see only 1-2% of blocks from the automatic list:
Reason Count % First Last CIARMY 2416 45 Dec 6 00:00 Dec 6 23:59 DSHIELD 1353 25 Dec 6 00:00 Dec 6 23:59 INPUT 1294 24 Dec 6 00:00 Dec 6 23:59 AUTOBLACKLIST 122 2 Dec 6 00:20 Dec 6 16:28 BLOCKLIST_DE 89 2 Dec 6 00:20 Dec 6 23:46
and sometimes none at all, but one one occasion it blocked over 8000 packets. Again I'm aware this is for a home system, which is rather different than from a Web server.
If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not seem to be problematic, as they are hosted on 3rd party servers, too).
One of them will be, sooner or later. And one is enough I suppose.
DShield (https://dshield.org/api/#threatfeeds) and firehol (http://iplists.firehol.org/) seem to host copies of most of the lists as well.
I do not really want to overthink this - we didn’t do this with clamav for example either. But it probably is a decision to be made by the user what they want to enable and we should not enable anything by default. So no data will be leaked as long as the user does not consent.
-Michael
Thanks, and best regards, Peter Müller
<Screenshot_2019-12-16_IP_Address_Blacklists.png>
Hi,
On 17 Dec 2019, at 18:21, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Hi,
I think we've agreed on the following:
- Checks to be made every 15 minutes, subject to per-list minimum rate.
- Modify download to use If-Modified-Since rather than separate HEAD
request.
- Remove automatic blacklist.
I thought someone wants to counter my argument. This is not a dictatorship :)
other comments are below.
Tim
On 16/12/2019 22:20, Michael Tremer wrote:
Hi,
On 16 Dec 2019, at 20:06, Tim FitzGeorge lists@tfitzgeorge.me.uk wrote:
Hi,
I've attached the current GUI screenshot.
Thanks for that.
I have a couple of suggestions/concerns about it:
a) I am not sure what the value is about the top box that is called “Status”. It is basically a summary of the iptables output, but will it help the user? A blacklist with more or fewer entries is not better or worse, blocking more packets isn’t better than blocking fewer. It is about the quality of the blocks.
I agree that the size of the lists is not all that useful, but the information about the number of blocked packets can be an indication of compromise. It depends on the type of the list, but a list that targets C&C channels (e.g. the FEODO lists) should not show any input packets being blocked, unless one of the computers being protected is infected. Apart from the BOGON lists, none of the lists should show any output packets being blocked; again it's a potential indication of infection if otherwise.
The Shodan list will definitely show some activity unless you are behind CG NAT.
Should we not have this rather in the logging section?
And these counters here will be reset when the firewall is being reloaded. Should this data therefore not be collected from the logs so that you can go back in time? That would be very important to identify any start of compromise.
b) I would prefer to move the settings box to the top. If you like you can show the size of the blacklists there and when they have been updated, too.
I put the status at the top since it's usually what I want to see when I go to the page. Under normal circumstances I don't change the settings, so I'd rather not have to scroll past them to get to the information I'm actually interested in. The update times are just useful to show that the lists are actually being updated. The size is interesting to me, but as you say it probably won't help the average user, especially if you don't know which lists block individual addresses and which block nets (BOGON, DSHIELD and EMERGING_COMPROMISED).
See my comment on the order above. Splitting the page might make sense.
When you show size, does that mean lines in the blacklists? So it could be networks or single IP addresses? Does it not make sense to count IP addresses (i.e. 256 for a /24 network and so on) and put it in the table?
I think I would only be interested in the size for two things:
a) Is there any chance for overblocking? i.e. does the list have 2 billion entries which would be half the public IPv4 address space or so. I cannot really come up with a good example here.
Or
b) What is the performance impact/memory consumption of the list?
I originally had the status and settings in one list, but I decided that it was getting a bit messy and difficult to follow. I've attached a screenshot of a quickly hacked page. (Note that the status area only appears once the blacklists have been enabled.)
You are right. This is getting quite tight there.
You could write packets in/out into on column like 2k/1023, but see my concerns about this above.
c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
I agree that it's it's an over simplistic summary of the lists. I note that Tom Rymes has asked to keep it, but I think its only value is if it suggests to users that things are more complicated than they appear and therefore prompts them to read the Wiki.
I will reply to that separately.
d) I do not understand what the “update check rate” means. I suppose we should show the update interval of the lists in the table and just refresh them according to that.
It's the minimum check period for updates. The system checks for updates at either the rate specific to the list or this rate, whichever is slower. This is a hangover from the early days, copied from the IPS updates. I suppose it could have a value for some people who may wish to lower the check rate, but I'm not sure that it has sufficient value to be worth keeping.
No, I would not let the user decide when and how often to update the lists.
Generally there is this argument internally how often some databases should be updated. People with slow internet connections or volume based data plans will find this rather expensive to update those lists too often.
If we ever want to go ahead with that in IPFire 2, there should be a global switch for a “low data mode” which then delays updating these things.
e) Do we want to keep the automatic blacklists now? I do not think it will actually improve the security of IPFire.
I'll remove them. I think they do have a use, in that they can detect someone doing a port scan and block them before they find an open port, but as you say there is also the possibility of a DOS.
How would this block a DOS?
-Michael
I'll update the code based on our discussions, and submit an updated set of patches - I imagine there will have to be at least one more iteration.
Let’s wait until we have come to decisions :)
-Michael
Tim
(No additional comments below)
On 13/12/2019 23:11, Michael Tremer wrote:
Hi,
Again my apologies for my late reply. Busy busy weeks.
On 8 Dec 2019, at 20:50, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Hello,
It's my turn to apologise for being slow to respond - I've had a busy week, but I should have plenty of time over the next couple of weeks.
No worries. Turns out we all do :)
I've made most of the comments inline, however I think Michael had a question (which I can't find now) about what happens if someone enables all the lists. One thing which would perhaps make this less likely is that the WUI tags the available lists with whether they're safe or not, with a footnote that safe means that the list only blocks malicious traffic. This won't guarantee that a user won't still try to enable all the lists, but it should make them realise that they should think first.
We had a couple of features going slightly wrong or being “misunderstood” by some users. People still seem to panic when they see “local recursor”. To this day I do not know why.
We cannot make everything idiot-proof. And when some user if of that category, they probably should shutdown their IPFire box, educate themselves and then come back again. So I do not want to limit people, but make things as easy as possible.
If someone enables all the lists, good luck with passing packets :)
I have considered replacing this tag with a risk high/medium/low and maybe adding a category (invalid/application/scanner/C&C or something like that), but that may provide too much information and dissuade them from actually following the links to checkout what the list actually does.
Can we have a screenshot of the GUI right now? I didn’t run the code, yet.
We should document the lists like we do it with the rulesets of the IPS. People might ignore this, but that is on them.
Tim
On 05/12/2019 22:25, Michael Tremer wrote:
Hello,
> On 4 Dec 2019, at 17:05, Peter Müller peter.mueller@ipfire.org wrote: > > Hello Tim, hello Michael, > > please see my responses inline... >>>> >>>> We could periodically update the blacklists on our main mirror (and >>>> wait for the network to sync it), make sure it is signed and write >>>> a small downloader that fetches, validates and installs them. >>>> >>>> @All: Thoughts on this? >>> >>> I think there are a number of points here. >>> >>> Firstly, from the point of a third party using IPFire, is this really >>> solving the privacy disclosure problem? There's no way round disclosing >>> your public IP address to someone you're downloading from; all this does >>> is change who that information is being disclosed to. For the user >>> there's no way of knowing whether the source is more or less protective >>> of the user's privacy than the blacklist provider. Indeed it won't be >>> possible to know who the lists are being downloaded from until the >>> download starts. >> >> There is a way: Tor. But that is a totally different story. > Well, I see a third option on this: Use the mirror infrastructure we already > have. Every IPFire installation discloses its public IP address to one of these > servers sooner or later, so we do not disclose additional data if the blacklists > were fetched from these. > > Needless to say, Tor (hidden services) would be better, but that is a different > story indeed. :-) >> >> The point is rather that a forget list can be sent instead of the “real” one. > I did not get this. Forget? Forged?? ???
Yes, I meant to write forged, but auto-correct didn’t let me.
>>> Secondly, latency; some of the lists are updated every 5 minutes. While >>> I've limited the maximum check rate to hourly, will the updates >>> propagate quickly enough. For reference on my main system the 24 >>> updates on the CIARMY list made 143 498 changes (additions or >>> deletions). I've seem it do over 200 000. > Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not > document a recommended update interval anywhere, so we can only guess. > > Personally, more static lists seem to be preferable for packet filtering. Highly > dynamic ones such as CIArmy should be done via DNSBL queries or something similar > - do we really want to have that list here?
It is not really an option to implement a DNSBL into a packet filter, but I get your point.
One of the 'selling points' for an IP address blacklist is that it can respond quickly to new threats - or rather new attackers. While a new IDS/IPS rule needs time to analyse the threat, generate a rule and check it, it's easy to add an address to a list. So, I think the CIArmy list is potentially useful for protecting home systems etc. with budget hardware, but I would be very careful about using it for a protecting a general access website.
If they are very volatile, we should honour that and update them often, too.
It probably is more about false-positives being removed very quickly instead of adding threats very quickly. The average IPFire user is probably not under threats like these to need to react very quickly.
So I do not see much value in adding those lists and then updating once a day. Does it have to be every 5 min? No. I would suggest 15 which should be good enough for everyone.
Other lists should of course not be updated every 15 minutes when not needed.
Running every 15 minutes would allow us to retry downloading lists that are on an hourly schedule if the download failed.
>> How did you come up with the hour? Will it be retried more often if the download was not successful? > One hour is the most common interval indeed, but adding some random time might > be useful in order to reduce load on the servers providing a blacklist.
Yes, definitely. Otherwise we will shoot down our mirrors.
When I implemented that section of code, specifying the minimum check period in hours seemed to provide a convenient way of allowing a check period covering a wide range, with an hour as the fastest and a week as the slowest. I didn't looked at the CIArmy list until much later. Most of the lists don't change nearly as much, but the CIArmy list is described as one that deliberately responds quickly.
From my production system, for yesterday:
The following block lists were updated: BLOCKLIST_DE: 24 Time(s) - 9341 change(s) BOGON_FULL: 1 Time(s) - 10 change(s) CIARMY: 24 Time(s) - 159134 change(s) DSHIELD: 7 Time(s) - 18 change(s) EMERGING_FWRULE: 1 Time(s) - 50 change(s) FEODO_AGGRESIVE: 24 Time(s) - 13 change(s) SHODAN: 1 Time(s) - 0 change(s) SPAMHAUS_DROP: 1 Time(s) - 0 change(s) TOR_EXIT: 24 Time(s) - 162 change(s)
Very interesting statistics.
and my test system:
The following block lists were updated: ALIENVAULT: 19 Time(s) - 5331 change(s) EMERGING_COMPROMISED: 1 Time(s) - 26 change(s) TALOS_MALICIOUS: 1 Time(s) - 36 change(s)
That covers most of the lists. From the WUI, since 1 Dec:
Blacklist Entries pkts bytes Last updated in in AUTOBLACKLIST 0 731 51144 Sun Dec 8 18:40:02 2019 BLOCKLIST_DE 28020 857 46735 Sun Dec 8 18:40:02 2019 BOGON_FULL 214 5255 189K Sun Dec 8 16:50:04 2019 CIARMY 15000 19774 976K Sun Dec 8 18:04:01 2019 DSHIELD 20 7992 321K Sun Dec 8 16:45:13 2019 EMERGING_FWRULE 1647 197 8383 Fri Dec 6 05:29:07 2019 FEODO_AGGRESIVE 7169 0 0 Sun Dec 8 18:50:07 2019 SHODAN 32 34 1530 Sun Dec 8 17:54:09 2019 SPAMHAUS_DROP 823 0 0 Fri Dec 6 18:22:35 2019 SPAMHAUS_EDROP 111 82 16433 Thu Dec 5 17:27:09 2019 TOR_EXIT 1055 0 0 Sun Dec 8 18:31:02 2019
This as well.
Those are more packets than I would have expected.
(I've left out the pkts/bytes out fields which were all 0)
Note that where possible I do a HEAD request first and then only download the list if the modification time has changed since the last check. For dynamically generated lists this isn't possible.
You won’t need a HEAD request for it. You can include it in the GET request.
Have a look at location downloader where I use that. The server will respond with 304 and not send any payload.
https://git.ipfire.org/?p=location/libloc.git;a=blob;f=src/python/location-d...
If the download isn't successful it just gives up and waits for the next attempt (apart from the usual retries in the library). I probably should to change that so that it only applies the per list minimum update period in this case (specified in the sources file) rather than the user specified value as well.
I think it is not the worst if an update fails. It might just happen every once in a while.
So I would suggest to just re-run the script more often and when the mtime of the file is older than the threshold, a download is attempted. You can use that timestamp for the GET request.
I already use a time offset on the downloads - when it's started from boot, backup restore or WUI enable, it checks to see if it's installed in the fcrontab, and if not adds itself at a randomly generated offset in the hour.
That should potentially go to red.up, or if we can settle on 15 minutes, I would consider that often enough to quickly update all lists after a reboot.
>>> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is >>> the largest at a few MB), there are going to be a lot of them. How will >>> this affect the willingness of people to mirror IPFire? > I do not consider this being a problem as we do not generate that much traffic > to them. Of course, that depends on the update interval again.
That depends on your point of view.
I do not have a problem with this at all in my data center, but there are plenty of people with a volume-based LTE plan or simply a 128 kBit/s connection. It will take a longer time to download the lists for them. We need to mind that.
>>> >>>> >>>> Talking about the preference of packet filter and IPS, I prefer to >>>> use the latter as well as it gains more insight in what kind of malicious >>>> traffic tried to pass a firewall machine. On systems with low resources, >>>> this might be problematic and removing load from the IPS can be preferred >>>> (make this configurable?!), on others, people might want to have both >>>> results. >>>> >>> You're only going to get one result for a packet whichever way round the >>> IP blacklist and IPS are since whichever comes first will drop the >>> packet before it reaches the second (well it would be possible to put >>> the IP blacklist first and get it to log and mark packets which are then >>> dropped after the IPS, but I think that's getting a little complicated. >>> In addition I've seen the messages about the trouble marking was >>> causing in the QoS). >>> >>> I think it's a 50/50 choice as to which is more valuable first; it's >>> probably going to differ from packet to packet. For me the possibility >>> of reducing the IPS load means I prefer putting the IP blacklist first >>> >>> It should be fairly easy to add the choice of where to put the IP >>> blacklist. I think it'll have to be in the main firewall script, so >>> it'll require a firewall restart, but it's not something that'll be >>> changed often. >> >> I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier. > Agreed. > >> >> I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads. >> >> We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily. > So dropping packets from blacklisted IP addresses/networks before IPS is it, then. >> >> I do not even think it makes sense to swap the order in the outgoing direction. > Me too. > >> >> What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this. > Based on my personal experience, firewall hits observed on a single machine exposed > to the internet are interesting, but the overall situation across multiple machines > is even more interesting. Very quickly, you'll end on something like a centralised > logging server and custom statistical analysis here...
Probably a project for IPFire 4.0 :)
Or use one of the existing services, like the DSHIELD client https://dshield.org/howto.html (subject to privacy concerns again).
>> >>>> >>>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default. >>>> I would love to see the bogon ruleset here, too (think about 8chan successor >>>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause >>>> interference if RED does not have a public IP address assigned. >>> >>> I can add a field to the options file that controls whether a list is >>> enabled by default. > Thank you. :-) >> >> To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat. > This is my only objection against this patchset. Now, what can we do about it? > One possibility is to apply the patchset now and implement a custom download > source thing later on, or do that before releasing Core Update 139 (or which version > the patchset will be to) after we agreed on something.
I do not see this being merged for 139. But that is not important. We need to get it right first and then release it.
As far as I know, nobody has tested this, yet.
There are a number of people who have been running an earlier version which I shared on GitHub. There were a few early issues, but it seems to be OK now.
https://forum.ipfire.org/viewtopic.php?f=27&t=21845
This version wasn't integrated into IPFire, so (for example) it inserted itself into the INPUT IPTables chain rather than having it's chains created as part of the firewall start-up script.
I have huge concerns about the automatic blacklist. @Peter: What is your opinion on this?
While I implemented it, I'm aware of its potential to cause problems, which is why it has to be separately enabled. It's not caused me any issues at the default settings (blocks at over 10 packets per hour until 1 hour has passed without seeing packets from the address), but I've not used it on a site with publicly announced services. If I was going to use it on a web site I would want to, at the very minimum, drop the block period drastically.
I suppose this is entirely unusable on an IPFire box in a data center that hosts things. Let’s say our rack in Hanover.
You will have hits from some broken IP stacks and people might just end up on this without doing anything wrong.
You can denial-of-service easily as well and I suppose without aggregation of data from many many systems, it does not make sense to instantly block addresses. And even then you probably would block an entire subnet.
So I am not sure how I should feel about it. I do not think that it adds anything because the packets would have been blocked anyways.
On the other hand, it's good at responding quickly. Usually I see only 1-2% of blocks from the automatic list:
Reason Count % First Last CIARMY 2416 45 Dec 6 00:00 Dec 6 23:59 DSHIELD 1353 25 Dec 6 00:00 Dec 6 23:59 INPUT 1294 24 Dec 6 00:00 Dec 6 23:59 AUTOBLACKLIST 122 2 Dec 6 00:20 Dec 6 16:28 BLOCKLIST_DE 89 2 Dec 6 00:20 Dec 6 23:46
and sometimes none at all, but one one occasion it blocked over 8000 packets. Again I'm aware this is for a home system, which is rather different than from a Web server.
> If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not > seem to be problematic, as they are hosted on 3rd party servers, too).
One of them will be, sooner or later. And one is enough I suppose.
DShield (https://dshield.org/api/#threatfeeds) and firehol (http://iplists.firehol.org/) seem to host copies of most of the lists as well.
I do not really want to overthink this - we didn’t do this with clamav for example either. But it probably is a decision to be made by the user what they want to enable and we should not enable anything by default. So no data will be leaked as long as the user does not consent.
-Michael
> > Thanks, and best regards, > Peter Müller
<Screenshot_2019-12-16_IP_Address_Blacklists.png>
<Screenshot_2019-12-17_IP_Address_Blacklists.png>
Hi,
I've got my systems checking for downloads at 15 minute intervals and using If-Modified-Since. This all seems to be working OK.
Responses to comments below:
On 18/12/2019 12:07, Michael Tremer wrote:
Hi,
On 17 Dec 2019, at 18:21, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Hi,
I think we've agreed on the following:
- Checks to be made every 15 minutes, subject to per-list minimum rate.
- Modify download to use If-Modified-Since rather than separate HEAD
request.
- Remove automatic blacklist.
I thought someone wants to counter my argument. This is not a dictatorship :)
other comments are below.
Tim
On 16/12/2019 22:20, Michael Tremer wrote:
Hi,
On 16 Dec 2019, at 20:06, Tim FitzGeorge lists@tfitzgeorge.me.uk wrote:
Hi,
I've attached the current GUI screenshot.
Thanks for that.
I have a couple of suggestions/concerns about it:
a) I am not sure what the value is about the top box that is called “Status”. It is basically a summary of the iptables output, but will it help the user? A blacklist with more or fewer entries is not better or worse, blocking more packets isn’t better than blocking fewer. It is about the quality of the blocks.
I agree that the size of the lists is not all that useful, but the information about the number of blocked packets can be an indication of compromise. It depends on the type of the list, but a list that targets C&C channels (e.g. the FEODO lists) should not show any input packets being blocked, unless one of the computers being protected is infected. Apart from the BOGON lists, none of the lists should show any output packets being blocked; again it's a potential indication of infection if otherwise.
The Shodan list will definitely show some activity unless you are behind CG NAT.
Should we not have this rather in the logging section?
And these counters here will be reset when the firewall is being reloaded. Should this data therefore not be collected from the logs so that you can go back in time? That would be very important to identify any start of compromise.
That makes sense. I'll remove the status from this page and create a log page for the information. If anyone actually wants the iptables information it's available in from Firewall > iptables anyway.
Do you think it makes more sense to show packets blocked in/out or packets blocked per source interface? The latter would of (limited) help in tracking down potential compromise.
b) I would prefer to move the settings box to the top. If you like you can show the size of the blacklists there and when they have been updated, too.
I put the status at the top since it's usually what I want to see when I go to the page. Under normal circumstances I don't change the settings, so I'd rather not have to scroll past them to get to the information I'm actually interested in. The update times are just useful to show that the lists are actually being updated. The size is interesting to me, but as you say it probably won't help the average user, especially if you don't know which lists block individual addresses and which block nets (BOGON, DSHIELD and EMERGING_COMPROMISED).
See my comment on the order above. Splitting the page might make sense.
When you show size, does that mean lines in the blacklists? So it could be networks or single IP addresses? Does it not make sense to count IP addresses (i.e. 256 for a /24 network and so on) and put it in the table?
It's lines in the blacklist.
I think I would only be interested in the size for two things:
a) Is there any chance for overblocking? i.e. does the list have 2 billion entries which would be half the public IPv4 address space or so. I cannot really come up with a good example here.
Or
b) What is the performance impact/memory consumption of the list?
It would be interesting to show the type (single address or net), number of entries and number of addresses; the first two affect performance, the second memory use and the latter the number of blocked addresses, but that's really only going to be of interest to a very small number of people, who can always do 'ipset list -t' to get the information. So I'll remove the size column.
I originally had the status and settings in one list, but I decided that it was getting a bit messy and difficult to follow. I've attached a screenshot of a quickly hacked page. (Note that the status area only appears once the blacklists have been enabled.)
You are right. This is getting quite tight there.
You could write packets in/out into on column like 2k/1023, but see my concerns about this above.
Actually I've come to the conclusion that the number of bytes blocked isn't all that important; it's sufficient to show the number of packets.
c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
I agree that it's it's an over simplistic summary of the lists. I note that Tom Rymes has asked to keep it, but I think its only value is if it suggests to users that things are more complicated than they appear and therefore prompts them to read the Wiki.
I will reply to that separately.
Looking at that reply, I think it makes sense to include a category column in both the settings and the log summary page - the former to help selection of which lists to use, and the latter as a help when deciding whether the information could indicate compromise or not.
d) I do not understand what the “update check rate” means. I suppose we should show the update interval of the lists in the table and just refresh them according to that.
It's the minimum check period for updates. The system checks for updates at either the rate specific to the list or this rate, whichever is slower. This is a hangover from the early days, copied from the IPS updates. I suppose it could have a value for some people who may wish to lower the check rate, but I'm not sure that it has sufficient value to be worth keeping.
No, I would not let the user decide when and how often to update the lists.
OK. I'll remove it.
Generally there is this argument internally how often some databases should be updated. People with slow internet connections or volume based data plans will find this rather expensive to update those lists too often.
If we ever want to go ahead with that in IPFire 2, there should be a global switch for a “low data mode” which then delays updating these things.
That makes sense. Maybe also the ability to set when downloads take place - but as you say that's for the future.
e) Do we want to keep the automatic blacklists now? I do not think it will actually improve the security of IPFire.
I'll remove them. I think they do have a use, in that they can detect someone doing a port scan and block them before they find an open port, but as you say there is also the possibility of a DOS.
How would this block a DOS?
Sorry, I didn't make myself clear - there's the possibility that someone could cause a DOS with the automatic list by faking the source IP address and sending a few packets on a random port.
-Michael
Tim
I'll update the code based on our discussions, and submit an updated set of patches - I imagine there will have to be at least one more iteration.
Let’s wait until we have come to decisions :)
-Michael
Tim
(No additional comments below)
On 13/12/2019 23:11, Michael Tremer wrote:
Hi,
Again my apologies for my late reply. Busy busy weeks.
On 8 Dec 2019, at 20:50, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Hello,
It's my turn to apologise for being slow to respond - I've had a busy week, but I should have plenty of time over the next couple of weeks.
No worries. Turns out we all do :)
I've made most of the comments inline, however I think Michael had a question (which I can't find now) about what happens if someone enables all the lists. One thing which would perhaps make this less likely is that the WUI tags the available lists with whether they're safe or not, with a footnote that safe means that the list only blocks malicious traffic. This won't guarantee that a user won't still try to enable all the lists, but it should make them realise that they should think first.
We had a couple of features going slightly wrong or being “misunderstood” by some users. People still seem to panic when they see “local recursor”. To this day I do not know why.
We cannot make everything idiot-proof. And when some user if of that category, they probably should shutdown their IPFire box, educate themselves and then come back again. So I do not want to limit people, but make things as easy as possible.
If someone enables all the lists, good luck with passing packets :)
I have considered replacing this tag with a risk high/medium/low and maybe adding a category (invalid/application/scanner/C&C or something like that), but that may provide too much information and dissuade them from actually following the links to checkout what the list actually does.
Can we have a screenshot of the GUI right now? I didn’t run the code, yet.
We should document the lists like we do it with the rulesets of the IPS. People might ignore this, but that is on them.
Tim
On 05/12/2019 22:25, Michael Tremer wrote: > Hello, > >> On 4 Dec 2019, at 17:05, Peter Müller peter.mueller@ipfire.org wrote: >> >> Hello Tim, hello Michael, >> >> please see my responses inline... >>>>> >>>>> We could periodically update the blacklists on our main mirror (and >>>>> wait for the network to sync it), make sure it is signed and write >>>>> a small downloader that fetches, validates and installs them. >>>>> >>>>> @All: Thoughts on this? >>>> >>>> I think there are a number of points here. >>>> >>>> Firstly, from the point of a third party using IPFire, is this really >>>> solving the privacy disclosure problem? There's no way round disclosing >>>> your public IP address to someone you're downloading from; all this does >>>> is change who that information is being disclosed to. For the user >>>> there's no way of knowing whether the source is more or less protective >>>> of the user's privacy than the blacklist provider. Indeed it won't be >>>> possible to know who the lists are being downloaded from until the >>>> download starts. >>> >>> There is a way: Tor. But that is a totally different story. >> Well, I see a third option on this: Use the mirror infrastructure we already >> have. Every IPFire installation discloses its public IP address to one of these >> servers sooner or later, so we do not disclose additional data if the blacklists >> were fetched from these. >> >> Needless to say, Tor (hidden services) would be better, but that is a different >> story indeed. :-) >>> >>> The point is rather that a forget list can be sent instead of the “real” one. >> I did not get this. Forget? Forged?? ??? > > Yes, I meant to write forged, but auto-correct didn’t let me. > >>>> Secondly, latency; some of the lists are updated every 5 minutes. While >>>> I've limited the maximum check rate to hourly, will the updates >>>> propagate quickly enough. For reference on my main system the 24 >>>> updates on the CIARMY list made 143 498 changes (additions or >>>> deletions). I've seem it do over 200 000. >> Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not >> document a recommended update interval anywhere, so we can only guess. >> >> Personally, more static lists seem to be preferable for packet filtering. Highly >> dynamic ones such as CIArmy should be done via DNSBL queries or something similar >> - do we really want to have that list here? > > It is not really an option to implement a DNSBL into a packet filter, but I get your point.
One of the 'selling points' for an IP address blacklist is that it can respond quickly to new threats - or rather new attackers. While a new IDS/IPS rule needs time to analyse the threat, generate a rule and check it, it's easy to add an address to a list. So, I think the CIArmy list is potentially useful for protecting home systems etc. with budget hardware, but I would be very careful about using it for a protecting a general access website.
If they are very volatile, we should honour that and update them often, too.
It probably is more about false-positives being removed very quickly instead of adding threats very quickly. The average IPFire user is probably not under threats like these to need to react very quickly.
So I do not see much value in adding those lists and then updating once a day. Does it have to be every 5 min? No. I would suggest 15 which should be good enough for everyone.
Other lists should of course not be updated every 15 minutes when not needed.
Running every 15 minutes would allow us to retry downloading lists that are on an hourly schedule if the download failed.
> >>> How did you come up with the hour? Will it be retried more often if the download was not successful? >> One hour is the most common interval indeed, but adding some random time might >> be useful in order to reduce load on the servers providing a blacklist. > > Yes, definitely. Otherwise we will shoot down our mirrors.
When I implemented that section of code, specifying the minimum check period in hours seemed to provide a convenient way of allowing a check period covering a wide range, with an hour as the fastest and a week as the slowest. I didn't looked at the CIArmy list until much later. Most of the lists don't change nearly as much, but the CIArmy list is described as one that deliberately responds quickly.
From my production system, for yesterday:
The following block lists were updated: BLOCKLIST_DE: 24 Time(s) - 9341 change(s) BOGON_FULL: 1 Time(s) - 10 change(s) CIARMY: 24 Time(s) - 159134 change(s) DSHIELD: 7 Time(s) - 18 change(s) EMERGING_FWRULE: 1 Time(s) - 50 change(s) FEODO_AGGRESIVE: 24 Time(s) - 13 change(s) SHODAN: 1 Time(s) - 0 change(s) SPAMHAUS_DROP: 1 Time(s) - 0 change(s) TOR_EXIT: 24 Time(s) - 162 change(s)
Very interesting statistics.
and my test system:
The following block lists were updated: ALIENVAULT: 19 Time(s) - 5331 change(s) EMERGING_COMPROMISED: 1 Time(s) - 26 change(s) TALOS_MALICIOUS: 1 Time(s) - 36 change(s)
That covers most of the lists. From the WUI, since 1 Dec:
Blacklist Entries pkts bytes Last updated in in AUTOBLACKLIST 0 731 51144 Sun Dec 8 18:40:02 2019 BLOCKLIST_DE 28020 857 46735 Sun Dec 8 18:40:02 2019 BOGON_FULL 214 5255 189K Sun Dec 8 16:50:04 2019 CIARMY 15000 19774 976K Sun Dec 8 18:04:01 2019 DSHIELD 20 7992 321K Sun Dec 8 16:45:13 2019 EMERGING_FWRULE 1647 197 8383 Fri Dec 6 05:29:07 2019 FEODO_AGGRESIVE 7169 0 0 Sun Dec 8 18:50:07 2019 SHODAN 32 34 1530 Sun Dec 8 17:54:09 2019 SPAMHAUS_DROP 823 0 0 Fri Dec 6 18:22:35 2019 SPAMHAUS_EDROP 111 82 16433 Thu Dec 5 17:27:09 2019 TOR_EXIT 1055 0 0 Sun Dec 8 18:31:02 2019
This as well.
Those are more packets than I would have expected.
(I've left out the pkts/bytes out fields which were all 0)
Note that where possible I do a HEAD request first and then only download the list if the modification time has changed since the last check. For dynamically generated lists this isn't possible.
You won’t need a HEAD request for it. You can include it in the GET request.
Have a look at location downloader where I use that. The server will respond with 304 and not send any payload.
https://git.ipfire.org/?p=location/libloc.git;a=blob;f=src/python/location-d...
If the download isn't successful it just gives up and waits for the next attempt (apart from the usual retries in the library). I probably should to change that so that it only applies the per list minimum update period in this case (specified in the sources file) rather than the user specified value as well.
I think it is not the worst if an update fails. It might just happen every once in a while.
So I would suggest to just re-run the script more often and when the mtime of the file is older than the threshold, a download is attempted. You can use that timestamp for the GET request.
I already use a time offset on the downloads - when it's started from boot, backup restore or WUI enable, it checks to see if it's installed in the fcrontab, and if not adds itself at a randomly generated offset in the hour.
That should potentially go to red.up, or if we can settle on 15 minutes, I would consider that often enough to quickly update all lists after a reboot.
> >>>> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is >>>> the largest at a few MB), there are going to be a lot of them. How will >>>> this affect the willingness of people to mirror IPFire? >> I do not consider this being a problem as we do not generate that much traffic >> to them. Of course, that depends on the update interval again. > > That depends on your point of view. > > I do not have a problem with this at all in my data center, but there are plenty of people with a volume-based LTE plan or simply a 128 kBit/s connection. It will take a longer time to download the lists for them. We need to mind that. > >>>> >>>>> >>>>> Talking about the preference of packet filter and IPS, I prefer to >>>>> use the latter as well as it gains more insight in what kind of malicious >>>>> traffic tried to pass a firewall machine. On systems with low resources, >>>>> this might be problematic and removing load from the IPS can be preferred >>>>> (make this configurable?!), on others, people might want to have both >>>>> results. >>>>> >>>> You're only going to get one result for a packet whichever way round the >>>> IP blacklist and IPS are since whichever comes first will drop the >>>> packet before it reaches the second (well it would be possible to put >>>> the IP blacklist first and get it to log and mark packets which are then >>>> dropped after the IPS, but I think that's getting a little complicated. >>>> In addition I've seen the messages about the trouble marking was >>>> causing in the QoS). >>>> >>>> I think it's a 50/50 choice as to which is more valuable first; it's >>>> probably going to differ from packet to packet. For me the possibility >>>> of reducing the IPS load means I prefer putting the IP blacklist first >>>> >>>> It should be fairly easy to add the choice of where to put the IP >>>> blacklist. I think it'll have to be in the main firewall script, so >>>> it'll require a firewall restart, but it's not something that'll be >>>> changed often. >>> >>> I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier. >> Agreed. >> >>> >>> I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads. >>> >>> We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily. >> So dropping packets from blacklisted IP addresses/networks before IPS is it, then. >>> >>> I do not even think it makes sense to swap the order in the outgoing direction. >> Me too. >> >>> >>> What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this. >> Based on my personal experience, firewall hits observed on a single machine exposed >> to the internet are interesting, but the overall situation across multiple machines >> is even more interesting. Very quickly, you'll end on something like a centralised >> logging server and custom statistical analysis here... > > Probably a project for IPFire 4.0 :) > Or use one of the existing services, like the DSHIELD client https://dshield.org/howto.html (subject to privacy concerns again). >>> >>>>> >>>>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default. >>>>> I would love to see the bogon ruleset here, too (think about 8chan successor >>>>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause >>>>> interference if RED does not have a public IP address assigned. >>>> >>>> I can add a field to the options file that controls whether a list is >>>> enabled by default. >> Thank you. :-) >>> >>> To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat. >> This is my only objection against this patchset. Now, what can we do about it? >> One possibility is to apply the patchset now and implement a custom download >> source thing later on, or do that before releasing Core Update 139 (or which version >> the patchset will be to) after we agreed on something. > > I do not see this being merged for 139. But that is not important. We need to get it right first and then release it. > > As far as I know, nobody has tested this, yet.
There are a number of people who have been running an earlier version which I shared on GitHub. There were a few early issues, but it seems to be OK now.
https://forum.ipfire.org/viewtopic.php?f=27&t=21845
This version wasn't integrated into IPFire, so (for example) it inserted itself into the INPUT IPTables chain rather than having it's chains created as part of the firewall start-up script.
> > I have huge concerns about the automatic blacklist. @Peter: What is your opinion on this? >
While I implemented it, I'm aware of its potential to cause problems, which is why it has to be separately enabled. It's not caused me any issues at the default settings (blocks at over 10 packets per hour until 1 hour has passed without seeing packets from the address), but I've not used it on a site with publicly announced services. If I was going to use it on a web site I would want to, at the very minimum, drop the block period drastically.
I suppose this is entirely unusable on an IPFire box in a data center that hosts things. Let’s say our rack in Hanover.
You will have hits from some broken IP stacks and people might just end up on this without doing anything wrong.
You can denial-of-service easily as well and I suppose without aggregation of data from many many systems, it does not make sense to instantly block addresses. And even then you probably would block an entire subnet.
So I am not sure how I should feel about it. I do not think that it adds anything because the packets would have been blocked anyways.
On the other hand, it's good at responding quickly. Usually I see only 1-2% of blocks from the automatic list:
Reason Count % First Last CIARMY 2416 45 Dec 6 00:00 Dec 6 23:59 DSHIELD 1353 25 Dec 6 00:00 Dec 6 23:59 INPUT 1294 24 Dec 6 00:00 Dec 6 23:59 AUTOBLACKLIST 122 2 Dec 6 00:20 Dec 6 16:28 BLOCKLIST_DE 89 2 Dec 6 00:20 Dec 6 23:46
and sometimes none at all, but one one occasion it blocked over 8000 packets. Again I'm aware this is for a home system, which is rather different than from a Web server.
>> If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not >> seem to be problematic, as they are hosted on 3rd party servers, too). > > One of them will be, sooner or later. And one is enough I suppose.
DShield (https://dshield.org/api/#threatfeeds) and firehol (http://iplists.firehol.org/) seem to host copies of most of the lists as well. > > I do not really want to overthink this - we didn’t do this with clamav for example either. But it probably is a decision to be made by the user what they want to enable and we should not enable anything by default. So no data will be leaked as long as the user does not consent. > > -Michael > >> >> Thanks, and best regards, >> Peter Müller
<Screenshot_2019-12-16_IP_Address_Blacklists.png>
<Screenshot_2019-12-17_IP_Address_Blacklists.png>
Hi,
On 21 Dec 2019, at 19:34, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Hi,
I've got my systems checking for downloads at 15 minute intervals and using If-Modified-Since. This all seems to be working OK.
Great!
Responses to comments below:
On 18/12/2019 12:07, Michael Tremer wrote:
Hi,
On 17 Dec 2019, at 18:21, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Hi,
I think we've agreed on the following:
- Checks to be made every 15 minutes, subject to per-list minimum rate.
- Modify download to use If-Modified-Since rather than separate HEAD
request.
- Remove automatic blacklist.
I thought someone wants to counter my argument. This is not a dictatorship :)
other comments are below.
Tim
On 16/12/2019 22:20, Michael Tremer wrote:
Hi,
On 16 Dec 2019, at 20:06, Tim FitzGeorge lists@tfitzgeorge.me.uk wrote:
Hi,
I've attached the current GUI screenshot.
Thanks for that.
I have a couple of suggestions/concerns about it:
a) I am not sure what the value is about the top box that is called “Status”. It is basically a summary of the iptables output, but will it help the user? A blacklist with more or fewer entries is not better or worse, blocking more packets isn’t better than blocking fewer. It is about the quality of the blocks.
I agree that the size of the lists is not all that useful, but the information about the number of blocked packets can be an indication of compromise. It depends on the type of the list, but a list that targets C&C channels (e.g. the FEODO lists) should not show any input packets being blocked, unless one of the computers being protected is infected. Apart from the BOGON lists, none of the lists should show any output packets being blocked; again it's a potential indication of infection if otherwise.
The Shodan list will definitely show some activity unless you are behind CG NAT.
Should we not have this rather in the logging section?
And these counters here will be reset when the firewall is being reloaded. Should this data therefore not be collected from the logs so that you can go back in time? That would be very important to identify any start of compromise.
That makes sense. I'll remove the status from this page and create a log page for the information. If anyone actually wants the iptables information it's available in from Firewall > iptables anyway.
Do you think it makes more sense to show packets blocked in/out or packets blocked per source interface? The latter would of (limited) help in tracking down potential compromise.
I would only care for packets.
The size of the packets would only be interesting in a DoS, but it is still worthless information unless you say in what time it happened. i.e. you would want to show MBit/s or something like it, so you can estimate how much bandwidth is left. I would consider that out of scope here.
b) I would prefer to move the settings box to the top. If you like you can show the size of the blacklists there and when they have been updated, too.
I put the status at the top since it's usually what I want to see when I go to the page. Under normal circumstances I don't change the settings, so I'd rather not have to scroll past them to get to the information I'm actually interested in. The update times are just useful to show that the lists are actually being updated. The size is interesting to me, but as you say it probably won't help the average user, especially if you don't know which lists block individual addresses and which block nets (BOGON, DSHIELD and EMERGING_COMPROMISED).
See my comment on the order above. Splitting the page might make sense.
When you show size, does that mean lines in the blacklists? So it could be networks or single IP addresses? Does it not make sense to count IP addresses (i.e. 256 for a /24 network and so on) and put it in the table?
It's lines in the blacklist.
Okay, do we think this is what the user will expect?
I think I would only be interested in the size for two things:
a) Is there any chance for overblocking? i.e. does the list have 2 billion entries which would be half the public IPv4 address space or so. I cannot really come up with a good example here.
Or
b) What is the performance impact/memory consumption of the list?
It would be interesting to show the type (single address or net), number of entries and number of addresses; the first two affect performance, the second memory use and the latter the number of blocked addresses, but that's really only going to be of interest to a very small number of people, who can always do 'ipset list -t' to get the information. So I'll remove the size column.
I would consider this all to be entirely irrelevant for choice of hardware.
It is really cheap to throw the blacklists into ipset (unless somebody else knows better than me).
I originally had the status and settings in one list, but I decided that it was getting a bit messy and difficult to follow. I've attached a screenshot of a quickly hacked page. (Note that the status area only appears once the blacklists have been enabled.)
You are right. This is getting quite tight there.
You could write packets in/out into on column like 2k/1023, but see my concerns about this above.
Actually I've come to the conclusion that the number of bytes blocked isn't all that important; it's sufficient to show the number of packets.
Cool :)
c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
I agree that it's it's an over simplistic summary of the lists. I note that Tom Rymes has asked to keep it, but I think its only value is if it suggests to users that things are more complicated than they appear and therefore prompts them to read the Wiki.
I will reply to that separately.
Looking at that reply, I think it makes sense to include a category column in both the settings and the log summary page - the former to help selection of which lists to use, and the latter as a help when deciding whether the information could indicate compromise or not.
Agreed.
d) I do not understand what the “update check rate” means. I suppose we should show the update interval of the lists in the table and just refresh them according to that.
It's the minimum check period for updates. The system checks for updates at either the rate specific to the list or this rate, whichever is slower. This is a hangover from the early days, copied from the IPS updates. I suppose it could have a value for some people who may wish to lower the check rate, but I'm not sure that it has sufficient value to be worth keeping.
No, I would not let the user decide when and how often to update the lists.
OK. I'll remove it.
Generally there is this argument internally how often some databases should be updated. People with slow internet connections or volume based data plans will find this rather expensive to update those lists too often.
If we ever want to go ahead with that in IPFire 2, there should be a global switch for a “low data mode” which then delays updating these things.
That makes sense. Maybe also the ability to set when downloads take place - but as you say that's for the future.
Why would that be relevant to the user?
e) Do we want to keep the automatic blacklists now? I do not think it will actually improve the security of IPFire.
I'll remove them. I think they do have a use, in that they can detect someone doing a port scan and block them before they find an open port, but as you say there is also the possibility of a DOS.
How would this block a DOS?
Sorry, I didn't make myself clear - there's the possibility that someone could cause a DOS with the automatic list by faking the source IP address and sending a few packets on a random port.
Yes, that is what I would consider the biggest risk here as well - especially when hosting public services behind the firewall. Those triggers are probably not malicious, but rather bugs in network stacks, browsers being too keen on opening too many connections at the same time, etc.
I think this is all taking great shape now and I am very happy about this feature and cannot wait to have it in IPFire!
Best, -Michael
-Michael
Tim
I'll update the code based on our discussions, and submit an updated set of patches - I imagine there will have to be at least one more iteration.
Let’s wait until we have come to decisions :)
-Michael
Tim
(No additional comments below)
On 13/12/2019 23:11, Michael Tremer wrote:
Hi,
Again my apologies for my late reply. Busy busy weeks.
> On 8 Dec 2019, at 20:50, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote: > > Hello, > > It's my turn to apologise for being slow to respond - I've had a busy > week, but I should have plenty of time over the next couple of weeks.
No worries. Turns out we all do :)
> I've made most of the comments inline, however I think Michael had a > question (which I can't find now) about what happens if someone enables > all the lists. One thing which would perhaps make this less likely is > that the WUI tags the available lists with whether they're safe or not, > with a footnote that safe means that the list only blocks malicious > traffic. This won't guarantee that a user won't still try to enable all > the lists, but it should make them realise that they should think first.
We had a couple of features going slightly wrong or being “misunderstood” by some users. People still seem to panic when they see “local recursor”. To this day I do not know why.
We cannot make everything idiot-proof. And when some user if of that category, they probably should shutdown their IPFire box, educate themselves and then come back again. So I do not want to limit people, but make things as easy as possible.
If someone enables all the lists, good luck with passing packets :)
> I have considered replacing this tag with a risk high/medium/low and > maybe adding a category (invalid/application/scanner/C&C or something > like that), but that may provide too much information and dissuade them > from actually following the links to checkout what the list actually does.
Can we have a screenshot of the GUI right now? I didn’t run the code, yet.
We should document the lists like we do it with the rulesets of the IPS. People might ignore this, but that is on them.
> > Tim > > > On 05/12/2019 22:25, Michael Tremer wrote: >> Hello, >> >>> On 4 Dec 2019, at 17:05, Peter Müller peter.mueller@ipfire.org wrote: >>> >>> Hello Tim, hello Michael, >>> >>> please see my responses inline... >>>>>> >>>>>> We could periodically update the blacklists on our main mirror (and >>>>>> wait for the network to sync it), make sure it is signed and write >>>>>> a small downloader that fetches, validates and installs them. >>>>>> >>>>>> @All: Thoughts on this? >>>>> >>>>> I think there are a number of points here. >>>>> >>>>> Firstly, from the point of a third party using IPFire, is this really >>>>> solving the privacy disclosure problem? There's no way round disclosing >>>>> your public IP address to someone you're downloading from; all this does >>>>> is change who that information is being disclosed to. For the user >>>>> there's no way of knowing whether the source is more or less protective >>>>> of the user's privacy than the blacklist provider. Indeed it won't be >>>>> possible to know who the lists are being downloaded from until the >>>>> download starts. >>>> >>>> There is a way: Tor. But that is a totally different story. >>> Well, I see a third option on this: Use the mirror infrastructure we already >>> have. Every IPFire installation discloses its public IP address to one of these >>> servers sooner or later, so we do not disclose additional data if the blacklists >>> were fetched from these. >>> >>> Needless to say, Tor (hidden services) would be better, but that is a different >>> story indeed. :-) >>>> >>>> The point is rather that a forget list can be sent instead of the “real” one. >>> I did not get this. Forget? Forged?? ??? >> >> Yes, I meant to write forged, but auto-correct didn’t let me. >> >>>>> Secondly, latency; some of the lists are updated every 5 minutes. While >>>>> I've limited the maximum check rate to hourly, will the updates >>>>> propagate quickly enough. For reference on my main system the 24 >>>>> updates on the CIARMY list made 143 498 changes (additions or >>>>> deletions). I've seem it do over 200 000. >>> Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not >>> document a recommended update interval anywhere, so we can only guess. >>> >>> Personally, more static lists seem to be preferable for packet filtering. Highly >>> dynamic ones such as CIArmy should be done via DNSBL queries or something similar >>> - do we really want to have that list here? >> >> It is not really an option to implement a DNSBL into a packet filter, but I get your point. > > One of the 'selling points' for an IP address blacklist is that it can > respond quickly to new threats - or rather new attackers. While a new > IDS/IPS rule needs time to analyse the threat, generate a rule and check > it, it's easy to add an address to a list. So, I think the CIArmy list > is potentially useful for protecting home systems etc. with budget > hardware, but I would be very careful about using it for a protecting a > general access website.
If they are very volatile, we should honour that and update them often, too.
It probably is more about false-positives being removed very quickly instead of adding threats very quickly. The average IPFire user is probably not under threats like these to need to react very quickly.
So I do not see much value in adding those lists and then updating once a day. Does it have to be every 5 min? No. I would suggest 15 which should be good enough for everyone.
Other lists should of course not be updated every 15 minutes when not needed.
Running every 15 minutes would allow us to retry downloading lists that are on an hourly schedule if the download failed.
>> >>>> How did you come up with the hour? Will it be retried more often if the download was not successful? >>> One hour is the most common interval indeed, but adding some random time might >>> be useful in order to reduce load on the servers providing a blacklist. >> >> Yes, definitely. Otherwise we will shoot down our mirrors. > > When I implemented that section of code, specifying the minimum check > period in hours seemed to provide a convenient way of allowing a check > period covering a wide range, with an hour as the fastest and a week as > the slowest. I didn't looked at the CIArmy list until much later. Most > of the lists don't change nearly as much, but the CIArmy list is > described as one that deliberately responds quickly. > > From my production system, for yesterday: > > The following block lists were updated: > BLOCKLIST_DE: 24 Time(s) - 9341 change(s) > BOGON_FULL: 1 Time(s) - 10 change(s) > CIARMY: 24 Time(s) - 159134 change(s) > DSHIELD: 7 Time(s) - 18 change(s) > EMERGING_FWRULE: 1 Time(s) - 50 change(s) > FEODO_AGGRESIVE: 24 Time(s) - 13 change(s) > SHODAN: 1 Time(s) - 0 change(s) > SPAMHAUS_DROP: 1 Time(s) - 0 change(s) > TOR_EXIT: 24 Time(s) - 162 change(s)
Very interesting statistics.
> > and my test system: > > The following block lists were updated: > ALIENVAULT: 19 Time(s) - 5331 change(s) > EMERGING_COMPROMISED: 1 Time(s) - 26 change(s) > TALOS_MALICIOUS: 1 Time(s) - 36 change(s) > > That covers most of the lists. From the WUI, since 1 Dec: > > Blacklist Entries pkts bytes Last updated > in in > AUTOBLACKLIST 0 731 51144 Sun Dec 8 18:40:02 2019 > BLOCKLIST_DE 28020 857 46735 Sun Dec 8 18:40:02 2019 > BOGON_FULL 214 5255 189K Sun Dec 8 16:50:04 2019 > CIARMY 15000 19774 976K Sun Dec 8 18:04:01 2019 > DSHIELD 20 7992 321K Sun Dec 8 16:45:13 2019 > EMERGING_FWRULE 1647 197 8383 Fri Dec 6 05:29:07 2019 > FEODO_AGGRESIVE 7169 0 0 Sun Dec 8 18:50:07 2019 > SHODAN 32 34 1530 Sun Dec 8 17:54:09 2019 > SPAMHAUS_DROP 823 0 0 Fri Dec 6 18:22:35 2019 > SPAMHAUS_EDROP 111 82 16433 Thu Dec 5 17:27:09 2019 > TOR_EXIT 1055 0 0 Sun Dec 8 18:31:02 2019
This as well.
Those are more packets than I would have expected.
> > (I've left out the pkts/bytes out fields which were all 0) > > Note that where possible I do a HEAD request first and then only > download the list if the modification time has changed since the last > check. For dynamically generated lists this isn't possible.
You won’t need a HEAD request for it. You can include it in the GET request.
Have a look at location downloader where I use that. The server will respond with 304 and not send any payload.
https://git.ipfire.org/?p=location/libloc.git;a=blob;f=src/python/location-d...
> > If the download isn't successful it just gives up and waits for the next > attempt (apart from the usual retries in the library). I probably > should to change that so that it only applies the per list minimum > update period in this case (specified in the sources file) rather than > the user specified value as well.
I think it is not the worst if an update fails. It might just happen every once in a while.
So I would suggest to just re-run the script more often and when the mtime of the file is older than the threshold, a download is attempted. You can use that timestamp for the GET request.
> I already use a time offset on the downloads - when it's started from > boot, backup restore or WUI enable, it checks to see if it's installed > in the fcrontab, and if not adds itself at a randomly generated offset > in the hour.
That should potentially go to red.up, or if we can settle on 15 minutes, I would consider that often enough to quickly update all lists after a reboot.
> >> >>>>> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is >>>>> the largest at a few MB), there are going to be a lot of them. How will >>>>> this affect the willingness of people to mirror IPFire? >>> I do not consider this being a problem as we do not generate that much traffic >>> to them. Of course, that depends on the update interval again. >> >> That depends on your point of view. >> >> I do not have a problem with this at all in my data center, but there are plenty of people with a volume-based LTE plan or simply a 128 kBit/s connection. It will take a longer time to download the lists for them. We need to mind that. >> >>>>> >>>>>> >>>>>> Talking about the preference of packet filter and IPS, I prefer to >>>>>> use the latter as well as it gains more insight in what kind of malicious >>>>>> traffic tried to pass a firewall machine. On systems with low resources, >>>>>> this might be problematic and removing load from the IPS can be preferred >>>>>> (make this configurable?!), on others, people might want to have both >>>>>> results. >>>>>> >>>>> You're only going to get one result for a packet whichever way round the >>>>> IP blacklist and IPS are since whichever comes first will drop the >>>>> packet before it reaches the second (well it would be possible to put >>>>> the IP blacklist first and get it to log and mark packets which are then >>>>> dropped after the IPS, but I think that's getting a little complicated. >>>>> In addition I've seen the messages about the trouble marking was >>>>> causing in the QoS). >>>>> >>>>> I think it's a 50/50 choice as to which is more valuable first; it's >>>>> probably going to differ from packet to packet. For me the possibility >>>>> of reducing the IPS load means I prefer putting the IP blacklist first >>>>> >>>>> It should be fairly easy to add the choice of where to put the IP >>>>> blacklist. I think it'll have to be in the main firewall script, so >>>>> it'll require a firewall restart, but it's not something that'll be >>>>> changed often. >>>> >>>> I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier. >>> Agreed. >>> >>>> >>>> I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads. >>>> >>>> We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily. >>> So dropping packets from blacklisted IP addresses/networks before IPS is it, then. >>>> >>>> I do not even think it makes sense to swap the order in the outgoing direction. >>> Me too. >>> >>>> >>>> What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this. >>> Based on my personal experience, firewall hits observed on a single machine exposed >>> to the internet are interesting, but the overall situation across multiple machines >>> is even more interesting. Very quickly, you'll end on something like a centralised >>> logging server and custom statistical analysis here... >> >> Probably a project for IPFire 4.0 :) >> > Or use one of the existing services, like the DSHIELD client > https://dshield.org/howto.html (subject to privacy concerns again). >>>> >>>>>> >>>>>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default. >>>>>> I would love to see the bogon ruleset here, too (think about 8chan successor >>>>>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause >>>>>> interference if RED does not have a public IP address assigned. >>>>> >>>>> I can add a field to the options file that controls whether a list is >>>>> enabled by default. >>> Thank you. :-) >>>> >>>> To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat. >>> This is my only objection against this patchset. Now, what can we do about it? >>> One possibility is to apply the patchset now and implement a custom download >>> source thing later on, or do that before releasing Core Update 139 (or which version >>> the patchset will be to) after we agreed on something. >> >> I do not see this being merged for 139. But that is not important. We need to get it right first and then release it. >> >> As far as I know, nobody has tested this, yet. > > There are a number of people who have been running an earlier version > which I shared on GitHub. There were a few early issues, but it seems > to be OK now. > > https://forum.ipfire.org/viewtopic.php?f=27&t=21845 > > This version wasn't integrated into IPFire, so (for example) it inserted > itself into the INPUT IPTables chain rather than having it's chains > created as part of the firewall start-up script. > >> >> I have huge concerns about the automatic blacklist. @Peter: What is your opinion on this? >> > > While I implemented it, I'm aware of its potential to cause problems, > which is why it has to be separately enabled. It's not caused me any > issues at the default settings (blocks at over 10 packets per hour until > 1 hour has passed without seeing packets from the address), but I've not > used it on a site with publicly announced services. If I was going to > use it on a web site I would want to, at the very minimum, drop the > block period drastically.
I suppose this is entirely unusable on an IPFire box in a data center that hosts things. Let’s say our rack in Hanover.
You will have hits from some broken IP stacks and people might just end up on this without doing anything wrong.
You can denial-of-service easily as well and I suppose without aggregation of data from many many systems, it does not make sense to instantly block addresses. And even then you probably would block an entire subnet.
So I am not sure how I should feel about it. I do not think that it adds anything because the packets would have been blocked anyways.
> > On the other hand, it's good at responding quickly. Usually I see only > 1-2% of blocks from the automatic list: > > Reason Count % First Last > CIARMY 2416 45 Dec 6 00:00 Dec 6 23:59 > DSHIELD 1353 25 Dec 6 00:00 Dec 6 23:59 > INPUT 1294 24 Dec 6 00:00 Dec 6 23:59 > AUTOBLACKLIST 122 2 Dec 6 00:20 Dec 6 16:28 > BLOCKLIST_DE 89 2 Dec 6 00:20 Dec 6 23:46 > > and sometimes none at all, but one one occasion it blocked over 8000 > packets. Again I'm aware this is for a home system, which is rather > different than from a Web server. > >>> If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not >>> seem to be problematic, as they are hosted on 3rd party servers, too). >> >> One of them will be, sooner or later. And one is enough I suppose. > > DShield (https://dshield.org/api/#threatfeeds) and firehol > (http://iplists.firehol.org/) seem to host copies of most of the lists > as well. >> >> I do not really want to overthink this - we didn’t do this with clamav for example either. But it probably is a decision to be made by the user what they want to enable and we should not enable anything by default. So no data will be leaked as long as the user does not consent. >> >> -Michael >> >>> >>> Thanks, and best regards, >>> Peter Müller
<Screenshot_2019-12-16_IP_Address_Blacklists.png>
<Screenshot_2019-12-17_IP_Address_Blacklists.png>
Hi,
I'll code and test what we've agreed on. In the meantime I'm going to reply on the sub-thread started by Tom Rymes as to what we call the list categories, so that we can keep that discussion separate from further functionality etc. discussion.
Tim
On 21/12/2019 18:34, Tim FitzGeorge wrote:
Hi,
I've got my systems checking for downloads at 15 minute intervals and using If-Modified-Since. This all seems to be working OK.
Responses to comments below:
On 18/12/2019 12:07, Michael Tremer wrote:
Hi,
On 17 Dec 2019, at 18:21, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Hi,
I think we've agreed on the following:
- Checks to be made every 15 minutes, subject to per-list minimum rate.
- Modify download to use If-Modified-Since rather than separate HEAD
request.
- Remove automatic blacklist.
I thought someone wants to counter my argument. This is not a dictatorship :)
other comments are below.
Tim
On 16/12/2019 22:20, Michael Tremer wrote:
Hi,
On 16 Dec 2019, at 20:06, Tim FitzGeorge lists@tfitzgeorge.me.uk wrote:
Hi,
I've attached the current GUI screenshot.
Thanks for that.
I have a couple of suggestions/concerns about it:
a) I am not sure what the value is about the top box that is called “Status”. It is basically a summary of the iptables output, but will it help the user? A blacklist with more or fewer entries is not better or worse, blocking more packets isn’t better than blocking fewer. It is about the quality of the blocks.
I agree that the size of the lists is not all that useful, but the information about the number of blocked packets can be an indication of compromise. It depends on the type of the list, but a list that targets C&C channels (e.g. the FEODO lists) should not show any input packets being blocked, unless one of the computers being protected is infected. Apart from the BOGON lists, none of the lists should show any output packets being blocked; again it's a potential indication of infection if otherwise.
The Shodan list will definitely show some activity unless you are behind CG NAT.
Should we not have this rather in the logging section?
And these counters here will be reset when the firewall is being reloaded. Should this data therefore not be collected from the logs so that you can go back in time? That would be very important to identify any start of compromise.
That makes sense. I'll remove the status from this page and create a log page for the information. If anyone actually wants the iptables information it's available in from Firewall > iptables anyway.
Do you think it makes more sense to show packets blocked in/out or packets blocked per source interface? The latter would of (limited) help in tracking down potential compromise.
b) I would prefer to move the settings box to the top. If you like you can show the size of the blacklists there and when they have been updated, too.
I put the status at the top since it's usually what I want to see when I go to the page. Under normal circumstances I don't change the settings, so I'd rather not have to scroll past them to get to the information I'm actually interested in. The update times are just useful to show that the lists are actually being updated. The size is interesting to me, but as you say it probably won't help the average user, especially if you don't know which lists block individual addresses and which block nets (BOGON, DSHIELD and EMERGING_COMPROMISED).
See my comment on the order above. Splitting the page might make sense.
When you show size, does that mean lines in the blacklists? So it could be networks or single IP addresses? Does it not make sense to count IP addresses (i.e. 256 for a /24 network and so on) and put it in the table?
It's lines in the blacklist.
I think I would only be interested in the size for two things:
a) Is there any chance for overblocking? i.e. does the list have 2 billion entries which would be half the public IPv4 address space or so. I cannot really come up with a good example here.
Or
b) What is the performance impact/memory consumption of the list?
It would be interesting to show the type (single address or net), number of entries and number of addresses; the first two affect performance, the second memory use and the latter the number of blocked addresses, but that's really only going to be of interest to a very small number of people, who can always do 'ipset list -t' to get the information. So I'll remove the size column.
I originally had the status and settings in one list, but I decided that it was getting a bit messy and difficult to follow. I've attached a screenshot of a quickly hacked page. (Note that the status area only appears once the blacklists have been enabled.)
You are right. This is getting quite tight there.
You could write packets in/out into on column like 2k/1023, but see my concerns about this above.
Actually I've come to the conclusion that the number of bytes blocked isn't all that important; it's sufficient to show the number of packets.
c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
I agree that it's it's an over simplistic summary of the lists. I note that Tom Rymes has asked to keep it, but I think its only value is if it suggests to users that things are more complicated than they appear and therefore prompts them to read the Wiki.
I will reply to that separately.
Looking at that reply, I think it makes sense to include a category column in both the settings and the log summary page - the former to help selection of which lists to use, and the latter as a help when deciding whether the information could indicate compromise or not.
d) I do not understand what the “update check rate” means. I suppose we should show the update interval of the lists in the table and just refresh them according to that.
It's the minimum check period for updates. The system checks for updates at either the rate specific to the list or this rate, whichever is slower. This is a hangover from the early days, copied from the IPS updates. I suppose it could have a value for some people who may wish to lower the check rate, but I'm not sure that it has sufficient value to be worth keeping.
No, I would not let the user decide when and how often to update the lists.
OK. I'll remove it.
Generally there is this argument internally how often some databases should be updated. People with slow internet connections or volume based data plans will find this rather expensive to update those lists too often.
If we ever want to go ahead with that in IPFire 2, there should be a global switch for a “low data mode” which then delays updating these things.
That makes sense. Maybe also the ability to set when downloads take place - but as you say that's for the future.
e) Do we want to keep the automatic blacklists now? I do not think it will actually improve the security of IPFire.
I'll remove them. I think they do have a use, in that they can detect someone doing a port scan and block them before they find an open port, but as you say there is also the possibility of a DOS.
How would this block a DOS?
Sorry, I didn't make myself clear - there's the possibility that someone could cause a DOS with the automatic list by faking the source IP address and sending a few packets on a random port.
-Michael
Tim
I'll update the code based on our discussions, and submit an updated set of patches - I imagine there will have to be at least one more iteration.
Let’s wait until we have come to decisions :)
-Michael
Tim
(No additional comments below)
On 13/12/2019 23:11, Michael Tremer wrote:
Hi,
Again my apologies for my late reply. Busy busy weeks.
> On 8 Dec 2019, at 20:50, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote: > > Hello, > > It's my turn to apologise for being slow to respond - I've had a busy > week, but I should have plenty of time over the next couple of weeks.
No worries. Turns out we all do :)
> I've made most of the comments inline, however I think Michael had a > question (which I can't find now) about what happens if someone enables > all the lists. One thing which would perhaps make this less likely is > that the WUI tags the available lists with whether they're safe or not, > with a footnote that safe means that the list only blocks malicious > traffic. This won't guarantee that a user won't still try to enable all > the lists, but it should make them realise that they should think first.
We had a couple of features going slightly wrong or being “misunderstood” by some users. People still seem to panic when they see “local recursor”. To this day I do not know why.
We cannot make everything idiot-proof. And when some user if of that category, they probably should shutdown their IPFire box, educate themselves and then come back again. So I do not want to limit people, but make things as easy as possible.
If someone enables all the lists, good luck with passing packets :)
> I have considered replacing this tag with a risk high/medium/low and > maybe adding a category (invalid/application/scanner/C&C or something > like that), but that may provide too much information and dissuade them > from actually following the links to checkout what the list actually does.
Can we have a screenshot of the GUI right now? I didn’t run the code, yet.
We should document the lists like we do it with the rulesets of the IPS. People might ignore this, but that is on them.
> > Tim > > > On 05/12/2019 22:25, Michael Tremer wrote: >> Hello, >> >>> On 4 Dec 2019, at 17:05, Peter Müller peter.mueller@ipfire.org wrote: >>> >>> Hello Tim, hello Michael, >>> >>> please see my responses inline... >>>>>> >>>>>> We could periodically update the blacklists on our main mirror (and >>>>>> wait for the network to sync it), make sure it is signed and write >>>>>> a small downloader that fetches, validates and installs them. >>>>>> >>>>>> @All: Thoughts on this? >>>>> >>>>> I think there are a number of points here. >>>>> >>>>> Firstly, from the point of a third party using IPFire, is this really >>>>> solving the privacy disclosure problem? There's no way round disclosing >>>>> your public IP address to someone you're downloading from; all this does >>>>> is change who that information is being disclosed to. For the user >>>>> there's no way of knowing whether the source is more or less protective >>>>> of the user's privacy than the blacklist provider. Indeed it won't be >>>>> possible to know who the lists are being downloaded from until the >>>>> download starts. >>>> >>>> There is a way: Tor. But that is a totally different story. >>> Well, I see a third option on this: Use the mirror infrastructure we already >>> have. Every IPFire installation discloses its public IP address to one of these >>> servers sooner or later, so we do not disclose additional data if the blacklists >>> were fetched from these. >>> >>> Needless to say, Tor (hidden services) would be better, but that is a different >>> story indeed. :-) >>>> >>>> The point is rather that a forget list can be sent instead of the “real” one. >>> I did not get this. Forget? Forged?? ??? >> >> Yes, I meant to write forged, but auto-correct didn’t let me. >> >>>>> Secondly, latency; some of the lists are updated every 5 minutes. While >>>>> I've limited the maximum check rate to hourly, will the updates >>>>> propagate quickly enough. For reference on my main system the 24 >>>>> updates on the CIARMY list made 143 498 changes (additions or >>>>> deletions). I've seem it do over 200 000. >>> Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not >>> document a recommended update interval anywhere, so we can only guess. >>> >>> Personally, more static lists seem to be preferable for packet filtering. Highly >>> dynamic ones such as CIArmy should be done via DNSBL queries or something similar >>> - do we really want to have that list here? >> >> It is not really an option to implement a DNSBL into a packet filter, but I get your point. > > One of the 'selling points' for an IP address blacklist is that it can > respond quickly to new threats - or rather new attackers. While a new > IDS/IPS rule needs time to analyse the threat, generate a rule and check > it, it's easy to add an address to a list. So, I think the CIArmy list > is potentially useful for protecting home systems etc. with budget > hardware, but I would be very careful about using it for a protecting a > general access website.
If they are very volatile, we should honour that and update them often, too.
It probably is more about false-positives being removed very quickly instead of adding threats very quickly. The average IPFire user is probably not under threats like these to need to react very quickly.
So I do not see much value in adding those lists and then updating once a day. Does it have to be every 5 min? No. I would suggest 15 which should be good enough for everyone.
Other lists should of course not be updated every 15 minutes when not needed.
Running every 15 minutes would allow us to retry downloading lists that are on an hourly schedule if the download failed.
>> >>>> How did you come up with the hour? Will it be retried more often if the download was not successful? >>> One hour is the most common interval indeed, but adding some random time might >>> be useful in order to reduce load on the servers providing a blacklist. >> >> Yes, definitely. Otherwise we will shoot down our mirrors. > > When I implemented that section of code, specifying the minimum check > period in hours seemed to provide a convenient way of allowing a check > period covering a wide range, with an hour as the fastest and a week as > the slowest. I didn't looked at the CIArmy list until much later. Most > of the lists don't change nearly as much, but the CIArmy list is > described as one that deliberately responds quickly. > > From my production system, for yesterday: > > The following block lists were updated: > BLOCKLIST_DE: 24 Time(s) - 9341 change(s) > BOGON_FULL: 1 Time(s) - 10 change(s) > CIARMY: 24 Time(s) - 159134 change(s) > DSHIELD: 7 Time(s) - 18 change(s) > EMERGING_FWRULE: 1 Time(s) - 50 change(s) > FEODO_AGGRESIVE: 24 Time(s) - 13 change(s) > SHODAN: 1 Time(s) - 0 change(s) > SPAMHAUS_DROP: 1 Time(s) - 0 change(s) > TOR_EXIT: 24 Time(s) - 162 change(s)
Very interesting statistics.
> > and my test system: > > The following block lists were updated: > ALIENVAULT: 19 Time(s) - 5331 change(s) > EMERGING_COMPROMISED: 1 Time(s) - 26 change(s) > TALOS_MALICIOUS: 1 Time(s) - 36 change(s) > > That covers most of the lists. From the WUI, since 1 Dec: > > Blacklist Entries pkts bytes Last updated > in in > AUTOBLACKLIST 0 731 51144 Sun Dec 8 18:40:02 2019 > BLOCKLIST_DE 28020 857 46735 Sun Dec 8 18:40:02 2019 > BOGON_FULL 214 5255 189K Sun Dec 8 16:50:04 2019 > CIARMY 15000 19774 976K Sun Dec 8 18:04:01 2019 > DSHIELD 20 7992 321K Sun Dec 8 16:45:13 2019 > EMERGING_FWRULE 1647 197 8383 Fri Dec 6 05:29:07 2019 > FEODO_AGGRESIVE 7169 0 0 Sun Dec 8 18:50:07 2019 > SHODAN 32 34 1530 Sun Dec 8 17:54:09 2019 > SPAMHAUS_DROP 823 0 0 Fri Dec 6 18:22:35 2019 > SPAMHAUS_EDROP 111 82 16433 Thu Dec 5 17:27:09 2019 > TOR_EXIT 1055 0 0 Sun Dec 8 18:31:02 2019
This as well.
Those are more packets than I would have expected.
> > (I've left out the pkts/bytes out fields which were all 0) > > Note that where possible I do a HEAD request first and then only > download the list if the modification time has changed since the last > check. For dynamically generated lists this isn't possible.
You won’t need a HEAD request for it. You can include it in the GET request.
Have a look at location downloader where I use that. The server will respond with 304 and not send any payload.
https://git.ipfire.org/?p=location/libloc.git;a=blob;f=src/python/location-d...
> > If the download isn't successful it just gives up and waits for the next > attempt (apart from the usual retries in the library). I probably > should to change that so that it only applies the per list minimum > update period in this case (specified in the sources file) rather than > the user specified value as well.
I think it is not the worst if an update fails. It might just happen every once in a while.
So I would suggest to just re-run the script more often and when the mtime of the file is older than the threshold, a download is attempted. You can use that timestamp for the GET request.
> I already use a time offset on the downloads - when it's started from > boot, backup restore or WUI enable, it checks to see if it's installed > in the fcrontab, and if not adds itself at a randomly generated offset > in the hour.
That should potentially go to red.up, or if we can settle on 15 minutes, I would consider that often enough to quickly update all lists after a reboot.
> >> >>>>> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is >>>>> the largest at a few MB), there are going to be a lot of them. How will >>>>> this affect the willingness of people to mirror IPFire? >>> I do not consider this being a problem as we do not generate that much traffic >>> to them. Of course, that depends on the update interval again. >> >> That depends on your point of view. >> >> I do not have a problem with this at all in my data center, but there are plenty of people with a volume-based LTE plan or simply a 128 kBit/s connection. It will take a longer time to download the lists for them. We need to mind that. >> >>>>> >>>>>> >>>>>> Talking about the preference of packet filter and IPS, I prefer to >>>>>> use the latter as well as it gains more insight in what kind of malicious >>>>>> traffic tried to pass a firewall machine. On systems with low resources, >>>>>> this might be problematic and removing load from the IPS can be preferred >>>>>> (make this configurable?!), on others, people might want to have both >>>>>> results. >>>>>> >>>>> You're only going to get one result for a packet whichever way round the >>>>> IP blacklist and IPS are since whichever comes first will drop the >>>>> packet before it reaches the second (well it would be possible to put >>>>> the IP blacklist first and get it to log and mark packets which are then >>>>> dropped after the IPS, but I think that's getting a little complicated. >>>>> In addition I've seen the messages about the trouble marking was >>>>> causing in the QoS). >>>>> >>>>> I think it's a 50/50 choice as to which is more valuable first; it's >>>>> probably going to differ from packet to packet. For me the possibility >>>>> of reducing the IPS load means I prefer putting the IP blacklist first >>>>> >>>>> It should be fairly easy to add the choice of where to put the IP >>>>> blacklist. I think it'll have to be in the main firewall script, so >>>>> it'll require a firewall restart, but it's not something that'll be >>>>> changed often. >>>> >>>> I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier. >>> Agreed. >>> >>>> >>>> I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads. >>>> >>>> We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily. >>> So dropping packets from blacklisted IP addresses/networks before IPS is it, then. >>>> >>>> I do not even think it makes sense to swap the order in the outgoing direction. >>> Me too. >>> >>>> >>>> What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this. >>> Based on my personal experience, firewall hits observed on a single machine exposed >>> to the internet are interesting, but the overall situation across multiple machines >>> is even more interesting. Very quickly, you'll end on something like a centralised >>> logging server and custom statistical analysis here... >> >> Probably a project for IPFire 4.0 :) >> > Or use one of the existing services, like the DSHIELD client > https://dshield.org/howto.html (subject to privacy concerns again). >>>> >>>>>> >>>>>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default. >>>>>> I would love to see the bogon ruleset here, too (think about 8chan successor >>>>>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause >>>>>> interference if RED does not have a public IP address assigned. >>>>> >>>>> I can add a field to the options file that controls whether a list is >>>>> enabled by default. >>> Thank you. :-) >>>> >>>> To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat. >>> This is my only objection against this patchset. Now, what can we do about it? >>> One possibility is to apply the patchset now and implement a custom download >>> source thing later on, or do that before releasing Core Update 139 (or which version >>> the patchset will be to) after we agreed on something. >> >> I do not see this being merged for 139. But that is not important. We need to get it right first and then release it. >> >> As far as I know, nobody has tested this, yet. > > There are a number of people who have been running an earlier version > which I shared on GitHub. There were a few early issues, but it seems > to be OK now. > > https://forum.ipfire.org/viewtopic.php?f=27&t=21845 > > This version wasn't integrated into IPFire, so (for example) it inserted > itself into the INPUT IPTables chain rather than having it's chains > created as part of the firewall start-up script. > >> >> I have huge concerns about the automatic blacklist. @Peter: What is your opinion on this? >> > > While I implemented it, I'm aware of its potential to cause problems, > which is why it has to be separately enabled. It's not caused me any > issues at the default settings (blocks at over 10 packets per hour until > 1 hour has passed without seeing packets from the address), but I've not > used it on a site with publicly announced services. If I was going to > use it on a web site I would want to, at the very minimum, drop the > block period drastically.
I suppose this is entirely unusable on an IPFire box in a data center that hosts things. Let’s say our rack in Hanover.
You will have hits from some broken IP stacks and people might just end up on this without doing anything wrong.
You can denial-of-service easily as well and I suppose without aggregation of data from many many systems, it does not make sense to instantly block addresses. And even then you probably would block an entire subnet.
So I am not sure how I should feel about it. I do not think that it adds anything because the packets would have been blocked anyways.
> > On the other hand, it's good at responding quickly. Usually I see only > 1-2% of blocks from the automatic list: > > Reason Count % First Last > CIARMY 2416 45 Dec 6 00:00 Dec 6 23:59 > DSHIELD 1353 25 Dec 6 00:00 Dec 6 23:59 > INPUT 1294 24 Dec 6 00:00 Dec 6 23:59 > AUTOBLACKLIST 122 2 Dec 6 00:20 Dec 6 16:28 > BLOCKLIST_DE 89 2 Dec 6 00:20 Dec 6 23:46 > > and sometimes none at all, but one one occasion it blocked over 8000 > packets. Again I'm aware this is for a home system, which is rather > different than from a Web server. > >>> If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not >>> seem to be problematic, as they are hosted on 3rd party servers, too). >> >> One of them will be, sooner or later. And one is enough I suppose. > > DShield (https://dshield.org/api/#threatfeeds) and firehol > (http://iplists.firehol.org/) seem to host copies of most of the lists > as well. >> >> I do not really want to overthink this - we didn’t do this with clamav for example either. But it probably is a decision to be made by the user what they want to enable and we should not enable anything by default. So no data will be leaked as long as the user does not consent. >> >> -Michael >> >>> >>> Thanks, and best regards, >>> Peter Müller
<Screenshot_2019-12-16_IP_Address_Blacklists.png>
<Screenshot_2019-12-17_IP_Address_Blacklists.png>
Hello,
I've now updated the code - screenshots are attached.
IP_Address_Blacklists.png
This is the settings page. I don't think there's anything unexpected here.
IP_Address_Blacklist_Logs.png
The log page, replacing the status information that was previously at the top of the settings page. I've sorted the lists alphabetically, rather than by the number of hits since the worst threat is not necessary the largest number of blocked packets (which could implied by sorting it by number).
Firewall_log_blacklist.png
Details page accessed from the log. Almost identical to the other similar pages.
Log_Summary.png
This is an extract from the Log Summary page produced by logwatch.
I still need to do some tidying, remove unused language strings etc., and then update and test the patches, so it'll be some time before I'm ready to submit updated patches.
Tim
On 28/12/2019 20:59, Tim FitzGeorge wrote:
Hi,
I'll code and test what we've agreed on. In the meantime I'm going to reply on the sub-thread started by Tom Rymes as to what we call the list categories, so that we can keep that discussion separate from further functionality etc. discussion.
Tim
On 21/12/2019 18:34, Tim FitzGeorge wrote:
Hi,
I've got my systems checking for downloads at 15 minute intervals and using If-Modified-Since. This all seems to be working OK.
Responses to comments below:
On 18/12/2019 12:07, Michael Tremer wrote:
Hi,
On 17 Dec 2019, at 18:21, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Hi,
I think we've agreed on the following:
- Checks to be made every 15 minutes, subject to per-list minimum rate.
- Modify download to use If-Modified-Since rather than separate HEAD
request.
- Remove automatic blacklist.
I thought someone wants to counter my argument. This is not a dictatorship :)
other comments are below.
Tim
On 16/12/2019 22:20, Michael Tremer wrote:
Hi,
On 16 Dec 2019, at 20:06, Tim FitzGeorge lists@tfitzgeorge.me.uk wrote:
Hi,
I've attached the current GUI screenshot.
Thanks for that.
I have a couple of suggestions/concerns about it:
a) I am not sure what the value is about the top box that is called “Status”. It is basically a summary of the iptables output, but will it help the user? A blacklist with more or fewer entries is not better or worse, blocking more packets isn’t better than blocking fewer. It is about the quality of the blocks.
I agree that the size of the lists is not all that useful, but the information about the number of blocked packets can be an indication of compromise. It depends on the type of the list, but a list that targets C&C channels (e.g. the FEODO lists) should not show any input packets being blocked, unless one of the computers being protected is infected. Apart from the BOGON lists, none of the lists should show any output packets being blocked; again it's a potential indication of infection if otherwise.
The Shodan list will definitely show some activity unless you are behind CG NAT.
Should we not have this rather in the logging section?
And these counters here will be reset when the firewall is being reloaded. Should this data therefore not be collected from the logs so that you can go back in time? That would be very important to identify any start of compromise.
That makes sense. I'll remove the status from this page and create a log page for the information. If anyone actually wants the iptables information it's available in from Firewall > iptables anyway.
Do you think it makes more sense to show packets blocked in/out or packets blocked per source interface? The latter would of (limited) help in tracking down potential compromise.
b) I would prefer to move the settings box to the top. If you like you can show the size of the blacklists there and when they have been updated, too.
I put the status at the top since it's usually what I want to see when I go to the page. Under normal circumstances I don't change the settings, so I'd rather not have to scroll past them to get to the information I'm actually interested in. The update times are just useful to show that the lists are actually being updated. The size is interesting to me, but as you say it probably won't help the average user, especially if you don't know which lists block individual addresses and which block nets (BOGON, DSHIELD and EMERGING_COMPROMISED).
See my comment on the order above. Splitting the page might make sense.
When you show size, does that mean lines in the blacklists? So it could be networks or single IP addresses? Does it not make sense to count IP addresses (i.e. 256 for a /24 network and so on) and put it in the table?
It's lines in the blacklist.
I think I would only be interested in the size for two things:
a) Is there any chance for overblocking? i.e. does the list have 2 billion entries which would be half the public IPv4 address space or so. I cannot really come up with a good example here.
Or
b) What is the performance impact/memory consumption of the list?
It would be interesting to show the type (single address or net), number of entries and number of addresses; the first two affect performance, the second memory use and the latter the number of blocked addresses, but that's really only going to be of interest to a very small number of people, who can always do 'ipset list -t' to get the information. So I'll remove the size column.
I originally had the status and settings in one list, but I decided that it was getting a bit messy and difficult to follow. I've attached a screenshot of a quickly hacked page. (Note that the status area only appears once the blacklists have been enabled.)
You are right. This is getting quite tight there.
You could write packets in/out into on column like 2k/1023, but see my concerns about this above.
Actually I've come to the conclusion that the number of bytes blocked isn't all that important; it's sufficient to show the number of packets.
c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
I agree that it's it's an over simplistic summary of the lists. I note that Tom Rymes has asked to keep it, but I think its only value is if it suggests to users that things are more complicated than they appear and therefore prompts them to read the Wiki.
I will reply to that separately.
Looking at that reply, I think it makes sense to include a category column in both the settings and the log summary page - the former to help selection of which lists to use, and the latter as a help when deciding whether the information could indicate compromise or not.
d) I do not understand what the “update check rate” means. I suppose we should show the update interval of the lists in the table and just refresh them according to that.
It's the minimum check period for updates. The system checks for updates at either the rate specific to the list or this rate, whichever is slower. This is a hangover from the early days, copied from the IPS updates. I suppose it could have a value for some people who may wish to lower the check rate, but I'm not sure that it has sufficient value to be worth keeping.
No, I would not let the user decide when and how often to update the lists.
OK. I'll remove it.
Generally there is this argument internally how often some databases should be updated. People with slow internet connections or volume based data plans will find this rather expensive to update those lists too often.
If we ever want to go ahead with that in IPFire 2, there should be a global switch for a “low data mode” which then delays updating these things.
That makes sense. Maybe also the ability to set when downloads take place - but as you say that's for the future.
e) Do we want to keep the automatic blacklists now? I do not think it will actually improve the security of IPFire.
I'll remove them. I think they do have a use, in that they can detect someone doing a port scan and block them before they find an open port, but as you say there is also the possibility of a DOS.
How would this block a DOS?
Sorry, I didn't make myself clear - there's the possibility that someone could cause a DOS with the automatic list by faking the source IP address and sending a few packets on a random port.
-Michael
Tim
I'll update the code based on our discussions, and submit an updated set of patches - I imagine there will have to be at least one more iteration.
Let’s wait until we have come to decisions :)
-Michael
Tim
(No additional comments below)
On 13/12/2019 23:11, Michael Tremer wrote: > Hi, > > Again my apologies for my late reply. Busy busy weeks. > >> On 8 Dec 2019, at 20:50, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote: >> >> Hello, >> >> It's my turn to apologise for being slow to respond - I've had a busy >> week, but I should have plenty of time over the next couple of weeks. > > No worries. Turns out we all do :) > >> I've made most of the comments inline, however I think Michael had a >> question (which I can't find now) about what happens if someone enables >> all the lists. One thing which would perhaps make this less likely is >> that the WUI tags the available lists with whether they're safe or not, >> with a footnote that safe means that the list only blocks malicious >> traffic. This won't guarantee that a user won't still try to enable all >> the lists, but it should make them realise that they should think first. > > We had a couple of features going slightly wrong or being “misunderstood” by some users. People still seem to panic when they see “local recursor”. To this day I do not know why. > > We cannot make everything idiot-proof. And when some user if of that category, they probably should shutdown their IPFire box, educate themselves and then come back again. So I do not want to limit people, but make things as easy as possible. > > If someone enables all the lists, good luck with passing packets :) > >> I have considered replacing this tag with a risk high/medium/low and >> maybe adding a category (invalid/application/scanner/C&C or something >> like that), but that may provide too much information and dissuade them >> from actually following the links to checkout what the list actually does. > > Can we have a screenshot of the GUI right now? I didn’t run the code, yet. > > We should document the lists like we do it with the rulesets of the IPS. People might ignore this, but that is on them. > >> >> Tim >> >> >> On 05/12/2019 22:25, Michael Tremer wrote: >>> Hello, >>> >>>> On 4 Dec 2019, at 17:05, Peter Müller peter.mueller@ipfire.org wrote: >>>> >>>> Hello Tim, hello Michael, >>>> >>>> please see my responses inline... >>>>>>> >>>>>>> We could periodically update the blacklists on our main mirror (and >>>>>>> wait for the network to sync it), make sure it is signed and write >>>>>>> a small downloader that fetches, validates and installs them. >>>>>>> >>>>>>> @All: Thoughts on this? >>>>>> >>>>>> I think there are a number of points here. >>>>>> >>>>>> Firstly, from the point of a third party using IPFire, is this really >>>>>> solving the privacy disclosure problem? There's no way round disclosing >>>>>> your public IP address to someone you're downloading from; all this does >>>>>> is change who that information is being disclosed to. For the user >>>>>> there's no way of knowing whether the source is more or less protective >>>>>> of the user's privacy than the blacklist provider. Indeed it won't be >>>>>> possible to know who the lists are being downloaded from until the >>>>>> download starts. >>>>> >>>>> There is a way: Tor. But that is a totally different story. >>>> Well, I see a third option on this: Use the mirror infrastructure we already >>>> have. Every IPFire installation discloses its public IP address to one of these >>>> servers sooner or later, so we do not disclose additional data if the blacklists >>>> were fetched from these. >>>> >>>> Needless to say, Tor (hidden services) would be better, but that is a different >>>> story indeed. :-) >>>>> >>>>> The point is rather that a forget list can be sent instead of the “real” one. >>>> I did not get this. Forget? Forged?? ??? >>> >>> Yes, I meant to write forged, but auto-correct didn’t let me. >>> >>>>>> Secondly, latency; some of the lists are updated every 5 minutes. While >>>>>> I've limited the maximum check rate to hourly, will the updates >>>>>> propagate quickly enough. For reference on my main system the 24 >>>>>> updates on the CIARMY list made 143 498 changes (additions or >>>>>> deletions). I've seem it do over 200 000. >>>> Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not >>>> document a recommended update interval anywhere, so we can only guess. >>>> >>>> Personally, more static lists seem to be preferable for packet filtering. Highly >>>> dynamic ones such as CIArmy should be done via DNSBL queries or something similar >>>> - do we really want to have that list here? >>> >>> It is not really an option to implement a DNSBL into a packet filter, but I get your point. >> >> One of the 'selling points' for an IP address blacklist is that it can >> respond quickly to new threats - or rather new attackers. While a new >> IDS/IPS rule needs time to analyse the threat, generate a rule and check >> it, it's easy to add an address to a list. So, I think the CIArmy list >> is potentially useful for protecting home systems etc. with budget >> hardware, but I would be very careful about using it for a protecting a >> general access website. > > If they are very volatile, we should honour that and update them often, too. > > It probably is more about false-positives being removed very quickly instead of adding threats very quickly. The average IPFire user is probably not under threats like these to need to react very quickly. > > So I do not see much value in adding those lists and then updating once a day. Does it have to be every 5 min? No. I would suggest 15 which should be good enough for everyone. > > Other lists should of course not be updated every 15 minutes when not needed. > > Running every 15 minutes would allow us to retry downloading lists that are on an hourly schedule if the download failed. > >>> >>>>> How did you come up with the hour? Will it be retried more often if the download was not successful? >>>> One hour is the most common interval indeed, but adding some random time might >>>> be useful in order to reduce load on the servers providing a blacklist. >>> >>> Yes, definitely. Otherwise we will shoot down our mirrors. >> >> When I implemented that section of code, specifying the minimum check >> period in hours seemed to provide a convenient way of allowing a check >> period covering a wide range, with an hour as the fastest and a week as >> the slowest. I didn't looked at the CIArmy list until much later. Most >> of the lists don't change nearly as much, but the CIArmy list is >> described as one that deliberately responds quickly. >> >> From my production system, for yesterday: >> >> The following block lists were updated: >> BLOCKLIST_DE: 24 Time(s) - 9341 change(s) >> BOGON_FULL: 1 Time(s) - 10 change(s) >> CIARMY: 24 Time(s) - 159134 change(s) >> DSHIELD: 7 Time(s) - 18 change(s) >> EMERGING_FWRULE: 1 Time(s) - 50 change(s) >> FEODO_AGGRESIVE: 24 Time(s) - 13 change(s) >> SHODAN: 1 Time(s) - 0 change(s) >> SPAMHAUS_DROP: 1 Time(s) - 0 change(s) >> TOR_EXIT: 24 Time(s) - 162 change(s) > > Very interesting statistics. > >> >> and my test system: >> >> The following block lists were updated: >> ALIENVAULT: 19 Time(s) - 5331 change(s) >> EMERGING_COMPROMISED: 1 Time(s) - 26 change(s) >> TALOS_MALICIOUS: 1 Time(s) - 36 change(s) >> >> That covers most of the lists. From the WUI, since 1 Dec: >> >> Blacklist Entries pkts bytes Last updated >> in in >> AUTOBLACKLIST 0 731 51144 Sun Dec 8 18:40:02 2019 >> BLOCKLIST_DE 28020 857 46735 Sun Dec 8 18:40:02 2019 >> BOGON_FULL 214 5255 189K Sun Dec 8 16:50:04 2019 >> CIARMY 15000 19774 976K Sun Dec 8 18:04:01 2019 >> DSHIELD 20 7992 321K Sun Dec 8 16:45:13 2019 >> EMERGING_FWRULE 1647 197 8383 Fri Dec 6 05:29:07 2019 >> FEODO_AGGRESIVE 7169 0 0 Sun Dec 8 18:50:07 2019 >> SHODAN 32 34 1530 Sun Dec 8 17:54:09 2019 >> SPAMHAUS_DROP 823 0 0 Fri Dec 6 18:22:35 2019 >> SPAMHAUS_EDROP 111 82 16433 Thu Dec 5 17:27:09 2019 >> TOR_EXIT 1055 0 0 Sun Dec 8 18:31:02 2019 > > This as well. > > Those are more packets than I would have expected. > >> >> (I've left out the pkts/bytes out fields which were all 0) >> >> Note that where possible I do a HEAD request first and then only >> download the list if the modification time has changed since the last >> check. For dynamically generated lists this isn't possible. > > You won’t need a HEAD request for it. You can include it in the GET request. > > Have a look at location downloader where I use that. The server will respond with 304 and not send any payload. > > https://git.ipfire.org/?p=location/libloc.git;a=blob;f=src/python/location-d... > >> >> If the download isn't successful it just gives up and waits for the next >> attempt (apart from the usual retries in the library). I probably >> should to change that so that it only applies the per list minimum >> update period in this case (specified in the sources file) rather than >> the user specified value as well. > > I think it is not the worst if an update fails. It might just happen every once in a while. > > So I would suggest to just re-run the script more often and when the mtime of the file is older than the threshold, a download is attempted. You can use that timestamp for the GET request. > >> I already use a time offset on the downloads - when it's started from >> boot, backup restore or WUI enable, it checks to see if it's installed >> in the fcrontab, and if not adds itself at a randomly generated offset >> in the hour. > > That should potentially go to red.up, or if we can settle on 15 minutes, I would consider that often enough to quickly update all lists after a reboot. > >> >>> >>>>>> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is >>>>>> the largest at a few MB), there are going to be a lot of them. How will >>>>>> this affect the willingness of people to mirror IPFire? >>>> I do not consider this being a problem as we do not generate that much traffic >>>> to them. Of course, that depends on the update interval again. >>> >>> That depends on your point of view. >>> >>> I do not have a problem with this at all in my data center, but there are plenty of people with a volume-based LTE plan or simply a 128 kBit/s connection. It will take a longer time to download the lists for them. We need to mind that. >>> >>>>>> >>>>>>> >>>>>>> Talking about the preference of packet filter and IPS, I prefer to >>>>>>> use the latter as well as it gains more insight in what kind of malicious >>>>>>> traffic tried to pass a firewall machine. On systems with low resources, >>>>>>> this might be problematic and removing load from the IPS can be preferred >>>>>>> (make this configurable?!), on others, people might want to have both >>>>>>> results. >>>>>>> >>>>>> You're only going to get one result for a packet whichever way round the >>>>>> IP blacklist and IPS are since whichever comes first will drop the >>>>>> packet before it reaches the second (well it would be possible to put >>>>>> the IP blacklist first and get it to log and mark packets which are then >>>>>> dropped after the IPS, but I think that's getting a little complicated. >>>>>> In addition I've seen the messages about the trouble marking was >>>>>> causing in the QoS). >>>>>> >>>>>> I think it's a 50/50 choice as to which is more valuable first; it's >>>>>> probably going to differ from packet to packet. For me the possibility >>>>>> of reducing the IPS load means I prefer putting the IP blacklist first >>>>>> >>>>>> It should be fairly easy to add the choice of where to put the IP >>>>>> blacklist. I think it'll have to be in the main firewall script, so >>>>>> it'll require a firewall restart, but it's not something that'll be >>>>>> changed often. >>>>> >>>>> I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier. >>>> Agreed. >>>> >>>>> >>>>> I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads. >>>>> >>>>> We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily. >>>> So dropping packets from blacklisted IP addresses/networks before IPS is it, then. >>>>> >>>>> I do not even think it makes sense to swap the order in the outgoing direction. >>>> Me too. >>>> >>>>> >>>>> What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this. >>>> Based on my personal experience, firewall hits observed on a single machine exposed >>>> to the internet are interesting, but the overall situation across multiple machines >>>> is even more interesting. Very quickly, you'll end on something like a centralised >>>> logging server and custom statistical analysis here... >>> >>> Probably a project for IPFire 4.0 :) >>> >> Or use one of the existing services, like the DSHIELD client >> https://dshield.org/howto.html (subject to privacy concerns again). >>>>> >>>>>>> >>>>>>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default. >>>>>>> I would love to see the bogon ruleset here, too (think about 8chan successor >>>>>>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause >>>>>>> interference if RED does not have a public IP address assigned. >>>>>> >>>>>> I can add a field to the options file that controls whether a list is >>>>>> enabled by default. >>>> Thank you. :-) >>>>> >>>>> To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat. >>>> This is my only objection against this patchset. Now, what can we do about it? >>>> One possibility is to apply the patchset now and implement a custom download >>>> source thing later on, or do that before releasing Core Update 139 (or which version >>>> the patchset will be to) after we agreed on something. >>> >>> I do not see this being merged for 139. But that is not important. We need to get it right first and then release it. >>> >>> As far as I know, nobody has tested this, yet. >> >> There are a number of people who have been running an earlier version >> which I shared on GitHub. There were a few early issues, but it seems >> to be OK now. >> >> https://forum.ipfire.org/viewtopic.php?f=27&t=21845 >> >> This version wasn't integrated into IPFire, so (for example) it inserted >> itself into the INPUT IPTables chain rather than having it's chains >> created as part of the firewall start-up script. >> >>> >>> I have huge concerns about the automatic blacklist. @Peter: What is your opinion on this? >>> >> >> While I implemented it, I'm aware of its potential to cause problems, >> which is why it has to be separately enabled. It's not caused me any >> issues at the default settings (blocks at over 10 packets per hour until >> 1 hour has passed without seeing packets from the address), but I've not >> used it on a site with publicly announced services. If I was going to >> use it on a web site I would want to, at the very minimum, drop the >> block period drastically. > > I suppose this is entirely unusable on an IPFire box in a data center that hosts things. Let’s say our rack in Hanover. > > You will have hits from some broken IP stacks and people might just end up on this without doing anything wrong. > > You can denial-of-service easily as well and I suppose without aggregation of data from many many systems, it does not make sense to instantly block addresses. And even then you probably would block an entire subnet. > > So I am not sure how I should feel about it. I do not think that it adds anything because the packets would have been blocked anyways. > >> >> On the other hand, it's good at responding quickly. Usually I see only >> 1-2% of blocks from the automatic list: >> >> Reason Count % First Last >> CIARMY 2416 45 Dec 6 00:00 Dec 6 23:59 >> DSHIELD 1353 25 Dec 6 00:00 Dec 6 23:59 >> INPUT 1294 24 Dec 6 00:00 Dec 6 23:59 >> AUTOBLACKLIST 122 2 Dec 6 00:20 Dec 6 16:28 >> BLOCKLIST_DE 89 2 Dec 6 00:20 Dec 6 23:46 >> >> and sometimes none at all, but one one occasion it blocked over 8000 >> packets. Again I'm aware this is for a home system, which is rather >> different than from a Web server. >> >>>> If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not >>>> seem to be problematic, as they are hosted on 3rd party servers, too). >>> >>> One of them will be, sooner or later. And one is enough I suppose. >> >> DShield (https://dshield.org/api/#threatfeeds) and firehol >> (http://iplists.firehol.org/) seem to host copies of most of the lists >> as well. >>> >>> I do not really want to overthink this - we didn’t do this with clamav for example either. But it probably is a decision to be made by the user what they want to enable and we should not enable anything by default. So no data will be leaked as long as the user does not consent. >>> >>> -Michael >>> >>>> >>>> Thanks, and best regards, >>>> Peter Müller >
<Screenshot_2019-12-16_IP_Address_Blacklists.png>
<Screenshot_2019-12-17_IP_Address_Blacklists.png>
Hello all,
On 4 Jan 2020, at 19:02, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Hello,
I've now updated the code - screenshots are attached.
IP_Address_Blacklists.png
This is the settings page. I don't think there's anything unexpected here.
Yes, that looks a lot like the other IPFire pages :)
IP_Address_Blacklist_Logs.png
The log page, replacing the status information that was previously at the top of the settings page.
Looks great!
I've sorted the lists alphabetically, rather than by the number of hits since the worst threat is not necessary the largest number of blocked packets (which could implied by sorting it by number).
Yes, that is indeed a very good idea.
You could add a % sign in the percentage column, so it cannot be confused so easily with the other column.
Firewall_log_blacklist.png
Details page accessed from the log. Almost identical to the other similar pages.
*thumbs up*
Log_Summary.png
This is an extract from the Log Summary page produced by logwatch.
I still need to do some tidying, remove unused language strings etc., and then update and test the patches, so it'll be some time before I'm ready to submit updated patches.
Feel free to send some RFC to the list. It might be slightly noisy, but it is easier to review changes bit after bit.
Great great work!
-Michael
Tim
On 28/12/2019 20:59, Tim FitzGeorge wrote:
Hi,
I'll code and test what we've agreed on. In the meantime I'm going to reply on the sub-thread started by Tom Rymes as to what we call the list categories, so that we can keep that discussion separate from further functionality etc. discussion.
Tim
On 21/12/2019 18:34, Tim FitzGeorge wrote:
Hi,
I've got my systems checking for downloads at 15 minute intervals and using If-Modified-Since. This all seems to be working OK.
Responses to comments below:
On 18/12/2019 12:07, Michael Tremer wrote:
Hi,
On 17 Dec 2019, at 18:21, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Hi,
I think we've agreed on the following:
- Checks to be made every 15 minutes, subject to per-list minimum rate.
- Modify download to use If-Modified-Since rather than separate HEAD
request.
- Remove automatic blacklist.
I thought someone wants to counter my argument. This is not a dictatorship :)
other comments are below.
Tim
On 16/12/2019 22:20, Michael Tremer wrote:
Hi,
> On 16 Dec 2019, at 20:06, Tim FitzGeorge lists@tfitzgeorge.me.uk wrote: > > Hi, > > I've attached the current GUI screenshot.
Thanks for that.
I have a couple of suggestions/concerns about it:
a) I am not sure what the value is about the top box that is called “Status”. It is basically a summary of the iptables output, but will it help the user? A blacklist with more or fewer entries is not better or worse, blocking more packets isn’t better than blocking fewer. It is about the quality of the blocks.
I agree that the size of the lists is not all that useful, but the information about the number of blocked packets can be an indication of compromise. It depends on the type of the list, but a list that targets C&C channels (e.g. the FEODO lists) should not show any input packets being blocked, unless one of the computers being protected is infected. Apart from the BOGON lists, none of the lists should show any output packets being blocked; again it's a potential indication of infection if otherwise.
The Shodan list will definitely show some activity unless you are behind CG NAT.
Should we not have this rather in the logging section?
And these counters here will be reset when the firewall is being reloaded. Should this data therefore not be collected from the logs so that you can go back in time? That would be very important to identify any start of compromise.
That makes sense. I'll remove the status from this page and create a log page for the information. If anyone actually wants the iptables information it's available in from Firewall > iptables anyway.
Do you think it makes more sense to show packets blocked in/out or packets blocked per source interface? The latter would of (limited) help in tracking down potential compromise.
b) I would prefer to move the settings box to the top. If you like you can show the size of the blacklists there and when they have been updated, too.
I put the status at the top since it's usually what I want to see when I go to the page. Under normal circumstances I don't change the settings, so I'd rather not have to scroll past them to get to the information I'm actually interested in. The update times are just useful to show that the lists are actually being updated. The size is interesting to me, but as you say it probably won't help the average user, especially if you don't know which lists block individual addresses and which block nets (BOGON, DSHIELD and EMERGING_COMPROMISED).
See my comment on the order above. Splitting the page might make sense.
When you show size, does that mean lines in the blacklists? So it could be networks or single IP addresses? Does it not make sense to count IP addresses (i.e. 256 for a /24 network and so on) and put it in the table?
It's lines in the blacklist.
I think I would only be interested in the size for two things:
a) Is there any chance for overblocking? i.e. does the list have 2 billion entries which would be half the public IPv4 address space or so. I cannot really come up with a good example here.
Or
b) What is the performance impact/memory consumption of the list?
It would be interesting to show the type (single address or net), number of entries and number of addresses; the first two affect performance, the second memory use and the latter the number of blocked addresses, but that's really only going to be of interest to a very small number of people, who can always do 'ipset list -t' to get the information. So I'll remove the size column.
I originally had the status and settings in one list, but I decided that it was getting a bit messy and difficult to follow. I've attached a screenshot of a quickly hacked page. (Note that the status area only appears once the blacklists have been enabled.)
You are right. This is getting quite tight there.
You could write packets in/out into on column like 2k/1023, but see my concerns about this above.
Actually I've come to the conclusion that the number of bytes blocked isn't all that important; it's sufficient to show the number of packets.
c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
I agree that it's it's an over simplistic summary of the lists. I note that Tom Rymes has asked to keep it, but I think its only value is if it suggests to users that things are more complicated than they appear and therefore prompts them to read the Wiki.
I will reply to that separately.
Looking at that reply, I think it makes sense to include a category column in both the settings and the log summary page - the former to help selection of which lists to use, and the latter as a help when deciding whether the information could indicate compromise or not.
d) I do not understand what the “update check rate” means. I suppose we should show the update interval of the lists in the table and just refresh them according to that.
It's the minimum check period for updates. The system checks for updates at either the rate specific to the list or this rate, whichever is slower. This is a hangover from the early days, copied from the IPS updates. I suppose it could have a value for some people who may wish to lower the check rate, but I'm not sure that it has sufficient value to be worth keeping.
No, I would not let the user decide when and how often to update the lists.
OK. I'll remove it.
Generally there is this argument internally how often some databases should be updated. People with slow internet connections or volume based data plans will find this rather expensive to update those lists too often.
If we ever want to go ahead with that in IPFire 2, there should be a global switch for a “low data mode” which then delays updating these things.
That makes sense. Maybe also the ability to set when downloads take place - but as you say that's for the future.
e) Do we want to keep the automatic blacklists now? I do not think it will actually improve the security of IPFire.
I'll remove them. I think they do have a use, in that they can detect someone doing a port scan and block them before they find an open port, but as you say there is also the possibility of a DOS.
How would this block a DOS?
Sorry, I didn't make myself clear - there's the possibility that someone could cause a DOS with the automatic list by faking the source IP address and sending a few packets on a random port.
-Michael
Tim
> > I'll update the code based on our discussions, and submit an updated set > of patches - I imagine there will have to be at least one more iteration.
Let’s wait until we have come to decisions :)
-Michael
> > Tim > > (No additional comments below) > > On 13/12/2019 23:11, Michael Tremer wrote: >> Hi, >> >> Again my apologies for my late reply. Busy busy weeks. >> >>> On 8 Dec 2019, at 20:50, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote: >>> >>> Hello, >>> >>> It's my turn to apologise for being slow to respond - I've had a busy >>> week, but I should have plenty of time over the next couple of weeks. >> >> No worries. Turns out we all do :) >> >>> I've made most of the comments inline, however I think Michael had a >>> question (which I can't find now) about what happens if someone enables >>> all the lists. One thing which would perhaps make this less likely is >>> that the WUI tags the available lists with whether they're safe or not, >>> with a footnote that safe means that the list only blocks malicious >>> traffic. This won't guarantee that a user won't still try to enable all >>> the lists, but it should make them realise that they should think first. >> >> We had a couple of features going slightly wrong or being “misunderstood” by some users. People still seem to panic when they see “local recursor”. To this day I do not know why. >> >> We cannot make everything idiot-proof. And when some user if of that category, they probably should shutdown their IPFire box, educate themselves and then come back again. So I do not want to limit people, but make things as easy as possible. >> >> If someone enables all the lists, good luck with passing packets :) >> >>> I have considered replacing this tag with a risk high/medium/low and >>> maybe adding a category (invalid/application/scanner/C&C or something >>> like that), but that may provide too much information and dissuade them >>> from actually following the links to checkout what the list actually does. >> >> Can we have a screenshot of the GUI right now? I didn’t run the code, yet. >> >> We should document the lists like we do it with the rulesets of the IPS. People might ignore this, but that is on them. >> >>> >>> Tim >>> >>> >>> On 05/12/2019 22:25, Michael Tremer wrote: >>>> Hello, >>>> >>>>> On 4 Dec 2019, at 17:05, Peter Müller peter.mueller@ipfire.org wrote: >>>>> >>>>> Hello Tim, hello Michael, >>>>> >>>>> please see my responses inline... >>>>>>>> >>>>>>>> We could periodically update the blacklists on our main mirror (and >>>>>>>> wait for the network to sync it), make sure it is signed and write >>>>>>>> a small downloader that fetches, validates and installs them. >>>>>>>> >>>>>>>> @All: Thoughts on this? >>>>>>> >>>>>>> I think there are a number of points here. >>>>>>> >>>>>>> Firstly, from the point of a third party using IPFire, is this really >>>>>>> solving the privacy disclosure problem? There's no way round disclosing >>>>>>> your public IP address to someone you're downloading from; all this does >>>>>>> is change who that information is being disclosed to. For the user >>>>>>> there's no way of knowing whether the source is more or less protective >>>>>>> of the user's privacy than the blacklist provider. Indeed it won't be >>>>>>> possible to know who the lists are being downloaded from until the >>>>>>> download starts. >>>>>> >>>>>> There is a way: Tor. But that is a totally different story. >>>>> Well, I see a third option on this: Use the mirror infrastructure we already >>>>> have. Every IPFire installation discloses its public IP address to one of these >>>>> servers sooner or later, so we do not disclose additional data if the blacklists >>>>> were fetched from these. >>>>> >>>>> Needless to say, Tor (hidden services) would be better, but that is a different >>>>> story indeed. :-) >>>>>> >>>>>> The point is rather that a forget list can be sent instead of the “real” one. >>>>> I did not get this. Forget? Forged?? ??? >>>> >>>> Yes, I meant to write forged, but auto-correct didn’t let me. >>>> >>>>>>> Secondly, latency; some of the lists are updated every 5 minutes. While >>>>>>> I've limited the maximum check rate to hourly, will the updates >>>>>>> propagate quickly enough. For reference on my main system the 24 >>>>>>> updates on the CIARMY list made 143 498 changes (additions or >>>>>>> deletions). I've seem it do over 200 000. >>>>> Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not >>>>> document a recommended update interval anywhere, so we can only guess. >>>>> >>>>> Personally, more static lists seem to be preferable for packet filtering. Highly >>>>> dynamic ones such as CIArmy should be done via DNSBL queries or something similar >>>>> - do we really want to have that list here? >>>> >>>> It is not really an option to implement a DNSBL into a packet filter, but I get your point. >>> >>> One of the 'selling points' for an IP address blacklist is that it can >>> respond quickly to new threats - or rather new attackers. While a new >>> IDS/IPS rule needs time to analyse the threat, generate a rule and check >>> it, it's easy to add an address to a list. So, I think the CIArmy list >>> is potentially useful for protecting home systems etc. with budget >>> hardware, but I would be very careful about using it for a protecting a >>> general access website. >> >> If they are very volatile, we should honour that and update them often, too. >> >> It probably is more about false-positives being removed very quickly instead of adding threats very quickly. The average IPFire user is probably not under threats like these to need to react very quickly. >> >> So I do not see much value in adding those lists and then updating once a day. Does it have to be every 5 min? No. I would suggest 15 which should be good enough for everyone. >> >> Other lists should of course not be updated every 15 minutes when not needed. >> >> Running every 15 minutes would allow us to retry downloading lists that are on an hourly schedule if the download failed. >> >>>> >>>>>> How did you come up with the hour? Will it be retried more often if the download was not successful? >>>>> One hour is the most common interval indeed, but adding some random time might >>>>> be useful in order to reduce load on the servers providing a blacklist. >>>> >>>> Yes, definitely. Otherwise we will shoot down our mirrors. >>> >>> When I implemented that section of code, specifying the minimum check >>> period in hours seemed to provide a convenient way of allowing a check >>> period covering a wide range, with an hour as the fastest and a week as >>> the slowest. I didn't looked at the CIArmy list until much later. Most >>> of the lists don't change nearly as much, but the CIArmy list is >>> described as one that deliberately responds quickly. >>> >>> From my production system, for yesterday: >>> >>> The following block lists were updated: >>> BLOCKLIST_DE: 24 Time(s) - 9341 change(s) >>> BOGON_FULL: 1 Time(s) - 10 change(s) >>> CIARMY: 24 Time(s) - 159134 change(s) >>> DSHIELD: 7 Time(s) - 18 change(s) >>> EMERGING_FWRULE: 1 Time(s) - 50 change(s) >>> FEODO_AGGRESIVE: 24 Time(s) - 13 change(s) >>> SHODAN: 1 Time(s) - 0 change(s) >>> SPAMHAUS_DROP: 1 Time(s) - 0 change(s) >>> TOR_EXIT: 24 Time(s) - 162 change(s) >> >> Very interesting statistics. >> >>> >>> and my test system: >>> >>> The following block lists were updated: >>> ALIENVAULT: 19 Time(s) - 5331 change(s) >>> EMERGING_COMPROMISED: 1 Time(s) - 26 change(s) >>> TALOS_MALICIOUS: 1 Time(s) - 36 change(s) >>> >>> That covers most of the lists. From the WUI, since 1 Dec: >>> >>> Blacklist Entries pkts bytes Last updated >>> in in >>> AUTOBLACKLIST 0 731 51144 Sun Dec 8 18:40:02 2019 >>> BLOCKLIST_DE 28020 857 46735 Sun Dec 8 18:40:02 2019 >>> BOGON_FULL 214 5255 189K Sun Dec 8 16:50:04 2019 >>> CIARMY 15000 19774 976K Sun Dec 8 18:04:01 2019 >>> DSHIELD 20 7992 321K Sun Dec 8 16:45:13 2019 >>> EMERGING_FWRULE 1647 197 8383 Fri Dec 6 05:29:07 2019 >>> FEODO_AGGRESIVE 7169 0 0 Sun Dec 8 18:50:07 2019 >>> SHODAN 32 34 1530 Sun Dec 8 17:54:09 2019 >>> SPAMHAUS_DROP 823 0 0 Fri Dec 6 18:22:35 2019 >>> SPAMHAUS_EDROP 111 82 16433 Thu Dec 5 17:27:09 2019 >>> TOR_EXIT 1055 0 0 Sun Dec 8 18:31:02 2019 >> >> This as well. >> >> Those are more packets than I would have expected. >> >>> >>> (I've left out the pkts/bytes out fields which were all 0) >>> >>> Note that where possible I do a HEAD request first and then only >>> download the list if the modification time has changed since the last >>> check. For dynamically generated lists this isn't possible. >> >> You won’t need a HEAD request for it. You can include it in the GET request. >> >> Have a look at location downloader where I use that. The server will respond with 304 and not send any payload. >> >> https://git.ipfire.org/?p=location/libloc.git;a=blob;f=src/python/location-d... >> >>> >>> If the download isn't successful it just gives up and waits for the next >>> attempt (apart from the usual retries in the library). I probably >>> should to change that so that it only applies the per list minimum >>> update period in this case (specified in the sources file) rather than >>> the user specified value as well. >> >> I think it is not the worst if an update fails. It might just happen every once in a while. >> >> So I would suggest to just re-run the script more often and when the mtime of the file is older than the threshold, a download is attempted. You can use that timestamp for the GET request. >> >>> I already use a time offset on the downloads - when it's started from >>> boot, backup restore or WUI enable, it checks to see if it's installed >>> in the fcrontab, and if not adds itself at a randomly generated offset >>> in the hour. >> >> That should potentially go to red.up, or if we can settle on 15 minutes, I would consider that often enough to quickly update all lists after a reboot. >> >>> >>>> >>>>>>> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is >>>>>>> the largest at a few MB), there are going to be a lot of them. How will >>>>>>> this affect the willingness of people to mirror IPFire? >>>>> I do not consider this being a problem as we do not generate that much traffic >>>>> to them. Of course, that depends on the update interval again. >>>> >>>> That depends on your point of view. >>>> >>>> I do not have a problem with this at all in my data center, but there are plenty of people with a volume-based LTE plan or simply a 128 kBit/s connection. It will take a longer time to download the lists for them. We need to mind that. >>>> >>>>>>> >>>>>>>> >>>>>>>> Talking about the preference of packet filter and IPS, I prefer to >>>>>>>> use the latter as well as it gains more insight in what kind of malicious >>>>>>>> traffic tried to pass a firewall machine. On systems with low resources, >>>>>>>> this might be problematic and removing load from the IPS can be preferred >>>>>>>> (make this configurable?!), on others, people might want to have both >>>>>>>> results. >>>>>>>> >>>>>>> You're only going to get one result for a packet whichever way round the >>>>>>> IP blacklist and IPS are since whichever comes first will drop the >>>>>>> packet before it reaches the second (well it would be possible to put >>>>>>> the IP blacklist first and get it to log and mark packets which are then >>>>>>> dropped after the IPS, but I think that's getting a little complicated. >>>>>>> In addition I've seen the messages about the trouble marking was >>>>>>> causing in the QoS). >>>>>>> >>>>>>> I think it's a 50/50 choice as to which is more valuable first; it's >>>>>>> probably going to differ from packet to packet. For me the possibility >>>>>>> of reducing the IPS load means I prefer putting the IP blacklist first >>>>>>> >>>>>>> It should be fairly easy to add the choice of where to put the IP >>>>>>> blacklist. I think it'll have to be in the main firewall script, so >>>>>>> it'll require a firewall restart, but it's not something that'll be >>>>>>> changed often. >>>>>> >>>>>> I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier. >>>>> Agreed. >>>>> >>>>>> >>>>>> I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads. >>>>>> >>>>>> We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily. >>>>> So dropping packets from blacklisted IP addresses/networks before IPS is it, then. >>>>>> >>>>>> I do not even think it makes sense to swap the order in the outgoing direction. >>>>> Me too. >>>>> >>>>>> >>>>>> What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this. >>>>> Based on my personal experience, firewall hits observed on a single machine exposed >>>>> to the internet are interesting, but the overall situation across multiple machines >>>>> is even more interesting. Very quickly, you'll end on something like a centralised >>>>> logging server and custom statistical analysis here... >>>> >>>> Probably a project for IPFire 4.0 :) >>>> >>> Or use one of the existing services, like the DSHIELD client >>> https://dshield.org/howto.html (subject to privacy concerns again). >>>>>> >>>>>>>> >>>>>>>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default. >>>>>>>> I would love to see the bogon ruleset here, too (think about 8chan successor >>>>>>>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause >>>>>>>> interference if RED does not have a public IP address assigned. >>>>>>> >>>>>>> I can add a field to the options file that controls whether a list is >>>>>>> enabled by default. >>>>> Thank you. :-) >>>>>> >>>>>> To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat. >>>>> This is my only objection against this patchset. Now, what can we do about it? >>>>> One possibility is to apply the patchset now and implement a custom download >>>>> source thing later on, or do that before releasing Core Update 139 (or which version >>>>> the patchset will be to) after we agreed on something. >>>> >>>> I do not see this being merged for 139. But that is not important. We need to get it right first and then release it. >>>> >>>> As far as I know, nobody has tested this, yet. >>> >>> There are a number of people who have been running an earlier version >>> which I shared on GitHub. There were a few early issues, but it seems >>> to be OK now. >>> >>> https://forum.ipfire.org/viewtopic.php?f=27&t=21845 >>> >>> This version wasn't integrated into IPFire, so (for example) it inserted >>> itself into the INPUT IPTables chain rather than having it's chains >>> created as part of the firewall start-up script. >>> >>>> >>>> I have huge concerns about the automatic blacklist. @Peter: What is your opinion on this? >>>> >>> >>> While I implemented it, I'm aware of its potential to cause problems, >>> which is why it has to be separately enabled. It's not caused me any >>> issues at the default settings (blocks at over 10 packets per hour until >>> 1 hour has passed without seeing packets from the address), but I've not >>> used it on a site with publicly announced services. If I was going to >>> use it on a web site I would want to, at the very minimum, drop the >>> block period drastically. >> >> I suppose this is entirely unusable on an IPFire box in a data center that hosts things. Let’s say our rack in Hanover. >> >> You will have hits from some broken IP stacks and people might just end up on this without doing anything wrong. >> >> You can denial-of-service easily as well and I suppose without aggregation of data from many many systems, it does not make sense to instantly block addresses. And even then you probably would block an entire subnet. >> >> So I am not sure how I should feel about it. I do not think that it adds anything because the packets would have been blocked anyways. >> >>> >>> On the other hand, it's good at responding quickly. Usually I see only >>> 1-2% of blocks from the automatic list: >>> >>> Reason Count % First Last >>> CIARMY 2416 45 Dec 6 00:00 Dec 6 23:59 >>> DSHIELD 1353 25 Dec 6 00:00 Dec 6 23:59 >>> INPUT 1294 24 Dec 6 00:00 Dec 6 23:59 >>> AUTOBLACKLIST 122 2 Dec 6 00:20 Dec 6 16:28 >>> BLOCKLIST_DE 89 2 Dec 6 00:20 Dec 6 23:46 >>> >>> and sometimes none at all, but one one occasion it blocked over 8000 >>> packets. Again I'm aware this is for a home system, which is rather >>> different than from a Web server. >>> >>>>> If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not >>>>> seem to be problematic, as they are hosted on 3rd party servers, too). >>>> >>>> One of them will be, sooner or later. And one is enough I suppose. >>> >>> DShield (https://dshield.org/api/#threatfeeds) and firehol >>> (http://iplists.firehol.org/) seem to host copies of most of the lists >>> as well. >>>> >>>> I do not really want to overthink this - we didn’t do this with clamav for example either. But it probably is a decision to be made by the user what they want to enable and we should not enable anything by default. So no data will be leaked as long as the user does not consent. >>>> >>>> -Michael >>>> >>>>> >>>>> Thanks, and best regards, >>>>> Peter Müller >> > > <Screenshot_2019-12-16_IP_Address_Blacklists.png>
<Screenshot_2019-12-17_IP_Address_Blacklists.png>
<IP_Address_Blacklists.png><IP_Address_Blacklist_Logs.png><Firewall_log_blacklist.png><Log_Summary.png>
I'm now nearly ready to submit the updated patches, but before then I'd like to check that you're happy with some of my design decisions.
_IPTables_Chains_
When setting up the firewall (in iptables_init()), four new chains are created, IPBLACKLISTIN, IPBLACKLISTOUT, IPBLACKLISTREDIN and IPBLACKLISTREDOUT. The first two are inserted into the INPUT, OUTPUT and FORWARD chains, before the IPS chains. The main script is called to restore the previously saved blacklists, creating the rules in the latter two chains.
In iptables_red_up(), rules are created to forward traffic to/from the red interface from IPBLACKLISTIN/IPBLACKLISTOUT to IPBLACKLISTREDIN/IPBLACKLISTREDOUT.
I did consider putting the blacklist rules into the IPBLACKLISTIN/IPBLACKLISTOUT, but this couldn't be done until red comes up and would require much more processing at this point - not a problem if your red interface comes up and stays up, but if you've got one that disconnects on no traffic you probably don't want the extra overhead. The way I've done it also has the side effects of 1) the main script doesn't have to worry about what the red interface is, and 2) if you're using multiple blacklists, the check for the red interface is only done once.
_Log_WUI_Pages_
The two Log WUI pages are edited from existing pages. I've resisted the temptation to reformat them, which means that they can be diff'ed with the corresponding pages.
_Update_
The update is fairly simple - it calculates whether the check period is passed, tries a download specifying the If-Modified-Since and parses the returned file. It then compares the new list with the existing list, adding new addresses/networks and deleting those that have been removed. There's a check to handle a list changing from a list of addresses to a list of networks, in which case it changes the type of the IPSet - this should be a vary rare event, but could happen for a composite list.
_Enabling/Disabling_Blacklists_
For changes to the existing firewall rules, the firewall is shut down and restarted from scratch. The blacklist script does this on the fly, since the possibilities are simpler than all the options available for the firewall and also the necessary code for doing updates means that adding this code is fairly simple. This includes turning logging on and off.
_Logging_
The main scripts logs to syslog as ipblacklist, but if you run it from the command line it'll also output to the terminal. There's also some debug code which can be turned on by setting a value in the settings file - it's turned on at the lowest level automatically if you run from the command line.
Tim
On 06/01/2020 11:27, Michael Tremer wrote:
Hello all,
On 4 Jan 2020, at 19:02, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Hello,
I've now updated the code - screenshots are attached.
IP_Address_Blacklists.png
This is the settings page. I don't think there's anything unexpected here.
Yes, that looks a lot like the other IPFire pages :)
IP_Address_Blacklist_Logs.png
The log page, replacing the status information that was previously at the top of the settings page.
Looks great!
I've sorted the lists alphabetically, rather than by the number of hits since the worst threat is not necessary the largest number of blocked packets (which could implied by sorting it by number).
Yes, that is indeed a very good idea.
You could add a % sign in the percentage column, so it cannot be confused so easily with the other column.
Firewall_log_blacklist.png
Details page accessed from the log. Almost identical to the other similar pages.
*thumbs up*
Log_Summary.png
This is an extract from the Log Summary page produced by logwatch.
I still need to do some tidying, remove unused language strings etc., and then update and test the patches, so it'll be some time before I'm ready to submit updated patches.
Feel free to send some RFC to the list. It might be slightly noisy, but it is easier to review changes bit after bit.
Great great work!
-Michael
Tim
On 28/12/2019 20:59, Tim FitzGeorge wrote:
Hi,
I'll code and test what we've agreed on. In the meantime I'm going to reply on the sub-thread started by Tom Rymes as to what we call the list categories, so that we can keep that discussion separate from further functionality etc. discussion.
Tim
On 21/12/2019 18:34, Tim FitzGeorge wrote:
Hi,
I've got my systems checking for downloads at 15 minute intervals and using If-Modified-Since. This all seems to be working OK.
Responses to comments below:
On 18/12/2019 12:07, Michael Tremer wrote:
Hi,
On 17 Dec 2019, at 18:21, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Hi,
I think we've agreed on the following:
- Checks to be made every 15 minutes, subject to per-list minimum rate.
- Modify download to use If-Modified-Since rather than separate HEAD
request.
- Remove automatic blacklist.
I thought someone wants to counter my argument. This is not a dictatorship :)
other comments are below.
Tim
On 16/12/2019 22:20, Michael Tremer wrote: > Hi, > >> On 16 Dec 2019, at 20:06, Tim FitzGeorge lists@tfitzgeorge.me.uk wrote: >> >> Hi, >> >> I've attached the current GUI screenshot. > > Thanks for that. > > I have a couple of suggestions/concerns about it: > > a) I am not sure what the value is about the top box that is called “Status”. It is basically a summary of the iptables output, but will it help the user? A blacklist with more or fewer entries is not better or worse, blocking more packets isn’t better than blocking fewer. It is about the quality of the blocks.
I agree that the size of the lists is not all that useful, but the information about the number of blocked packets can be an indication of compromise. It depends on the type of the list, but a list that targets C&C channels (e.g. the FEODO lists) should not show any input packets being blocked, unless one of the computers being protected is infected. Apart from the BOGON lists, none of the lists should show any output packets being blocked; again it's a potential indication of infection if otherwise.
The Shodan list will definitely show some activity unless you are behind CG NAT.
Should we not have this rather in the logging section?
And these counters here will be reset when the firewall is being reloaded. Should this data therefore not be collected from the logs so that you can go back in time? That would be very important to identify any start of compromise.
That makes sense. I'll remove the status from this page and create a log page for the information. If anyone actually wants the iptables information it's available in from Firewall > iptables anyway.
Do you think it makes more sense to show packets blocked in/out or packets blocked per source interface? The latter would of (limited) help in tracking down potential compromise.
> > b) I would prefer to move the settings box to the top. If you like you can show the size of the blacklists there and when they have been updated, too.
I put the status at the top since it's usually what I want to see when I go to the page. Under normal circumstances I don't change the settings, so I'd rather not have to scroll past them to get to the information I'm actually interested in. The update times are just useful to show that the lists are actually being updated. The size is interesting to me, but as you say it probably won't help the average user, especially if you don't know which lists block individual addresses and which block nets (BOGON, DSHIELD and EMERGING_COMPROMISED).
See my comment on the order above. Splitting the page might make sense.
When you show size, does that mean lines in the blacklists? So it could be networks or single IP addresses? Does it not make sense to count IP addresses (i.e. 256 for a /24 network and so on) and put it in the table?
It's lines in the blacklist.
I think I would only be interested in the size for two things:
a) Is there any chance for overblocking? i.e. does the list have 2 billion entries which would be half the public IPv4 address space or so. I cannot really come up with a good example here.
Or
b) What is the performance impact/memory consumption of the list?
It would be interesting to show the type (single address or net), number of entries and number of addresses; the first two affect performance, the second memory use and the latter the number of blocked addresses, but that's really only going to be of interest to a very small number of people, who can always do 'ipset list -t' to get the information. So I'll remove the size column.
I originally had the status and settings in one list, but I decided that it was getting a bit messy and difficult to follow. I've attached a screenshot of a quickly hacked page. (Note that the status area only appears once the blacklists have been enabled.)
You are right. This is getting quite tight there.
You could write packets in/out into on column like 2k/1023, but see my concerns about this above.
Actually I've come to the conclusion that the number of bytes blocked isn't all that important; it's sufficient to show the number of packets.
> > c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
I agree that it's it's an over simplistic summary of the lists. I note that Tom Rymes has asked to keep it, but I think its only value is if it suggests to users that things are more complicated than they appear and therefore prompts them to read the Wiki.
I will reply to that separately.
Looking at that reply, I think it makes sense to include a category column in both the settings and the log summary page - the former to help selection of which lists to use, and the latter as a help when deciding whether the information could indicate compromise or not.
> > d) I do not understand what the “update check rate” means. I suppose we should show the update interval of the lists in the table and just refresh them according to that.
It's the minimum check period for updates. The system checks for updates at either the rate specific to the list or this rate, whichever is slower. This is a hangover from the early days, copied from the IPS updates. I suppose it could have a value for some people who may wish to lower the check rate, but I'm not sure that it has sufficient value to be worth keeping.
No, I would not let the user decide when and how often to update the lists.
OK. I'll remove it.
Generally there is this argument internally how often some databases should be updated. People with slow internet connections or volume based data plans will find this rather expensive to update those lists too often.
If we ever want to go ahead with that in IPFire 2, there should be a global switch for a “low data mode” which then delays updating these things.
That makes sense. Maybe also the ability to set when downloads take place - but as you say that's for the future.
> > e) Do we want to keep the automatic blacklists now? I do not think it will actually improve the security of IPFire. >
I'll remove them. I think they do have a use, in that they can detect someone doing a port scan and block them before they find an open port, but as you say there is also the possibility of a DOS.
How would this block a DOS?
Sorry, I didn't make myself clear - there's the possibility that someone could cause a DOS with the automatic list by faking the source IP address and sending a few packets on a random port.
-Michael
Tim
>> >> I'll update the code based on our discussions, and submit an updated set >> of patches - I imagine there will have to be at least one more iteration. > > Let’s wait until we have come to decisions :) > > -Michael > >> >> Tim >> >> (No additional comments below) >> >> On 13/12/2019 23:11, Michael Tremer wrote: >>> Hi, >>> >>> Again my apologies for my late reply. Busy busy weeks. >>> >>>> On 8 Dec 2019, at 20:50, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote: >>>> >>>> Hello, >>>> >>>> It's my turn to apologise for being slow to respond - I've had a busy >>>> week, but I should have plenty of time over the next couple of weeks. >>> >>> No worries. Turns out we all do :) >>> >>>> I've made most of the comments inline, however I think Michael had a >>>> question (which I can't find now) about what happens if someone enables >>>> all the lists. One thing which would perhaps make this less likely is >>>> that the WUI tags the available lists with whether they're safe or not, >>>> with a footnote that safe means that the list only blocks malicious >>>> traffic. This won't guarantee that a user won't still try to enable all >>>> the lists, but it should make them realise that they should think first. >>> >>> We had a couple of features going slightly wrong or being “misunderstood” by some users. People still seem to panic when they see “local recursor”. To this day I do not know why. >>> >>> We cannot make everything idiot-proof. And when some user if of that category, they probably should shutdown their IPFire box, educate themselves and then come back again. So I do not want to limit people, but make things as easy as possible. >>> >>> If someone enables all the lists, good luck with passing packets :) >>> >>>> I have considered replacing this tag with a risk high/medium/low and >>>> maybe adding a category (invalid/application/scanner/C&C or something >>>> like that), but that may provide too much information and dissuade them >>>> from actually following the links to checkout what the list actually does. >>> >>> Can we have a screenshot of the GUI right now? I didn’t run the code, yet. >>> >>> We should document the lists like we do it with the rulesets of the IPS. People might ignore this, but that is on them. >>> >>>> >>>> Tim >>>> >>>> >>>> On 05/12/2019 22:25, Michael Tremer wrote: >>>>> Hello, >>>>> >>>>>> On 4 Dec 2019, at 17:05, Peter Müller peter.mueller@ipfire.org wrote: >>>>>> >>>>>> Hello Tim, hello Michael, >>>>>> >>>>>> please see my responses inline... >>>>>>>>> >>>>>>>>> We could periodically update the blacklists on our main mirror (and >>>>>>>>> wait for the network to sync it), make sure it is signed and write >>>>>>>>> a small downloader that fetches, validates and installs them. >>>>>>>>> >>>>>>>>> @All: Thoughts on this? >>>>>>>> >>>>>>>> I think there are a number of points here. >>>>>>>> >>>>>>>> Firstly, from the point of a third party using IPFire, is this really >>>>>>>> solving the privacy disclosure problem? There's no way round disclosing >>>>>>>> your public IP address to someone you're downloading from; all this does >>>>>>>> is change who that information is being disclosed to. For the user >>>>>>>> there's no way of knowing whether the source is more or less protective >>>>>>>> of the user's privacy than the blacklist provider. Indeed it won't be >>>>>>>> possible to know who the lists are being downloaded from until the >>>>>>>> download starts. >>>>>>> >>>>>>> There is a way: Tor. But that is a totally different story. >>>>>> Well, I see a third option on this: Use the mirror infrastructure we already >>>>>> have. Every IPFire installation discloses its public IP address to one of these >>>>>> servers sooner or later, so we do not disclose additional data if the blacklists >>>>>> were fetched from these. >>>>>> >>>>>> Needless to say, Tor (hidden services) would be better, but that is a different >>>>>> story indeed. :-) >>>>>>> >>>>>>> The point is rather that a forget list can be sent instead of the “real” one. >>>>>> I did not get this. Forget? Forged?? ??? >>>>> >>>>> Yes, I meant to write forged, but auto-correct didn’t let me. >>>>> >>>>>>>> Secondly, latency; some of the lists are updated every 5 minutes. While >>>>>>>> I've limited the maximum check rate to hourly, will the updates >>>>>>>> propagate quickly enough. For reference on my main system the 24 >>>>>>>> updates on the CIARMY list made 143 498 changes (additions or >>>>>>>> deletions). I've seem it do over 200 000. >>>>>> Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not >>>>>> document a recommended update interval anywhere, so we can only guess. >>>>>> >>>>>> Personally, more static lists seem to be preferable for packet filtering. Highly >>>>>> dynamic ones such as CIArmy should be done via DNSBL queries or something similar >>>>>> - do we really want to have that list here? >>>>> >>>>> It is not really an option to implement a DNSBL into a packet filter, but I get your point. >>>> >>>> One of the 'selling points' for an IP address blacklist is that it can >>>> respond quickly to new threats - or rather new attackers. While a new >>>> IDS/IPS rule needs time to analyse the threat, generate a rule and check >>>> it, it's easy to add an address to a list. So, I think the CIArmy list >>>> is potentially useful for protecting home systems etc. with budget >>>> hardware, but I would be very careful about using it for a protecting a >>>> general access website. >>> >>> If they are very volatile, we should honour that and update them often, too. >>> >>> It probably is more about false-positives being removed very quickly instead of adding threats very quickly. The average IPFire user is probably not under threats like these to need to react very quickly. >>> >>> So I do not see much value in adding those lists and then updating once a day. Does it have to be every 5 min? No. I would suggest 15 which should be good enough for everyone. >>> >>> Other lists should of course not be updated every 15 minutes when not needed. >>> >>> Running every 15 minutes would allow us to retry downloading lists that are on an hourly schedule if the download failed. >>> >>>>> >>>>>>> How did you come up with the hour? Will it be retried more often if the download was not successful? >>>>>> One hour is the most common interval indeed, but adding some random time might >>>>>> be useful in order to reduce load on the servers providing a blacklist. >>>>> >>>>> Yes, definitely. Otherwise we will shoot down our mirrors. >>>> >>>> When I implemented that section of code, specifying the minimum check >>>> period in hours seemed to provide a convenient way of allowing a check >>>> period covering a wide range, with an hour as the fastest and a week as >>>> the slowest. I didn't looked at the CIArmy list until much later. Most >>>> of the lists don't change nearly as much, but the CIArmy list is >>>> described as one that deliberately responds quickly. >>>> >>>> From my production system, for yesterday: >>>> >>>> The following block lists were updated: >>>> BLOCKLIST_DE: 24 Time(s) - 9341 change(s) >>>> BOGON_FULL: 1 Time(s) - 10 change(s) >>>> CIARMY: 24 Time(s) - 159134 change(s) >>>> DSHIELD: 7 Time(s) - 18 change(s) >>>> EMERGING_FWRULE: 1 Time(s) - 50 change(s) >>>> FEODO_AGGRESIVE: 24 Time(s) - 13 change(s) >>>> SHODAN: 1 Time(s) - 0 change(s) >>>> SPAMHAUS_DROP: 1 Time(s) - 0 change(s) >>>> TOR_EXIT: 24 Time(s) - 162 change(s) >>> >>> Very interesting statistics. >>> >>>> >>>> and my test system: >>>> >>>> The following block lists were updated: >>>> ALIENVAULT: 19 Time(s) - 5331 change(s) >>>> EMERGING_COMPROMISED: 1 Time(s) - 26 change(s) >>>> TALOS_MALICIOUS: 1 Time(s) - 36 change(s) >>>> >>>> That covers most of the lists. From the WUI, since 1 Dec: >>>> >>>> Blacklist Entries pkts bytes Last updated >>>> in in >>>> AUTOBLACKLIST 0 731 51144 Sun Dec 8 18:40:02 2019 >>>> BLOCKLIST_DE 28020 857 46735 Sun Dec 8 18:40:02 2019 >>>> BOGON_FULL 214 5255 189K Sun Dec 8 16:50:04 2019 >>>> CIARMY 15000 19774 976K Sun Dec 8 18:04:01 2019 >>>> DSHIELD 20 7992 321K Sun Dec 8 16:45:13 2019 >>>> EMERGING_FWRULE 1647 197 8383 Fri Dec 6 05:29:07 2019 >>>> FEODO_AGGRESIVE 7169 0 0 Sun Dec 8 18:50:07 2019 >>>> SHODAN 32 34 1530 Sun Dec 8 17:54:09 2019 >>>> SPAMHAUS_DROP 823 0 0 Fri Dec 6 18:22:35 2019 >>>> SPAMHAUS_EDROP 111 82 16433 Thu Dec 5 17:27:09 2019 >>>> TOR_EXIT 1055 0 0 Sun Dec 8 18:31:02 2019 >>> >>> This as well. >>> >>> Those are more packets than I would have expected. >>> >>>> >>>> (I've left out the pkts/bytes out fields which were all 0) >>>> >>>> Note that where possible I do a HEAD request first and then only >>>> download the list if the modification time has changed since the last >>>> check. For dynamically generated lists this isn't possible. >>> >>> You won’t need a HEAD request for it. You can include it in the GET request. >>> >>> Have a look at location downloader where I use that. The server will respond with 304 and not send any payload. >>> >>> https://git.ipfire.org/?p=location/libloc.git;a=blob;f=src/python/location-d... >>> >>>> >>>> If the download isn't successful it just gives up and waits for the next >>>> attempt (apart from the usual retries in the library). I probably >>>> should to change that so that it only applies the per list minimum >>>> update period in this case (specified in the sources file) rather than >>>> the user specified value as well. >>> >>> I think it is not the worst if an update fails. It might just happen every once in a while. >>> >>> So I would suggest to just re-run the script more often and when the mtime of the file is older than the threshold, a download is attempted. You can use that timestamp for the GET request. >>> >>>> I already use a time offset on the downloads - when it's started from >>>> boot, backup restore or WUI enable, it checks to see if it's installed >>>> in the fcrontab, and if not adds itself at a randomly generated offset >>>> in the hour. >>> >>> That should potentially go to red.up, or if we can settle on 15 minutes, I would consider that often enough to quickly update all lists after a reboot. >>> >>>> >>>>> >>>>>>>> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is >>>>>>>> the largest at a few MB), there are going to be a lot of them. How will >>>>>>>> this affect the willingness of people to mirror IPFire? >>>>>> I do not consider this being a problem as we do not generate that much traffic >>>>>> to them. Of course, that depends on the update interval again. >>>>> >>>>> That depends on your point of view. >>>>> >>>>> I do not have a problem with this at all in my data center, but there are plenty of people with a volume-based LTE plan or simply a 128 kBit/s connection. It will take a longer time to download the lists for them. We need to mind that. >>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Talking about the preference of packet filter and IPS, I prefer to >>>>>>>>> use the latter as well as it gains more insight in what kind of malicious >>>>>>>>> traffic tried to pass a firewall machine. On systems with low resources, >>>>>>>>> this might be problematic and removing load from the IPS can be preferred >>>>>>>>> (make this configurable?!), on others, people might want to have both >>>>>>>>> results. >>>>>>>>> >>>>>>>> You're only going to get one result for a packet whichever way round the >>>>>>>> IP blacklist and IPS are since whichever comes first will drop the >>>>>>>> packet before it reaches the second (well it would be possible to put >>>>>>>> the IP blacklist first and get it to log and mark packets which are then >>>>>>>> dropped after the IPS, but I think that's getting a little complicated. >>>>>>>> In addition I've seen the messages about the trouble marking was >>>>>>>> causing in the QoS). >>>>>>>> >>>>>>>> I think it's a 50/50 choice as to which is more valuable first; it's >>>>>>>> probably going to differ from packet to packet. For me the possibility >>>>>>>> of reducing the IPS load means I prefer putting the IP blacklist first >>>>>>>> >>>>>>>> It should be fairly easy to add the choice of where to put the IP >>>>>>>> blacklist. I think it'll have to be in the main firewall script, so >>>>>>>> it'll require a firewall restart, but it's not something that'll be >>>>>>>> changed often. >>>>>>> >>>>>>> I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier. >>>>>> Agreed. >>>>>> >>>>>>> >>>>>>> I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads. >>>>>>> >>>>>>> We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily. >>>>>> So dropping packets from blacklisted IP addresses/networks before IPS is it, then. >>>>>>> >>>>>>> I do not even think it makes sense to swap the order in the outgoing direction. >>>>>> Me too. >>>>>> >>>>>>> >>>>>>> What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this. >>>>>> Based on my personal experience, firewall hits observed on a single machine exposed >>>>>> to the internet are interesting, but the overall situation across multiple machines >>>>>> is even more interesting. Very quickly, you'll end on something like a centralised >>>>>> logging server and custom statistical analysis here... >>>>> >>>>> Probably a project for IPFire 4.0 :) >>>>> >>>> Or use one of the existing services, like the DSHIELD client >>>> https://dshield.org/howto.html (subject to privacy concerns again). >>>>>>> >>>>>>>>> >>>>>>>>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default. >>>>>>>>> I would love to see the bogon ruleset here, too (think about 8chan successor >>>>>>>>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause >>>>>>>>> interference if RED does not have a public IP address assigned. >>>>>>>> >>>>>>>> I can add a field to the options file that controls whether a list is >>>>>>>> enabled by default. >>>>>> Thank you. :-) >>>>>>> >>>>>>> To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat. >>>>>> This is my only objection against this patchset. Now, what can we do about it? >>>>>> One possibility is to apply the patchset now and implement a custom download >>>>>> source thing later on, or do that before releasing Core Update 139 (or which version >>>>>> the patchset will be to) after we agreed on something. >>>>> >>>>> I do not see this being merged for 139. But that is not important. We need to get it right first and then release it. >>>>> >>>>> As far as I know, nobody has tested this, yet. >>>> >>>> There are a number of people who have been running an earlier version >>>> which I shared on GitHub. There were a few early issues, but it seems >>>> to be OK now. >>>> >>>> https://forum.ipfire.org/viewtopic.php?f=27&t=21845 >>>> >>>> This version wasn't integrated into IPFire, so (for example) it inserted >>>> itself into the INPUT IPTables chain rather than having it's chains >>>> created as part of the firewall start-up script. >>>> >>>>> >>>>> I have huge concerns about the automatic blacklist. @Peter: What is your opinion on this? >>>>> >>>> >>>> While I implemented it, I'm aware of its potential to cause problems, >>>> which is why it has to be separately enabled. It's not caused me any >>>> issues at the default settings (blocks at over 10 packets per hour until >>>> 1 hour has passed without seeing packets from the address), but I've not >>>> used it on a site with publicly announced services. If I was going to >>>> use it on a web site I would want to, at the very minimum, drop the >>>> block period drastically. >>> >>> I suppose this is entirely unusable on an IPFire box in a data center that hosts things. Let’s say our rack in Hanover. >>> >>> You will have hits from some broken IP stacks and people might just end up on this without doing anything wrong. >>> >>> You can denial-of-service easily as well and I suppose without aggregation of data from many many systems, it does not make sense to instantly block addresses. And even then you probably would block an entire subnet. >>> >>> So I am not sure how I should feel about it. I do not think that it adds anything because the packets would have been blocked anyways. >>> >>>> >>>> On the other hand, it's good at responding quickly. Usually I see only >>>> 1-2% of blocks from the automatic list: >>>> >>>> Reason Count % First Last >>>> CIARMY 2416 45 Dec 6 00:00 Dec 6 23:59 >>>> DSHIELD 1353 25 Dec 6 00:00 Dec 6 23:59 >>>> INPUT 1294 24 Dec 6 00:00 Dec 6 23:59 >>>> AUTOBLACKLIST 122 2 Dec 6 00:20 Dec 6 16:28 >>>> BLOCKLIST_DE 89 2 Dec 6 00:20 Dec 6 23:46 >>>> >>>> and sometimes none at all, but one one occasion it blocked over 8000 >>>> packets. Again I'm aware this is for a home system, which is rather >>>> different than from a Web server. >>>> >>>>>> If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not >>>>>> seem to be problematic, as they are hosted on 3rd party servers, too). >>>>> >>>>> One of them will be, sooner or later. And one is enough I suppose. >>>> >>>> DShield (https://dshield.org/api/#threatfeeds) and firehol >>>> (http://iplists.firehol.org/) seem to host copies of most of the lists >>>> as well. >>>>> >>>>> I do not really want to overthink this - we didn’t do this with clamav for example either. But it probably is a decision to be made by the user what they want to enable and we should not enable anything by default. So no data will be leaked as long as the user does not consent. >>>>> >>>>> -Michael >>>>> >>>>>> >>>>>> Thanks, and best regards, >>>>>> Peter Müller >>> >> >> <Screenshot_2019-12-16_IP_Address_Blacklists.png> >
<Screenshot_2019-12-17_IP_Address_Blacklists.png>
<IP_Address_Blacklists.png><IP_Address_Blacklist_Logs.png><Firewall_log_blacklist.png><Log_Summary.png>
Hi,
Apologies for my late reply again. I marked this email in my inbox and then it scrolled down because of loads of new stuff coming in. Please feel free to ping me if I do not respond in time.
On 24 Jan 2020, at 19:40, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
I'm now nearly ready to submit the updated patches, but before then I'd like to check that you're happy with some of my design decisions.
Cool.
_IPTables_Chains_
When setting up the firewall (in iptables_init()), four new chains are created, IPBLACKLISTIN, IPBLACKLISTOUT, IPBLACKLISTREDIN and IPBLACKLISTREDOUT. The first two are inserted into the INPUT, OUTPUT and FORWARD chains, before the IPS chains. The main script is called to restore the previously saved blacklists, creating the rules in the latter two chains.
For readability you can simply call them BLACKLIST*. The IP is a given.
In iptables_red_up(), rules are created to forward traffic to/from the red interface from IPBLACKLISTIN/IPBLACKLISTOUT to IPBLACKLISTREDIN/IPBLACKLISTREDOUT. I did consider putting the blacklist rules into the IPBLACKLISTIN/IPBLACKLISTOUT, but this couldn't be done until red comes up and would require much more processing at this point - not a problem if your red interface comes up and stays up, but if you've got one that disconnects on no traffic you probably don't want the extra overhead. The way I've done it also has the side effects of 1) the main script doesn't have to worry about what the red interface is, and 2) if you're using multiple blacklists, the check for the red interface is only done once.
You would only want to check the interface once and then jump into the chain. Can we not realise it that way?
_Log_WUI_Pages_
The two Log WUI pages are edited from existing pages. I've resisted the temptation to reformat them, which means that they can be diff'ed with the corresponding pages.
Loads of the UI code is messy and deliberately not touched any more. I guess this is fine.
_Update_
The update is fairly simple - it calculates whether the check period is passed, tries a download specifying the If-Modified-Since and parses the returned file. It then compares the new list with the existing list, adding new addresses/networks and deleting those that have been removed. There's a check to handle a list changing from a list of addresses to a list of networks, in which case it changes the type of the IPSet - this should be a vary rare event, but could happen for a composite list.
Why do you need the diff? Has that been in the first patchset?
Just reloading the whole ipset should be easier.
_Enabling/Disabling_Blacklists_
For changes to the existing firewall rules, the firewall is shut down and restarted from scratch. The blacklist script does this on the fly, since the possibilities are simpler than all the options available for the firewall and also the necessary code for doing updates means that adding this code is fairly simple. This includes turning logging on and off.
What do you mean by that? /etc/init.d/firewall restart?
That is something we cannot do because IPsec and some other services create temporary rules that would be lost.
_Logging_
The main scripts logs to syslog as ipblacklist, but if you run it from the command line it'll also output to the terminal. There's also some debug code which can be turned on by setting a value in the settings file - it's turned on at the lowest level automatically if you run from the command line.
Yes that is fine.
Best, -Michael
Tim
On 06/01/2020 11:27, Michael Tremer wrote:
Hello all,
On 4 Jan 2020, at 19:02, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Hello,
I've now updated the code - screenshots are attached.
IP_Address_Blacklists.png
This is the settings page. I don't think there's anything unexpected here.
Yes, that looks a lot like the other IPFire pages :)
IP_Address_Blacklist_Logs.png
The log page, replacing the status information that was previously at the top of the settings page.
Looks great!
I've sorted the lists alphabetically, rather than by the number of hits since the worst threat is not necessary the largest number of blocked packets (which could implied by sorting it by number).
Yes, that is indeed a very good idea.
You could add a % sign in the percentage column, so it cannot be confused so easily with the other column.
Firewall_log_blacklist.png
Details page accessed from the log. Almost identical to the other similar pages.
*thumbs up*
Log_Summary.png
This is an extract from the Log Summary page produced by logwatch.
I still need to do some tidying, remove unused language strings etc., and then update and test the patches, so it'll be some time before I'm ready to submit updated patches.
Feel free to send some RFC to the list. It might be slightly noisy, but it is easier to review changes bit after bit.
Great great work!
-Michael
Tim
On 28/12/2019 20:59, Tim FitzGeorge wrote:
Hi,
I'll code and test what we've agreed on. In the meantime I'm going to reply on the sub-thread started by Tom Rymes as to what we call the list categories, so that we can keep that discussion separate from further functionality etc. discussion.
Tim
On 21/12/2019 18:34, Tim FitzGeorge wrote:
Hi,
I've got my systems checking for downloads at 15 minute intervals and using If-Modified-Since. This all seems to be working OK.
Responses to comments below:
On 18/12/2019 12:07, Michael Tremer wrote:
Hi,
> On 17 Dec 2019, at 18:21, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote: > > Hi, > > I think we've agreed on the following: > > - Checks to be made every 15 minutes, subject to per-list minimum rate. > - Modify download to use If-Modified-Since rather than separate HEAD > request. > - Remove automatic blacklist.
I thought someone wants to counter my argument. This is not a dictatorship :)
> > other comments are below. > > Tim > > On 16/12/2019 22:20, Michael Tremer wrote: >> Hi, >> >>> On 16 Dec 2019, at 20:06, Tim FitzGeorge lists@tfitzgeorge.me.uk wrote: >>> >>> Hi, >>> >>> I've attached the current GUI screenshot. >> >> Thanks for that. >> >> I have a couple of suggestions/concerns about it: >> >> a) I am not sure what the value is about the top box that is called “Status”. It is basically a summary of the iptables output, but will it help the user? A blacklist with more or fewer entries is not better or worse, blocking more packets isn’t better than blocking fewer. It is about the quality of the blocks. > > I agree that the size of the lists is not all that useful, but the > information about the number of blocked packets can be an indication of > compromise. It depends on the type of the list, but a list that targets > C&C channels (e.g. the FEODO lists) should not show any input packets > being blocked, unless one of the computers being protected is infected. > Apart from the BOGON lists, none of the lists should show any output > packets being blocked; again it's a potential indication of infection if > otherwise.
The Shodan list will definitely show some activity unless you are behind CG NAT.
Should we not have this rather in the logging section?
And these counters here will be reset when the firewall is being reloaded. Should this data therefore not be collected from the logs so that you can go back in time? That would be very important to identify any start of compromise.
That makes sense. I'll remove the status from this page and create a log page for the information. If anyone actually wants the iptables information it's available in from Firewall > iptables anyway.
Do you think it makes more sense to show packets blocked in/out or packets blocked per source interface? The latter would of (limited) help in tracking down potential compromise.
>> >> b) I would prefer to move the settings box to the top. If you like you can show the size of the blacklists there and when they have been updated, too. > > I put the status at the top since it's usually what I want to see when I > go to the page. Under normal circumstances I don't change the settings, > so I'd rather not have to scroll past them to get to the information I'm > actually interested in. The update times are just useful to show that > the lists are actually being updated. The size is interesting to me, > but as you say it probably won't help the average user, especially if > you don't know which lists block individual addresses and which block > nets (BOGON, DSHIELD and EMERGING_COMPROMISED).
See my comment on the order above. Splitting the page might make sense.
When you show size, does that mean lines in the blacklists? So it could be networks or single IP addresses? Does it not make sense to count IP addresses (i.e. 256 for a /24 network and so on) and put it in the table?
It's lines in the blacklist.
I think I would only be interested in the size for two things:
a) Is there any chance for overblocking? i.e. does the list have 2 billion entries which would be half the public IPv4 address space or so. I cannot really come up with a good example here.
Or
b) What is the performance impact/memory consumption of the list?
It would be interesting to show the type (single address or net), number of entries and number of addresses; the first two affect performance, the second memory use and the latter the number of blocked addresses, but that's really only going to be of interest to a very small number of people, who can always do 'ipset list -t' to get the information. So I'll remove the size column.
> I originally had the status and settings in one list, but I decided that > it was getting a bit messy and difficult to follow. I've attached a > screenshot of a quickly hacked page. (Note that the status area only > appears once the blacklists have been enabled.)
You are right. This is getting quite tight there.
You could write packets in/out into on column like 2k/1023, but see my concerns about this above.
Actually I've come to the conclusion that the number of bytes blocked isn't all that important; it's sufficient to show the number of packets.
>> >> c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out. > > I agree that it's it's an over simplistic summary of the lists. I note > that Tom Rymes has asked to keep it, but I think its only value is if it > suggests to users that things are more complicated than they appear and > therefore prompts them to read the Wiki.
I will reply to that separately.
Looking at that reply, I think it makes sense to include a category column in both the settings and the log summary page - the former to help selection of which lists to use, and the latter as a help when deciding whether the information could indicate compromise or not.
>> >> d) I do not understand what the “update check rate” means. I suppose we should show the update interval of the lists in the table and just refresh them according to that. > > It's the minimum check period for updates. The system checks for > updates at either the rate specific to the list or this rate, whichever > is slower. This is a hangover from the early days, copied from the IPS > updates. I suppose it could have a value for some people who may wish > to lower the check rate, but I'm not sure that it has sufficient value > to be worth keeping.
No, I would not let the user decide when and how often to update the lists.
OK. I'll remove it.
Generally there is this argument internally how often some databases should be updated. People with slow internet connections or volume based data plans will find this rather expensive to update those lists too often.
If we ever want to go ahead with that in IPFire 2, there should be a global switch for a “low data mode” which then delays updating these things.
That makes sense. Maybe also the ability to set when downloads take place - but as you say that's for the future.
>> >> e) Do we want to keep the automatic blacklists now? I do not think it will actually improve the security of IPFire. >> > > I'll remove them. I think they do have a use, in that they can detect > someone doing a port scan and block them before they find an open port, > but as you say there is also the possibility of a DOS.
How would this block a DOS?
Sorry, I didn't make myself clear - there's the possibility that someone could cause a DOS with the automatic list by faking the source IP address and sending a few packets on a random port.
-Michael
Tim
>>> >>> I'll update the code based on our discussions, and submit an updated set >>> of patches - I imagine there will have to be at least one more iteration. >> >> Let’s wait until we have come to decisions :) >> >> -Michael >> >>> >>> Tim >>> >>> (No additional comments below) >>> >>> On 13/12/2019 23:11, Michael Tremer wrote: >>>> Hi, >>>> >>>> Again my apologies for my late reply. Busy busy weeks. >>>> >>>>> On 8 Dec 2019, at 20:50, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote: >>>>> >>>>> Hello, >>>>> >>>>> It's my turn to apologise for being slow to respond - I've had a busy >>>>> week, but I should have plenty of time over the next couple of weeks. >>>> >>>> No worries. Turns out we all do :) >>>> >>>>> I've made most of the comments inline, however I think Michael had a >>>>> question (which I can't find now) about what happens if someone enables >>>>> all the lists. One thing which would perhaps make this less likely is >>>>> that the WUI tags the available lists with whether they're safe or not, >>>>> with a footnote that safe means that the list only blocks malicious >>>>> traffic. This won't guarantee that a user won't still try to enable all >>>>> the lists, but it should make them realise that they should think first. >>>> >>>> We had a couple of features going slightly wrong or being “misunderstood” by some users. People still seem to panic when they see “local recursor”. To this day I do not know why. >>>> >>>> We cannot make everything idiot-proof. And when some user if of that category, they probably should shutdown their IPFire box, educate themselves and then come back again. So I do not want to limit people, but make things as easy as possible. >>>> >>>> If someone enables all the lists, good luck with passing packets :) >>>> >>>>> I have considered replacing this tag with a risk high/medium/low and >>>>> maybe adding a category (invalid/application/scanner/C&C or something >>>>> like that), but that may provide too much information and dissuade them >>>>> from actually following the links to checkout what the list actually does. >>>> >>>> Can we have a screenshot of the GUI right now? I didn’t run the code, yet. >>>> >>>> We should document the lists like we do it with the rulesets of the IPS. People might ignore this, but that is on them. >>>> >>>>> >>>>> Tim >>>>> >>>>> >>>>> On 05/12/2019 22:25, Michael Tremer wrote: >>>>>> Hello, >>>>>> >>>>>>> On 4 Dec 2019, at 17:05, Peter Müller peter.mueller@ipfire.org wrote: >>>>>>> >>>>>>> Hello Tim, hello Michael, >>>>>>> >>>>>>> please see my responses inline... >>>>>>>>>> >>>>>>>>>> We could periodically update the blacklists on our main mirror (and >>>>>>>>>> wait for the network to sync it), make sure it is signed and write >>>>>>>>>> a small downloader that fetches, validates and installs them. >>>>>>>>>> >>>>>>>>>> @All: Thoughts on this? >>>>>>>>> >>>>>>>>> I think there are a number of points here. >>>>>>>>> >>>>>>>>> Firstly, from the point of a third party using IPFire, is this really >>>>>>>>> solving the privacy disclosure problem? There's no way round disclosing >>>>>>>>> your public IP address to someone you're downloading from; all this does >>>>>>>>> is change who that information is being disclosed to. For the user >>>>>>>>> there's no way of knowing whether the source is more or less protective >>>>>>>>> of the user's privacy than the blacklist provider. Indeed it won't be >>>>>>>>> possible to know who the lists are being downloaded from until the >>>>>>>>> download starts. >>>>>>>> >>>>>>>> There is a way: Tor. But that is a totally different story. >>>>>>> Well, I see a third option on this: Use the mirror infrastructure we already >>>>>>> have. Every IPFire installation discloses its public IP address to one of these >>>>>>> servers sooner or later, so we do not disclose additional data if the blacklists >>>>>>> were fetched from these. >>>>>>> >>>>>>> Needless to say, Tor (hidden services) would be better, but that is a different >>>>>>> story indeed. :-) >>>>>>>> >>>>>>>> The point is rather that a forget list can be sent instead of the “real” one. >>>>>>> I did not get this. Forget? Forged?? ??? >>>>>> >>>>>> Yes, I meant to write forged, but auto-correct didn’t let me. >>>>>> >>>>>>>>> Secondly, latency; some of the lists are updated every 5 minutes. While >>>>>>>>> I've limited the maximum check rate to hourly, will the updates >>>>>>>>> propagate quickly enough. For reference on my main system the 24 >>>>>>>>> updates on the CIARMY list made 143 498 changes (additions or >>>>>>>>> deletions). I've seem it do over 200 000. >>>>>>> Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not >>>>>>> document a recommended update interval anywhere, so we can only guess. >>>>>>> >>>>>>> Personally, more static lists seem to be preferable for packet filtering. Highly >>>>>>> dynamic ones such as CIArmy should be done via DNSBL queries or something similar >>>>>>> - do we really want to have that list here? >>>>>> >>>>>> It is not really an option to implement a DNSBL into a packet filter, but I get your point. >>>>> >>>>> One of the 'selling points' for an IP address blacklist is that it can >>>>> respond quickly to new threats - or rather new attackers. While a new >>>>> IDS/IPS rule needs time to analyse the threat, generate a rule and check >>>>> it, it's easy to add an address to a list. So, I think the CIArmy list >>>>> is potentially useful for protecting home systems etc. with budget >>>>> hardware, but I would be very careful about using it for a protecting a >>>>> general access website. >>>> >>>> If they are very volatile, we should honour that and update them often, too. >>>> >>>> It probably is more about false-positives being removed very quickly instead of adding threats very quickly. The average IPFire user is probably not under threats like these to need to react very quickly. >>>> >>>> So I do not see much value in adding those lists and then updating once a day. Does it have to be every 5 min? No. I would suggest 15 which should be good enough for everyone. >>>> >>>> Other lists should of course not be updated every 15 minutes when not needed. >>>> >>>> Running every 15 minutes would allow us to retry downloading lists that are on an hourly schedule if the download failed. >>>> >>>>>> >>>>>>>> How did you come up with the hour? Will it be retried more often if the download was not successful? >>>>>>> One hour is the most common interval indeed, but adding some random time might >>>>>>> be useful in order to reduce load on the servers providing a blacklist. >>>>>> >>>>>> Yes, definitely. Otherwise we will shoot down our mirrors. >>>>> >>>>> When I implemented that section of code, specifying the minimum check >>>>> period in hours seemed to provide a convenient way of allowing a check >>>>> period covering a wide range, with an hour as the fastest and a week as >>>>> the slowest. I didn't looked at the CIArmy list until much later. Most >>>>> of the lists don't change nearly as much, but the CIArmy list is >>>>> described as one that deliberately responds quickly. >>>>> >>>>> From my production system, for yesterday: >>>>> >>>>> The following block lists were updated: >>>>> BLOCKLIST_DE: 24 Time(s) - 9341 change(s) >>>>> BOGON_FULL: 1 Time(s) - 10 change(s) >>>>> CIARMY: 24 Time(s) - 159134 change(s) >>>>> DSHIELD: 7 Time(s) - 18 change(s) >>>>> EMERGING_FWRULE: 1 Time(s) - 50 change(s) >>>>> FEODO_AGGRESIVE: 24 Time(s) - 13 change(s) >>>>> SHODAN: 1 Time(s) - 0 change(s) >>>>> SPAMHAUS_DROP: 1 Time(s) - 0 change(s) >>>>> TOR_EXIT: 24 Time(s) - 162 change(s) >>>> >>>> Very interesting statistics. >>>> >>>>> >>>>> and my test system: >>>>> >>>>> The following block lists were updated: >>>>> ALIENVAULT: 19 Time(s) - 5331 change(s) >>>>> EMERGING_COMPROMISED: 1 Time(s) - 26 change(s) >>>>> TALOS_MALICIOUS: 1 Time(s) - 36 change(s) >>>>> >>>>> That covers most of the lists. From the WUI, since 1 Dec: >>>>> >>>>> Blacklist Entries pkts bytes Last updated >>>>> in in >>>>> AUTOBLACKLIST 0 731 51144 Sun Dec 8 18:40:02 2019 >>>>> BLOCKLIST_DE 28020 857 46735 Sun Dec 8 18:40:02 2019 >>>>> BOGON_FULL 214 5255 189K Sun Dec 8 16:50:04 2019 >>>>> CIARMY 15000 19774 976K Sun Dec 8 18:04:01 2019 >>>>> DSHIELD 20 7992 321K Sun Dec 8 16:45:13 2019 >>>>> EMERGING_FWRULE 1647 197 8383 Fri Dec 6 05:29:07 2019 >>>>> FEODO_AGGRESIVE 7169 0 0 Sun Dec 8 18:50:07 2019 >>>>> SHODAN 32 34 1530 Sun Dec 8 17:54:09 2019 >>>>> SPAMHAUS_DROP 823 0 0 Fri Dec 6 18:22:35 2019 >>>>> SPAMHAUS_EDROP 111 82 16433 Thu Dec 5 17:27:09 2019 >>>>> TOR_EXIT 1055 0 0 Sun Dec 8 18:31:02 2019 >>>> >>>> This as well. >>>> >>>> Those are more packets than I would have expected. >>>> >>>>> >>>>> (I've left out the pkts/bytes out fields which were all 0) >>>>> >>>>> Note that where possible I do a HEAD request first and then only >>>>> download the list if the modification time has changed since the last >>>>> check. For dynamically generated lists this isn't possible. >>>> >>>> You won’t need a HEAD request for it. You can include it in the GET request. >>>> >>>> Have a look at location downloader where I use that. The server will respond with 304 and not send any payload. >>>> >>>> https://git.ipfire.org/?p=location/libloc.git;a=blob;f=src/python/location-d... >>>> >>>>> >>>>> If the download isn't successful it just gives up and waits for the next >>>>> attempt (apart from the usual retries in the library). I probably >>>>> should to change that so that it only applies the per list minimum >>>>> update period in this case (specified in the sources file) rather than >>>>> the user specified value as well. >>>> >>>> I think it is not the worst if an update fails. It might just happen every once in a while. >>>> >>>> So I would suggest to just re-run the script more often and when the mtime of the file is older than the threshold, a download is attempted. You can use that timestamp for the GET request. >>>> >>>>> I already use a time offset on the downloads - when it's started from >>>>> boot, backup restore or WUI enable, it checks to see if it's installed >>>>> in the fcrontab, and if not adds itself at a randomly generated offset >>>>> in the hour. >>>> >>>> That should potentially go to red.up, or if we can settle on 15 minutes, I would consider that often enough to quickly update all lists after a reboot. >>>> >>>>> >>>>>> >>>>>>>>> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is >>>>>>>>> the largest at a few MB), there are going to be a lot of them. How will >>>>>>>>> this affect the willingness of people to mirror IPFire? >>>>>>> I do not consider this being a problem as we do not generate that much traffic >>>>>>> to them. Of course, that depends on the update interval again. >>>>>> >>>>>> That depends on your point of view. >>>>>> >>>>>> I do not have a problem with this at all in my data center, but there are plenty of people with a volume-based LTE plan or simply a 128 kBit/s connection. It will take a longer time to download the lists for them. We need to mind that. >>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Talking about the preference of packet filter and IPS, I prefer to >>>>>>>>>> use the latter as well as it gains more insight in what kind of malicious >>>>>>>>>> traffic tried to pass a firewall machine. On systems with low resources, >>>>>>>>>> this might be problematic and removing load from the IPS can be preferred >>>>>>>>>> (make this configurable?!), on others, people might want to have both >>>>>>>>>> results. >>>>>>>>>> >>>>>>>>> You're only going to get one result for a packet whichever way round the >>>>>>>>> IP blacklist and IPS are since whichever comes first will drop the >>>>>>>>> packet before it reaches the second (well it would be possible to put >>>>>>>>> the IP blacklist first and get it to log and mark packets which are then >>>>>>>>> dropped after the IPS, but I think that's getting a little complicated. >>>>>>>>> In addition I've seen the messages about the trouble marking was >>>>>>>>> causing in the QoS). >>>>>>>>> >>>>>>>>> I think it's a 50/50 choice as to which is more valuable first; it's >>>>>>>>> probably going to differ from packet to packet. For me the possibility >>>>>>>>> of reducing the IPS load means I prefer putting the IP blacklist first >>>>>>>>> >>>>>>>>> It should be fairly easy to add the choice of where to put the IP >>>>>>>>> blacklist. I think it'll have to be in the main firewall script, so >>>>>>>>> it'll require a firewall restart, but it's not something that'll be >>>>>>>>> changed often. >>>>>>>> >>>>>>>> I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier. >>>>>>> Agreed. >>>>>>> >>>>>>>> >>>>>>>> I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads. >>>>>>>> >>>>>>>> We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily. >>>>>>> So dropping packets from blacklisted IP addresses/networks before IPS is it, then. >>>>>>>> >>>>>>>> I do not even think it makes sense to swap the order in the outgoing direction. >>>>>>> Me too. >>>>>>> >>>>>>>> >>>>>>>> What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this. >>>>>>> Based on my personal experience, firewall hits observed on a single machine exposed >>>>>>> to the internet are interesting, but the overall situation across multiple machines >>>>>>> is even more interesting. Very quickly, you'll end on something like a centralised >>>>>>> logging server and custom statistical analysis here... >>>>>> >>>>>> Probably a project for IPFire 4.0 :) >>>>>> >>>>> Or use one of the existing services, like the DSHIELD client >>>>> https://dshield.org/howto.html (subject to privacy concerns again). >>>>>>>> >>>>>>>>>> >>>>>>>>>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default. >>>>>>>>>> I would love to see the bogon ruleset here, too (think about 8chan successor >>>>>>>>>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause >>>>>>>>>> interference if RED does not have a public IP address assigned. >>>>>>>>> >>>>>>>>> I can add a field to the options file that controls whether a list is >>>>>>>>> enabled by default. >>>>>>> Thank you. :-) >>>>>>>> >>>>>>>> To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat. >>>>>>> This is my only objection against this patchset. Now, what can we do about it? >>>>>>> One possibility is to apply the patchset now and implement a custom download >>>>>>> source thing later on, or do that before releasing Core Update 139 (or which version >>>>>>> the patchset will be to) after we agreed on something. >>>>>> >>>>>> I do not see this being merged for 139. But that is not important. We need to get it right first and then release it. >>>>>> >>>>>> As far as I know, nobody has tested this, yet. >>>>> >>>>> There are a number of people who have been running an earlier version >>>>> which I shared on GitHub. There were a few early issues, but it seems >>>>> to be OK now. >>>>> >>>>> https://forum.ipfire.org/viewtopic.php?f=27&t=21845 >>>>> >>>>> This version wasn't integrated into IPFire, so (for example) it inserted >>>>> itself into the INPUT IPTables chain rather than having it's chains >>>>> created as part of the firewall start-up script. >>>>> >>>>>> >>>>>> I have huge concerns about the automatic blacklist. @Peter: What is your opinion on this? >>>>>> >>>>> >>>>> While I implemented it, I'm aware of its potential to cause problems, >>>>> which is why it has to be separately enabled. It's not caused me any >>>>> issues at the default settings (blocks at over 10 packets per hour until >>>>> 1 hour has passed without seeing packets from the address), but I've not >>>>> used it on a site with publicly announced services. If I was going to >>>>> use it on a web site I would want to, at the very minimum, drop the >>>>> block period drastically. >>>> >>>> I suppose this is entirely unusable on an IPFire box in a data center that hosts things. Let’s say our rack in Hanover. >>>> >>>> You will have hits from some broken IP stacks and people might just end up on this without doing anything wrong. >>>> >>>> You can denial-of-service easily as well and I suppose without aggregation of data from many many systems, it does not make sense to instantly block addresses. And even then you probably would block an entire subnet. >>>> >>>> So I am not sure how I should feel about it. I do not think that it adds anything because the packets would have been blocked anyways. >>>> >>>>> >>>>> On the other hand, it's good at responding quickly. Usually I see only >>>>> 1-2% of blocks from the automatic list: >>>>> >>>>> Reason Count % First Last >>>>> CIARMY 2416 45 Dec 6 00:00 Dec 6 23:59 >>>>> DSHIELD 1353 25 Dec 6 00:00 Dec 6 23:59 >>>>> INPUT 1294 24 Dec 6 00:00 Dec 6 23:59 >>>>> AUTOBLACKLIST 122 2 Dec 6 00:20 Dec 6 16:28 >>>>> BLOCKLIST_DE 89 2 Dec 6 00:20 Dec 6 23:46 >>>>> >>>>> and sometimes none at all, but one one occasion it blocked over 8000 >>>>> packets. Again I'm aware this is for a home system, which is rather >>>>> different than from a Web server. >>>>> >>>>>>> If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not >>>>>>> seem to be problematic, as they are hosted on 3rd party servers, too). >>>>>> >>>>>> One of them will be, sooner or later. And one is enough I suppose. >>>>> >>>>> DShield (https://dshield.org/api/#threatfeeds) and firehol >>>>> (http://iplists.firehol.org/) seem to host copies of most of the lists >>>>> as well. >>>>>> >>>>>> I do not really want to overthink this - we didn’t do this with clamav for example either. But it probably is a decision to be made by the user what they want to enable and we should not enable anything by default. So no data will be leaked as long as the user does not consent. >>>>>> >>>>>> -Michael >>>>>> >>>>>>> >>>>>>> Thanks, and best regards, >>>>>>> Peter Müller >>>> >>> >>> <Screenshot_2019-12-16_IP_Address_Blacklists.png> >> > > <Screenshot_2019-12-17_IP_Address_Blacklists.png>
<IP_Address_Blacklists.png><IP_Address_Blacklist_Logs.png><Firewall_log_blacklist.png><Log_Summary.png>
Hi,
On 28/01/2020 17:14, Michael Tremer wrote:
Hi,
Apologies for my late reply again. I marked this email in my inbox and then it scrolled down because of loads of new stuff coming in. Please feel free to ping me if I do not respond in time.
That's OK. It took me a little longer than expected to check the patches.
On 24 Jan 2020, at 19:40, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
I'm now nearly ready to submit the updated patches, but before then I'd like to check that you're happy with some of my design decisions.
Cool.
_IPTables_Chains_
When setting up the firewall (in iptables_init()), four new chains are created, IPBLACKLISTIN, IPBLACKLISTOUT, IPBLACKLISTREDIN and IPBLACKLISTREDOUT. The first two are inserted into the INPUT, OUTPUT and FORWARD chains, before the IPS chains. The main script is called to restore the previously saved blacklists, creating the rules in the latter two chains.
For readability you can simply call them BLACKLIST*. The IP is a given.
Good point. I'll make the change.
In iptables_red_up(), rules are created to forward traffic to/from the red interface from IPBLACKLISTIN/IPBLACKLISTOUT to IPBLACKLISTREDIN/IPBLACKLISTREDOUT. I did consider putting the blacklist rules into the IPBLACKLISTIN/IPBLACKLISTOUT, but this couldn't be done until red comes up and would require much more processing at this point - not a problem if your red interface comes up and stays up, but if you've got one that disconnects on no traffic you probably don't want the extra overhead. The way I've done it also has the side effects of 1) the main script doesn't have to worry about what the red interface is, and 2) if you're using multiple blacklists, the check for the red interface is only done once.
You would only want to check the interface once and then jump into the chain. Can we not realise it that way?
I'm not sure what you're suggesting. I insert BLACKLISTIN into INPUT & FORWARD and BLACKLISTOUT into OUTPUT & FORWARD. These receive all non-ICMP traffic. On red up traffic from the red interface is sent from BLACKLISTIN to BLACKLISTREDIN and traffic to the red interface is sent from BLACKLISTOUT to BLACKLISTREDOUT. The rules to check against the IPSets are put into the *RED* chain, checking against the source or destination addresses as appropriate.
_Log_WUI_Pages_
The two Log WUI pages are edited from existing pages. I've resisted the temptation to reformat them, which means that they can be diff'ed with the corresponding pages.
Loads of the UI code is messy and deliberately not touched any more. I guess this is fine.
Yes, it's not pretty, but it is pragmatic.
_Update_
The update is fairly simple - it calculates whether the check period is passed, tries a download specifying the If-Modified-Since and parses the returned file. It then compares the new list with the existing list, adding new addresses/networks and deleting those that have been removed. There's a check to handle a list changing from a list of addresses to a list of networks, in which case it changes the type of the IPSet - this should be a vary rare event, but could happen for a composite list.
Why do you need the diff? Has that been in the first patchset?
It was, yes.
Originally I did it for performance. It took over a minute to process the ALIENVAULT list (over 80 000 entries); just calling ipset with changes reduced this to a few seconds. I've since modified the code to pipe commands to a process running 'ipset restore', which has drastically speed up the processing, but it's still a little bit quicker to just do changes.
The second reason is that it doesn't feel right to flush the old list, thus removing its protection while the new one is loaded, especially if the new list is the same as the old list (not all sources implement IP-Modified-Since). The actual period of reduced protection may be short, but I prefer to avoid it.
Just reloading the whole ipset should be easier.
_Enabling/Disabling_Blacklists_
For changes to the existing firewall rules, the firewall is shut down and restarted from scratch. The blacklist script does this on the fly, since the possibilities are simpler than all the options available for the firewall and also the necessary code for doing updates means that adding this code is fairly simple. This includes turning logging on and off.
What do you mean by that? /etc/init.d/firewall restart?
That is something we cannot do because IPsec and some other services create temporary rules that would be lost.
No. I'm actually trying to say the opposite (badly). The existing IPFire code only implements some firewall changes by shutting down and restarting the firewall, but all the IP Blacklist functions (enable/disable/update lists, enable/disable logging) can be done without this - the necessary rules can be added to/deleted from the BLACKLISTRED* chains without affecting any of the rest of the firewall system.
Tim
_Logging_
The main scripts logs to syslog as ipblacklist, but if you run it from the command line it'll also output to the terminal. There's also some debug code which can be turned on by setting a value in the settings file - it's turned on at the lowest level automatically if you run from the command line.
Yes that is fine.
Best, -Michael
Tim
On 06/01/2020 11:27, Michael Tremer wrote:
Hello all,
On 4 Jan 2020, at 19:02, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Hello,
I've now updated the code - screenshots are attached.
IP_Address_Blacklists.png
This is the settings page. I don't think there's anything unexpected here.
Yes, that looks a lot like the other IPFire pages :)
IP_Address_Blacklist_Logs.png
The log page, replacing the status information that was previously at the top of the settings page.
Looks great!
I've sorted the lists alphabetically, rather than by the number of hits since the worst threat is not necessary the largest number of blocked packets (which could implied by sorting it by number).
Yes, that is indeed a very good idea.
You could add a % sign in the percentage column, so it cannot be confused so easily with the other column.
Firewall_log_blacklist.png
Details page accessed from the log. Almost identical to the other similar pages.
*thumbs up*
Log_Summary.png
This is an extract from the Log Summary page produced by logwatch.
I still need to do some tidying, remove unused language strings etc., and then update and test the patches, so it'll be some time before I'm ready to submit updated patches.
Feel free to send some RFC to the list. It might be slightly noisy, but it is easier to review changes bit after bit.
Great great work!
-Michael
Tim
On 28/12/2019 20:59, Tim FitzGeorge wrote:
Hi,
I'll code and test what we've agreed on. In the meantime I'm going to reply on the sub-thread started by Tom Rymes as to what we call the list categories, so that we can keep that discussion separate from further functionality etc. discussion.
Tim
On 21/12/2019 18:34, Tim FitzGeorge wrote:
Hi,
I've got my systems checking for downloads at 15 minute intervals and using If-Modified-Since. This all seems to be working OK.
Responses to comments below:
On 18/12/2019 12:07, Michael Tremer wrote: > Hi, > >> On 17 Dec 2019, at 18:21, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote: >> >> Hi, >> >> I think we've agreed on the following: >> >> - Checks to be made every 15 minutes, subject to per-list minimum rate. >> - Modify download to use If-Modified-Since rather than separate HEAD >> request. >> - Remove automatic blacklist. > > I thought someone wants to counter my argument. This is not a dictatorship :) > >> >> other comments are below. >> >> Tim >> >> On 16/12/2019 22:20, Michael Tremer wrote: >>> Hi, >>> >>>> On 16 Dec 2019, at 20:06, Tim FitzGeorge lists@tfitzgeorge.me.uk wrote: >>>> >>>> Hi, >>>> >>>> I've attached the current GUI screenshot. >>> >>> Thanks for that. >>> >>> I have a couple of suggestions/concerns about it: >>> >>> a) I am not sure what the value is about the top box that is called “Status”. It is basically a summary of the iptables output, but will it help the user? A blacklist with more or fewer entries is not better or worse, blocking more packets isn’t better than blocking fewer. It is about the quality of the blocks. >> >> I agree that the size of the lists is not all that useful, but the >> information about the number of blocked packets can be an indication of >> compromise. It depends on the type of the list, but a list that targets >> C&C channels (e.g. the FEODO lists) should not show any input packets >> being blocked, unless one of the computers being protected is infected. >> Apart from the BOGON lists, none of the lists should show any output >> packets being blocked; again it's a potential indication of infection if >> otherwise. > > The Shodan list will definitely show some activity unless you are behind CG NAT. > > Should we not have this rather in the logging section? > > And these counters here will be reset when the firewall is being reloaded. Should this data therefore not be collected from the logs so that you can go back in time? That would be very important to identify any start of compromise. >
That makes sense. I'll remove the status from this page and create a log page for the information. If anyone actually wants the iptables information it's available in from Firewall > iptables anyway.
Do you think it makes more sense to show packets blocked in/out or packets blocked per source interface? The latter would of (limited) help in tracking down potential compromise.
>>> >>> b) I would prefer to move the settings box to the top. If you like you can show the size of the blacklists there and when they have been updated, too. >> >> I put the status at the top since it's usually what I want to see when I >> go to the page. Under normal circumstances I don't change the settings, >> so I'd rather not have to scroll past them to get to the information I'm >> actually interested in. The update times are just useful to show that >> the lists are actually being updated. The size is interesting to me, >> but as you say it probably won't help the average user, especially if >> you don't know which lists block individual addresses and which block >> nets (BOGON, DSHIELD and EMERGING_COMPROMISED). > > See my comment on the order above. Splitting the page might make sense. > > When you show size, does that mean lines in the blacklists? So it could be networks or single IP addresses? Does it not make sense to count IP addresses (i.e. 256 for a /24 network and so on) and put it in the table? It's lines in the blacklist.
> > I think I would only be interested in the size for two things: > > a) Is there any chance for overblocking? i.e. does the list have 2 billion entries which would be half the public IPv4 address space or so. I cannot really come up with a good example here. > > Or > > b) What is the performance impact/memory consumption of the list?
It would be interesting to show the type (single address or net), number of entries and number of addresses; the first two affect performance, the second memory use and the latter the number of blocked addresses, but that's really only going to be of interest to a very small number of people, who can always do 'ipset list -t' to get the information. So I'll remove the size column.
> >> I originally had the status and settings in one list, but I decided that >> it was getting a bit messy and difficult to follow. I've attached a >> screenshot of a quickly hacked page. (Note that the status area only >> appears once the blacklists have been enabled.) > > You are right. This is getting quite tight there. > > You could write packets in/out into on column like 2k/1023, but see my concerns about this above.
Actually I've come to the conclusion that the number of bytes blocked isn't all that important; it's sufficient to show the number of packets.
> >>> >>> c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out. >> >> I agree that it's it's an over simplistic summary of the lists. I note >> that Tom Rymes has asked to keep it, but I think its only value is if it >> suggests to users that things are more complicated than they appear and >> therefore prompts them to read the Wiki. > > I will reply to that separately.
Looking at that reply, I think it makes sense to include a category column in both the settings and the log summary page - the former to help selection of which lists to use, and the latter as a help when deciding whether the information could indicate compromise or not.
> >>> >>> d) I do not understand what the “update check rate” means. I suppose we should show the update interval of the lists in the table and just refresh them according to that. >> >> It's the minimum check period for updates. The system checks for >> updates at either the rate specific to the list or this rate, whichever >> is slower. This is a hangover from the early days, copied from the IPS >> updates. I suppose it could have a value for some people who may wish >> to lower the check rate, but I'm not sure that it has sufficient value >> to be worth keeping. > > No, I would not let the user decide when and how often to update the lists.
OK. I'll remove it.
> > Generally there is this argument internally how often some databases should be updated. People with slow internet connections or volume based data plans will find this rather expensive to update those lists too often. > > If we ever want to go ahead with that in IPFire 2, there should be a global switch for a “low data mode” which then delays updating these things.
That makes sense. Maybe also the ability to set when downloads take place - but as you say that's for the future.
> >>> >>> e) Do we want to keep the automatic blacklists now? I do not think it will actually improve the security of IPFire. >>> >> >> I'll remove them. I think they do have a use, in that they can detect >> someone doing a port scan and block them before they find an open port, >> but as you say there is also the possibility of a DOS. > > How would this block a DOS?
Sorry, I didn't make myself clear - there's the possibility that someone could cause a DOS with the automatic list by faking the source IP address and sending a few packets on a random port.
> > -Michael
Tim > >>>> >>>> I'll update the code based on our discussions, and submit an updated set >>>> of patches - I imagine there will have to be at least one more iteration. >>> >>> Let’s wait until we have come to decisions :) >>> >>> -Michael >>> >>>> >>>> Tim >>>> >>>> (No additional comments below) >>>> >>>> On 13/12/2019 23:11, Michael Tremer wrote: >>>>> Hi, >>>>> >>>>> Again my apologies for my late reply. Busy busy weeks. >>>>> >>>>>> On 8 Dec 2019, at 20:50, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote: >>>>>> >>>>>> Hello, >>>>>> >>>>>> It's my turn to apologise for being slow to respond - I've had a busy >>>>>> week, but I should have plenty of time over the next couple of weeks. >>>>> >>>>> No worries. Turns out we all do :) >>>>> >>>>>> I've made most of the comments inline, however I think Michael had a >>>>>> question (which I can't find now) about what happens if someone enables >>>>>> all the lists. One thing which would perhaps make this less likely is >>>>>> that the WUI tags the available lists with whether they're safe or not, >>>>>> with a footnote that safe means that the list only blocks malicious >>>>>> traffic. This won't guarantee that a user won't still try to enable all >>>>>> the lists, but it should make them realise that they should think first. >>>>> >>>>> We had a couple of features going slightly wrong or being “misunderstood” by some users. People still seem to panic when they see “local recursor”. To this day I do not know why. >>>>> >>>>> We cannot make everything idiot-proof. And when some user if of that category, they probably should shutdown their IPFire box, educate themselves and then come back again. So I do not want to limit people, but make things as easy as possible. >>>>> >>>>> If someone enables all the lists, good luck with passing packets :) >>>>> >>>>>> I have considered replacing this tag with a risk high/medium/low and >>>>>> maybe adding a category (invalid/application/scanner/C&C or something >>>>>> like that), but that may provide too much information and dissuade them >>>>>> from actually following the links to checkout what the list actually does. >>>>> >>>>> Can we have a screenshot of the GUI right now? I didn’t run the code, yet. >>>>> >>>>> We should document the lists like we do it with the rulesets of the IPS. People might ignore this, but that is on them. >>>>> >>>>>> >>>>>> Tim >>>>>> >>>>>> >>>>>> On 05/12/2019 22:25, Michael Tremer wrote: >>>>>>> Hello, >>>>>>> >>>>>>>> On 4 Dec 2019, at 17:05, Peter Müller peter.mueller@ipfire.org wrote: >>>>>>>> >>>>>>>> Hello Tim, hello Michael, >>>>>>>> >>>>>>>> please see my responses inline... >>>>>>>>>>> >>>>>>>>>>> We could periodically update the blacklists on our main mirror (and >>>>>>>>>>> wait for the network to sync it), make sure it is signed and write >>>>>>>>>>> a small downloader that fetches, validates and installs them. >>>>>>>>>>> >>>>>>>>>>> @All: Thoughts on this? >>>>>>>>>> >>>>>>>>>> I think there are a number of points here. >>>>>>>>>> >>>>>>>>>> Firstly, from the point of a third party using IPFire, is this really >>>>>>>>>> solving the privacy disclosure problem? There's no way round disclosing >>>>>>>>>> your public IP address to someone you're downloading from; all this does >>>>>>>>>> is change who that information is being disclosed to. For the user >>>>>>>>>> there's no way of knowing whether the source is more or less protective >>>>>>>>>> of the user's privacy than the blacklist provider. Indeed it won't be >>>>>>>>>> possible to know who the lists are being downloaded from until the >>>>>>>>>> download starts. >>>>>>>>> >>>>>>>>> There is a way: Tor. But that is a totally different story. >>>>>>>> Well, I see a third option on this: Use the mirror infrastructure we already >>>>>>>> have. Every IPFire installation discloses its public IP address to one of these >>>>>>>> servers sooner or later, so we do not disclose additional data if the blacklists >>>>>>>> were fetched from these. >>>>>>>> >>>>>>>> Needless to say, Tor (hidden services) would be better, but that is a different >>>>>>>> story indeed. :-) >>>>>>>>> >>>>>>>>> The point is rather that a forget list can be sent instead of the “real” one. >>>>>>>> I did not get this. Forget? Forged?? ??? >>>>>>> >>>>>>> Yes, I meant to write forged, but auto-correct didn’t let me. >>>>>>> >>>>>>>>>> Secondly, latency; some of the lists are updated every 5 minutes. While >>>>>>>>>> I've limited the maximum check rate to hourly, will the updates >>>>>>>>>> propagate quickly enough. For reference on my main system the 24 >>>>>>>>>> updates on the CIARMY list made 143 498 changes (additions or >>>>>>>>>> deletions). I've seem it do over 200 000. >>>>>>>> Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not >>>>>>>> document a recommended update interval anywhere, so we can only guess. >>>>>>>> >>>>>>>> Personally, more static lists seem to be preferable for packet filtering. Highly >>>>>>>> dynamic ones such as CIArmy should be done via DNSBL queries or something similar >>>>>>>> - do we really want to have that list here? >>>>>>> >>>>>>> It is not really an option to implement a DNSBL into a packet filter, but I get your point. >>>>>> >>>>>> One of the 'selling points' for an IP address blacklist is that it can >>>>>> respond quickly to new threats - or rather new attackers. While a new >>>>>> IDS/IPS rule needs time to analyse the threat, generate a rule and check >>>>>> it, it's easy to add an address to a list. So, I think the CIArmy list >>>>>> is potentially useful for protecting home systems etc. with budget >>>>>> hardware, but I would be very careful about using it for a protecting a >>>>>> general access website. >>>>> >>>>> If they are very volatile, we should honour that and update them often, too. >>>>> >>>>> It probably is more about false-positives being removed very quickly instead of adding threats very quickly. The average IPFire user is probably not under threats like these to need to react very quickly. >>>>> >>>>> So I do not see much value in adding those lists and then updating once a day. Does it have to be every 5 min? No. I would suggest 15 which should be good enough for everyone. >>>>> >>>>> Other lists should of course not be updated every 15 minutes when not needed. >>>>> >>>>> Running every 15 minutes would allow us to retry downloading lists that are on an hourly schedule if the download failed. >>>>> >>>>>>> >>>>>>>>> How did you come up with the hour? Will it be retried more often if the download was not successful? >>>>>>>> One hour is the most common interval indeed, but adding some random time might >>>>>>>> be useful in order to reduce load on the servers providing a blacklist. >>>>>>> >>>>>>> Yes, definitely. Otherwise we will shoot down our mirrors. >>>>>> >>>>>> When I implemented that section of code, specifying the minimum check >>>>>> period in hours seemed to provide a convenient way of allowing a check >>>>>> period covering a wide range, with an hour as the fastest and a week as >>>>>> the slowest. I didn't looked at the CIArmy list until much later. Most >>>>>> of the lists don't change nearly as much, but the CIArmy list is >>>>>> described as one that deliberately responds quickly. >>>>>> >>>>>> From my production system, for yesterday: >>>>>> >>>>>> The following block lists were updated: >>>>>> BLOCKLIST_DE: 24 Time(s) - 9341 change(s) >>>>>> BOGON_FULL: 1 Time(s) - 10 change(s) >>>>>> CIARMY: 24 Time(s) - 159134 change(s) >>>>>> DSHIELD: 7 Time(s) - 18 change(s) >>>>>> EMERGING_FWRULE: 1 Time(s) - 50 change(s) >>>>>> FEODO_AGGRESIVE: 24 Time(s) - 13 change(s) >>>>>> SHODAN: 1 Time(s) - 0 change(s) >>>>>> SPAMHAUS_DROP: 1 Time(s) - 0 change(s) >>>>>> TOR_EXIT: 24 Time(s) - 162 change(s) >>>>> >>>>> Very interesting statistics. >>>>> >>>>>> >>>>>> and my test system: >>>>>> >>>>>> The following block lists were updated: >>>>>> ALIENVAULT: 19 Time(s) - 5331 change(s) >>>>>> EMERGING_COMPROMISED: 1 Time(s) - 26 change(s) >>>>>> TALOS_MALICIOUS: 1 Time(s) - 36 change(s) >>>>>> >>>>>> That covers most of the lists. From the WUI, since 1 Dec: >>>>>> >>>>>> Blacklist Entries pkts bytes Last updated >>>>>> in in >>>>>> AUTOBLACKLIST 0 731 51144 Sun Dec 8 18:40:02 2019 >>>>>> BLOCKLIST_DE 28020 857 46735 Sun Dec 8 18:40:02 2019 >>>>>> BOGON_FULL 214 5255 189K Sun Dec 8 16:50:04 2019 >>>>>> CIARMY 15000 19774 976K Sun Dec 8 18:04:01 2019 >>>>>> DSHIELD 20 7992 321K Sun Dec 8 16:45:13 2019 >>>>>> EMERGING_FWRULE 1647 197 8383 Fri Dec 6 05:29:07 2019 >>>>>> FEODO_AGGRESIVE 7169 0 0 Sun Dec 8 18:50:07 2019 >>>>>> SHODAN 32 34 1530 Sun Dec 8 17:54:09 2019 >>>>>> SPAMHAUS_DROP 823 0 0 Fri Dec 6 18:22:35 2019 >>>>>> SPAMHAUS_EDROP 111 82 16433 Thu Dec 5 17:27:09 2019 >>>>>> TOR_EXIT 1055 0 0 Sun Dec 8 18:31:02 2019 >>>>> >>>>> This as well. >>>>> >>>>> Those are more packets than I would have expected. >>>>> >>>>>> >>>>>> (I've left out the pkts/bytes out fields which were all 0) >>>>>> >>>>>> Note that where possible I do a HEAD request first and then only >>>>>> download the list if the modification time has changed since the last >>>>>> check. For dynamically generated lists this isn't possible. >>>>> >>>>> You won’t need a HEAD request for it. You can include it in the GET request. >>>>> >>>>> Have a look at location downloader where I use that. The server will respond with 304 and not send any payload. >>>>> >>>>> https://git.ipfire.org/?p=location/libloc.git;a=blob;f=src/python/location-d... >>>>> >>>>>> >>>>>> If the download isn't successful it just gives up and waits for the next >>>>>> attempt (apart from the usual retries in the library). I probably >>>>>> should to change that so that it only applies the per list minimum >>>>>> update period in this case (specified in the sources file) rather than >>>>>> the user specified value as well. >>>>> >>>>> I think it is not the worst if an update fails. It might just happen every once in a while. >>>>> >>>>> So I would suggest to just re-run the script more often and when the mtime of the file is older than the threshold, a download is attempted. You can use that timestamp for the GET request. >>>>> >>>>>> I already use a time offset on the downloads - when it's started from >>>>>> boot, backup restore or WUI enable, it checks to see if it's installed >>>>>> in the fcrontab, and if not adds itself at a randomly generated offset >>>>>> in the hour. >>>>> >>>>> That should potentially go to red.up, or if we can settle on 15 minutes, I would consider that often enough to quickly update all lists after a reboot. >>>>> >>>>>> >>>>>>> >>>>>>>>>> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is >>>>>>>>>> the largest at a few MB), there are going to be a lot of them. How will >>>>>>>>>> this affect the willingness of people to mirror IPFire? >>>>>>>> I do not consider this being a problem as we do not generate that much traffic >>>>>>>> to them. Of course, that depends on the update interval again. >>>>>>> >>>>>>> That depends on your point of view. >>>>>>> >>>>>>> I do not have a problem with this at all in my data center, but there are plenty of people with a volume-based LTE plan or simply a 128 kBit/s connection. It will take a longer time to download the lists for them. We need to mind that. >>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Talking about the preference of packet filter and IPS, I prefer to >>>>>>>>>>> use the latter as well as it gains more insight in what kind of malicious >>>>>>>>>>> traffic tried to pass a firewall machine. On systems with low resources, >>>>>>>>>>> this might be problematic and removing load from the IPS can be preferred >>>>>>>>>>> (make this configurable?!), on others, people might want to have both >>>>>>>>>>> results. >>>>>>>>>>> >>>>>>>>>> You're only going to get one result for a packet whichever way round the >>>>>>>>>> IP blacklist and IPS are since whichever comes first will drop the >>>>>>>>>> packet before it reaches the second (well it would be possible to put >>>>>>>>>> the IP blacklist first and get it to log and mark packets which are then >>>>>>>>>> dropped after the IPS, but I think that's getting a little complicated. >>>>>>>>>> In addition I've seen the messages about the trouble marking was >>>>>>>>>> causing in the QoS). >>>>>>>>>> >>>>>>>>>> I think it's a 50/50 choice as to which is more valuable first; it's >>>>>>>>>> probably going to differ from packet to packet. For me the possibility >>>>>>>>>> of reducing the IPS load means I prefer putting the IP blacklist first >>>>>>>>>> >>>>>>>>>> It should be fairly easy to add the choice of where to put the IP >>>>>>>>>> blacklist. I think it'll have to be in the main firewall script, so >>>>>>>>>> it'll require a firewall restart, but it's not something that'll be >>>>>>>>>> changed often. >>>>>>>>> >>>>>>>>> I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier. >>>>>>>> Agreed. >>>>>>>> >>>>>>>>> >>>>>>>>> I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads. >>>>>>>>> >>>>>>>>> We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily. >>>>>>>> So dropping packets from blacklisted IP addresses/networks before IPS is it, then. >>>>>>>>> >>>>>>>>> I do not even think it makes sense to swap the order in the outgoing direction. >>>>>>>> Me too. >>>>>>>> >>>>>>>>> >>>>>>>>> What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this. >>>>>>>> Based on my personal experience, firewall hits observed on a single machine exposed >>>>>>>> to the internet are interesting, but the overall situation across multiple machines >>>>>>>> is even more interesting. Very quickly, you'll end on something like a centralised >>>>>>>> logging server and custom statistical analysis here... >>>>>>> >>>>>>> Probably a project for IPFire 4.0 :) >>>>>>> >>>>>> Or use one of the existing services, like the DSHIELD client >>>>>> https://dshield.org/howto.html (subject to privacy concerns again). >>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default. >>>>>>>>>>> I would love to see the bogon ruleset here, too (think about 8chan successor >>>>>>>>>>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause >>>>>>>>>>> interference if RED does not have a public IP address assigned. >>>>>>>>>> >>>>>>>>>> I can add a field to the options file that controls whether a list is >>>>>>>>>> enabled by default. >>>>>>>> Thank you. :-) >>>>>>>>> >>>>>>>>> To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat. >>>>>>>> This is my only objection against this patchset. Now, what can we do about it? >>>>>>>> One possibility is to apply the patchset now and implement a custom download >>>>>>>> source thing later on, or do that before releasing Core Update 139 (or which version >>>>>>>> the patchset will be to) after we agreed on something. >>>>>>> >>>>>>> I do not see this being merged for 139. But that is not important. We need to get it right first and then release it. >>>>>>> >>>>>>> As far as I know, nobody has tested this, yet. >>>>>> >>>>>> There are a number of people who have been running an earlier version >>>>>> which I shared on GitHub. There were a few early issues, but it seems >>>>>> to be OK now. >>>>>> >>>>>> https://forum.ipfire.org/viewtopic.php?f=27&t=21845 >>>>>> >>>>>> This version wasn't integrated into IPFire, so (for example) it inserted >>>>>> itself into the INPUT IPTables chain rather than having it's chains >>>>>> created as part of the firewall start-up script. >>>>>> >>>>>>> >>>>>>> I have huge concerns about the automatic blacklist. @Peter: What is your opinion on this? >>>>>>> >>>>>> >>>>>> While I implemented it, I'm aware of its potential to cause problems, >>>>>> which is why it has to be separately enabled. It's not caused me any >>>>>> issues at the default settings (blocks at over 10 packets per hour until >>>>>> 1 hour has passed without seeing packets from the address), but I've not >>>>>> used it on a site with publicly announced services. If I was going to >>>>>> use it on a web site I would want to, at the very minimum, drop the >>>>>> block period drastically. >>>>> >>>>> I suppose this is entirely unusable on an IPFire box in a data center that hosts things. Let’s say our rack in Hanover. >>>>> >>>>> You will have hits from some broken IP stacks and people might just end up on this without doing anything wrong. >>>>> >>>>> You can denial-of-service easily as well and I suppose without aggregation of data from many many systems, it does not make sense to instantly block addresses. And even then you probably would block an entire subnet. >>>>> >>>>> So I am not sure how I should feel about it. I do not think that it adds anything because the packets would have been blocked anyways. >>>>> >>>>>> >>>>>> On the other hand, it's good at responding quickly. Usually I see only >>>>>> 1-2% of blocks from the automatic list: >>>>>> >>>>>> Reason Count % First Last >>>>>> CIARMY 2416 45 Dec 6 00:00 Dec 6 23:59 >>>>>> DSHIELD 1353 25 Dec 6 00:00 Dec 6 23:59 >>>>>> INPUT 1294 24 Dec 6 00:00 Dec 6 23:59 >>>>>> AUTOBLACKLIST 122 2 Dec 6 00:20 Dec 6 16:28 >>>>>> BLOCKLIST_DE 89 2 Dec 6 00:20 Dec 6 23:46 >>>>>> >>>>>> and sometimes none at all, but one one occasion it blocked over 8000 >>>>>> packets. Again I'm aware this is for a home system, which is rather >>>>>> different than from a Web server. >>>>>> >>>>>>>> If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not >>>>>>>> seem to be problematic, as they are hosted on 3rd party servers, too). >>>>>>> >>>>>>> One of them will be, sooner or later. And one is enough I suppose. >>>>>> >>>>>> DShield (https://dshield.org/api/#threatfeeds) and firehol >>>>>> (http://iplists.firehol.org/) seem to host copies of most of the lists >>>>>> as well. >>>>>>> >>>>>>> I do not really want to overthink this - we didn’t do this with clamav for example either. But it probably is a decision to be made by the user what they want to enable and we should not enable anything by default. So no data will be leaked as long as the user does not consent. >>>>>>> >>>>>>> -Michael >>>>>>> >>>>>>>> >>>>>>>> Thanks, and best regards, >>>>>>>> Peter Müller >>>>> >>>> >>>> <Screenshot_2019-12-16_IP_Address_Blacklists.png> >>> >> >> <Screenshot_2019-12-17_IP_Address_Blacklists.png> >
<IP_Address_Blacklists.png><IP_Address_Blacklist_Logs.png><Firewall_log_blacklist.png><Log_Summary.png>
Hi,
On 29 Jan 2020, at 20:40, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Hi,
On 28/01/2020 17:14, Michael Tremer wrote:
Hi,
Apologies for my late reply again. I marked this email in my inbox and then it scrolled down because of loads of new stuff coming in. Please feel free to ping me if I do not respond in time.
That's OK. It took me a little longer than expected to check the patches.
On 24 Jan 2020, at 19:40, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
I'm now nearly ready to submit the updated patches, but before then I'd like to check that you're happy with some of my design decisions.
Cool.
_IPTables_Chains_
When setting up the firewall (in iptables_init()), four new chains are created, IPBLACKLISTIN, IPBLACKLISTOUT, IPBLACKLISTREDIN and IPBLACKLISTREDOUT. The first two are inserted into the INPUT, OUTPUT and FORWARD chains, before the IPS chains. The main script is called to restore the previously saved blacklists, creating the rules in the latter two chains.
For readability you can simply call them BLACKLIST*. The IP is a given.
Good point. I'll make the change.
In iptables_red_up(), rules are created to forward traffic to/from the red interface from IPBLACKLISTIN/IPBLACKLISTOUT to IPBLACKLISTREDIN/IPBLACKLISTREDOUT. I did consider putting the blacklist rules into the IPBLACKLISTIN/IPBLACKLISTOUT, but this couldn't be done until red comes up and would require much more processing at this point - not a problem if your red interface comes up and stays up, but if you've got one that disconnects on no traffic you probably don't want the extra overhead. The way I've done it also has the side effects of 1) the main script doesn't have to worry about what the red interface is, and 2) if you're using multiple blacklists, the check for the red interface is only done once.
You would only want to check the interface once and then jump into the chain. Can we not realise it that way?
I'm not sure what you're suggesting. I insert BLACKLISTIN into INPUT & FORWARD and BLACKLISTOUT into OUTPUT & FORWARD. These receive all non-ICMP traffic. On red up traffic from the red interface is sent from BLACKLISTIN to BLACKLISTREDIN and traffic to the red interface is sent from BLACKLISTOUT to BLACKLISTREDOUT. The rules to check against the IPSets are put into the *RED* chain, checking against the source or destination addresses as appropriate.
This sounds very complicated to me.
Why can we not check this in BLACKLISTIN/OUT?
_Log_WUI_Pages_
The two Log WUI pages are edited from existing pages. I've resisted the temptation to reformat them, which means that they can be diff'ed with the corresponding pages.
Loads of the UI code is messy and deliberately not touched any more. I guess this is fine.
Yes, it's not pretty, but it is pragmatic.
_Update_
The update is fairly simple - it calculates whether the check period is passed, tries a download specifying the If-Modified-Since and parses the returned file. It then compares the new list with the existing list, adding new addresses/networks and deleting those that have been removed. There's a check to handle a list changing from a list of addresses to a list of networks, in which case it changes the type of the IPSet - this should be a vary rare event, but could happen for a composite list.
Why do you need the diff? Has that been in the first patchset?
It was, yes.
Originally I did it for performance. It took over a minute to process the ALIENVAULT list (over 80 000 entries); just calling ipset with changes reduced this to a few seconds. I've since modified the code to pipe commands to a process running 'ipset restore', which has drastically speed up the processing, but it's still a little bit quicker to just do changes.
The second reason is that it doesn't feel right to flush the old list, thus removing its protection while the new one is loaded, especially if the new list is the same as the old list (not all sources implement IP-Modified-Since). The actual period of reduced protection may be short, but I prefer to avoid it.
Okay. I was hoping that the “ipset restore” update would flush and add everything again in one large atomic operation.
Just reloading the whole ipset should be easier.
_Enabling/Disabling_Blacklists_
For changes to the existing firewall rules, the firewall is shut down and restarted from scratch. The blacklist script does this on the fly, since the possibilities are simpler than all the options available for the firewall and also the necessary code for doing updates means that adding this code is fairly simple. This includes turning logging on and off.
What do you mean by that? /etc/init.d/firewall restart?
That is something we cannot do because IPsec and some other services create temporary rules that would be lost.
No. I'm actually trying to say the opposite (badly). The existing IPFire code only implements some firewall changes by shutting down and restarting the firewall, but all the IP Blacklist functions (enable/disable/update lists, enable/disable logging) can be done without this - the necessary rules can be added to/deleted from the BLACKLISTRED* chains without affecting any of the rest of the firewall system.
Ah cool. That is how it needs to be.
-Michael
Tim
P.S. Is there actually a reason why you do not have a shell account on our servers and host your changed on our git server? I am not sure if I ever offered you one, but it would help me a lot to have a look at the code in a git repo :)
_Logging_
The main scripts logs to syslog as ipblacklist, but if you run it from the command line it'll also output to the terminal. There's also some debug code which can be turned on by setting a value in the settings file - it's turned on at the lowest level automatically if you run from the command line.
Yes that is fine.
Best, -Michael
Tim
On 06/01/2020 11:27, Michael Tremer wrote:
Hello all,
On 4 Jan 2020, at 19:02, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Hello,
I've now updated the code - screenshots are attached.
IP_Address_Blacklists.png
This is the settings page. I don't think there's anything unexpected here.
Yes, that looks a lot like the other IPFire pages :)
IP_Address_Blacklist_Logs.png
The log page, replacing the status information that was previously at the top of the settings page.
Looks great!
I've sorted the lists alphabetically, rather than by the number of hits since the worst threat is not necessary the largest number of blocked packets (which could implied by sorting it by number).
Yes, that is indeed a very good idea.
You could add a % sign in the percentage column, so it cannot be confused so easily with the other column.
Firewall_log_blacklist.png
Details page accessed from the log. Almost identical to the other similar pages.
*thumbs up*
Log_Summary.png
This is an extract from the Log Summary page produced by logwatch.
I still need to do some tidying, remove unused language strings etc., and then update and test the patches, so it'll be some time before I'm ready to submit updated patches.
Feel free to send some RFC to the list. It might be slightly noisy, but it is easier to review changes bit after bit.
Great great work!
-Michael
Tim
On 28/12/2019 20:59, Tim FitzGeorge wrote:
Hi,
I'll code and test what we've agreed on. In the meantime I'm going to reply on the sub-thread started by Tom Rymes as to what we call the list categories, so that we can keep that discussion separate from further functionality etc. discussion.
Tim
On 21/12/2019 18:34, Tim FitzGeorge wrote: > Hi, > > I've got my systems checking for downloads at 15 minute intervals and > using If-Modified-Since. This all seems to be working OK. > > Responses to comments below: > > On 18/12/2019 12:07, Michael Tremer wrote: >> Hi, >> >>> On 17 Dec 2019, at 18:21, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote: >>> >>> Hi, >>> >>> I think we've agreed on the following: >>> >>> - Checks to be made every 15 minutes, subject to per-list minimum rate. >>> - Modify download to use If-Modified-Since rather than separate HEAD >>> request. >>> - Remove automatic blacklist. >> >> I thought someone wants to counter my argument. This is not a dictatorship :) >> >>> >>> other comments are below. >>> >>> Tim >>> >>> On 16/12/2019 22:20, Michael Tremer wrote: >>>> Hi, >>>> >>>>> On 16 Dec 2019, at 20:06, Tim FitzGeorge lists@tfitzgeorge.me.uk wrote: >>>>> >>>>> Hi, >>>>> >>>>> I've attached the current GUI screenshot. >>>> >>>> Thanks for that. >>>> >>>> I have a couple of suggestions/concerns about it: >>>> >>>> a) I am not sure what the value is about the top box that is called “Status”. It is basically a summary of the iptables output, but will it help the user? A blacklist with more or fewer entries is not better or worse, blocking more packets isn’t better than blocking fewer. It is about the quality of the blocks. >>> >>> I agree that the size of the lists is not all that useful, but the >>> information about the number of blocked packets can be an indication of >>> compromise. It depends on the type of the list, but a list that targets >>> C&C channels (e.g. the FEODO lists) should not show any input packets >>> being blocked, unless one of the computers being protected is infected. >>> Apart from the BOGON lists, none of the lists should show any output >>> packets being blocked; again it's a potential indication of infection if >>> otherwise. >> >> The Shodan list will definitely show some activity unless you are behind CG NAT. >> >> Should we not have this rather in the logging section? >> >> And these counters here will be reset when the firewall is being reloaded. Should this data therefore not be collected from the logs so that you can go back in time? That would be very important to identify any start of compromise. >> > > That makes sense. I'll remove the status from this page and create a > log page for the information. If anyone actually wants the iptables > information it's available in from Firewall > iptables anyway. > > Do you think it makes more sense to show packets blocked in/out or > packets blocked per source interface? The latter would of (limited) > help in tracking down potential compromise. > >>>> >>>> b) I would prefer to move the settings box to the top. If you like you can show the size of the blacklists there and when they have been updated, too. >>> >>> I put the status at the top since it's usually what I want to see when I >>> go to the page. Under normal circumstances I don't change the settings, >>> so I'd rather not have to scroll past them to get to the information I'm >>> actually interested in. The update times are just useful to show that >>> the lists are actually being updated. The size is interesting to me, >>> but as you say it probably won't help the average user, especially if >>> you don't know which lists block individual addresses and which block >>> nets (BOGON, DSHIELD and EMERGING_COMPROMISED). >> >> See my comment on the order above. Splitting the page might make sense. >> >> When you show size, does that mean lines in the blacklists? So it could be networks or single IP addresses? Does it not make sense to count IP addresses (i.e. 256 for a /24 network and so on) and put it in the table? > It's lines in the blacklist. > >> >> I think I would only be interested in the size for two things: >> >> a) Is there any chance for overblocking? i.e. does the list have 2 billion entries which would be half the public IPv4 address space or so. I cannot really come up with a good example here. >> >> Or >> >> b) What is the performance impact/memory consumption of the list? > > It would be interesting to show the type (single address or net), number > of entries and number of addresses; the first two affect performance, > the second memory use and the latter the number of blocked addresses, > but that's really only going to be of interest to a very small number of > people, who can always do 'ipset list -t' to get the information. So > I'll remove the size column. > >> >>> I originally had the status and settings in one list, but I decided that >>> it was getting a bit messy and difficult to follow. I've attached a >>> screenshot of a quickly hacked page. (Note that the status area only >>> appears once the blacklists have been enabled.) >> >> You are right. This is getting quite tight there. >> >> You could write packets in/out into on column like 2k/1023, but see my concerns about this above. > > Actually I've come to the conclusion that the number of bytes blocked > isn't all that important; it's sufficient to show the number of packets. > >> >>>> >>>> c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out. >>> >>> I agree that it's it's an over simplistic summary of the lists. I note >>> that Tom Rymes has asked to keep it, but I think its only value is if it >>> suggests to users that things are more complicated than they appear and >>> therefore prompts them to read the Wiki. >> >> I will reply to that separately. > > Looking at that reply, I think it makes sense to include a category > column in both the settings and the log summary page - the former to > help selection of which lists to use, and the latter as a help when > deciding whether the information could indicate compromise or not. > >> >>>> >>>> d) I do not understand what the “update check rate” means. I suppose we should show the update interval of the lists in the table and just refresh them according to that. >>> >>> It's the minimum check period for updates. The system checks for >>> updates at either the rate specific to the list or this rate, whichever >>> is slower. This is a hangover from the early days, copied from the IPS >>> updates. I suppose it could have a value for some people who may wish >>> to lower the check rate, but I'm not sure that it has sufficient value >>> to be worth keeping. >> >> No, I would not let the user decide when and how often to update the lists. > > OK. I'll remove it. > >> >> Generally there is this argument internally how often some databases should be updated. People with slow internet connections or volume based data plans will find this rather expensive to update those lists too often. >> >> If we ever want to go ahead with that in IPFire 2, there should be a global switch for a “low data mode” which then delays updating these things. > > That makes sense. Maybe also the ability to set when downloads take > place - but as you say that's for the future. > >> >>>> >>>> e) Do we want to keep the automatic blacklists now? I do not think it will actually improve the security of IPFire. >>>> >>> >>> I'll remove them. I think they do have a use, in that they can detect >>> someone doing a port scan and block them before they find an open port, >>> but as you say there is also the possibility of a DOS. >> >> How would this block a DOS? > > Sorry, I didn't make myself clear - there's the possibility that someone > could cause a DOS with the automatic list by faking the source IP > address and sending a few packets on a random port. > >> >> -Michael > > Tim >> >>>>> >>>>> I'll update the code based on our discussions, and submit an updated set >>>>> of patches - I imagine there will have to be at least one more iteration. >>>> >>>> Let’s wait until we have come to decisions :) >>>> >>>> -Michael >>>> >>>>> >>>>> Tim >>>>> >>>>> (No additional comments below) >>>>> >>>>> On 13/12/2019 23:11, Michael Tremer wrote: >>>>>> Hi, >>>>>> >>>>>> Again my apologies for my late reply. Busy busy weeks. >>>>>> >>>>>>> On 8 Dec 2019, at 20:50, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote: >>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> It's my turn to apologise for being slow to respond - I've had a busy >>>>>>> week, but I should have plenty of time over the next couple of weeks. >>>>>> >>>>>> No worries. Turns out we all do :) >>>>>> >>>>>>> I've made most of the comments inline, however I think Michael had a >>>>>>> question (which I can't find now) about what happens if someone enables >>>>>>> all the lists. One thing which would perhaps make this less likely is >>>>>>> that the WUI tags the available lists with whether they're safe or not, >>>>>>> with a footnote that safe means that the list only blocks malicious >>>>>>> traffic. This won't guarantee that a user won't still try to enable all >>>>>>> the lists, but it should make them realise that they should think first. >>>>>> >>>>>> We had a couple of features going slightly wrong or being “misunderstood” by some users. People still seem to panic when they see “local recursor”. To this day I do not know why. >>>>>> >>>>>> We cannot make everything idiot-proof. And when some user if of that category, they probably should shutdown their IPFire box, educate themselves and then come back again. So I do not want to limit people, but make things as easy as possible. >>>>>> >>>>>> If someone enables all the lists, good luck with passing packets :) >>>>>> >>>>>>> I have considered replacing this tag with a risk high/medium/low and >>>>>>> maybe adding a category (invalid/application/scanner/C&C or something >>>>>>> like that), but that may provide too much information and dissuade them >>>>>>> from actually following the links to checkout what the list actually does. >>>>>> >>>>>> Can we have a screenshot of the GUI right now? I didn’t run the code, yet. >>>>>> >>>>>> We should document the lists like we do it with the rulesets of the IPS. People might ignore this, but that is on them. >>>>>> >>>>>>> >>>>>>> Tim >>>>>>> >>>>>>> >>>>>>> On 05/12/2019 22:25, Michael Tremer wrote: >>>>>>>> Hello, >>>>>>>> >>>>>>>>> On 4 Dec 2019, at 17:05, Peter Müller peter.mueller@ipfire.org wrote: >>>>>>>>> >>>>>>>>> Hello Tim, hello Michael, >>>>>>>>> >>>>>>>>> please see my responses inline... >>>>>>>>>>>> >>>>>>>>>>>> We could periodically update the blacklists on our main mirror (and >>>>>>>>>>>> wait for the network to sync it), make sure it is signed and write >>>>>>>>>>>> a small downloader that fetches, validates and installs them. >>>>>>>>>>>> >>>>>>>>>>>> @All: Thoughts on this? >>>>>>>>>>> >>>>>>>>>>> I think there are a number of points here. >>>>>>>>>>> >>>>>>>>>>> Firstly, from the point of a third party using IPFire, is this really >>>>>>>>>>> solving the privacy disclosure problem? There's no way round disclosing >>>>>>>>>>> your public IP address to someone you're downloading from; all this does >>>>>>>>>>> is change who that information is being disclosed to. For the user >>>>>>>>>>> there's no way of knowing whether the source is more or less protective >>>>>>>>>>> of the user's privacy than the blacklist provider. Indeed it won't be >>>>>>>>>>> possible to know who the lists are being downloaded from until the >>>>>>>>>>> download starts. >>>>>>>>>> >>>>>>>>>> There is a way: Tor. But that is a totally different story. >>>>>>>>> Well, I see a third option on this: Use the mirror infrastructure we already >>>>>>>>> have. Every IPFire installation discloses its public IP address to one of these >>>>>>>>> servers sooner or later, so we do not disclose additional data if the blacklists >>>>>>>>> were fetched from these. >>>>>>>>> >>>>>>>>> Needless to say, Tor (hidden services) would be better, but that is a different >>>>>>>>> story indeed. :-) >>>>>>>>>> >>>>>>>>>> The point is rather that a forget list can be sent instead of the “real” one. >>>>>>>>> I did not get this. Forget? Forged?? ??? >>>>>>>> >>>>>>>> Yes, I meant to write forged, but auto-correct didn’t let me. >>>>>>>> >>>>>>>>>>> Secondly, latency; some of the lists are updated every 5 minutes. While >>>>>>>>>>> I've limited the maximum check rate to hourly, will the updates >>>>>>>>>>> propagate quickly enough. For reference on my main system the 24 >>>>>>>>>>> updates on the CIARMY list made 143 498 changes (additions or >>>>>>>>>>> deletions). I've seem it do over 200 000. >>>>>>>>> Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not >>>>>>>>> document a recommended update interval anywhere, so we can only guess. >>>>>>>>> >>>>>>>>> Personally, more static lists seem to be preferable for packet filtering. Highly >>>>>>>>> dynamic ones such as CIArmy should be done via DNSBL queries or something similar >>>>>>>>> - do we really want to have that list here? >>>>>>>> >>>>>>>> It is not really an option to implement a DNSBL into a packet filter, but I get your point. >>>>>>> >>>>>>> One of the 'selling points' for an IP address blacklist is that it can >>>>>>> respond quickly to new threats - or rather new attackers. While a new >>>>>>> IDS/IPS rule needs time to analyse the threat, generate a rule and check >>>>>>> it, it's easy to add an address to a list. So, I think the CIArmy list >>>>>>> is potentially useful for protecting home systems etc. with budget >>>>>>> hardware, but I would be very careful about using it for a protecting a >>>>>>> general access website. >>>>>> >>>>>> If they are very volatile, we should honour that and update them often, too. >>>>>> >>>>>> It probably is more about false-positives being removed very quickly instead of adding threats very quickly. The average IPFire user is probably not under threats like these to need to react very quickly. >>>>>> >>>>>> So I do not see much value in adding those lists and then updating once a day. Does it have to be every 5 min? No. I would suggest 15 which should be good enough for everyone. >>>>>> >>>>>> Other lists should of course not be updated every 15 minutes when not needed. >>>>>> >>>>>> Running every 15 minutes would allow us to retry downloading lists that are on an hourly schedule if the download failed. >>>>>> >>>>>>>> >>>>>>>>>> How did you come up with the hour? Will it be retried more often if the download was not successful? >>>>>>>>> One hour is the most common interval indeed, but adding some random time might >>>>>>>>> be useful in order to reduce load on the servers providing a blacklist. >>>>>>>> >>>>>>>> Yes, definitely. Otherwise we will shoot down our mirrors. >>>>>>> >>>>>>> When I implemented that section of code, specifying the minimum check >>>>>>> period in hours seemed to provide a convenient way of allowing a check >>>>>>> period covering a wide range, with an hour as the fastest and a week as >>>>>>> the slowest. I didn't looked at the CIArmy list until much later. Most >>>>>>> of the lists don't change nearly as much, but the CIArmy list is >>>>>>> described as one that deliberately responds quickly. >>>>>>> >>>>>>> From my production system, for yesterday: >>>>>>> >>>>>>> The following block lists were updated: >>>>>>> BLOCKLIST_DE: 24 Time(s) - 9341 change(s) >>>>>>> BOGON_FULL: 1 Time(s) - 10 change(s) >>>>>>> CIARMY: 24 Time(s) - 159134 change(s) >>>>>>> DSHIELD: 7 Time(s) - 18 change(s) >>>>>>> EMERGING_FWRULE: 1 Time(s) - 50 change(s) >>>>>>> FEODO_AGGRESIVE: 24 Time(s) - 13 change(s) >>>>>>> SHODAN: 1 Time(s) - 0 change(s) >>>>>>> SPAMHAUS_DROP: 1 Time(s) - 0 change(s) >>>>>>> TOR_EXIT: 24 Time(s) - 162 change(s) >>>>>> >>>>>> Very interesting statistics. >>>>>> >>>>>>> >>>>>>> and my test system: >>>>>>> >>>>>>> The following block lists were updated: >>>>>>> ALIENVAULT: 19 Time(s) - 5331 change(s) >>>>>>> EMERGING_COMPROMISED: 1 Time(s) - 26 change(s) >>>>>>> TALOS_MALICIOUS: 1 Time(s) - 36 change(s) >>>>>>> >>>>>>> That covers most of the lists. From the WUI, since 1 Dec: >>>>>>> >>>>>>> Blacklist Entries pkts bytes Last updated >>>>>>> in in >>>>>>> AUTOBLACKLIST 0 731 51144 Sun Dec 8 18:40:02 2019 >>>>>>> BLOCKLIST_DE 28020 857 46735 Sun Dec 8 18:40:02 2019 >>>>>>> BOGON_FULL 214 5255 189K Sun Dec 8 16:50:04 2019 >>>>>>> CIARMY 15000 19774 976K Sun Dec 8 18:04:01 2019 >>>>>>> DSHIELD 20 7992 321K Sun Dec 8 16:45:13 2019 >>>>>>> EMERGING_FWRULE 1647 197 8383 Fri Dec 6 05:29:07 2019 >>>>>>> FEODO_AGGRESIVE 7169 0 0 Sun Dec 8 18:50:07 2019 >>>>>>> SHODAN 32 34 1530 Sun Dec 8 17:54:09 2019 >>>>>>> SPAMHAUS_DROP 823 0 0 Fri Dec 6 18:22:35 2019 >>>>>>> SPAMHAUS_EDROP 111 82 16433 Thu Dec 5 17:27:09 2019 >>>>>>> TOR_EXIT 1055 0 0 Sun Dec 8 18:31:02 2019 >>>>>> >>>>>> This as well. >>>>>> >>>>>> Those are more packets than I would have expected. >>>>>> >>>>>>> >>>>>>> (I've left out the pkts/bytes out fields which were all 0) >>>>>>> >>>>>>> Note that where possible I do a HEAD request first and then only >>>>>>> download the list if the modification time has changed since the last >>>>>>> check. For dynamically generated lists this isn't possible. >>>>>> >>>>>> You won’t need a HEAD request for it. You can include it in the GET request. >>>>>> >>>>>> Have a look at location downloader where I use that. The server will respond with 304 and not send any payload. >>>>>> >>>>>> https://git.ipfire.org/?p=location/libloc.git;a=blob;f=src/python/location-d... >>>>>> >>>>>>> >>>>>>> If the download isn't successful it just gives up and waits for the next >>>>>>> attempt (apart from the usual retries in the library). I probably >>>>>>> should to change that so that it only applies the per list minimum >>>>>>> update period in this case (specified in the sources file) rather than >>>>>>> the user specified value as well. >>>>>> >>>>>> I think it is not the worst if an update fails. It might just happen every once in a while. >>>>>> >>>>>> So I would suggest to just re-run the script more often and when the mtime of the file is older than the threshold, a download is attempted. You can use that timestamp for the GET request. >>>>>> >>>>>>> I already use a time offset on the downloads - when it's started from >>>>>>> boot, backup restore or WUI enable, it checks to see if it's installed >>>>>>> in the fcrontab, and if not adds itself at a randomly generated offset >>>>>>> in the hour. >>>>>> >>>>>> That should potentially go to red.up, or if we can settle on 15 minutes, I would consider that often enough to quickly update all lists after a reboot. >>>>>> >>>>>>> >>>>>>>> >>>>>>>>>>> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is >>>>>>>>>>> the largest at a few MB), there are going to be a lot of them. How will >>>>>>>>>>> this affect the willingness of people to mirror IPFire? >>>>>>>>> I do not consider this being a problem as we do not generate that much traffic >>>>>>>>> to them. Of course, that depends on the update interval again. >>>>>>>> >>>>>>>> That depends on your point of view. >>>>>>>> >>>>>>>> I do not have a problem with this at all in my data center, but there are plenty of people with a volume-based LTE plan or simply a 128 kBit/s connection. It will take a longer time to download the lists for them. We need to mind that. >>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Talking about the preference of packet filter and IPS, I prefer to >>>>>>>>>>>> use the latter as well as it gains more insight in what kind of malicious >>>>>>>>>>>> traffic tried to pass a firewall machine. On systems with low resources, >>>>>>>>>>>> this might be problematic and removing load from the IPS can be preferred >>>>>>>>>>>> (make this configurable?!), on others, people might want to have both >>>>>>>>>>>> results. >>>>>>>>>>>> >>>>>>>>>>> You're only going to get one result for a packet whichever way round the >>>>>>>>>>> IP blacklist and IPS are since whichever comes first will drop the >>>>>>>>>>> packet before it reaches the second (well it would be possible to put >>>>>>>>>>> the IP blacklist first and get it to log and mark packets which are then >>>>>>>>>>> dropped after the IPS, but I think that's getting a little complicated. >>>>>>>>>>> In addition I've seen the messages about the trouble marking was >>>>>>>>>>> causing in the QoS). >>>>>>>>>>> >>>>>>>>>>> I think it's a 50/50 choice as to which is more valuable first; it's >>>>>>>>>>> probably going to differ from packet to packet. For me the possibility >>>>>>>>>>> of reducing the IPS load means I prefer putting the IP blacklist first >>>>>>>>>>> >>>>>>>>>>> It should be fairly easy to add the choice of where to put the IP >>>>>>>>>>> blacklist. I think it'll have to be in the main firewall script, so >>>>>>>>>>> it'll require a firewall restart, but it's not something that'll be >>>>>>>>>>> changed often. >>>>>>>>>> >>>>>>>>>> I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier. >>>>>>>>> Agreed. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads. >>>>>>>>>> >>>>>>>>>> We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily. >>>>>>>>> So dropping packets from blacklisted IP addresses/networks before IPS is it, then. >>>>>>>>>> >>>>>>>>>> I do not even think it makes sense to swap the order in the outgoing direction. >>>>>>>>> Me too. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this. >>>>>>>>> Based on my personal experience, firewall hits observed on a single machine exposed >>>>>>>>> to the internet are interesting, but the overall situation across multiple machines >>>>>>>>> is even more interesting. Very quickly, you'll end on something like a centralised >>>>>>>>> logging server and custom statistical analysis here... >>>>>>>> >>>>>>>> Probably a project for IPFire 4.0 :) >>>>>>>> >>>>>>> Or use one of the existing services, like the DSHIELD client >>>>>>> https://dshield.org/howto.html (subject to privacy concerns again). >>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default. >>>>>>>>>>>> I would love to see the bogon ruleset here, too (think about 8chan successor >>>>>>>>>>>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause >>>>>>>>>>>> interference if RED does not have a public IP address assigned. >>>>>>>>>>> >>>>>>>>>>> I can add a field to the options file that controls whether a list is >>>>>>>>>>> enabled by default. >>>>>>>>> Thank you. :-) >>>>>>>>>> >>>>>>>>>> To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat. >>>>>>>>> This is my only objection against this patchset. Now, what can we do about it? >>>>>>>>> One possibility is to apply the patchset now and implement a custom download >>>>>>>>> source thing later on, or do that before releasing Core Update 139 (or which version >>>>>>>>> the patchset will be to) after we agreed on something. >>>>>>>> >>>>>>>> I do not see this being merged for 139. But that is not important. We need to get it right first and then release it. >>>>>>>> >>>>>>>> As far as I know, nobody has tested this, yet. >>>>>>> >>>>>>> There are a number of people who have been running an earlier version >>>>>>> which I shared on GitHub. There were a few early issues, but it seems >>>>>>> to be OK now. >>>>>>> >>>>>>> https://forum.ipfire.org/viewtopic.php?f=27&t=21845 >>>>>>> >>>>>>> This version wasn't integrated into IPFire, so (for example) it inserted >>>>>>> itself into the INPUT IPTables chain rather than having it's chains >>>>>>> created as part of the firewall start-up script. >>>>>>> >>>>>>>> >>>>>>>> I have huge concerns about the automatic blacklist. @Peter: What is your opinion on this? >>>>>>>> >>>>>>> >>>>>>> While I implemented it, I'm aware of its potential to cause problems, >>>>>>> which is why it has to be separately enabled. It's not caused me any >>>>>>> issues at the default settings (blocks at over 10 packets per hour until >>>>>>> 1 hour has passed without seeing packets from the address), but I've not >>>>>>> used it on a site with publicly announced services. If I was going to >>>>>>> use it on a web site I would want to, at the very minimum, drop the >>>>>>> block period drastically. >>>>>> >>>>>> I suppose this is entirely unusable on an IPFire box in a data center that hosts things. Let’s say our rack in Hanover. >>>>>> >>>>>> You will have hits from some broken IP stacks and people might just end up on this without doing anything wrong. >>>>>> >>>>>> You can denial-of-service easily as well and I suppose without aggregation of data from many many systems, it does not make sense to instantly block addresses. And even then you probably would block an entire subnet. >>>>>> >>>>>> So I am not sure how I should feel about it. I do not think that it adds anything because the packets would have been blocked anyways. >>>>>> >>>>>>> >>>>>>> On the other hand, it's good at responding quickly. Usually I see only >>>>>>> 1-2% of blocks from the automatic list: >>>>>>> >>>>>>> Reason Count % First Last >>>>>>> CIARMY 2416 45 Dec 6 00:00 Dec 6 23:59 >>>>>>> DSHIELD 1353 25 Dec 6 00:00 Dec 6 23:59 >>>>>>> INPUT 1294 24 Dec 6 00:00 Dec 6 23:59 >>>>>>> AUTOBLACKLIST 122 2 Dec 6 00:20 Dec 6 16:28 >>>>>>> BLOCKLIST_DE 89 2 Dec 6 00:20 Dec 6 23:46 >>>>>>> >>>>>>> and sometimes none at all, but one one occasion it blocked over 8000 >>>>>>> packets. Again I'm aware this is for a home system, which is rather >>>>>>> different than from a Web server. >>>>>>> >>>>>>>>> If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not >>>>>>>>> seem to be problematic, as they are hosted on 3rd party servers, too). >>>>>>>> >>>>>>>> One of them will be, sooner or later. And one is enough I suppose. >>>>>>> >>>>>>> DShield (https://dshield.org/api/#threatfeeds) and firehol >>>>>>> (http://iplists.firehol.org/) seem to host copies of most of the lists >>>>>>> as well. >>>>>>>> >>>>>>>> I do not really want to overthink this - we didn’t do this with clamav for example either. But it probably is a decision to be made by the user what they want to enable and we should not enable anything by default. So no data will be leaked as long as the user does not consent. >>>>>>>> >>>>>>>> -Michael >>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, and best regards, >>>>>>>>> Peter Müller >>>>>> >>>>> >>>>> <Screenshot_2019-12-16_IP_Address_Blacklists.png> >>>> >>> >>> <Screenshot_2019-12-17_IP_Address_Blacklists.png> >> >
<IP_Address_Blacklists.png><IP_Address_Blacklist_Logs.png><Firewall_log_blacklist.png><Log_Summary.png>
Hi,
On 30/01/2020 12:54, Michael Tremer wrote:
Hi,
On 29 Jan 2020, at 20:40, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Hi,
On 28/01/2020 17:14, Michael Tremer wrote:
Hi,
Apologies for my late reply again. I marked this email in my inbox and then it scrolled down because of loads of new stuff coming in. Please feel free to ping me if I do not respond in time.
That's OK. It took me a little longer than expected to check the patches.
On 24 Jan 2020, at 19:40, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
I'm now nearly ready to submit the updated patches, but before then I'd like to check that you're happy with some of my design decisions.
Cool.
_IPTables_Chains_
When setting up the firewall (in iptables_init()), four new chains are created, IPBLACKLISTIN, IPBLACKLISTOUT, IPBLACKLISTREDIN and IPBLACKLISTREDOUT. The first two are inserted into the INPUT, OUTPUT and FORWARD chains, before the IPS chains. The main script is called to restore the previously saved blacklists, creating the rules in the latter two chains.
For readability you can simply call them BLACKLIST*. The IP is a given.
Good point. I'll make the change.
In iptables_red_up(), rules are created to forward traffic to/from the red interface from IPBLACKLISTIN/IPBLACKLISTOUT to IPBLACKLISTREDIN/IPBLACKLISTREDOUT. I did consider putting the blacklist rules into the IPBLACKLISTIN/IPBLACKLISTOUT, but this couldn't be done until red comes up and would require much more processing at this point - not a problem if your red interface comes up and stays up, but if you've got one that disconnects on no traffic you probably don't want the extra overhead. The way I've done it also has the side effects of 1) the main script doesn't have to worry about what the red interface is, and 2) if you're using multiple blacklists, the check for the red interface is only done once.
You would only want to check the interface once and then jump into the chain. Can we not realise it that way?
I'm not sure what you're suggesting. I insert BLACKLISTIN into INPUT & FORWARD and BLACKLISTOUT into OUTPUT & FORWARD. These receive all non-ICMP traffic. On red up traffic from the red interface is sent from BLACKLISTIN to BLACKLISTREDIN and traffic to the red interface is sent from BLACKLISTOUT to BLACKLISTREDOUT. The rules to check against the IPSets are put into the *RED* chain, checking against the source or destination addresses as appropriate.
This sounds very complicated to me.
Why can we not check this in BLACKLISTIN/OUT?
Unfortunately at that point in the setup the red interface in /var/ipfire/red/iface is not always set, so it's not possible to check the interface when creating the rules to jump to BLACKLISTIN/OUT.
It would be possible to delay creating the rules to check against the IPSets until red comes up and then check the interface in the individual blacklist rules, but that duplicates the interface check. The current arrangement is the best I've come up with.
_Log_WUI_Pages_
The two Log WUI pages are edited from existing pages. I've resisted the temptation to reformat them, which means that they can be diff'ed with the corresponding pages.
Loads of the UI code is messy and deliberately not touched any more. I guess this is fine.
Yes, it's not pretty, but it is pragmatic.
_Update_
The update is fairly simple - it calculates whether the check period is passed, tries a download specifying the If-Modified-Since and parses the returned file. It then compares the new list with the existing list, adding new addresses/networks and deleting those that have been removed. There's a check to handle a list changing from a list of addresses to a list of networks, in which case it changes the type of the IPSet - this should be a vary rare event, but could happen for a composite list.
Why do you need the diff? Has that been in the first patchset?
It was, yes.
Originally I did it for performance. It took over a minute to process the ALIENVAULT list (over 80 000 entries); just calling ipset with changes reduced this to a few seconds. I've since modified the code to pipe commands to a process running 'ipset restore', which has drastically speed up the processing, but it's still a little bit quicker to just do changes.
The second reason is that it doesn't feel right to flush the old list, thus removing its protection while the new one is loaded, especially if the new list is the same as the old list (not all sources implement IP-Modified-Since). The actual period of reduced protection may be short, but I prefer to avoid it.
Okay. I was hoping that the “ipset restore” update would flush and add everything again in one large atomic operation.
It would be possible to send a 'flush' command to 'ipset restore', but the blacklists still have to be read and then the entries turned into the correct 'add' commands. There's also some additional processing to handle the hash type changing (between hash:ip and hash:net) and also for the size growing too large for the size of the IPSet. By the time this is all done, I don't think that there's any advantage to be gained in doing a flush followed by a bulk load of everything.
Just reloading the whole ipset should be easier.
_Enabling/Disabling_Blacklists_
For changes to the existing firewall rules, the firewall is shut down and restarted from scratch. The blacklist script does this on the fly, since the possibilities are simpler than all the options available for the firewall and also the necessary code for doing updates means that adding this code is fairly simple. This includes turning logging on and off.
What do you mean by that? /etc/init.d/firewall restart?
That is something we cannot do because IPsec and some other services create temporary rules that would be lost.
No. I'm actually trying to say the opposite (badly). The existing IPFire code only implements some firewall changes by shutting down and restarting the firewall, but all the IP Blacklist functions (enable/disable/update lists, enable/disable logging) can be done without this - the necessary rules can be added to/deleted from the BLACKLISTRED* chains without affecting any of the rest of the firewall system.
Ah cool. That is how it needs to be.
-Michael
Tim
P.S. Is there actually a reason why you do not have a shell account on our servers and host your changed on our git server? I am not sure if I ever offered you one, but it would help me a lot to have a look at the code in a git repo :)
You haven't previously offered me one (or I can't remember it anyway). I think it would make sense given the moderately large amount of code.
Tim
_Logging_
The main scripts logs to syslog as ipblacklist, but if you run it from the command line it'll also output to the terminal. There's also some debug code which can be turned on by setting a value in the settings file - it's turned on at the lowest level automatically if you run from the command line.
Yes that is fine.
Best, -Michael
Tim
On 06/01/2020 11:27, Michael Tremer wrote:
Hello all,
On 4 Jan 2020, at 19:02, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Hello,
I've now updated the code - screenshots are attached.
IP_Address_Blacklists.png
This is the settings page. I don't think there's anything unexpected here.
Yes, that looks a lot like the other IPFire pages :)
IP_Address_Blacklist_Logs.png
The log page, replacing the status information that was previously at the top of the settings page.
Looks great!
I've sorted the lists alphabetically, rather than by the number of hits since the worst threat is not necessary the largest number of blocked packets (which could implied by sorting it by number).
Yes, that is indeed a very good idea.
You could add a % sign in the percentage column, so it cannot be confused so easily with the other column.
Firewall_log_blacklist.png
Details page accessed from the log. Almost identical to the other similar pages.
*thumbs up*
Log_Summary.png
This is an extract from the Log Summary page produced by logwatch.
I still need to do some tidying, remove unused language strings etc., and then update and test the patches, so it'll be some time before I'm ready to submit updated patches.
Feel free to send some RFC to the list. It might be slightly noisy, but it is easier to review changes bit after bit.
Great great work!
-Michael
Tim
On 28/12/2019 20:59, Tim FitzGeorge wrote: > Hi, > > I'll code and test what we've agreed on. In the meantime I'm going to > reply on the sub-thread started by Tom Rymes as to what we call the list > categories, so that we can keep that discussion separate from further > functionality etc. discussion. > > Tim > > On 21/12/2019 18:34, Tim FitzGeorge wrote: >> Hi, >> >> I've got my systems checking for downloads at 15 minute intervals and >> using If-Modified-Since. This all seems to be working OK. >> >> Responses to comments below: >> >> On 18/12/2019 12:07, Michael Tremer wrote: >>> Hi, >>> >>>> On 17 Dec 2019, at 18:21, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote: >>>> >>>> Hi, >>>> >>>> I think we've agreed on the following: >>>> >>>> - Checks to be made every 15 minutes, subject to per-list minimum rate. >>>> - Modify download to use If-Modified-Since rather than separate HEAD >>>> request. >>>> - Remove automatic blacklist. >>> >>> I thought someone wants to counter my argument. This is not a dictatorship :) >>> >>>> >>>> other comments are below. >>>> >>>> Tim >>>> >>>> On 16/12/2019 22:20, Michael Tremer wrote: >>>>> Hi, >>>>> >>>>>> On 16 Dec 2019, at 20:06, Tim FitzGeorge lists@tfitzgeorge.me.uk wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> I've attached the current GUI screenshot. >>>>> >>>>> Thanks for that. >>>>> >>>>> I have a couple of suggestions/concerns about it: >>>>> >>>>> a) I am not sure what the value is about the top box that is called “Status”. It is basically a summary of the iptables output, but will it help the user? A blacklist with more or fewer entries is not better or worse, blocking more packets isn’t better than blocking fewer. It is about the quality of the blocks. >>>> >>>> I agree that the size of the lists is not all that useful, but the >>>> information about the number of blocked packets can be an indication of >>>> compromise. It depends on the type of the list, but a list that targets >>>> C&C channels (e.g. the FEODO lists) should not show any input packets >>>> being blocked, unless one of the computers being protected is infected. >>>> Apart from the BOGON lists, none of the lists should show any output >>>> packets being blocked; again it's a potential indication of infection if >>>> otherwise. >>> >>> The Shodan list will definitely show some activity unless you are behind CG NAT. >>> >>> Should we not have this rather in the logging section? >>> >>> And these counters here will be reset when the firewall is being reloaded. Should this data therefore not be collected from the logs so that you can go back in time? That would be very important to identify any start of compromise. >>> >> >> That makes sense. I'll remove the status from this page and create a >> log page for the information. If anyone actually wants the iptables >> information it's available in from Firewall > iptables anyway. >> >> Do you think it makes more sense to show packets blocked in/out or >> packets blocked per source interface? The latter would of (limited) >> help in tracking down potential compromise. >> >>>>> >>>>> b) I would prefer to move the settings box to the top. If you like you can show the size of the blacklists there and when they have been updated, too. >>>> >>>> I put the status at the top since it's usually what I want to see when I >>>> go to the page. Under normal circumstances I don't change the settings, >>>> so I'd rather not have to scroll past them to get to the information I'm >>>> actually interested in. The update times are just useful to show that >>>> the lists are actually being updated. The size is interesting to me, >>>> but as you say it probably won't help the average user, especially if >>>> you don't know which lists block individual addresses and which block >>>> nets (BOGON, DSHIELD and EMERGING_COMPROMISED). >>> >>> See my comment on the order above. Splitting the page might make sense. >>> >>> When you show size, does that mean lines in the blacklists? So it could be networks or single IP addresses? Does it not make sense to count IP addresses (i.e. 256 for a /24 network and so on) and put it in the table? >> It's lines in the blacklist. >> >>> >>> I think I would only be interested in the size for two things: >>> >>> a) Is there any chance for overblocking? i.e. does the list have 2 billion entries which would be half the public IPv4 address space or so. I cannot really come up with a good example here. >>> >>> Or >>> >>> b) What is the performance impact/memory consumption of the list? >> >> It would be interesting to show the type (single address or net), number >> of entries and number of addresses; the first two affect performance, >> the second memory use and the latter the number of blocked addresses, >> but that's really only going to be of interest to a very small number of >> people, who can always do 'ipset list -t' to get the information. So >> I'll remove the size column. >> >>> >>>> I originally had the status and settings in one list, but I decided that >>>> it was getting a bit messy and difficult to follow. I've attached a >>>> screenshot of a quickly hacked page. (Note that the status area only >>>> appears once the blacklists have been enabled.) >>> >>> You are right. This is getting quite tight there. >>> >>> You could write packets in/out into on column like 2k/1023, but see my concerns about this above. >> >> Actually I've come to the conclusion that the number of bytes blocked >> isn't all that important; it's sufficient to show the number of packets. >> >>> >>>>> >>>>> c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out. >>>> >>>> I agree that it's it's an over simplistic summary of the lists. I note >>>> that Tom Rymes has asked to keep it, but I think its only value is if it >>>> suggests to users that things are more complicated than they appear and >>>> therefore prompts them to read the Wiki. >>> >>> I will reply to that separately. >> >> Looking at that reply, I think it makes sense to include a category >> column in both the settings and the log summary page - the former to >> help selection of which lists to use, and the latter as a help when >> deciding whether the information could indicate compromise or not. >> >>> >>>>> >>>>> d) I do not understand what the “update check rate” means. I suppose we should show the update interval of the lists in the table and just refresh them according to that. >>>> >>>> It's the minimum check period for updates. The system checks for >>>> updates at either the rate specific to the list or this rate, whichever >>>> is slower. This is a hangover from the early days, copied from the IPS >>>> updates. I suppose it could have a value for some people who may wish >>>> to lower the check rate, but I'm not sure that it has sufficient value >>>> to be worth keeping. >>> >>> No, I would not let the user decide when and how often to update the lists. >> >> OK. I'll remove it. >> >>> >>> Generally there is this argument internally how often some databases should be updated. People with slow internet connections or volume based data plans will find this rather expensive to update those lists too often. >>> >>> If we ever want to go ahead with that in IPFire 2, there should be a global switch for a “low data mode” which then delays updating these things. >> >> That makes sense. Maybe also the ability to set when downloads take >> place - but as you say that's for the future. >> >>> >>>>> >>>>> e) Do we want to keep the automatic blacklists now? I do not think it will actually improve the security of IPFire. >>>>> >>>> >>>> I'll remove them. I think they do have a use, in that they can detect >>>> someone doing a port scan and block them before they find an open port, >>>> but as you say there is also the possibility of a DOS. >>> >>> How would this block a DOS? >> >> Sorry, I didn't make myself clear - there's the possibility that someone >> could cause a DOS with the automatic list by faking the source IP >> address and sending a few packets on a random port. >> >>> >>> -Michael >> >> Tim >>> >>>>>> >>>>>> I'll update the code based on our discussions, and submit an updated set >>>>>> of patches - I imagine there will have to be at least one more iteration. >>>>> >>>>> Let’s wait until we have come to decisions :) >>>>> >>>>> -Michael >>>>> >>>>>> >>>>>> Tim >>>>>> >>>>>> (No additional comments below) >>>>>> >>>>>> On 13/12/2019 23:11, Michael Tremer wrote: >>>>>>> Hi, >>>>>>> >>>>>>> Again my apologies for my late reply. Busy busy weeks. >>>>>>> >>>>>>>> On 8 Dec 2019, at 20:50, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote: >>>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> It's my turn to apologise for being slow to respond - I've had a busy >>>>>>>> week, but I should have plenty of time over the next couple of weeks. >>>>>>> >>>>>>> No worries. Turns out we all do :) >>>>>>> >>>>>>>> I've made most of the comments inline, however I think Michael had a >>>>>>>> question (which I can't find now) about what happens if someone enables >>>>>>>> all the lists. One thing which would perhaps make this less likely is >>>>>>>> that the WUI tags the available lists with whether they're safe or not, >>>>>>>> with a footnote that safe means that the list only blocks malicious >>>>>>>> traffic. This won't guarantee that a user won't still try to enable all >>>>>>>> the lists, but it should make them realise that they should think first. >>>>>>> >>>>>>> We had a couple of features going slightly wrong or being “misunderstood” by some users. People still seem to panic when they see “local recursor”. To this day I do not know why. >>>>>>> >>>>>>> We cannot make everything idiot-proof. And when some user if of that category, they probably should shutdown their IPFire box, educate themselves and then come back again. So I do not want to limit people, but make things as easy as possible. >>>>>>> >>>>>>> If someone enables all the lists, good luck with passing packets :) >>>>>>> >>>>>>>> I have considered replacing this tag with a risk high/medium/low and >>>>>>>> maybe adding a category (invalid/application/scanner/C&C or something >>>>>>>> like that), but that may provide too much information and dissuade them >>>>>>>> from actually following the links to checkout what the list actually does. >>>>>>> >>>>>>> Can we have a screenshot of the GUI right now? I didn’t run the code, yet. >>>>>>> >>>>>>> We should document the lists like we do it with the rulesets of the IPS. People might ignore this, but that is on them. >>>>>>> >>>>>>>> >>>>>>>> Tim >>>>>>>> >>>>>>>> >>>>>>>> On 05/12/2019 22:25, Michael Tremer wrote: >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>>> On 4 Dec 2019, at 17:05, Peter Müller peter.mueller@ipfire.org wrote: >>>>>>>>>> >>>>>>>>>> Hello Tim, hello Michael, >>>>>>>>>> >>>>>>>>>> please see my responses inline... >>>>>>>>>>>>> >>>>>>>>>>>>> We could periodically update the blacklists on our main mirror (and >>>>>>>>>>>>> wait for the network to sync it), make sure it is signed and write >>>>>>>>>>>>> a small downloader that fetches, validates and installs them. >>>>>>>>>>>>> >>>>>>>>>>>>> @All: Thoughts on this? >>>>>>>>>>>> >>>>>>>>>>>> I think there are a number of points here. >>>>>>>>>>>> >>>>>>>>>>>> Firstly, from the point of a third party using IPFire, is this really >>>>>>>>>>>> solving the privacy disclosure problem? There's no way round disclosing >>>>>>>>>>>> your public IP address to someone you're downloading from; all this does >>>>>>>>>>>> is change who that information is being disclosed to. For the user >>>>>>>>>>>> there's no way of knowing whether the source is more or less protective >>>>>>>>>>>> of the user's privacy than the blacklist provider. Indeed it won't be >>>>>>>>>>>> possible to know who the lists are being downloaded from until the >>>>>>>>>>>> download starts. >>>>>>>>>>> >>>>>>>>>>> There is a way: Tor. But that is a totally different story. >>>>>>>>>> Well, I see a third option on this: Use the mirror infrastructure we already >>>>>>>>>> have. Every IPFire installation discloses its public IP address to one of these >>>>>>>>>> servers sooner or later, so we do not disclose additional data if the blacklists >>>>>>>>>> were fetched from these. >>>>>>>>>> >>>>>>>>>> Needless to say, Tor (hidden services) would be better, but that is a different >>>>>>>>>> story indeed. :-) >>>>>>>>>>> >>>>>>>>>>> The point is rather that a forget list can be sent instead of the “real” one. >>>>>>>>>> I did not get this. Forget? Forged?? ??? >>>>>>>>> >>>>>>>>> Yes, I meant to write forged, but auto-correct didn’t let me. >>>>>>>>> >>>>>>>>>>>> Secondly, latency; some of the lists are updated every 5 minutes. While >>>>>>>>>>>> I've limited the maximum check rate to hourly, will the updates >>>>>>>>>>>> propagate quickly enough. For reference on my main system the 24 >>>>>>>>>>>> updates on the CIARMY list made 143 498 changes (additions or >>>>>>>>>>>> deletions). I've seem it do over 200 000. >>>>>>>>>> Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not >>>>>>>>>> document a recommended update interval anywhere, so we can only guess. >>>>>>>>>> >>>>>>>>>> Personally, more static lists seem to be preferable for packet filtering. Highly >>>>>>>>>> dynamic ones such as CIArmy should be done via DNSBL queries or something similar >>>>>>>>>> - do we really want to have that list here? >>>>>>>>> >>>>>>>>> It is not really an option to implement a DNSBL into a packet filter, but I get your point. >>>>>>>> >>>>>>>> One of the 'selling points' for an IP address blacklist is that it can >>>>>>>> respond quickly to new threats - or rather new attackers. While a new >>>>>>>> IDS/IPS rule needs time to analyse the threat, generate a rule and check >>>>>>>> it, it's easy to add an address to a list. So, I think the CIArmy list >>>>>>>> is potentially useful for protecting home systems etc. with budget >>>>>>>> hardware, but I would be very careful about using it for a protecting a >>>>>>>> general access website. >>>>>>> >>>>>>> If they are very volatile, we should honour that and update them often, too. >>>>>>> >>>>>>> It probably is more about false-positives being removed very quickly instead of adding threats very quickly. The average IPFire user is probably not under threats like these to need to react very quickly. >>>>>>> >>>>>>> So I do not see much value in adding those lists and then updating once a day. Does it have to be every 5 min? No. I would suggest 15 which should be good enough for everyone. >>>>>>> >>>>>>> Other lists should of course not be updated every 15 minutes when not needed. >>>>>>> >>>>>>> Running every 15 minutes would allow us to retry downloading lists that are on an hourly schedule if the download failed. >>>>>>> >>>>>>>>> >>>>>>>>>>> How did you come up with the hour? Will it be retried more often if the download was not successful? >>>>>>>>>> One hour is the most common interval indeed, but adding some random time might >>>>>>>>>> be useful in order to reduce load on the servers providing a blacklist. >>>>>>>>> >>>>>>>>> Yes, definitely. Otherwise we will shoot down our mirrors. >>>>>>>> >>>>>>>> When I implemented that section of code, specifying the minimum check >>>>>>>> period in hours seemed to provide a convenient way of allowing a check >>>>>>>> period covering a wide range, with an hour as the fastest and a week as >>>>>>>> the slowest. I didn't looked at the CIArmy list until much later. Most >>>>>>>> of the lists don't change nearly as much, but the CIArmy list is >>>>>>>> described as one that deliberately responds quickly. >>>>>>>> >>>>>>>> From my production system, for yesterday: >>>>>>>> >>>>>>>> The following block lists were updated: >>>>>>>> BLOCKLIST_DE: 24 Time(s) - 9341 change(s) >>>>>>>> BOGON_FULL: 1 Time(s) - 10 change(s) >>>>>>>> CIARMY: 24 Time(s) - 159134 change(s) >>>>>>>> DSHIELD: 7 Time(s) - 18 change(s) >>>>>>>> EMERGING_FWRULE: 1 Time(s) - 50 change(s) >>>>>>>> FEODO_AGGRESIVE: 24 Time(s) - 13 change(s) >>>>>>>> SHODAN: 1 Time(s) - 0 change(s) >>>>>>>> SPAMHAUS_DROP: 1 Time(s) - 0 change(s) >>>>>>>> TOR_EXIT: 24 Time(s) - 162 change(s) >>>>>>> >>>>>>> Very interesting statistics. >>>>>>> >>>>>>>> >>>>>>>> and my test system: >>>>>>>> >>>>>>>> The following block lists were updated: >>>>>>>> ALIENVAULT: 19 Time(s) - 5331 change(s) >>>>>>>> EMERGING_COMPROMISED: 1 Time(s) - 26 change(s) >>>>>>>> TALOS_MALICIOUS: 1 Time(s) - 36 change(s) >>>>>>>> >>>>>>>> That covers most of the lists. From the WUI, since 1 Dec: >>>>>>>> >>>>>>>> Blacklist Entries pkts bytes Last updated >>>>>>>> in in >>>>>>>> AUTOBLACKLIST 0 731 51144 Sun Dec 8 18:40:02 2019 >>>>>>>> BLOCKLIST_DE 28020 857 46735 Sun Dec 8 18:40:02 2019 >>>>>>>> BOGON_FULL 214 5255 189K Sun Dec 8 16:50:04 2019 >>>>>>>> CIARMY 15000 19774 976K Sun Dec 8 18:04:01 2019 >>>>>>>> DSHIELD 20 7992 321K Sun Dec 8 16:45:13 2019 >>>>>>>> EMERGING_FWRULE 1647 197 8383 Fri Dec 6 05:29:07 2019 >>>>>>>> FEODO_AGGRESIVE 7169 0 0 Sun Dec 8 18:50:07 2019 >>>>>>>> SHODAN 32 34 1530 Sun Dec 8 17:54:09 2019 >>>>>>>> SPAMHAUS_DROP 823 0 0 Fri Dec 6 18:22:35 2019 >>>>>>>> SPAMHAUS_EDROP 111 82 16433 Thu Dec 5 17:27:09 2019 >>>>>>>> TOR_EXIT 1055 0 0 Sun Dec 8 18:31:02 2019 >>>>>>> >>>>>>> This as well. >>>>>>> >>>>>>> Those are more packets than I would have expected. >>>>>>> >>>>>>>> >>>>>>>> (I've left out the pkts/bytes out fields which were all 0) >>>>>>>> >>>>>>>> Note that where possible I do a HEAD request first and then only >>>>>>>> download the list if the modification time has changed since the last >>>>>>>> check. For dynamically generated lists this isn't possible. >>>>>>> >>>>>>> You won’t need a HEAD request for it. You can include it in the GET request. >>>>>>> >>>>>>> Have a look at location downloader where I use that. The server will respond with 304 and not send any payload. >>>>>>> >>>>>>> https://git.ipfire.org/?p=location/libloc.git;a=blob;f=src/python/location-d... >>>>>>> >>>>>>>> >>>>>>>> If the download isn't successful it just gives up and waits for the next >>>>>>>> attempt (apart from the usual retries in the library). I probably >>>>>>>> should to change that so that it only applies the per list minimum >>>>>>>> update period in this case (specified in the sources file) rather than >>>>>>>> the user specified value as well. >>>>>>> >>>>>>> I think it is not the worst if an update fails. It might just happen every once in a while. >>>>>>> >>>>>>> So I would suggest to just re-run the script more often and when the mtime of the file is older than the threshold, a download is attempted. You can use that timestamp for the GET request. >>>>>>> >>>>>>>> I already use a time offset on the downloads - when it's started from >>>>>>>> boot, backup restore or WUI enable, it checks to see if it's installed >>>>>>>> in the fcrontab, and if not adds itself at a randomly generated offset >>>>>>>> in the hour. >>>>>>> >>>>>>> That should potentially go to red.up, or if we can settle on 15 minutes, I would consider that often enough to quickly update all lists after a reboot. >>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>>>>> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is >>>>>>>>>>>> the largest at a few MB), there are going to be a lot of them. How will >>>>>>>>>>>> this affect the willingness of people to mirror IPFire? >>>>>>>>>> I do not consider this being a problem as we do not generate that much traffic >>>>>>>>>> to them. Of course, that depends on the update interval again. >>>>>>>>> >>>>>>>>> That depends on your point of view. >>>>>>>>> >>>>>>>>> I do not have a problem with this at all in my data center, but there are plenty of people with a volume-based LTE plan or simply a 128 kBit/s connection. It will take a longer time to download the lists for them. We need to mind that. >>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Talking about the preference of packet filter and IPS, I prefer to >>>>>>>>>>>>> use the latter as well as it gains more insight in what kind of malicious >>>>>>>>>>>>> traffic tried to pass a firewall machine. On systems with low resources, >>>>>>>>>>>>> this might be problematic and removing load from the IPS can be preferred >>>>>>>>>>>>> (make this configurable?!), on others, people might want to have both >>>>>>>>>>>>> results. >>>>>>>>>>>>> >>>>>>>>>>>> You're only going to get one result for a packet whichever way round the >>>>>>>>>>>> IP blacklist and IPS are since whichever comes first will drop the >>>>>>>>>>>> packet before it reaches the second (well it would be possible to put >>>>>>>>>>>> the IP blacklist first and get it to log and mark packets which are then >>>>>>>>>>>> dropped after the IPS, but I think that's getting a little complicated. >>>>>>>>>>>> In addition I've seen the messages about the trouble marking was >>>>>>>>>>>> causing in the QoS). >>>>>>>>>>>> >>>>>>>>>>>> I think it's a 50/50 choice as to which is more valuable first; it's >>>>>>>>>>>> probably going to differ from packet to packet. For me the possibility >>>>>>>>>>>> of reducing the IPS load means I prefer putting the IP blacklist first >>>>>>>>>>>> >>>>>>>>>>>> It should be fairly easy to add the choice of where to put the IP >>>>>>>>>>>> blacklist. I think it'll have to be in the main firewall script, so >>>>>>>>>>>> it'll require a firewall restart, but it's not something that'll be >>>>>>>>>>>> changed often. >>>>>>>>>>> >>>>>>>>>>> I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier. >>>>>>>>>> Agreed. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads. >>>>>>>>>>> >>>>>>>>>>> We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily. >>>>>>>>>> So dropping packets from blacklisted IP addresses/networks before IPS is it, then. >>>>>>>>>>> >>>>>>>>>>> I do not even think it makes sense to swap the order in the outgoing direction. >>>>>>>>>> Me too. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this. >>>>>>>>>> Based on my personal experience, firewall hits observed on a single machine exposed >>>>>>>>>> to the internet are interesting, but the overall situation across multiple machines >>>>>>>>>> is even more interesting. Very quickly, you'll end on something like a centralised >>>>>>>>>> logging server and custom statistical analysis here... >>>>>>>>> >>>>>>>>> Probably a project for IPFire 4.0 :) >>>>>>>>> >>>>>>>> Or use one of the existing services, like the DSHIELD client >>>>>>>> https://dshield.org/howto.html (subject to privacy concerns again). >>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default. >>>>>>>>>>>>> I would love to see the bogon ruleset here, too (think about 8chan successor >>>>>>>>>>>>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause >>>>>>>>>>>>> interference if RED does not have a public IP address assigned. >>>>>>>>>>>> >>>>>>>>>>>> I can add a field to the options file that controls whether a list is >>>>>>>>>>>> enabled by default. >>>>>>>>>> Thank you. :-) >>>>>>>>>>> >>>>>>>>>>> To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat. >>>>>>>>>> This is my only objection against this patchset. Now, what can we do about it? >>>>>>>>>> One possibility is to apply the patchset now and implement a custom download >>>>>>>>>> source thing later on, or do that before releasing Core Update 139 (or which version >>>>>>>>>> the patchset will be to) after we agreed on something. >>>>>>>>> >>>>>>>>> I do not see this being merged for 139. But that is not important. We need to get it right first and then release it. >>>>>>>>> >>>>>>>>> As far as I know, nobody has tested this, yet. >>>>>>>> >>>>>>>> There are a number of people who have been running an earlier version >>>>>>>> which I shared on GitHub. There were a few early issues, but it seems >>>>>>>> to be OK now. >>>>>>>> >>>>>>>> https://forum.ipfire.org/viewtopic.php?f=27&t=21845 >>>>>>>> >>>>>>>> This version wasn't integrated into IPFire, so (for example) it inserted >>>>>>>> itself into the INPUT IPTables chain rather than having it's chains >>>>>>>> created as part of the firewall start-up script. >>>>>>>> >>>>>>>>> >>>>>>>>> I have huge concerns about the automatic blacklist. @Peter: What is your opinion on this? >>>>>>>>> >>>>>>>> >>>>>>>> While I implemented it, I'm aware of its potential to cause problems, >>>>>>>> which is why it has to be separately enabled. It's not caused me any >>>>>>>> issues at the default settings (blocks at over 10 packets per hour until >>>>>>>> 1 hour has passed without seeing packets from the address), but I've not >>>>>>>> used it on a site with publicly announced services. If I was going to >>>>>>>> use it on a web site I would want to, at the very minimum, drop the >>>>>>>> block period drastically. >>>>>>> >>>>>>> I suppose this is entirely unusable on an IPFire box in a data center that hosts things. Let’s say our rack in Hanover. >>>>>>> >>>>>>> You will have hits from some broken IP stacks and people might just end up on this without doing anything wrong. >>>>>>> >>>>>>> You can denial-of-service easily as well and I suppose without aggregation of data from many many systems, it does not make sense to instantly block addresses. And even then you probably would block an entire subnet. >>>>>>> >>>>>>> So I am not sure how I should feel about it. I do not think that it adds anything because the packets would have been blocked anyways. >>>>>>> >>>>>>>> >>>>>>>> On the other hand, it's good at responding quickly. Usually I see only >>>>>>>> 1-2% of blocks from the automatic list: >>>>>>>> >>>>>>>> Reason Count % First Last >>>>>>>> CIARMY 2416 45 Dec 6 00:00 Dec 6 23:59 >>>>>>>> DSHIELD 1353 25 Dec 6 00:00 Dec 6 23:59 >>>>>>>> INPUT 1294 24 Dec 6 00:00 Dec 6 23:59 >>>>>>>> AUTOBLACKLIST 122 2 Dec 6 00:20 Dec 6 16:28 >>>>>>>> BLOCKLIST_DE 89 2 Dec 6 00:20 Dec 6 23:46 >>>>>>>> >>>>>>>> and sometimes none at all, but one one occasion it blocked over 8000 >>>>>>>> packets. Again I'm aware this is for a home system, which is rather >>>>>>>> different than from a Web server. >>>>>>>> >>>>>>>>>> If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not >>>>>>>>>> seem to be problematic, as they are hosted on 3rd party servers, too). >>>>>>>>> >>>>>>>>> One of them will be, sooner or later. And one is enough I suppose. >>>>>>>> >>>>>>>> DShield (https://dshield.org/api/#threatfeeds) and firehol >>>>>>>> (http://iplists.firehol.org/) seem to host copies of most of the lists >>>>>>>> as well. >>>>>>>>> >>>>>>>>> I do not really want to overthink this - we didn’t do this with clamav for example either. But it probably is a decision to be made by the user what they want to enable and we should not enable anything by default. So no data will be leaked as long as the user does not consent. >>>>>>>>> >>>>>>>>> -Michael >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks, and best regards, >>>>>>>>>> Peter Müller >>>>>>> >>>>>> >>>>>> <Screenshot_2019-12-16_IP_Address_Blacklists.png> >>>>> >>>> >>>> <Screenshot_2019-12-17_IP_Address_Blacklists.png> >>> >> >
<IP_Address_Blacklists.png><IP_Address_Blacklist_Logs.png><Firewall_log_blacklist.png><Log_Summary.png>
Hi,
On 30 Jan 2020, at 20:24, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Hi,
On 30/01/2020 12:54, Michael Tremer wrote:
Hi,
On 29 Jan 2020, at 20:40, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Hi,
On 28/01/2020 17:14, Michael Tremer wrote:
Hi,
Apologies for my late reply again. I marked this email in my inbox and then it scrolled down because of loads of new stuff coming in. Please feel free to ping me if I do not respond in time.
That's OK. It took me a little longer than expected to check the patches.
On 24 Jan 2020, at 19:40, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
I'm now nearly ready to submit the updated patches, but before then I'd like to check that you're happy with some of my design decisions.
Cool.
_IPTables_Chains_
When setting up the firewall (in iptables_init()), four new chains are created, IPBLACKLISTIN, IPBLACKLISTOUT, IPBLACKLISTREDIN and IPBLACKLISTREDOUT. The first two are inserted into the INPUT, OUTPUT and FORWARD chains, before the IPS chains. The main script is called to restore the previously saved blacklists, creating the rules in the latter two chains.
For readability you can simply call them BLACKLIST*. The IP is a given.
Good point. I'll make the change.
In iptables_red_up(), rules are created to forward traffic to/from the red interface from IPBLACKLISTIN/IPBLACKLISTOUT to IPBLACKLISTREDIN/IPBLACKLISTREDOUT. I did consider putting the blacklist rules into the IPBLACKLISTIN/IPBLACKLISTOUT, but this couldn't be done until red comes up and would require much more processing at this point - not a problem if your red interface comes up and stays up, but if you've got one that disconnects on no traffic you probably don't want the extra overhead. The way I've done it also has the side effects of 1) the main script doesn't have to worry about what the red interface is, and 2) if you're using multiple blacklists, the check for the red interface is only done once.
You would only want to check the interface once and then jump into the chain. Can we not realise it that way?
I'm not sure what you're suggesting. I insert BLACKLISTIN into INPUT & FORWARD and BLACKLISTOUT into OUTPUT & FORWARD. These receive all non-ICMP traffic. On red up traffic from the red interface is sent from BLACKLISTIN to BLACKLISTREDIN and traffic to the red interface is sent from BLACKLISTOUT to BLACKLISTREDOUT. The rules to check against the IPSets are put into the *RED* chain, checking against the source or destination addresses as appropriate.
This sounds very complicated to me.
Why can we not check this in BLACKLISTIN/OUT?
Unfortunately at that point in the setup the red interface in /var/ipfire/red/iface is not always set, so it's not possible to check the interface when creating the rules to jump to BLACKLISTIN/OUT.
Yes, I agree with the delayed initialisation of that chain.
But where is the difference between the two chains here?
You would add rules like this:
iptables -A BLACKLISTIN -i $RED -m set --set some-blacklist src -j REJECT
Those can go into BLACKLISTIN. There is no point in jumping into BLACKLISTREDIN first and then having this rule:
iptables -A BLACKLISTREDIN -m set --set some-blacklist src -j REJECT
It would be possible to delay creating the rules to check against the IPSets until red comes up and then check the interface in the individual blacklist rules, but that duplicates the interface check. The current arrangement is the best I've come up with.
The interface check is cheap. It is an integer. I suppose jumping to another chain is worse, but not by much.
_Log_WUI_Pages_
The two Log WUI pages are edited from existing pages. I've resisted the temptation to reformat them, which means that they can be diff'ed with the corresponding pages.
Loads of the UI code is messy and deliberately not touched any more. I guess this is fine.
Yes, it's not pretty, but it is pragmatic.
_Update_
The update is fairly simple - it calculates whether the check period is passed, tries a download specifying the If-Modified-Since and parses the returned file. It then compares the new list with the existing list, adding new addresses/networks and deleting those that have been removed. There's a check to handle a list changing from a list of addresses to a list of networks, in which case it changes the type of the IPSet - this should be a vary rare event, but could happen for a composite list.
Why do you need the diff? Has that been in the first patchset?
It was, yes.
Originally I did it for performance. It took over a minute to process the ALIENVAULT list (over 80 000 entries); just calling ipset with changes reduced this to a few seconds. I've since modified the code to pipe commands to a process running 'ipset restore', which has drastically speed up the processing, but it's still a little bit quicker to just do changes.
The second reason is that it doesn't feel right to flush the old list, thus removing its protection while the new one is loaded, especially if the new list is the same as the old list (not all sources implement IP-Modified-Since). The actual period of reduced protection may be short, but I prefer to avoid it.
Okay. I was hoping that the “ipset restore” update would flush and add everything again in one large atomic operation.
It would be possible to send a 'flush' command to 'ipset restore', but the blacklists still have to be read and then the entries turned into the correct 'add' commands. There's also some additional processing to handle the hash type changing (between hash:ip and hash:net) and also for the size growing too large for the size of the IPSet. By the time this is all done, I don't think that there's any advantage to be gained in doing a flush followed by a bulk load of everything.
Just reloading the whole ipset should be easier.
_Enabling/Disabling_Blacklists_
For changes to the existing firewall rules, the firewall is shut down and restarted from scratch. The blacklist script does this on the fly, since the possibilities are simpler than all the options available for the firewall and also the necessary code for doing updates means that adding this code is fairly simple. This includes turning logging on and off.
What do you mean by that? /etc/init.d/firewall restart?
That is something we cannot do because IPsec and some other services create temporary rules that would be lost.
No. I'm actually trying to say the opposite (badly). The existing IPFire code only implements some firewall changes by shutting down and restarting the firewall, but all the IP Blacklist functions (enable/disable/update lists, enable/disable logging) can be done without this - the necessary rules can be added to/deleted from the BLACKLISTRED* chains without affecting any of the rest of the firewall system.
Ah cool. That is how it needs to be.
-Michael
Tim
P.S. Is there actually a reason why you do not have a shell account on our servers and host your changed on our git server? I am not sure if I ever offered you one, but it would help me a lot to have a look at the code in a git repo :)
You haven't previously offered me one (or I can't remember it anyway). I think it would make sense given the moderately large amount of code.
Okay, I will contact you privately about this. Sorry for not doing this earlier.
Best, -Michael
Tim
_Logging_
The main scripts logs to syslog as ipblacklist, but if you run it from the command line it'll also output to the terminal. There's also some debug code which can be turned on by setting a value in the settings file - it's turned on at the lowest level automatically if you run from the command line.
Yes that is fine.
Best, -Michael
Tim
On 06/01/2020 11:27, Michael Tremer wrote:
Hello all,
> On 4 Jan 2020, at 19:02, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote: > > Hello, > > I've now updated the code - screenshots are attached. > > IP_Address_Blacklists.png > > This is the settings page. I don't think there's anything unexpected here.
Yes, that looks a lot like the other IPFire pages :)
> IP_Address_Blacklist_Logs.png > > The log page, replacing the status information that was previously at > the top of the settings page.
Looks great!
> I've sorted the lists alphabetically, rather than by the number of hits > since the worst threat is not necessary the largest number of blocked > packets (which could implied by sorting it by number).
Yes, that is indeed a very good idea.
You could add a % sign in the percentage column, so it cannot be confused so easily with the other column.
> Firewall_log_blacklist.png > > Details page accessed from the log. Almost identical to the other > similar pages.
*thumbs up*
> Log_Summary.png > > This is an extract from the Log Summary page produced by logwatch. > > I still need to do some tidying, remove unused language strings etc., > and then update and test the patches, so it'll be some time before I'm > ready to submit updated patches.
Feel free to send some RFC to the list. It might be slightly noisy, but it is easier to review changes bit after bit.
Great great work!
-Michael
> > Tim > > On 28/12/2019 20:59, Tim FitzGeorge wrote: >> Hi, >> >> I'll code and test what we've agreed on. In the meantime I'm going to >> reply on the sub-thread started by Tom Rymes as to what we call the list >> categories, so that we can keep that discussion separate from further >> functionality etc. discussion. >> >> Tim >> >> On 21/12/2019 18:34, Tim FitzGeorge wrote: >>> Hi, >>> >>> I've got my systems checking for downloads at 15 minute intervals and >>> using If-Modified-Since. This all seems to be working OK. >>> >>> Responses to comments below: >>> >>> On 18/12/2019 12:07, Michael Tremer wrote: >>>> Hi, >>>> >>>>> On 17 Dec 2019, at 18:21, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote: >>>>> >>>>> Hi, >>>>> >>>>> I think we've agreed on the following: >>>>> >>>>> - Checks to be made every 15 minutes, subject to per-list minimum rate. >>>>> - Modify download to use If-Modified-Since rather than separate HEAD >>>>> request. >>>>> - Remove automatic blacklist. >>>> >>>> I thought someone wants to counter my argument. This is not a dictatorship :) >>>> >>>>> >>>>> other comments are below. >>>>> >>>>> Tim >>>>> >>>>> On 16/12/2019 22:20, Michael Tremer wrote: >>>>>> Hi, >>>>>> >>>>>>> On 16 Dec 2019, at 20:06, Tim FitzGeorge lists@tfitzgeorge.me.uk wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I've attached the current GUI screenshot. >>>>>> >>>>>> Thanks for that. >>>>>> >>>>>> I have a couple of suggestions/concerns about it: >>>>>> >>>>>> a) I am not sure what the value is about the top box that is called “Status”. It is basically a summary of the iptables output, but will it help the user? A blacklist with more or fewer entries is not better or worse, blocking more packets isn’t better than blocking fewer. It is about the quality of the blocks. >>>>> >>>>> I agree that the size of the lists is not all that useful, but the >>>>> information about the number of blocked packets can be an indication of >>>>> compromise. It depends on the type of the list, but a list that targets >>>>> C&C channels (e.g. the FEODO lists) should not show any input packets >>>>> being blocked, unless one of the computers being protected is infected. >>>>> Apart from the BOGON lists, none of the lists should show any output >>>>> packets being blocked; again it's a potential indication of infection if >>>>> otherwise. >>>> >>>> The Shodan list will definitely show some activity unless you are behind CG NAT. >>>> >>>> Should we not have this rather in the logging section? >>>> >>>> And these counters here will be reset when the firewall is being reloaded. Should this data therefore not be collected from the logs so that you can go back in time? That would be very important to identify any start of compromise. >>>> >>> >>> That makes sense. I'll remove the status from this page and create a >>> log page for the information. If anyone actually wants the iptables >>> information it's available in from Firewall > iptables anyway. >>> >>> Do you think it makes more sense to show packets blocked in/out or >>> packets blocked per source interface? The latter would of (limited) >>> help in tracking down potential compromise. >>> >>>>>> >>>>>> b) I would prefer to move the settings box to the top. If you like you can show the size of the blacklists there and when they have been updated, too. >>>>> >>>>> I put the status at the top since it's usually what I want to see when I >>>>> go to the page. Under normal circumstances I don't change the settings, >>>>> so I'd rather not have to scroll past them to get to the information I'm >>>>> actually interested in. The update times are just useful to show that >>>>> the lists are actually being updated. The size is interesting to me, >>>>> but as you say it probably won't help the average user, especially if >>>>> you don't know which lists block individual addresses and which block >>>>> nets (BOGON, DSHIELD and EMERGING_COMPROMISED). >>>> >>>> See my comment on the order above. Splitting the page might make sense. >>>> >>>> When you show size, does that mean lines in the blacklists? So it could be networks or single IP addresses? Does it not make sense to count IP addresses (i.e. 256 for a /24 network and so on) and put it in the table? >>> It's lines in the blacklist. >>> >>>> >>>> I think I would only be interested in the size for two things: >>>> >>>> a) Is there any chance for overblocking? i.e. does the list have 2 billion entries which would be half the public IPv4 address space or so. I cannot really come up with a good example here. >>>> >>>> Or >>>> >>>> b) What is the performance impact/memory consumption of the list? >>> >>> It would be interesting to show the type (single address or net), number >>> of entries and number of addresses; the first two affect performance, >>> the second memory use and the latter the number of blocked addresses, >>> but that's really only going to be of interest to a very small number of >>> people, who can always do 'ipset list -t' to get the information. So >>> I'll remove the size column. >>> >>>> >>>>> I originally had the status and settings in one list, but I decided that >>>>> it was getting a bit messy and difficult to follow. I've attached a >>>>> screenshot of a quickly hacked page. (Note that the status area only >>>>> appears once the blacklists have been enabled.) >>>> >>>> You are right. This is getting quite tight there. >>>> >>>> You could write packets in/out into on column like 2k/1023, but see my concerns about this above. >>> >>> Actually I've come to the conclusion that the number of bytes blocked >>> isn't all that important; it's sufficient to show the number of packets. >>> >>>> >>>>>> >>>>>> c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out. >>>>> >>>>> I agree that it's it's an over simplistic summary of the lists. I note >>>>> that Tom Rymes has asked to keep it, but I think its only value is if it >>>>> suggests to users that things are more complicated than they appear and >>>>> therefore prompts them to read the Wiki. >>>> >>>> I will reply to that separately. >>> >>> Looking at that reply, I think it makes sense to include a category >>> column in both the settings and the log summary page - the former to >>> help selection of which lists to use, and the latter as a help when >>> deciding whether the information could indicate compromise or not. >>> >>>> >>>>>> >>>>>> d) I do not understand what the “update check rate” means. I suppose we should show the update interval of the lists in the table and just refresh them according to that. >>>>> >>>>> It's the minimum check period for updates. The system checks for >>>>> updates at either the rate specific to the list or this rate, whichever >>>>> is slower. This is a hangover from the early days, copied from the IPS >>>>> updates. I suppose it could have a value for some people who may wish >>>>> to lower the check rate, but I'm not sure that it has sufficient value >>>>> to be worth keeping. >>>> >>>> No, I would not let the user decide when and how often to update the lists. >>> >>> OK. I'll remove it. >>> >>>> >>>> Generally there is this argument internally how often some databases should be updated. People with slow internet connections or volume based data plans will find this rather expensive to update those lists too often. >>>> >>>> If we ever want to go ahead with that in IPFire 2, there should be a global switch for a “low data mode” which then delays updating these things. >>> >>> That makes sense. Maybe also the ability to set when downloads take >>> place - but as you say that's for the future. >>> >>>> >>>>>> >>>>>> e) Do we want to keep the automatic blacklists now? I do not think it will actually improve the security of IPFire. >>>>>> >>>>> >>>>> I'll remove them. I think they do have a use, in that they can detect >>>>> someone doing a port scan and block them before they find an open port, >>>>> but as you say there is also the possibility of a DOS. >>>> >>>> How would this block a DOS? >>> >>> Sorry, I didn't make myself clear - there's the possibility that someone >>> could cause a DOS with the automatic list by faking the source IP >>> address and sending a few packets on a random port. >>> >>>> >>>> -Michael >>> >>> Tim >>>> >>>>>>> >>>>>>> I'll update the code based on our discussions, and submit an updated set >>>>>>> of patches - I imagine there will have to be at least one more iteration. >>>>>> >>>>>> Let’s wait until we have come to decisions :) >>>>>> >>>>>> -Michael >>>>>> >>>>>>> >>>>>>> Tim >>>>>>> >>>>>>> (No additional comments below) >>>>>>> >>>>>>> On 13/12/2019 23:11, Michael Tremer wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> Again my apologies for my late reply. Busy busy weeks. >>>>>>>> >>>>>>>>> On 8 Dec 2019, at 20:50, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote: >>>>>>>>> >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> It's my turn to apologise for being slow to respond - I've had a busy >>>>>>>>> week, but I should have plenty of time over the next couple of weeks. >>>>>>>> >>>>>>>> No worries. Turns out we all do :) >>>>>>>> >>>>>>>>> I've made most of the comments inline, however I think Michael had a >>>>>>>>> question (which I can't find now) about what happens if someone enables >>>>>>>>> all the lists. One thing which would perhaps make this less likely is >>>>>>>>> that the WUI tags the available lists with whether they're safe or not, >>>>>>>>> with a footnote that safe means that the list only blocks malicious >>>>>>>>> traffic. This won't guarantee that a user won't still try to enable all >>>>>>>>> the lists, but it should make them realise that they should think first. >>>>>>>> >>>>>>>> We had a couple of features going slightly wrong or being “misunderstood” by some users. People still seem to panic when they see “local recursor”. To this day I do not know why. >>>>>>>> >>>>>>>> We cannot make everything idiot-proof. And when some user if of that category, they probably should shutdown their IPFire box, educate themselves and then come back again. So I do not want to limit people, but make things as easy as possible. >>>>>>>> >>>>>>>> If someone enables all the lists, good luck with passing packets :) >>>>>>>> >>>>>>>>> I have considered replacing this tag with a risk high/medium/low and >>>>>>>>> maybe adding a category (invalid/application/scanner/C&C or something >>>>>>>>> like that), but that may provide too much information and dissuade them >>>>>>>>> from actually following the links to checkout what the list actually does. >>>>>>>> >>>>>>>> Can we have a screenshot of the GUI right now? I didn’t run the code, yet. >>>>>>>> >>>>>>>> We should document the lists like we do it with the rulesets of the IPS. People might ignore this, but that is on them. >>>>>>>> >>>>>>>>> >>>>>>>>> Tim >>>>>>>>> >>>>>>>>> >>>>>>>>> On 05/12/2019 22:25, Michael Tremer wrote: >>>>>>>>>> Hello, >>>>>>>>>> >>>>>>>>>>> On 4 Dec 2019, at 17:05, Peter Müller peter.mueller@ipfire.org wrote: >>>>>>>>>>> >>>>>>>>>>> Hello Tim, hello Michael, >>>>>>>>>>> >>>>>>>>>>> please see my responses inline... >>>>>>>>>>>>>> >>>>>>>>>>>>>> We could periodically update the blacklists on our main mirror (and >>>>>>>>>>>>>> wait for the network to sync it), make sure it is signed and write >>>>>>>>>>>>>> a small downloader that fetches, validates and installs them. >>>>>>>>>>>>>> >>>>>>>>>>>>>> @All: Thoughts on this? >>>>>>>>>>>>> >>>>>>>>>>>>> I think there are a number of points here. >>>>>>>>>>>>> >>>>>>>>>>>>> Firstly, from the point of a third party using IPFire, is this really >>>>>>>>>>>>> solving the privacy disclosure problem? There's no way round disclosing >>>>>>>>>>>>> your public IP address to someone you're downloading from; all this does >>>>>>>>>>>>> is change who that information is being disclosed to. For the user >>>>>>>>>>>>> there's no way of knowing whether the source is more or less protective >>>>>>>>>>>>> of the user's privacy than the blacklist provider. Indeed it won't be >>>>>>>>>>>>> possible to know who the lists are being downloaded from until the >>>>>>>>>>>>> download starts. >>>>>>>>>>>> >>>>>>>>>>>> There is a way: Tor. But that is a totally different story. >>>>>>>>>>> Well, I see a third option on this: Use the mirror infrastructure we already >>>>>>>>>>> have. Every IPFire installation discloses its public IP address to one of these >>>>>>>>>>> servers sooner or later, so we do not disclose additional data if the blacklists >>>>>>>>>>> were fetched from these. >>>>>>>>>>> >>>>>>>>>>> Needless to say, Tor (hidden services) would be better, but that is a different >>>>>>>>>>> story indeed. :-) >>>>>>>>>>>> >>>>>>>>>>>> The point is rather that a forget list can be sent instead of the “real” one. >>>>>>>>>>> I did not get this. Forget? Forged?? ??? >>>>>>>>>> >>>>>>>>>> Yes, I meant to write forged, but auto-correct didn’t let me. >>>>>>>>>> >>>>>>>>>>>>> Secondly, latency; some of the lists are updated every 5 minutes. While >>>>>>>>>>>>> I've limited the maximum check rate to hourly, will the updates >>>>>>>>>>>>> propagate quickly enough. For reference on my main system the 24 >>>>>>>>>>>>> updates on the CIARMY list made 143 498 changes (additions or >>>>>>>>>>>>> deletions). I've seem it do over 200 000. >>>>>>>>>>> Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not >>>>>>>>>>> document a recommended update interval anywhere, so we can only guess. >>>>>>>>>>> >>>>>>>>>>> Personally, more static lists seem to be preferable for packet filtering. Highly >>>>>>>>>>> dynamic ones such as CIArmy should be done via DNSBL queries or something similar >>>>>>>>>>> - do we really want to have that list here? >>>>>>>>>> >>>>>>>>>> It is not really an option to implement a DNSBL into a packet filter, but I get your point. >>>>>>>>> >>>>>>>>> One of the 'selling points' for an IP address blacklist is that it can >>>>>>>>> respond quickly to new threats - or rather new attackers. While a new >>>>>>>>> IDS/IPS rule needs time to analyse the threat, generate a rule and check >>>>>>>>> it, it's easy to add an address to a list. So, I think the CIArmy list >>>>>>>>> is potentially useful for protecting home systems etc. with budget >>>>>>>>> hardware, but I would be very careful about using it for a protecting a >>>>>>>>> general access website. >>>>>>>> >>>>>>>> If they are very volatile, we should honour that and update them often, too. >>>>>>>> >>>>>>>> It probably is more about false-positives being removed very quickly instead of adding threats very quickly. The average IPFire user is probably not under threats like these to need to react very quickly. >>>>>>>> >>>>>>>> So I do not see much value in adding those lists and then updating once a day. Does it have to be every 5 min? No. I would suggest 15 which should be good enough for everyone. >>>>>>>> >>>>>>>> Other lists should of course not be updated every 15 minutes when not needed. >>>>>>>> >>>>>>>> Running every 15 minutes would allow us to retry downloading lists that are on an hourly schedule if the download failed. >>>>>>>> >>>>>>>>>> >>>>>>>>>>>> How did you come up with the hour? Will it be retried more often if the download was not successful? >>>>>>>>>>> One hour is the most common interval indeed, but adding some random time might >>>>>>>>>>> be useful in order to reduce load on the servers providing a blacklist. >>>>>>>>>> >>>>>>>>>> Yes, definitely. Otherwise we will shoot down our mirrors. >>>>>>>>> >>>>>>>>> When I implemented that section of code, specifying the minimum check >>>>>>>>> period in hours seemed to provide a convenient way of allowing a check >>>>>>>>> period covering a wide range, with an hour as the fastest and a week as >>>>>>>>> the slowest. I didn't looked at the CIArmy list until much later. Most >>>>>>>>> of the lists don't change nearly as much, but the CIArmy list is >>>>>>>>> described as one that deliberately responds quickly. >>>>>>>>> >>>>>>>>> From my production system, for yesterday: >>>>>>>>> >>>>>>>>> The following block lists were updated: >>>>>>>>> BLOCKLIST_DE: 24 Time(s) - 9341 change(s) >>>>>>>>> BOGON_FULL: 1 Time(s) - 10 change(s) >>>>>>>>> CIARMY: 24 Time(s) - 159134 change(s) >>>>>>>>> DSHIELD: 7 Time(s) - 18 change(s) >>>>>>>>> EMERGING_FWRULE: 1 Time(s) - 50 change(s) >>>>>>>>> FEODO_AGGRESIVE: 24 Time(s) - 13 change(s) >>>>>>>>> SHODAN: 1 Time(s) - 0 change(s) >>>>>>>>> SPAMHAUS_DROP: 1 Time(s) - 0 change(s) >>>>>>>>> TOR_EXIT: 24 Time(s) - 162 change(s) >>>>>>>> >>>>>>>> Very interesting statistics. >>>>>>>> >>>>>>>>> >>>>>>>>> and my test system: >>>>>>>>> >>>>>>>>> The following block lists were updated: >>>>>>>>> ALIENVAULT: 19 Time(s) - 5331 change(s) >>>>>>>>> EMERGING_COMPROMISED: 1 Time(s) - 26 change(s) >>>>>>>>> TALOS_MALICIOUS: 1 Time(s) - 36 change(s) >>>>>>>>> >>>>>>>>> That covers most of the lists. From the WUI, since 1 Dec: >>>>>>>>> >>>>>>>>> Blacklist Entries pkts bytes Last updated >>>>>>>>> in in >>>>>>>>> AUTOBLACKLIST 0 731 51144 Sun Dec 8 18:40:02 2019 >>>>>>>>> BLOCKLIST_DE 28020 857 46735 Sun Dec 8 18:40:02 2019 >>>>>>>>> BOGON_FULL 214 5255 189K Sun Dec 8 16:50:04 2019 >>>>>>>>> CIARMY 15000 19774 976K Sun Dec 8 18:04:01 2019 >>>>>>>>> DSHIELD 20 7992 321K Sun Dec 8 16:45:13 2019 >>>>>>>>> EMERGING_FWRULE 1647 197 8383 Fri Dec 6 05:29:07 2019 >>>>>>>>> FEODO_AGGRESIVE 7169 0 0 Sun Dec 8 18:50:07 2019 >>>>>>>>> SHODAN 32 34 1530 Sun Dec 8 17:54:09 2019 >>>>>>>>> SPAMHAUS_DROP 823 0 0 Fri Dec 6 18:22:35 2019 >>>>>>>>> SPAMHAUS_EDROP 111 82 16433 Thu Dec 5 17:27:09 2019 >>>>>>>>> TOR_EXIT 1055 0 0 Sun Dec 8 18:31:02 2019 >>>>>>>> >>>>>>>> This as well. >>>>>>>> >>>>>>>> Those are more packets than I would have expected. >>>>>>>> >>>>>>>>> >>>>>>>>> (I've left out the pkts/bytes out fields which were all 0) >>>>>>>>> >>>>>>>>> Note that where possible I do a HEAD request first and then only >>>>>>>>> download the list if the modification time has changed since the last >>>>>>>>> check. For dynamically generated lists this isn't possible. >>>>>>>> >>>>>>>> You won’t need a HEAD request for it. You can include it in the GET request. >>>>>>>> >>>>>>>> Have a look at location downloader where I use that. The server will respond with 304 and not send any payload. >>>>>>>> >>>>>>>> https://git.ipfire.org/?p=location/libloc.git;a=blob;f=src/python/location-d... >>>>>>>> >>>>>>>>> >>>>>>>>> If the download isn't successful it just gives up and waits for the next >>>>>>>>> attempt (apart from the usual retries in the library). I probably >>>>>>>>> should to change that so that it only applies the per list minimum >>>>>>>>> update period in this case (specified in the sources file) rather than >>>>>>>>> the user specified value as well. >>>>>>>> >>>>>>>> I think it is not the worst if an update fails. It might just happen every once in a while. >>>>>>>> >>>>>>>> So I would suggest to just re-run the script more often and when the mtime of the file is older than the threshold, a download is attempted. You can use that timestamp for the GET request. >>>>>>>> >>>>>>>>> I already use a time offset on the downloads - when it's started from >>>>>>>>> boot, backup restore or WUI enable, it checks to see if it's installed >>>>>>>>> in the fcrontab, and if not adds itself at a randomly generated offset >>>>>>>>> in the hour. >>>>>>>> >>>>>>>> That should potentially go to red.up, or if we can settle on 15 minutes, I would consider that often enough to quickly update all lists after a reboot. >>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is >>>>>>>>>>>>> the largest at a few MB), there are going to be a lot of them. How will >>>>>>>>>>>>> this affect the willingness of people to mirror IPFire? >>>>>>>>>>> I do not consider this being a problem as we do not generate that much traffic >>>>>>>>>>> to them. Of course, that depends on the update interval again. >>>>>>>>>> >>>>>>>>>> That depends on your point of view. >>>>>>>>>> >>>>>>>>>> I do not have a problem with this at all in my data center, but there are plenty of people with a volume-based LTE plan or simply a 128 kBit/s connection. It will take a longer time to download the lists for them. We need to mind that. >>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Talking about the preference of packet filter and IPS, I prefer to >>>>>>>>>>>>>> use the latter as well as it gains more insight in what kind of malicious >>>>>>>>>>>>>> traffic tried to pass a firewall machine. On systems with low resources, >>>>>>>>>>>>>> this might be problematic and removing load from the IPS can be preferred >>>>>>>>>>>>>> (make this configurable?!), on others, people might want to have both >>>>>>>>>>>>>> results. >>>>>>>>>>>>>> >>>>>>>>>>>>> You're only going to get one result for a packet whichever way round the >>>>>>>>>>>>> IP blacklist and IPS are since whichever comes first will drop the >>>>>>>>>>>>> packet before it reaches the second (well it would be possible to put >>>>>>>>>>>>> the IP blacklist first and get it to log and mark packets which are then >>>>>>>>>>>>> dropped after the IPS, but I think that's getting a little complicated. >>>>>>>>>>>>> In addition I've seen the messages about the trouble marking was >>>>>>>>>>>>> causing in the QoS). >>>>>>>>>>>>> >>>>>>>>>>>>> I think it's a 50/50 choice as to which is more valuable first; it's >>>>>>>>>>>>> probably going to differ from packet to packet. For me the possibility >>>>>>>>>>>>> of reducing the IPS load means I prefer putting the IP blacklist first >>>>>>>>>>>>> >>>>>>>>>>>>> It should be fairly easy to add the choice of where to put the IP >>>>>>>>>>>>> blacklist. I think it'll have to be in the main firewall script, so >>>>>>>>>>>>> it'll require a firewall restart, but it's not something that'll be >>>>>>>>>>>>> changed often. >>>>>>>>>>>> >>>>>>>>>>>> I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier. >>>>>>>>>>> Agreed. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads. >>>>>>>>>>>> >>>>>>>>>>>> We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily. >>>>>>>>>>> So dropping packets from blacklisted IP addresses/networks before IPS is it, then. >>>>>>>>>>>> >>>>>>>>>>>> I do not even think it makes sense to swap the order in the outgoing direction. >>>>>>>>>>> Me too. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this. >>>>>>>>>>> Based on my personal experience, firewall hits observed on a single machine exposed >>>>>>>>>>> to the internet are interesting, but the overall situation across multiple machines >>>>>>>>>>> is even more interesting. Very quickly, you'll end on something like a centralised >>>>>>>>>>> logging server and custom statistical analysis here... >>>>>>>>>> >>>>>>>>>> Probably a project for IPFire 4.0 :) >>>>>>>>>> >>>>>>>>> Or use one of the existing services, like the DSHIELD client >>>>>>>>> https://dshield.org/howto.html (subject to privacy concerns again). >>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default. >>>>>>>>>>>>>> I would love to see the bogon ruleset here, too (think about 8chan successor >>>>>>>>>>>>>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause >>>>>>>>>>>>>> interference if RED does not have a public IP address assigned. >>>>>>>>>>>>> >>>>>>>>>>>>> I can add a field to the options file that controls whether a list is >>>>>>>>>>>>> enabled by default. >>>>>>>>>>> Thank you. :-) >>>>>>>>>>>> >>>>>>>>>>>> To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat. >>>>>>>>>>> This is my only objection against this patchset. Now, what can we do about it? >>>>>>>>>>> One possibility is to apply the patchset now and implement a custom download >>>>>>>>>>> source thing later on, or do that before releasing Core Update 139 (or which version >>>>>>>>>>> the patchset will be to) after we agreed on something. >>>>>>>>>> >>>>>>>>>> I do not see this being merged for 139. But that is not important. We need to get it right first and then release it. >>>>>>>>>> >>>>>>>>>> As far as I know, nobody has tested this, yet. >>>>>>>>> >>>>>>>>> There are a number of people who have been running an earlier version >>>>>>>>> which I shared on GitHub. There were a few early issues, but it seems >>>>>>>>> to be OK now. >>>>>>>>> >>>>>>>>> https://forum.ipfire.org/viewtopic.php?f=27&t=21845 >>>>>>>>> >>>>>>>>> This version wasn't integrated into IPFire, so (for example) it inserted >>>>>>>>> itself into the INPUT IPTables chain rather than having it's chains >>>>>>>>> created as part of the firewall start-up script. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> I have huge concerns about the automatic blacklist. @Peter: What is your opinion on this? >>>>>>>>>> >>>>>>>>> >>>>>>>>> While I implemented it, I'm aware of its potential to cause problems, >>>>>>>>> which is why it has to be separately enabled. It's not caused me any >>>>>>>>> issues at the default settings (blocks at over 10 packets per hour until >>>>>>>>> 1 hour has passed without seeing packets from the address), but I've not >>>>>>>>> used it on a site with publicly announced services. If I was going to >>>>>>>>> use it on a web site I would want to, at the very minimum, drop the >>>>>>>>> block period drastically. >>>>>>>> >>>>>>>> I suppose this is entirely unusable on an IPFire box in a data center that hosts things. Let’s say our rack in Hanover. >>>>>>>> >>>>>>>> You will have hits from some broken IP stacks and people might just end up on this without doing anything wrong. >>>>>>>> >>>>>>>> You can denial-of-service easily as well and I suppose without aggregation of data from many many systems, it does not make sense to instantly block addresses. And even then you probably would block an entire subnet. >>>>>>>> >>>>>>>> So I am not sure how I should feel about it. I do not think that it adds anything because the packets would have been blocked anyways. >>>>>>>> >>>>>>>>> >>>>>>>>> On the other hand, it's good at responding quickly. Usually I see only >>>>>>>>> 1-2% of blocks from the automatic list: >>>>>>>>> >>>>>>>>> Reason Count % First Last >>>>>>>>> CIARMY 2416 45 Dec 6 00:00 Dec 6 23:59 >>>>>>>>> DSHIELD 1353 25 Dec 6 00:00 Dec 6 23:59 >>>>>>>>> INPUT 1294 24 Dec 6 00:00 Dec 6 23:59 >>>>>>>>> AUTOBLACKLIST 122 2 Dec 6 00:20 Dec 6 16:28 >>>>>>>>> BLOCKLIST_DE 89 2 Dec 6 00:20 Dec 6 23:46 >>>>>>>>> >>>>>>>>> and sometimes none at all, but one one occasion it blocked over 8000 >>>>>>>>> packets. Again I'm aware this is for a home system, which is rather >>>>>>>>> different than from a Web server. >>>>>>>>> >>>>>>>>>>> If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not >>>>>>>>>>> seem to be problematic, as they are hosted on 3rd party servers, too). >>>>>>>>>> >>>>>>>>>> One of them will be, sooner or later. And one is enough I suppose. >>>>>>>>> >>>>>>>>> DShield (https://dshield.org/api/#threatfeeds) and firehol >>>>>>>>> (http://iplists.firehol.org/) seem to host copies of most of the lists >>>>>>>>> as well. >>>>>>>>>> >>>>>>>>>> I do not really want to overthink this - we didn’t do this with clamav for example either. But it probably is a decision to be made by the user what they want to enable and we should not enable anything by default. So no data will be leaked as long as the user does not consent. >>>>>>>>>> >>>>>>>>>> -Michael >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks, and best regards, >>>>>>>>>>> Peter Müller >>>>>>>> >>>>>>> >>>>>>> <Screenshot_2019-12-16_IP_Address_Blacklists.png> >>>>>> >>>>> >>>>> <Screenshot_2019-12-17_IP_Address_Blacklists.png> >>>> >>> >> > > <IP_Address_Blacklists.png><IP_Address_Blacklist_Logs.png><Firewall_log_blacklist.png><Log_Summary.png>
-
Michael Tremer -
Tim FitzGeorge -
Tom Rymes