From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: [PATCH 0/5] ipblacklist: IP Address Blacklists Date: Mon, 02 Dec 2019 11:17:23 +0000 Message-ID: In-Reply-To: <25bec64a-a8db-0182-680d-8ebebf7a9184@tfitzgeorge.me.uk> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============6230047775427988728==" List-Id: --===============6230047775427988728== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi, > On 29 Nov 2019, at 23:25, Tim FitzGeorge wrote: >=20 > Hello Peter and Michael, >=20 > My responses to both of your emails are below: >=20 > On 28/11/2019 21:39, Peter M=C3=BCller wrote: >> Hello Tim, hello Michael, hello *, >>=20 >> first, I'd like to apologise for the late response. I used to >> be more active on this mailing list than I am today, but hopefully >> this will be a temporary situation only. >>=20 >> Michael is right, I did not have time to mention my concerns about >> fetching the blacklists from external sources. In my opinion, this >> causes privacy issues as we disclose the public IP addresses of >> all IPFire installations using a blacklist to its provider/vendor. >>=20 >> In my humble opinion, we can rely on a diverse and robust mirror >> infrastructure for our Core Updates and packages, so why not use them >> for other data such as blacklists or upcoming libloc as well? >>=20 >> We could periodically update the blacklists on our main mirror (and >> wait for the network to sync it), make sure it is signed and write >> a small downloader that fetches, validates and installs them. >>=20 >> @All: Thoughts on this? >=20 > I think there are a number of points here. >=20 > Firstly, from the point of a third party using IPFire, is this really > solving the privacy disclosure problem? There's no way round disclosing > your public IP address to someone you're downloading from; all this does > is change who that information is being disclosed to. For the user > there's no way of knowing whether the source is more or less protective > of the user's privacy than the blacklist provider. Indeed it won't be > possible to know who the lists are being downloaded from until the > download starts. There is a way: Tor. But that is a totally different story. The point is rather that a forget list can be sent instead of the =E2=80=9Cre= al=E2=80=9D one. > Secondly, latency; some of the lists are updated every 5 minutes. While > I've limited the maximum check rate to hourly, will the updates > propagate quickly enough. For reference on my main system the 24 > updates on the CIARMY list made 143 498 changes (additions or > deletions). I've seem it do over 200 000. How did you come up with the hour? Will it be retried more often if the downl= oad was not successful? > Third, bandwidth; while the downloads are fairly small (ALIENVAULT is > the largest at a few MB), there are going to be a lot of them. How will > this affect the willingness of people to mirror IPFire? >=20 >>=20 >> Talking about the preference of packet filter and IPS, I prefer to >> use the latter as well as it gains more insight in what kind of malicious >> traffic tried to pass a firewall machine. On systems with low resources, >> this might be problematic and removing load from the IPS can be preferred >> (make this configurable?!), on others, people might want to have both >> results. >>=20 > You're only going to get one result for a packet whichever way round the > IP blacklist and IPS are since whichever comes first will drop the > packet before it reaches the second (well it would be possible to put > the IP blacklist first and get it to log and mark packets which are then > dropped after the IPS, but I think that's getting a little complicated. > In addition I've seen the messages about the trouble marking was > causing in the QoS). >=20 > I think it's a 50/50 choice as to which is more valuable first; it's > probably going to differ from packet to packet. For me the possibility > of reducing the IPS load means I prefer putting the IP blacklist first >=20 > It should be fairly easy to add the choice of where to put the IP > blacklist. I think it'll have to be in the main firewall script, so > it'll require a firewall restart, but it's not something that'll be > changed often. I do not think that the user should choose this. If we cannot easily make a d= ecision, how can our users do this? Not saying they are stupid here, we are j= ust giving them something so that they do not have to put the thought and res= earch into things themselves and make their jobs easier. See my comments in my previous email about this. I think performance matters. And if the IPS comes first, the most likely case= would be that we are seeing a SYN packet that is being scanned and after tha= t being dropped by the blacklist. It was pointless to even scan the empty pac= ket (TCP fast open aside). This is only different for other packets with payl= oads. We would protect the IPS from a SYN flooding attack here at least and from sc= anning more packets unnecessarily. I do not even think it makes sense to swap the order in the outgoing directio= n. >> Regarding the GeoIP block feature, this has always been my strongest >> point against it: Neither were dropped packets logged (hope this is >> still true, as I have not tried for quite a while), nor is more fine >> grained filtering possible - e.g. by limiting this to certain services -, >> nor are IPS results available for packets dropped. >>=20 >> We should _always_ log dropped packets (no matter which feature drops them= ), >> and leave it to the user to decide whether traffic from/to blacklisted >> IP addresses should traverse the IPS or not. >=20 > I agree. I added the possibility to turn off logging because of user > requests for the earlier version I made available on GitHub, but logging > is enabled by default. What IPFire is lacking is a statistical analysis for those logs. Collecting m= ore and more data isn=E2=80=99t helpful from my point of view. Only if you ar= e looking at a very specific thing. >>=20 >> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by defaul= t. >> I would love to see the bogon ruleset here, too (think about 8chan success= or >> hosted at unallocated RIPE space in Saint Petersburg), but that will likel= y cause >> interference if RED does not have a public IP address assigned. >=20 > I can add a field to the options file that controls whether a list is > enabled by default. To stress the point from above again: We would then share all public IP addre= sses of all IPFire systems in the world with Spamhaus and who is hosting thei= r infrastructure. That can be considered a threat. -Michael >=20 >>=20 >> Regarding the code quality, I agree with Michael. Thanks again for providi= ng >> this. :-) As soon as we agree on answers to the questions raised meanwhile= , I >> will add my "Reviewed-by"-tag. >>=20 >> Thanks, and best regards, >> Peter M=C3=BCller >>=20 >>=20 >>> Hello Tim, >>>=20 >>> Thank you for sending in this patchset. >>>=20 >>> I think as well that this functionality would be a great addition to the = IPS that we have right now and we have talked about this on numerous occasion= s - including implementation details. >>>=20 >>> Now, you have done the whole thing. Well done. >>>=20 >>> Before we dive into the code, can I ask a couple of high-level questions? >>>=20 >>> Peter is always bringing up that downloading blacklists isn=E2=80=99t a g= ood idea. It has actually become one of the biggest obstacles in our conversa= tions and I am surprised he didn=E2=80=99t bring it up again :) >>>=20 >>> The automatic blacklist feature. What is the objective here? We saw value= in having traffic even from malicious sources passed to the IPS so that it w= ill examine it and log it. The idea was to have a better picture of the threa= ts instead of just silencing them. Not sure what is best in the end. >=20 > It's intended to provide a quick local response to portscans based on > packets which are dropped due to the red input policy. >=20 >>>=20 >>> I am unsure how users will deal with this and turn on =E2=80=9Call the li= sts=E2=80=9D(TM) and suddenly things do not work any more. How are they meant= to figure out a good threshold? Should we not make that decision for them in= stead? >=20 > Indeed. I've seen the messages from people who have turned on all the > IPS rules. In addition I know that at least one person who used the > version on GitHub turned on all the lists, even the ones where one is a > subset of another list (I added code to prevent this). I don't believe > that it caused major problems since, unlike the IPS, it doesn't have > rules that can block what for some people is normal activity. It's > still a bad idea. I think it makes sense to deliver it with a default > configuration, which might hint that it's not meant to have all the > lists enabled. >=20 > I'll obviously cover this in the Wiki page(s), but that doesn't mean > people will read it... >>>=20 >>> About the implementation: Your code is very very clean as always. There a= re a couple of things that I would like to see changed around how those iptab= les rules are being inserted into the existing chains. You are adding somethi= ng into the POLICYIN chain with surgical precision which might break when the= chain is being modified. This is potentially all minor stuff and can be fixe= d in minutes. >=20 > This is one of the areas I was unsure about. I've looked at the > existing code, but I obviously don't have the detailed knowledge of how > the different parts of the system interact when setting up the firewall. >=20 > The rule added to POLICYIN is the one that adds addresses to the > autoblacklist; it's not all that critical where it goes in POLICYIN, but > the ideal place is just before the LOG. Normally the code in > firewall-policy will put it in the right place, but I think I know how > to place it better in ipblacklist. >=20 > (Aside - I write onboard software for spacecraft and we always try to > bear in mind that it could be use that has to modify it in a hurry after > working on something else for ten years. It does tend to encourage > being tidy). >=20 >>>=20 >>> Well done! >>>=20 >>> -Michael >=20 > If you let me know what you think, I can make some modifications and > then send a V2. >=20 > Tim >=20 >>>=20 >>>> On 25 Nov 2019, at 20:13, Tim FitzGeorge wrot= e: >>>>=20 >>>> Implements downloading of IP address blacklists and implementing >>>> them as IPSets. A separate IPSet is used for each blacklist; this >>>> simplifies handling of overlaps between different lists. Traffic >>>> to or from the red0/ppp0 interface is checked against the IPSets. >>>> The check is placed before the IPS check as the IPSet check is >>>> much lighter on CPU use which means that overall CPU use is >>>> reduced. >>>>=20 >>>> The available lists are defined in a separate file. A WUI page >>>> allows the desired lists to be enabled and the interval between >>>> checks for updates to be defined. A minimum update check interval >>>> is defined for each blacklist in the definition file. >>>>=20 >>>> Optionally, an automatically updating blacklist can be enabled. >>>> This adds addresses to an IPSet if the rate of packets dropped by >>>> the default red0/ppp0 input policy exceeds a user defined threshold. >>>> The addresses are kept in the IPSet until a user defined period >>>> without packets from the blocked address has passed. >>>>=20 >>>> Tim FitzGeorge (5): >>>> ipblacklist: Main script >>>> ipblacklist: WUI and language file >>>> ipblacklist: Ancillary files >>>> ipblacklist: Modifications to system >>>> ipblacklist: Build infrastructure >>>>=20 >>>> config/backup/backup.pl | 1 + >>>> config/backup/include | 2 + >>>> config/firewall/firewall-policy | 5 + >>>> config/ipblacklist/sources | 151 +++ >>>> config/logwatch/ipblacklist | 103 ++ >>>> config/logwatch/ipblacklist.conf | 34 + >>>> config/menu/50-firewall.menu | 5 + >>>> config/rootfiles/common/aarch64/stage2 | 1 + >>>> config/rootfiles/common/configroot | 2 + >>>> config/rootfiles/common/ipblacklist-sources | 1 + >>>> config/rootfiles/common/logwatch | 2 + >>>> config/rootfiles/common/misc-progs | 2 + >>>> config/rootfiles/common/stage2 | 1 + >>>> config/rootfiles/common/web-user-interface | 1 + >>>> config/rootfiles/common/x86_64/stage2 | 1 + >>>> html/cgi-bin/ipblacklist.cgi | 725 +++++++++++++ >>>> html/cgi-bin/logs.cgi/log.dat | 2 + >>>> langs/en/cgi-bin/en.pl | 31 + >>>> lfs/configroot | 4 +- >>>> lfs/ipblacklist-sources | 53 + >>>> lfs/logwatch | 2 + >>>> make.sh | 11 +- >>>> src/initscripts/system/firewall | 20 + >>>> src/misc-progs/Makefile | 2 +- >>>> src/misc-progs/getipsetstat.c | 28 + >>>> src/misc-progs/ipblacklistctrl.c | 52 + >>>> src/scripts/ipblacklist | 1558 +++++++++++++++++++++= ++++++ >>>> 27 files changed, 2792 insertions(+), 8 deletions(-) >>>> create mode 100644 config/ipblacklist/sources >>>> create mode 100644 config/logwatch/ipblacklist >>>> create mode 100644 config/logwatch/ipblacklist.conf >>>> create mode 100644 config/rootfiles/common/ipblacklist-sources >>>> create mode 100644 html/cgi-bin/ipblacklist.cgi >>>> create mode 100644 lfs/ipblacklist-sources >>>> create mode 100644 src/misc-progs/getipsetstat.c >>>> create mode 100644 src/misc-progs/ipblacklistctrl.c >>>> create mode 100755 src/scripts/ipblacklist >>>>=20 >>>> --=20 >>>> 2.16.4 --===============6230047775427988728==--