From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: Logging Locationblock packets Date: Sat, 23 Nov 2024 10:26:07 +0100 Message-ID: <074EA3F9-8A01-4C67-9EF1-0FC556AC3BB8@ipfire.org> In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============8328940383832986242==" List-Id: --===============8328940383832986242== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello, > On 21 Nov 2024, at 00:08, Rob Brewer wrote: >=20 > Hi Michael and Adolf, >=20 > I'll combine my reply to both of you. >=20 > On Wed, 20 Nov 2024 15:21:28 +0000, Michael Tremer wrote: >=20 >> Hello, >>=20 >>> On 20 Nov 2024, at 13:36, Rob Brewer >>> wrote: >>>=20 >>> On Tue, 19 Nov 2024 17:10:24 +0000, Michael Tremer wrote: >>>=20 >>> Hi Michael, thank for your response, I was beginning to think there was >>> little interest in this topic. >>=20 >> I was traveling and so have a little bit of a back log as usual :) >>=20 >> I can=E2=80=99t speak for the others, though. >>=20 >=20 > Not a problem, I should have said that I previously raised this a topic on = > the forum and Abolf suggested that I raised it on the mailing list as=20 > well. Yes, I really prefer this here, because the forum is very overwhelming someti= mes. There is too many threads, too many people=E2=80=A6 Mailing lists are gr= eat for development. >>=20 >>> See below... >>>=20 >>>> Hello Rob, >>>>=20 >>>> Good to hear from you again. >>>>=20 >>>>> On 9 Nov 2024, at 14:00, Rob Brewer >>>>> wrote: >>>>>=20 >>>>> As a contributor to the SANS ISC, I email my IPTABLES logs hourly >>>>> to the Dshield database, however the default IPFire IPTABLES logs are >>>>> incomplete since packets dropped by Locationblock are not logged to >>>>> syslog. >>>>=20 >>>> This is correct. >>>>=20 >>>>> As a workaround I edit the locationblock function of rules.pl to add >>>>> a IPTABLES rule to log the dropped packets. This edit does not of >>>>> course survive any Core-Updates that upgrade rules.pl and the edit >>>>> needs to be re-applied. >>>>>=20 >>>>> In addition, unless the Locationblock packets are logged to syslog >>>>> the data displayed in the Firewall logs sections are incomplete as >>>>> the packets dropped by Location Block are not displayed. >>>>>=20 >>>>> I understand that the original intension of the Location Block >>>>> feature was to reduce the amount of log messages on installations >>>>> running on extremely cheap flash storage. I suspect that these >>>>> installations are now almost zero especially since IPFire no longer >>>>> supports 32 bit versions. >>>>>=20 >>>>> The obvious solution to this to make Location Block logging >>>>> selectable as a firewall option and my quick look at making a patches >>>>> for this would seem to be fairly simple task to rules.pl and and >>>>> optionsfw.cgi. >>>>>=20 >>>>> I would be happy to provide the necessary patches should this be >>>>> acceptable. >>>>=20 >>>> I think a couple of things have changed since we introduced this >>>> feature and we have also learned a couple of lessons here: >>>>=20 >>>> * Yes it is true that this was introduced to keep the logs smaller, >>>> but not necessarily to avoid using storage, mainly to keep them >>>> readable by a human. I was always a bit doubtful about this and >>>> believe that apart from this, there is no more value in a general >>>> location blocking feature. The firewall will drop those packets >>>> anyway. >>>>=20 >>>>=20 >>> IPFire does a very good job in protecting the internal network from >>> unwanted probes from the WAN, and I agree that for most users, location >>> block does not enhance this protection. >>>=20 >>> As I run a mail server, I use locationblock as part of my filtering to >>> prevent bad actors from accessing my mail server and I think would be >>> useful for anyone who are running servers on their networks. >>>=20 >>> I don't like blocking traffic, but I have witnessed over the last few >>> years a substantial rise in abuse from certain countries from where I >>> would never expect to receive emails and locationblock works very well >>> for me in filtering out those unwanted connections. >>>=20 >>> I currently have 21 countries blocked including XD Hostile networks. >>> All of these blocked countries have been seen to support abusive >>> networks. >>>=20 >>> It would be possible to generate individual rules for these networks >>> which in some cases I do with Banish, but it is sometimes easier just >>> to drop the whole country. >>=20 >> Easy yes. I am just not sure what we are actually getting out of this. >>=20 >> I don=E2=80=99t like to become political on this list and so I won=E2=80= =99t, but I know >> that some people hold the following beliefs: >>=20 >> * There are =E2=80=9Cbad=E2=80=9D countries out there. In the western worl= d those are >> Russia, China, and more that fall into this kind of =E2=80=9Ccategory=E2= =80=9D. Nothing >> is going to stop them to open up an account on AWS and suddenly they >> have access to resources anywhere in the world and so actually, if those >> players were up to anything =E2=80=9Cbad=E2=80=9D, they can come out of an= y IP space >> they want. There is also Tor which allows this. >>=20 >=20 > I have been submitting my iptables logs to dshield for over 20 years and=20 > in the early days I would be sending a lot less than 1000 a day (this was=20 > with IPCop of course). >=20 > Over the last few years the number of firewall iptables logs have =20 > escalated until it had reached dshield's daily limit of 100,000 /day and=20 > I started looking at ways to limit the logging of some of the 'bad actors' = > firewall probes. What do you use to send them the logs? The Python script or one of those alte= rnatives? >> * Looking at actual data, the bullet proof ISPs that annoy us are not >> that far away. They are hosted in Europe and very prominently so in the >> Netherlands - a =E2=80=9Cgood=E2=80=9D country. >>=20 >=20 > I use Banish for this as it can block by ASN and can target bullet proof=20 > ISPs not only in NL but also in the USA and elsewhere. Using the 'location = > function' the blocked ASN's are updated when the database is updated. >=20 >> This is just an example to explain that we all have our idea of the >> world in our head, but the Internet doesn=E2=80=99t know these borders. >> Geopolitics is complicated and I don=E2=80=99t think that this is a tool t= hat >> helps to fight this kind of problem. I hope this is clear what I am >> trying to say. >>=20 >=20 > Yes my approach is to use multiple tools and Locationblock is just one of=20 > the tools I am using. Maybe my approach is over the top though. Probably not massively, but maybe slightly. Wouldn=E2=80=99t it be nicer to have everything in one place instead of havin= g multiple tools that work together in concert? >> I personally trust in Spamhaus which is feeding data into our =E2=80=9CHos= tile >> Networks=E2=80=9D blocker and they are looking at the actual traffic that = they >> see from those subnets that they are blocking. They have a clear set of >> rules and they don=E2=80=99t look just at the country and say =E2=80=9Cah = yes, the bad >> guys again=E2=80=9D. >>=20 >=20 > For emails I also drop packets if listed in a number of DNS blackhole=20 > lists using Auto-Dnsbl which I adapted for IPFire. See: >=20 > https://people.ipfire.org/~helix/auto-dnsbl/README > for info. >=20 > I have been using this for about a year and can confirm that it is very=20 > effective in removing spammers. >=20 >> I have stated this recently in the RPZ discussion that I feel that we >> are lacking good sources for this kind of data. For blocking anything. >> Loads of people believe =E2=80=9Cmore is more=E2=80=9D and to be fair=E2= =80=A6 once you get to >> blocking 90% of the internet you probably have blocked 90% of the noise. >> However this won=E2=80=99t do anything for targeted attacks. I personally = would >> rather have more quality here. >>=20 >=20 > I only use a few blocklists with ipblocklist and rely mainly on my locally = > derived data from failed smtp connections or from data from dshield's=20 > daily report. >=20 >>>> * People seem to misunderstand this feature: The block as many >>>> countries as they can click on and think that this makes things =E2=80= =9Cmore >>>> secure=E2=80=9D, >>>> which of course it doesn=E2=80=99t. >>>>=20 >>>>=20 >>> I think this is education and I suspect that very few of these people >>> are running servers on their system and need location block. >>=20 >> I am personally not here to educate people. I am happy to help out and >> give my opinion, but I suppose considering that people think as I have >> outlined above, who am I to educate them? >>=20 >=20 > Sorry I didn't express my reply well, I agree that IPFire users should be=20 > responsible for their own actions. Naturally. But we should also not give them sharp knives when we don=E2=80=99= t expect them to use them properly. >> It is just very regretful to me when people are ab-using a feature just >> to =E2=80=9Cfeel more safe=E2=80=9D. >>=20 >>>> * What I personally like as a feature is to apply location filters to >>>> individual rules. That allows to apply rate limits in a few places and >>>> easily select some larger groups of potentially less interesting >>>> traffic. >>>=20 >>> I agree and is similar to what I am doing for ports 25 and 587 although >>> with location block. >>>=20 >>>=20 >>>> * Back when this was introduced there was also a lot less use of >>>> botnets as we know them now. Pretty much all recent DoS attacks I >>>> remember have been coming from all over the world because systems >>>> where compromised wherever they were. This renders the filter >>>> absolutely useless. >>>=20 >>> For this, I am currently testing a dynamic block list on my system >>> which adds an entry to an ipset when an ip address exceeds a threshold >>> in the previous 24 hours. Currently there are about 7000 entries in >>> this set which have been detected over the previous month. Not for >>> everyone because it needs a Whitelist to prevent some 'push' >>> technologies such as Facebook and to protect from bulk inputs such as >>> mailing lists. I also blacklist a large number of ip addresses which >>> are associated with port scanners. >>>=20 >>> This would work well as a botnet filter but it needs an understanding >>> of the blocking process, so it would not be suitable for everyone. >>=20 >> We have talked about this on the last team call and came to the >> conclusion that the whole set of blocking features is a total mess. >> There are several places where things can be blocked, they all heavily >> overlap. Not to be going on about Spamhaus again, but the DROP list is >> in the =E2=80=9CHostile Networks=E2=80=9D feature, there is a special coun= try code for >> it, pretty much every IPS ruleset contains it, the IP blocklist feature >> supports it. Now there are people throwing this all into DNS as well. I >> believe this is all getting a little bit out of hand. >>=20 >=20 > I don't use the IPS mainly due to bad experiences when trying to use Snort = > with IPCop. I find it difficult to set up and select the appropriate=20 > Rulsets for my requirements. I suspect the IPS would be very effective if=20 > I could get it set up correctly. This has massively improved in recent times. And it is FAST! >> I personally would prefer *one* place and we really make this work well. >>=20 >> Some people find the IPS has too much overhead - I am beginning to >> disagree. >>=20 >> Some people add all sorts of blacklists that are clearly for >> experimental/research use and suddenly complain about things not working >> anymore. It is hard to figure out what is actually blocking traffic when >> there are about five places to look for. The result often is to disable >> the IPS, then something else, and after a while all of this is turned >> off and the firewall is configured to a bare minimum. We don=E2=80=99t want >> people to run it like that. >>=20 >>>> * Infrastructure is becoming more centralised. Some people attempt to >>>> block the US and pretty much nothing works any more. Well=E2=80=A6 >>>>=20 >>>>=20 >>> :) >>>=20 >>>> Therefore I am happy to rethink this feature - personally I would even >>>> remove it entirely, but I don=E2=80=99t like to handle the stress about = this >>>> now. >>>=20 >>> No I think you should keep it. Locationblock is very useful, but needs >>> to be used in a considered way. I could live without it but its >>> availability makes my life easier. Also being updated on a weekly basis >>> makes keeping track of abusers easier. >>=20 >> We issue a new location database once a day, but the IPFire systems >> update it only once a week. We deemed that as sufficient as there are >> not that many changes from one version to another one. >>=20 >>>> I think this should log now. I don=E2=80=99t think there is any legitima= te >>>> reason to disable firewall logging at all. This is the paper trail >>>> that you want when debugging anything or doing forensics on an attack. >>>> Certainly bad flash is not a valid reason for me. >>>>=20 >>>>=20 >>> I have started generating some patches to make 'locationblock logging' >>> as a selectable feature and could post them here in the next day or so >>> if of any use. >>=20 >> What would this basically do? Log it as a simple rule for all, or do you >> include the selected country? I think the latter would make more sense. >>=20 >=20 > OK - I have modified location-block.cgi to add an additional checkbox at=20 > the top of the page titled "Log dropped packets", very similar to=20 > ipblocklist.cgi. This adds a entry to the /var/ipfire/firewall/ > locationblock settings file ON or OFF. >=20 > In /usr/lib/firewall/rules.pl I add an additional Iptables LOCATIONBLOCK=20 > log rule with the prefix 'LOCBLOCK-$location '. This is selectable from=20 > the Log dropped packets entry in the locationblock settings file. >=20 > This generates kernel messages such as: >=20 > LOCBLOCK-CN IN=3Dppp0 OUT=3Dppp0 MAC=3D SRC=3D114.139.42.71 DST=3D81.187.17= 5.103=20 > LEN=3D60 TOS=3D0x00 PREC=3D0x00 TTL=3D47 ID=3D34664 DF PROTO=3DTCP SPT=3D46= 837 DPT=3D23=20 > WINDOW=3D29040 RES=3D0x00 SYN URGP=3D0 >=20 > As you can see the prefix contains the country code of the dropped=20 > packets. Cool. So I think coding-wise this is pretty straight forward. The more crucial question for me is below=E2=80=A6 >> What is the rationale to turn this off? Why don=E2=80=99t we turn this on = for >> everybody all of the time? >>=20 >=20 > I would be happy with that I was suggesting that we use selectable logging = > to be compatible with earlier IPFire versions. What do we have to be compatible with? Just IPFire=E2=80=99s behaviour from l= ast year? I would be happy to break that if we have a good reason, because the world so= metimes moves on. What we have released 10 years ago probably doesn=E2=80=99t= have good defaults any more. Sometimes we have to revisit those and sometime= s we even have to be pushy with some changes if we want some long-standing us= ers to adopt them. IPFire users usually have no reason to do a reinstall and = we don=E2=80=99t want to leave the people behind who have been running IPFire= perfectly well for the last 10 years or even longer. So at least I would lik= e to discuss what we push a little harder. >>> In fact I think that IPFire's logging environment, which was inherited >>> from IPCop needs a re-visit. >>=20 >> Yes. But this is probably not a thing I would be looking at in IPFire 2. >>=20 >> I am also happy to think a little bit more outside of the box and see >> what we would fundamentally change in the future, but not implement in >> IPFire 2. >>=20 >>> I was pleased to see that a while back there was a discussion of >>> replacing sylogd with rsyslogd which I think would be a very useful >>> upgrade and enable kernel messages, like on most current linux systems, >>> to be logged to alternative log files instead of everything ending up >>> in /var/log/ messages. >>=20 >> The currentl syslog server could do that. The problem is more that we >> don=E2=80=99t have good tools to view the logs on the web UI. So there is = simply >> one place where most things go into and we will then filter it out when >> users use the web UI. This is tremendously slow. >>=20 >=20 > I did look at this a while back and believe that with syslog all kernel=20 > messages must go to the same log file and cannot be filtered off as can be = > done with rsyslog. I use this facility in rsyslog.d on my server to filter = > off SYN Flood logs to a separate log file with ":msg,contains". I don't=20 > believe that syslog has this option. Different messages could be sent to different =E2=80=9Cfacilities=E2=80=9D an= d then we can filter them into individual files. I would not want to split firewall hits into different files though. I just w= ouldn=E2=80=99t mind if firewall hits were no longer in /var/log/messages. >>> The replacement with rsyslgd could also inhibit the "last message >>> repeated x times" in syslog and which are ignored by IPFire's Log menu >>> and causes erroneous results in the 'Count' values. >>=20 >> This might be a setting that we can turn off in the current >> implementation. >>=20 >=20 > I understand that this is hard coded into syslogd and can't be changed. In = > rsyslog it defaults to "off". >=20 > https://www.rsyslog.com/doc/configuration/action/ > rsconf1_repeatedmsgreduction.html What impact does this have for you? I never noticed any problem by summarisin= g the logs. >> We could also think about moving all firewall hits into >> /var/log/firewall.log or something. That should free up enough in >> /var/log/messages to make debugging easier. >>=20 >=20 > That would be a sensible option, currently I filter out firewall hits from = > the messages.log with a perl regex.=20 >=20 >>>> I would be happy to hear some more opinions on this from the list. >>>>=20 >>>> -Michael >>>=20 >>> I hope these comments are useful and would be pleased to help where I >>> can. >>=20 >> Of course :) >>=20 >> -Michael >>=20 >>=20 >>>=20 >>> Rob -Michael --===============8328940383832986242==--