From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: Logging Locationblock packets Date: Mon, 25 Nov 2024 10:49:24 +0100 Message-ID: <1929202F-18F8-4D42-88F9-AEA310189B83@ipfire.org> In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============9003613368277310438==" List-Id: --===============9003613368277310438== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello > On 23 Nov 2024, at 15:47, Adolf Belka wrote: >=20 > Hi Rob, >=20 > On 23/11/2024 15:36, Rob Brewer wrote: >> On Sat, 23 Nov 2024 10:26:07 +0100, Michael Tremer wrote: >>=20 >>> Hello, >>>=20 >>>> On 21 Nov 2024, at 00:08, Rob Brewer >> wrote: >>>> Hi Michael and Adolf, >>>>=20 >>>> I'll combine my reply to both of you. >>>>=20 >>>> On Wed, 20 Nov 2024 15:21:28 +0000, Michael Tremer wrote: >>>>=20 >>>>> Hello, >>>>>=20 >>>>>> On 20 Nov 2024, at 13:36, Rob Brewer >>>>>> wrote: >>>>>>=20 >>>>>> On Tue, 19 Nov 2024 17:10:24 +0000, Michael Tremer wrote: >>>>>>=20 >>>>>> Hi Michael, thank for your response, I was beginning to think there >> was >>>>>> little interest in this topic. >>>>> I was traveling and so have a little bit of a back log as usual :) >>>>>=20 >>>>> I can=E2=80=99t speak for the others, though. >>>>>=20 >>>> Not a problem, I should have said that I previously raised this a >> topic >> on >>>> the forum and Adolf suggested that I raised it on the mailing list as >>>> well. >>> Yes, I really prefer this here, because the forum is very overwhelming >> sometimes. There is too many threads, too many people=E2=80=A6 Mailing lis= ts are >> great for development. >>=20 >> That's fine, one query though you don't seem to snip older parts of the >> post, is this your preference? >>=20 >>>>>> See below... >>>>>>=20 >>>>>>> Hello Rob, >>>>>>>=20 >>>>>>> Good to hear from you again. >>>>>>>=20 >>>>>>>> On 9 Nov 2024, at 14:00, Rob Brewer >>>>>>>> wrote: >>>>>>>>=20 >>>>>>>> As a contributor to the SANS ISC, I email my IPTABLES logs hourly >>>>>>>> to the Dshield database, however the default IPFire IPTABLES logs >> are >>>>>>>> incomplete since packets dropped by Locationblock are not logged to >>>>>>>> syslog. >>>>>>> This is correct. >>>>>>>=20 >>>>>>>> As a workaround I edit the locationblock function of rules.pl to >> add >>>>>>>> a IPTABLES rule to log the dropped packets. This edit does not of >>>>>>>> course survive any Core-Updates that upgrade rules.pl and the edit >>>>>>>> needs to be re-applied. >>>>>>>>=20 >>>>>>>> In addition, unless the Locationblock packets are logged to syslog >>>>>>>> the data displayed in the Firewall logs sections are incomplete as >>>>>>>> the packets dropped by Location Block are not displayed. >>>>>>>>=20 >>>>>>>> I understand that the original intension of the Location Block >>>>>>>> feature was to reduce the amount of log messages on installations >>>>>>>> running on extremely cheap flash storage. I suspect that these >>>>>>>> installations are now almost zero especially since IPFire no longer >>>>>>>> supports 32 bit versions. >>>>>>>>=20 >>>>>>>> The obvious solution to this to make Location Block logging >>>>>>>> selectable as a firewall option and my quick look at making a >> patches >>>>>>>> for this would seem to be fairly simple task to rules.pl and and >>>>>>>> optionsfw.cgi. >>>>>>>>=20 >>>>>>>> I would be happy to provide the necessary patches should this be >>>>>>>> acceptable. >>>>>>> I think a couple of things have changed since we introduced this >>>>>>> feature and we have also learned a couple of lessons here: >>>>>>>=20 >>>>>>> * Yes it is true that this was introduced to keep the logs smaller, >>>>>>> but not necessarily to avoid using storage, mainly to keep them >>>>>>> readable by a human. I was always a bit doubtful about this and >>>>>>> believe that apart from this, there is no more value in a general >>>>>>> location blocking feature. The firewall will drop those packets >>>>>>> anyway. >>>>>>>=20 >>>>>>>=20 >>>>>> IPFire does a very good job in protecting the internal network from >>>>>> unwanted probes from the WAN, and I agree that for most users, >> location >>>>>> block does not enhance this protection. >>>>>>=20 >>>>>> As I run a mail server, I use locationblock as part of my filtering >> to >>>>>> prevent bad actors from accessing my mail server and I think would be >>>>>> useful for anyone who are running servers on their networks. >>>>>>=20 >>>>>> I don't like blocking traffic, but I have witnessed over the last few >>>>>> years a substantial rise in abuse from certain countries from where I >>>>>> would never expect to receive emails and locationblock works very >> well >>>>>> for me in filtering out those unwanted connections. >>>>>>=20 >>>>>> I currently have 21 countries blocked including XD Hostile networks. >>>>>> All of these blocked countries have been seen to support abusive >>>>>> networks. >>>>>>=20 >>>>>> It would be possible to generate individual rules for these networks >>>>>> which in some cases I do with Banish, but it is sometimes easier just >>>>>> to drop the whole country. >>>>> Easy yes. I am just not sure what we are actually getting out of this. >>>>>=20 >>>>> I don=E2=80=99t like to become political on this list and so I won=E2= =80=99t, but I >> know >>>>> that some people hold the following beliefs: >>>>>=20 >>>>> * There are =E2=80=9Cbad=E2=80=9D countries out there. In the western w= orld those are >>>>> Russia, China, and more that fall into this kind of =E2=80=9Ccategory= =E2=80=9D. >> Nothing >>>>> is going to stop them to open up an account on AWS and suddenly they >>>>> have access to resources anywhere in the world and so actually, if >> those >>>>> players were up to anything =E2=80=9Cbad=E2=80=9D, they can come out of= any IP space >>>>> they want. There is also Tor which allows this. >>>>>=20 >>>> I have been submitting my iptables logs to dshield for over 20 years >> and >>>> in the early days I would be sending a lot less than 1000 a day (this >> was >>>> with IPCop of course). >>>>=20 >>>> Over the last few years the number of firewall iptables logs have >>>> escalated until it had reached dshield's daily limit of 100,000 /day >> and >>>> I started looking at ways to limit the logging of some of the 'bad >> actors' >>>> firewall probes. >>> What do you use to send them the logs? The Python script or one of >> those >> alternatives? >>=20 >> I use a modified Perl script (iptables.pl) which is about 20 years old was >> originally written for Smoothwall and Ipcop1. This was part of the Logsend >> add-on and I modified for IPCop2 about 15 years ago. >>=20 >> Now modified for IPFire and using DMA email instead of the original >> sendEmail. I've uploaded my version to people.ipfire.org/~helix/dshield. >>=20 >> I had intended to write an install/uninstall script but is in the todo >> list. Do you think it may be of interest to anyone if I do? >>=20 >>>>> * Looking at actual data, the bullet proof ISPs that annoy us are not >>>>> that far away. They are hosted in Europe and very prominently so in >> the >>>>> Netherlands - a =E2=80=9Cgood=E2=80=9D country. >>>>>=20 >>>> I use Banish for this as it can block by ASN and can target bullet >> proof >>>> ISPs not only in NL but also in the USA and elsewhere. Using the >> 'location >>>> function' the blocked ASN's are updated when the database is updated. >>>>=20 >>>>> This is just an example to explain that we all have our idea of the >>>>> world in our head, but the Internet doesn=E2=80=99t know these borders. >>>>> Geopolitics is complicated and I don=E2=80=99t think that this is a too= l that >>>>> helps to fight this kind of problem. I hope this is clear what I am >>>>> trying to say. >>>>>=20 >>>> Yes my approach is to use multiple tools and Locationblock is just one >> of >>>> the tools I am using. Maybe my approach is over the top though. >>> Probably not massively, but maybe slightly. >>>=20 >>> Wouldn=E2=80=99t it be nicer to have everything in one place instead of h= aving >> multiple tools that work together in concert? >> I think that would be a lot of work, would you consider that for IPFire2 >> or 3? >>=20 >>>>> I personally trust in Spamhaus which is feeding data into our =E2=80=9C= Hostile >>>>> Networks=E2=80=9D blocker and they are looking at the actual traffic th= at they >>>>> see from those subnets that they are blocking. They have a clear set >> of >>>>> rules and they don=E2=80=99t look just at the country and say =E2=80=9C= ah yes, the bad >>>>> guys again=E2=80=9D. >>>>>=20 >>>> For emails I also drop packets if listed in a number of DNS blackhole >>>> lists using Auto-Dnsbl which I adapted for IPFire. See: >>>>=20 >>>> https://people.ipfire.org/~helix/auto-dnsbl/README >>>> for info. >>>>=20 >>>> I have been using this for about a year and can confirm that it is >> very >>>> effective in removing spammers. >>>>=20 >>>>> I have stated this recently in the RPZ discussion that I feel that we >>>>> are lacking good sources for this kind of data. For blocking anything. >>>>> Loads of people believe =E2=80=9Cmore is more=E2=80=9D and to be fair= =E2=80=A6 once you get to >>>>> blocking 90% of the internet you probably have blocked 90% of the >> noise. >>>>> However this won=E2=80=99t do anything for targeted attacks. I personal= ly >> would >>>>> rather have more quality here. >>>>>=20 >>>> I only use a few blocklists with ipblocklist and rely mainly on my >> locally >>>> derived data from failed smtp connections or from data from dshield's >>>> daily report. >>>>=20 >>>>>>> * People seem to misunderstand this feature: The block as many >>>>>>> countries as they can click on and think that this makes things >> =E2=80=9Cmore >>>>>>> secure=E2=80=9D, >>>>>>> which of course it doesn=E2=80=99t. >>>>>>>=20 >>>>>>>=20 >>>>>> I think this is education and I suspect that very few of these people >>>>>> are running servers on their system and need location block. >>>>> I am personally not here to educate people. I am happy to help out and >>>>> give my opinion, but I suppose considering that people think as I have >>>>> outlined above, who am I to educate them? >>>>>=20 >>>> Sorry I didn't express my reply well, I agree that IPFire users should >> be >>>> responsible for their own actions. >>> Naturally. But we should also not give them sharp knives when we don=E2= =80=99t >> expect them to use them properly. >>>>> It is just very regretful to me when people are ab-using a feature >> just >>>>> to =E2=80=9Cfeel more safe=E2=80=9D. >>>>>=20 >>>>>>> * What I personally like as a feature is to apply location filters >> to >>>>>>> individual rules. That allows to apply rate limits in a few places >> and >>>>>>> easily select some larger groups of potentially less interesting >>>>>>> traffic. >>>>>> I agree and is similar to what I am doing for ports 25 and 587 >> although >>>>>> with location block. >>>>>>=20 >>>>>>=20 >>>>>>> * Back when this was introduced there was also a lot less use of >>>>>>> botnets as we know them now. Pretty much all recent DoS attacks I >>>>>>> remember have been coming from all over the world because systems >>>>>>> where compromised wherever they were. This renders the filter >>>>>>> absolutely useless. >>>>>> For this, I am currently testing a dynamic block list on my system >>>>>> which adds an entry to an ipset when an ip address exceeds a >> threshold >>>>>> in the previous 24 hours. Currently there are about 7000 entries in >>>>>> this set which have been detected over the previous month. Not for >>>>>> everyone because it needs a Whitelist to prevent some 'push' >>>>>> technologies such as Facebook and to protect from bulk inputs such as >>>>>> mailing lists. I also blacklist a large number of ip addresses which >>>>>> are associated with port scanners. >>>>>>=20 >>>>>> This would work well as a botnet filter but it needs an understanding >>>>>> of the blocking process, so it would not be suitable for everyone. >>>>> We have talked about this on the last team call and came to the >>>>> conclusion that the whole set of blocking features is a total mess. >>>>> There are several places where things can be blocked, they all heavily >>>>> overlap. Not to be going on about Spamhaus again, but the DROP list is >>>>> in the =E2=80=9CHostile Networks=E2=80=9D feature, there is a special c= ountry code for >>>>> it, pretty much every IPS ruleset contains it, the IP blocklist >> feature >>>>> supports it. Now there are people throwing this all into DNS as well. >> I >>>>> believe this is all getting a little bit out of hand. >>>>>=20 >>>> I don't use the IPS mainly due to bad experiences when trying to use >> Snort >>>> with IPCop. I find it difficult to set up and select the appropriate >>>> Rulsets for my requirements. I suspect the IPS would be very effective >> if >>>> I could get it set up correctly. >>> This has massively improved in recent times. And it is FAST! >>>=20 >> I'll have another look, but my current set-up is working well, so I'm >> reluctant to change. >>=20 >>>>> I personally would prefer *one* place and we really make this work >> well. >>>>> Some people find the IPS has too much overhead - I am beginning to >>>>> disagree. >>>>>=20 >>>>> Some people add all sorts of blacklists that are clearly for >>>>> experimental/research use and suddenly complain about things not >> working >>>>> anymore. It is hard to figure out what is actually blocking traffic >> when >>>>> there are about five places to look for. The result often is to >> disable >>>>> the IPS, then something else, and after a while all of this is turned >>>>> off and the firewall is configured to a bare minimum. We don=E2=80=99t = want >>>>> people to run it like that. >>>>>=20 >>>>>>> * Infrastructure is becoming more centralised. Some people attempt >> to >>>>>>> block the US and pretty much nothing works any more. Well=E2=80=A6 >>>>>>>=20 >>>>>>>=20 >>>>>> :) >>>>>>=20 >>>>>>> Therefore I am happy to rethink this feature - personally I would >> even >>>>>>> remove it entirely, but I don=E2=80=99t like to handle the stress abo= ut this >>>>>>> now. >>>>>> No I think you should keep it. Locationblock is very useful, but >> needs >>>>>> to be used in a considered way. I could live without it but its >>>>>> availability makes my life easier. Also being updated on a weekly >> basis >>>>>> makes keeping track of abusers easier. >>>>> We issue a new location database once a day, but the IPFire systems >>>>> update it only once a week. We deemed that as sufficient as there are >>>>> not that many changes from one version to another one. >>>>>=20 >>>>>>> I think this should log now. I don=E2=80=99t think there is any legit= imate >>>>>>> reason to disable firewall logging at all. This is the paper trail >>>>>>> that you want when debugging anything or doing forensics on an >> attack. >>>>>>> Certainly bad flash is not a valid reason for me. >>>>>>>=20 >>>>>>>=20 >>>>>> I have started generating some patches to make 'locationblock >> logging' >>>>>> as a selectable feature and could post them here in the next day or >> so >>>>>> if of any use. >>>>> What would this basically do? Log it as a simple rule for all, or do >> you >>>>> include the selected country? I think the latter would make more >> sense. >>>> OK - I have modified location-block.cgi to add an additional checkbox >> at >>>> the top of the page titled "Log dropped packets", very similar to >>>> ipblocklist.cgi. This adds a entry to the /var/ipfire/firewall/ >>>> locationblock settings file ON or OFF. >>>>=20 >>>> In /usr/lib/firewall/rules.pl I add an additional Iptables >> LOCATIONBLOCK >>>> log rule with the prefix 'LOCBLOCK-$location '. This is selectable >> from >>>> the Log dropped packets entry in the locationblock settings file. >>>>=20 >>>> This generates kernel messages such as: >>>>=20 >>>> LOCBLOCK-CN IN=3Dppp0 OUT=3Dppp0 MAC=3D SRC=3D114.139.42.71 DST=3D81.187= .175.103 >>>> LEN=3D60 TOS=3D0x00 PREC=3D0x00 TTL=3D47 ID=3D34664 DF PROTO=3DTCP SPT= =3D46837 >> DPT=3D23 >>>> WINDOW=3D29040 RES=3D0x00 SYN URGP=3D0 >>>>=20 >>>> As you can see the prefix contains the country code of the dropped >>>> packets. >>> Cool. So I think coding-wise this is pretty straight forward. >>>=20 >>> The more crucial question for me is below=E2=80=A6 >>>=20 >> Do have a look at the patches I posted and let me have any comments. If >> it's not needed and we can add locationblock logging without selection >> then that's what I have been doing since LB was first implemented. >>=20 >>>>> What is the rationale to turn this off? Why don=E2=80=99t we turn this = on for >>>>> everybody all of the time? >>>>>=20 >>>> I would be happy with that I was suggesting that we use selectable >> logging >>>> to be compatible with earlier IPFire versions. >>> What do we have to be compatible with? Just IPFire=E2=80=99s behaviour fr= om >> last >> year? >>> I would be happy to break that if we have a good reason, because the >> world sometimes moves on. What we have released 10 years ago probably >> doesn=E2=80=99t have good defaults any more. Sometimes we have to revisit = those >> and sometimes we even have to be pushy with some changes if we want some >> long-standing users to adopt them. IPFire users usually have no reason to >> do a reinstall and we don=E2=80=99t want to leave the people behind who ha= ve been >> running IPFire perfectly well for the last 10 years or even longer. So at >> least I would like to discuss what we push a little harder. >> As Adolf pointed out in the Forum,the WIKI stated the rational for not >> logging was to protect flash memory from excessive use. I don't think that >> is appropriate now. >>=20 >>>>>> In fact I think that IPFire's logging environment, which was >> inherited >>>>>> from IPCop needs a re-visit. >>>>> Yes. But this is probably not a thing I would be looking at in IPFire >> 2. >>>>> I am also happy to think a little bit more outside of the box and see >>>>> what we would fundamentally change in the future, but not implement in >>>>> IPFire 2. >>>>>=20 >>>>>> I was pleased to see that a while back there was a discussion of >>>>>> replacing sylogd with rsyslogd which I think would be a very useful >>>>>> upgrade and enable kernel messages, like on most current linux >> systems, >>>>>> to be logged to alternative log files instead of everything ending up >>>>>> in /var/log/ messages. >>>>> The currentl syslog server could do that. The problem is more that we >>>>> don=E2=80=99t have good tools to view the logs on the web UI. So there = is >> simply >>>>> one place where most things go into and we will then filter it out >> when >>>>> users use the web UI. This is tremendously slow. >>>>>=20 >>>> I did look at this a while back and believe that with syslog all >> kernel >>>> messages must go to the same log file and cannot be filtered off as >> can >> be >>>> done with rsyslog. I use this facility in rsyslog.d on my server to >> filter >>>> off SYN Flood logs to a separate log file with ":msg,contains". I >> don't >>>> believe that syslog has this option. >>> Different messages could be sent to different =E2=80=9Cfacilities=E2=80= =9D and then we >> can filter them into individual files. >>> I would not want to split firewall hits into different files though. I >> just wouldn=E2=80=99t mind if firewall hits were no longer in /var/log/mes= sages. >>=20 >>=20 >> You need to be careful with the LOG* menus which would need to look in 2 >> or more places for historical logs and so the cgi's would need to be >> modified. >>=20 >> I would however welcome kernel message filtering since it would allow my >> use of a separate log directory for my auto-dnsbl program to monitor for >> non-dropped incoming SMTP connections without appearing in 'messages' log >> file. >>=20 >>=20 >>>>>> The replacement with rsyslgd could also inhibit the "last message >>>>>> repeated x times" in syslog and which are ignored by IPFire's Log >> menu >>>>>> and causes erroneous results in the 'Count' values. >>>>> This might be a setting that we can turn off in the current >>>>> implementation. >>>>>=20 >>>> I understand that this is hard coded into syslogd and can't be >> changed. >> In >>>> rsyslog it defaults to "off". >>>>=20 >>>> https://www.rsyslog.com/doc/configuration/action/ >>>> rsconf1_repeatedmsgreduction.html >>> What impact does this have for you? I never noticed any problem by >> summarising the logs. >>=20 >> I came across this and it illustrates the point well: >>=20 >> https://dcid.me/notes/syslog-last-message-repeated-x-times-rant >>=20 >> and makes the point that kernel messages can be delayed by several seconds >> and could affect any DOS filtering that may be in use. >>=20 >> The dshield iptables.pl script tries to recover the lost repeated messages >> but looses the time-stamp. >>=20 >> When comparing the dshield logs with IPFire's logs there is invariably a >> discrepancy with the IPFire's count sometimes considerably less especially >> with multiple port-scans over a minute or so. >>=20 >> Any chance of looking again at rsyslog? IPFire is the only linux system I >> know of still using syslog. >>=20 >> I seem to remember that Adolf had completed all the necessary changes to >> introduce rsyslog a years or so ago but was not implemented, >=20 > It wasn't me. It was Erik Kapfer. I have looked at it to have a go at it bu= t there have been a lot of other things that are on my plate that I am still = working through and of course sometimes things come along that have to be add= ressed and jump ahead in the queue. Story of my life :) > Regards, >=20 > Adolf. >=20 >> =20 >>>>> We could also think about moving all firewall hits into >>>>> /var/log/firewall.log or something. That should free up enough in >>>>> /var/log/messages to make debugging easier. >>>>>=20 >>>> That would be a sensible option, currently I filter out firewall hits >> from >>>> the messages.log with a perl regex. >>>>=20 >>>>>>> I would be happy to hear some more opinions on this from the list. >>>>>>>=20 >>>>>>> -Michael >>>>>> I hope these comments are useful and would be pleased to help where I >>>>>> can. >>>>> Of course :) >>>>>=20 >>>>> -Michael >>>>>=20 >>>>>=20 >>>>>> Rob >>> -Michael >> Rob --===============9003613368277310438==--