From mboxrd@z Thu Jan 1 00:00:00 1970 From: Adolf Belka To: development@lists.ipfire.org Subject: Re: Logging Locationblock packets Date: Sat, 23 Nov 2024 15:47:01 +0100 Message-ID: In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1600524375450439135==" List-Id: --===============1600524375450439135== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi Rob, On 23/11/2024 15:36, Rob Brewer wrote: > On Sat, 23 Nov 2024 10:26:07 +0100, Michael Tremer wrote: > >> Hello, >> >>> On 21 Nov 2024, at 00:08, Rob Brewer > wrote: >>> Hi Michael and Adolf, >>> >>> I'll combine my reply to both of you. >>> >>> On Wed, 20 Nov 2024 15:21:28 +0000, Michael Tremer wrote: >>> >>>> Hello, >>>> >>>>> On 20 Nov 2024, at 13:36, Rob Brewer >>>>> wrote: >>>>> >>>>> On Tue, 19 Nov 2024 17:10:24 +0000, Michael Tremer wrote: >>>>> >>>>> Hi Michael, thank for your response, I was beginning to think there > was >>>>> little interest in this topic. >>>> I was traveling and so have a little bit of a back log as usual :) >>>> >>>> I can=E2=80=99t speak for the others, though. >>>> >>> Not a problem, I should have said that I previously raised this a > topic > on >>> the forum and Adolf suggested that I raised it on the mailing list as >>> well. >> Yes, I really prefer this here, because the forum is very overwhelming > sometimes. There is too many threads, too many people=E2=80=A6 Mailing list= s are > great for development. > > That's fine, one query though you don't seem to snip older parts of the > post, is this your preference? > >>>>> See below... >>>>> >>>>>> Hello Rob, >>>>>> >>>>>> Good to hear from you again. >>>>>> >>>>>>> On 9 Nov 2024, at 14:00, Rob Brewer >>>>>>> wrote: >>>>>>> >>>>>>> As a contributor to the SANS ISC, I email my IPTABLES logs hourly >>>>>>> to the Dshield database, however the default IPFire IPTABLES logs > are >>>>>>> incomplete since packets dropped by Locationblock are not logged to >>>>>>> syslog. >>>>>> This is correct. >>>>>> >>>>>>> As a workaround I edit the locationblock function of rules.pl to > add >>>>>>> a IPTABLES rule to log the dropped packets. This edit does not of >>>>>>> course survive any Core-Updates that upgrade rules.pl and the edit >>>>>>> needs to be re-applied. >>>>>>> >>>>>>> In addition, unless the Locationblock packets are logged to syslog >>>>>>> the data displayed in the Firewall logs sections are incomplete as >>>>>>> the packets dropped by Location Block are not displayed. >>>>>>> >>>>>>> I understand that the original intension of the Location Block >>>>>>> feature was to reduce the amount of log messages on installations >>>>>>> running on extremely cheap flash storage. I suspect that these >>>>>>> installations are now almost zero especially since IPFire no longer >>>>>>> supports 32 bit versions. >>>>>>> >>>>>>> The obvious solution to this to make Location Block logging >>>>>>> selectable as a firewall option and my quick look at making a > patches >>>>>>> for this would seem to be fairly simple task to rules.pl and and >>>>>>> optionsfw.cgi. >>>>>>> >>>>>>> I would be happy to provide the necessary patches should this be >>>>>>> acceptable. >>>>>> I think a couple of things have changed since we introduced this >>>>>> feature and we have also learned a couple of lessons here: >>>>>> >>>>>> * Yes it is true that this was introduced to keep the logs smaller, >>>>>> but not necessarily to avoid using storage, mainly to keep them >>>>>> readable by a human. I was always a bit doubtful about this and >>>>>> believe that apart from this, there is no more value in a general >>>>>> location blocking feature. The firewall will drop those packets >>>>>> anyway. >>>>>> >>>>>> >>>>> IPFire does a very good job in protecting the internal network from >>>>> unwanted probes from the WAN, and I agree that for most users, > location >>>>> block does not enhance this protection. >>>>> >>>>> As I run a mail server, I use locationblock as part of my filtering > to >>>>> prevent bad actors from accessing my mail server and I think would be >>>>> useful for anyone who are running servers on their networks. >>>>> >>>>> I don't like blocking traffic, but I have witnessed over the last few >>>>> years a substantial rise in abuse from certain countries from where I >>>>> would never expect to receive emails and locationblock works very > well >>>>> for me in filtering out those unwanted connections. >>>>> >>>>> I currently have 21 countries blocked including XD Hostile networks. >>>>> All of these blocked countries have been seen to support abusive >>>>> networks. >>>>> >>>>> It would be possible to generate individual rules for these networks >>>>> which in some cases I do with Banish, but it is sometimes easier just >>>>> to drop the whole country. >>>> Easy yes. I am just not sure what we are actually getting out of this. >>>> >>>> I don=E2=80=99t like to become political on this list and so I won=E2=80= =99t, but I > know >>>> that some people hold the following beliefs: >>>> >>>> * There are =E2=80=9Cbad=E2=80=9D countries out there. In the western wo= rld those are >>>> Russia, China, and more that fall into this kind of =E2=80=9Ccategory=E2= =80=9D. > Nothing >>>> is going to stop them to open up an account on AWS and suddenly they >>>> have access to resources anywhere in the world and so actually, if > those >>>> players were up to anything =E2=80=9Cbad=E2=80=9D, they can come out of = any IP space >>>> they want. There is also Tor which allows this. >>>> >>> I have been submitting my iptables logs to dshield for over 20 years > and >>> in the early days I would be sending a lot less than 1000 a day (this > was >>> with IPCop of course). >>> >>> Over the last few years the number of firewall iptables logs have >>> escalated until it had reached dshield's daily limit of 100,000 /day > and >>> I started looking at ways to limit the logging of some of the 'bad > actors' >>> firewall probes. >> What do you use to send them the logs? The Python script or one of > those > alternatives? > > I use a modified Perl script (iptables.pl) which is about 20 years old was > originally written for Smoothwall and Ipcop1. This was part of the Logsend > add-on and I modified for IPCop2 about 15 years ago. > > Now modified for IPFire and using DMA email instead of the original > sendEmail. I've uploaded my version to people.ipfire.org/~helix/dshield. > > I had intended to write an install/uninstall script but is in the todo > list. Do you think it may be of interest to anyone if I do? > >>>> * Looking at actual data, the bullet proof ISPs that annoy us are not >>>> that far away. They are hosted in Europe and very prominently so in > the >>>> Netherlands - a =E2=80=9Cgood=E2=80=9D country. >>>> >>> I use Banish for this as it can block by ASN and can target bullet > proof >>> ISPs not only in NL but also in the USA and elsewhere. Using the > 'location >>> function' the blocked ASN's are updated when the database is updated. >>> >>>> This is just an example to explain that we all have our idea of the >>>> world in our head, but the Internet doesn=E2=80=99t know these borders. >>>> Geopolitics is complicated and I don=E2=80=99t think that this is a tool= that >>>> helps to fight this kind of problem. I hope this is clear what I am >>>> trying to say. >>>> >>> Yes my approach is to use multiple tools and Locationblock is just one > of >>> the tools I am using. Maybe my approach is over the top though. >> Probably not massively, but maybe slightly. >> >> Wouldn=E2=80=99t it be nicer to have everything in one place instead of ha= ving > multiple tools that work together in concert? > I think that would be a lot of work, would you consider that for IPFire2 > or 3? > >>>> I personally trust in Spamhaus which is feeding data into our =E2=80=9CH= ostile >>>> Networks=E2=80=9D blocker and they are looking at the actual traffic tha= t they >>>> see from those subnets that they are blocking. They have a clear set > of >>>> rules and they don=E2=80=99t look just at the country and say =E2=80=9Ca= h yes, the bad >>>> guys again=E2=80=9D. >>>> >>> For emails I also drop packets if listed in a number of DNS blackhole >>> lists using Auto-Dnsbl which I adapted for IPFire. See: >>> >>> https://people.ipfire.org/~helix/auto-dnsbl/README >>> for info. >>> >>> I have been using this for about a year and can confirm that it is > very >>> effective in removing spammers. >>> >>>> I have stated this recently in the RPZ discussion that I feel that we >>>> are lacking good sources for this kind of data. For blocking anything. >>>> Loads of people believe =E2=80=9Cmore is more=E2=80=9D and to be fair=E2= =80=A6 once you get to >>>> blocking 90% of the internet you probably have blocked 90% of the > noise. >>>> However this won=E2=80=99t do anything for targeted attacks. I personally > would >>>> rather have more quality here. >>>> >>> I only use a few blocklists with ipblocklist and rely mainly on my > locally >>> derived data from failed smtp connections or from data from dshield's >>> daily report. >>> >>>>>> * People seem to misunderstand this feature: The block as many >>>>>> countries as they can click on and think that this makes things > =E2=80=9Cmore >>>>>> secure=E2=80=9D, >>>>>> which of course it doesn=E2=80=99t. >>>>>> >>>>>> >>>>> I think this is education and I suspect that very few of these people >>>>> are running servers on their system and need location block. >>>> I am personally not here to educate people. I am happy to help out and >>>> give my opinion, but I suppose considering that people think as I have >>>> outlined above, who am I to educate them? >>>> >>> Sorry I didn't express my reply well, I agree that IPFire users should > be >>> responsible for their own actions. >> Naturally. But we should also not give them sharp knives when we don=E2=80= =99t > expect them to use them properly. >>>> It is just very regretful to me when people are ab-using a feature > just >>>> to =E2=80=9Cfeel more safe=E2=80=9D. >>>> >>>>>> * What I personally like as a feature is to apply location filters > to >>>>>> individual rules. That allows to apply rate limits in a few places > and >>>>>> easily select some larger groups of potentially less interesting >>>>>> traffic. >>>>> I agree and is similar to what I am doing for ports 25 and 587 > although >>>>> with location block. >>>>> >>>>> >>>>>> * Back when this was introduced there was also a lot less use of >>>>>> botnets as we know them now. Pretty much all recent DoS attacks I >>>>>> remember have been coming from all over the world because systems >>>>>> where compromised wherever they were. This renders the filter >>>>>> absolutely useless. >>>>> For this, I am currently testing a dynamic block list on my system >>>>> which adds an entry to an ipset when an ip address exceeds a > threshold >>>>> in the previous 24 hours. Currently there are about 7000 entries in >>>>> this set which have been detected over the previous month. Not for >>>>> everyone because it needs a Whitelist to prevent some 'push' >>>>> technologies such as Facebook and to protect from bulk inputs such as >>>>> mailing lists. I also blacklist a large number of ip addresses which >>>>> are associated with port scanners. >>>>> >>>>> This would work well as a botnet filter but it needs an understanding >>>>> of the blocking process, so it would not be suitable for everyone. >>>> We have talked about this on the last team call and came to the >>>> conclusion that the whole set of blocking features is a total mess. >>>> There are several places where things can be blocked, they all heavily >>>> overlap. Not to be going on about Spamhaus again, but the DROP list is >>>> in the =E2=80=9CHostile Networks=E2=80=9D feature, there is a special co= untry code for >>>> it, pretty much every IPS ruleset contains it, the IP blocklist > feature >>>> supports it. Now there are people throwing this all into DNS as well. > I >>>> believe this is all getting a little bit out of hand. >>>> >>> I don't use the IPS mainly due to bad experiences when trying to use > Snort >>> with IPCop. I find it difficult to set up and select the appropriate >>> Rulsets for my requirements. I suspect the IPS would be very effective > if >>> I could get it set up correctly. >> This has massively improved in recent times. And it is FAST! >> > I'll have another look, but my current set-up is working well, so I'm > reluctant to change. > >>>> I personally would prefer *one* place and we really make this work > well. >>>> Some people find the IPS has too much overhead - I am beginning to >>>> disagree. >>>> >>>> Some people add all sorts of blacklists that are clearly for >>>> experimental/research use and suddenly complain about things not > working >>>> anymore. It is hard to figure out what is actually blocking traffic > when >>>> there are about five places to look for. The result often is to > disable >>>> the IPS, then something else, and after a while all of this is turned >>>> off and the firewall is configured to a bare minimum. We don=E2=80=99t w= ant >>>> people to run it like that. >>>> >>>>>> * Infrastructure is becoming more centralised. Some people attempt > to >>>>>> block the US and pretty much nothing works any more. Well=E2=80=A6 >>>>>> >>>>>> >>>>> :) >>>>> >>>>>> Therefore I am happy to rethink this feature - personally I would > even >>>>>> remove it entirely, but I don=E2=80=99t like to handle the stress abou= t this >>>>>> now. >>>>> No I think you should keep it. Locationblock is very useful, but > needs >>>>> to be used in a considered way. I could live without it but its >>>>> availability makes my life easier. Also being updated on a weekly > basis >>>>> makes keeping track of abusers easier. >>>> We issue a new location database once a day, but the IPFire systems >>>> update it only once a week. We deemed that as sufficient as there are >>>> not that many changes from one version to another one. >>>> >>>>>> I think this should log now. I don=E2=80=99t think there is any legiti= mate >>>>>> reason to disable firewall logging at all. This is the paper trail >>>>>> that you want when debugging anything or doing forensics on an > attack. >>>>>> Certainly bad flash is not a valid reason for me. >>>>>> >>>>>> >>>>> I have started generating some patches to make 'locationblock > logging' >>>>> as a selectable feature and could post them here in the next day or > so >>>>> if of any use. >>>> What would this basically do? Log it as a simple rule for all, or do > you >>>> include the selected country? I think the latter would make more > sense. >>> OK - I have modified location-block.cgi to add an additional checkbox > at >>> the top of the page titled "Log dropped packets", very similar to >>> ipblocklist.cgi. This adds a entry to the /var/ipfire/firewall/ >>> locationblock settings file ON or OFF. >>> >>> In /usr/lib/firewall/rules.pl I add an additional Iptables > LOCATIONBLOCK >>> log rule with the prefix 'LOCBLOCK-$location '. This is selectable > from >>> the Log dropped packets entry in the locationblock settings file. >>> >>> This generates kernel messages such as: >>> >>> LOCBLOCK-CN IN=3Dppp0 OUT=3Dppp0 MAC=3D SRC=3D114.139.42.71 DST=3D81.187.= 175.103 >>> LEN=3D60 TOS=3D0x00 PREC=3D0x00 TTL=3D47 ID=3D34664 DF PROTO=3DTCP SPT=3D= 46837 > DPT=3D23 >>> WINDOW=3D29040 RES=3D0x00 SYN URGP=3D0 >>> >>> As you can see the prefix contains the country code of the dropped >>> packets. >> Cool. So I think coding-wise this is pretty straight forward. >> >> The more crucial question for me is below=E2=80=A6 >> > Do have a look at the patches I posted and let me have any comments. If > it's not needed and we can add locationblock logging without selection > then that's what I have been doing since LB was first implemented. > >>>> What is the rationale to turn this off? Why don=E2=80=99t we turn this o= n for >>>> everybody all of the time? >>>> >>> I would be happy with that I was suggesting that we use selectable > logging >>> to be compatible with earlier IPFire versions. >> What do we have to be compatible with? Just IPFire=E2=80=99s behaviour from > last > year? >> I would be happy to break that if we have a good reason, because the > world sometimes moves on. What we have released 10 years ago probably > doesn=E2=80=99t have good defaults any more. Sometimes we have to revisit t= hose > and sometimes we even have to be pushy with some changes if we want some > long-standing users to adopt them. IPFire users usually have no reason to > do a reinstall and we don=E2=80=99t want to leave the people behind who hav= e been > running IPFire perfectly well for the last 10 years or even longer. So at > least I would like to discuss what we push a little harder. > As Adolf pointed out in the Forum,the WIKI stated the rational for not > logging was to protect flash memory from excessive use. I don't think that > is appropriate now. > >>>>> In fact I think that IPFire's logging environment, which was > inherited >>>>> from IPCop needs a re-visit. >>>> Yes. But this is probably not a thing I would be looking at in IPFire > 2. >>>> I am also happy to think a little bit more outside of the box and see >>>> what we would fundamentally change in the future, but not implement in >>>> IPFire 2. >>>> >>>>> I was pleased to see that a while back there was a discussion of >>>>> replacing sylogd with rsyslogd which I think would be a very useful >>>>> upgrade and enable kernel messages, like on most current linux > systems, >>>>> to be logged to alternative log files instead of everything ending up >>>>> in /var/log/ messages. >>>> The currentl syslog server could do that. The problem is more that we >>>> don=E2=80=99t have good tools to view the logs on the web UI. So there is > simply >>>> one place where most things go into and we will then filter it out > when >>>> users use the web UI. This is tremendously slow. >>>> >>> I did look at this a while back and believe that with syslog all > kernel >>> messages must go to the same log file and cannot be filtered off as > can > be >>> done with rsyslog. I use this facility in rsyslog.d on my server to > filter >>> off SYN Flood logs to a separate log file with ":msg,contains". I > don't >>> believe that syslog has this option. >> Different messages could be sent to different =E2=80=9Cfacilities=E2=80=9D= and then we > can filter them into individual files. >> I would not want to split firewall hits into different files though. I > just wouldn=E2=80=99t mind if firewall hits were no longer in /var/log/mess= ages. > > > You need to be careful with the LOG* menus which would need to look in 2 > or more places for historical logs and so the cgi's would need to be > modified. > > I would however welcome kernel message filtering since it would allow my > use of a separate log directory for my auto-dnsbl program to monitor for > non-dropped incoming SMTP connections without appearing in 'messages' log > file. > > >>>>> The replacement with rsyslgd could also inhibit the "last message >>>>> repeated x times" in syslog and which are ignored by IPFire's Log > menu >>>>> and causes erroneous results in the 'Count' values. >>>> This might be a setting that we can turn off in the current >>>> implementation. >>>> >>> I understand that this is hard coded into syslogd and can't be > changed. > In >>> rsyslog it defaults to "off". >>> >>> https://www.rsyslog.com/doc/configuration/action/ >>> rsconf1_repeatedmsgreduction.html >> What impact does this have for you? I never noticed any problem by > summarising the logs. > > I came across this and it illustrates the point well: > > https://dcid.me/notes/syslog-last-message-repeated-x-times-rant > > and makes the point that kernel messages can be delayed by several seconds > and could affect any DOS filtering that may be in use. > > The dshield iptables.pl script tries to recover the lost repeated messages > but looses the time-stamp. > > When comparing the dshield logs with IPFire's logs there is invariably a > discrepancy with the IPFire's count sometimes considerably less especially > with multiple port-scans over a minute or so. > > Any chance of looking again at rsyslog? IPFire is the only linux system I > know of still using syslog. > > I seem to remember that Adolf had completed all the necessary changes to > introduce rsyslog a years or so ago but was not implemented, It wasn't me. It was Erik Kapfer. I have looked at it to have a go at it but = there have been a lot of other things that are on my plate that I am still wo= rking through and of course sometimes things come along that have to be addre= ssed and jump ahead in the queue. Regards, Adolf. > =20 > >>>> We could also think about moving all firewall hits into >>>> /var/log/firewall.log or something. That should free up enough in >>>> /var/log/messages to make debugging easier. >>>> >>> That would be a sensible option, currently I filter out firewall hits > from >>> the messages.log with a perl regex. >>> >>>>>> I would be happy to hear some more opinions on this from the list. >>>>>> >>>>>> -Michael >>>>> I hope these comments are useful and would be pleased to help where I >>>>> can. >>>> Of course :) >>>> >>>> -Michael >>>> >>>> >>>>> Rob >> -Michael > Rob --===============1600524375450439135==--