From mboxrd@z Thu Jan 1 00:00:00 1970 From: Rob Brewer To: development@lists.ipfire.org Subject: Re: Logging Locationblock packets Date: Sat, 23 Nov 2024 14:36:54 +0000 Message-ID: In-Reply-To: <074EA3F9-8A01-4C67-9EF1-0FC556AC3BB8@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2453050562232315898==" List-Id: --===============2453050562232315898== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable On Sat, 23 Nov 2024 10:26:07 +0100, Michael Tremer wrote: > Hello, >=20 >> On 21 Nov 2024, at 00:08, Rob Brewer =20 wrote: >>=20 >> Hi Michael and Adolf, >>=20 >> I'll combine my reply to both of you. >>=20 >> On Wed, 20 Nov 2024 15:21:28 +0000, Michael Tremer wrote: >>=20 >>> Hello, >>>=20 >>>> On 20 Nov 2024, at 13:36, Rob Brewer >>>> wrote: >>>>=20 >>>> On Tue, 19 Nov 2024 17:10:24 +0000, Michael Tremer wrote: >>>>=20 >>>> Hi Michael, thank for your response, I was beginning to think there=20 was >>>> little interest in this topic. >>>=20 >>> I was traveling and so have a little bit of a back log as usual :) >>>=20 >>> I can=E2=80=99t speak for the others, though. >>>=20 >>=20 >> Not a problem, I should have said that I previously raised this a=20 topic=20 on=20 >> the forum and Adolf suggested that I raised it on the mailing list as=20 >> well. >=20 > Yes, I really prefer this here, because the forum is very overwhelming=20 sometimes. There is too many threads, too many people=E2=80=A6 Mailing lists = are=20 great for development. That's fine, one query though you don't seem to snip older parts of the=20 post, is this your preference? >=20 >>>=20 >>>> See below... >>>>=20 >>>>> Hello Rob, >>>>>=20 >>>>> Good to hear from you again. >>>>>=20 >>>>>> On 9 Nov 2024, at 14:00, Rob Brewer >>>>>> wrote: >>>>>>=20 >>>>>> As a contributor to the SANS ISC, I email my IPTABLES logs hourly >>>>>> to the Dshield database, however the default IPFire IPTABLES logs=20 are >>>>>> incomplete since packets dropped by Locationblock are not logged to >>>>>> syslog. >>>>>=20 >>>>> This is correct. >>>>>=20 >>>>>> As a workaround I edit the locationblock function of rules.pl to=20 add >>>>>> a IPTABLES rule to log the dropped packets. This edit does not of >>>>>> course survive any Core-Updates that upgrade rules.pl and the edit >>>>>> needs to be re-applied. >>>>>>=20 >>>>>> In addition, unless the Locationblock packets are logged to syslog >>>>>> the data displayed in the Firewall logs sections are incomplete as >>>>>> the packets dropped by Location Block are not displayed. >>>>>>=20 >>>>>> I understand that the original intension of the Location Block >>>>>> feature was to reduce the amount of log messages on installations >>>>>> running on extremely cheap flash storage. I suspect that these >>>>>> installations are now almost zero especially since IPFire no longer >>>>>> supports 32 bit versions. >>>>>>=20 >>>>>> The obvious solution to this to make Location Block logging >>>>>> selectable as a firewall option and my quick look at making a=20 patches >>>>>> for this would seem to be fairly simple task to rules.pl and and >>>>>> optionsfw.cgi. >>>>>>=20 >>>>>> I would be happy to provide the necessary patches should this be >>>>>> acceptable. >>>>>=20 >>>>> I think a couple of things have changed since we introduced this >>>>> feature and we have also learned a couple of lessons here: >>>>>=20 >>>>> * Yes it is true that this was introduced to keep the logs smaller, >>>>> but not necessarily to avoid using storage, mainly to keep them >>>>> readable by a human. I was always a bit doubtful about this and >>>>> believe that apart from this, there is no more value in a general >>>>> location blocking feature. The firewall will drop those packets >>>>> anyway. >>>>>=20 >>>>>=20 >>>> IPFire does a very good job in protecting the internal network from >>>> unwanted probes from the WAN, and I agree that for most users,=20 location >>>> block does not enhance this protection. >>>>=20 >>>> As I run a mail server, I use locationblock as part of my filtering=20 to >>>> prevent bad actors from accessing my mail server and I think would be >>>> useful for anyone who are running servers on their networks. >>>>=20 >>>> I don't like blocking traffic, but I have witnessed over the last few >>>> years a substantial rise in abuse from certain countries from where I >>>> would never expect to receive emails and locationblock works very=20 well >>>> for me in filtering out those unwanted connections. >>>>=20 >>>> I currently have 21 countries blocked including XD Hostile networks. >>>> All of these blocked countries have been seen to support abusive >>>> networks. >>>>=20 >>>> It would be possible to generate individual rules for these networks >>>> which in some cases I do with Banish, but it is sometimes easier just >>>> to drop the whole country. >>>=20 >>> Easy yes. I am just not sure what we are actually getting out of this. >>>=20 >>> I don=E2=80=99t like to become political on this list and so I won=E2=80= =99t, but I=20 know >>> that some people hold the following beliefs: >>>=20 >>> * There are =E2=80=9Cbad=E2=80=9D countries out there. In the western wor= ld those are >>> Russia, China, and more that fall into this kind of =E2=80=9Ccategory=E2= =80=9D.=20 Nothing >>> is going to stop them to open up an account on AWS and suddenly they >>> have access to resources anywhere in the world and so actually, if=20 those >>> players were up to anything =E2=80=9Cbad=E2=80=9D, they can come out of a= ny IP space >>> they want. There is also Tor which allows this. >>>=20 >>=20 >> I have been submitting my iptables logs to dshield for over 20 years=20 and=20 >> in the early days I would be sending a lot less than 1000 a day (this=20 was=20 >> with IPCop of course). >>=20 >> Over the last few years the number of firewall iptables logs have =20 >> escalated until it had reached dshield's daily limit of 100,000 /day =20 and=20 >> I started looking at ways to limit the logging of some of the 'bad=20 actors'=20 >> firewall probes. >=20 > What do you use to send them the logs? The Python script or one of=20 those=20 alternatives? I use a modified Perl script (iptables.pl) which is about 20 years old was=20 originally written for Smoothwall and Ipcop1. This was part of the Logsend=20 add-on and I modified for IPCop2 about 15 years ago.=20 Now modified for IPFire and using DMA email instead of the original=20 sendEmail. I've uploaded my version to people.ipfire.org/~helix/dshield.=20 I had intended to write an install/uninstall script but is in the todo=20 list. Do you think it may be of interest to anyone if I do? >=20 >>> * Looking at actual data, the bullet proof ISPs that annoy us are not >>> that far away. They are hosted in Europe and very prominently so in=20 the >>> Netherlands - a =E2=80=9Cgood=E2=80=9D country. >>>=20 >>=20 >> I use Banish for this as it can block by ASN and can target bullet=20 proof=20 >> ISPs not only in NL but also in the USA and elsewhere. Using the=20 'location=20 >> function' the blocked ASN's are updated when the database is updated. >>=20 >>> This is just an example to explain that we all have our idea of the >>> world in our head, but the Internet doesn=E2=80=99t know these borders. >>> Geopolitics is complicated and I don=E2=80=99t think that this is a tool = that >>> helps to fight this kind of problem. I hope this is clear what I am >>> trying to say. >>>=20 >>=20 >> Yes my approach is to use multiple tools and Locationblock is just one=20 of=20 >> the tools I am using. Maybe my approach is over the top though. >=20 > Probably not massively, but maybe slightly. >=20 > Wouldn=E2=80=99t it be nicer to have everything in one place instead of hav= ing=20 multiple tools that work together in concert? >=20 I think that would be a lot of work, would you consider that for IPFire2=20 or 3? >>> I personally trust in Spamhaus which is feeding data into our =E2=80=9CHo= stile >>> Networks=E2=80=9D blocker and they are looking at the actual traffic that= they >>> see from those subnets that they are blocking. They have a clear set=20 of >>> rules and they don=E2=80=99t look just at the country and say =E2=80=9Cah= yes, the bad >>> guys again=E2=80=9D. >>>=20 >>=20 >> For emails I also drop packets if listed in a number of DNS blackhole=20 >> lists using Auto-Dnsbl which I adapted for IPFire. See: >>=20 >> https://people.ipfire.org/~helix/auto-dnsbl/README >> for info. >>=20 >> I have been using this for about a year and can confirm that it is=20 very=20 >> effective in removing spammers. >>=20 >>> I have stated this recently in the RPZ discussion that I feel that we >>> are lacking good sources for this kind of data. For blocking anything. >>> Loads of people believe =E2=80=9Cmore is more=E2=80=9D and to be fair=E2= =80=A6 once you get to >>> blocking 90% of the internet you probably have blocked 90% of the=20 noise. >>> However this won=E2=80=99t do anything for targeted attacks. I personally= =20 would >>> rather have more quality here. >>>=20 >>=20 >> I only use a few blocklists with ipblocklist and rely mainly on my=20 locally=20 >> derived data from failed smtp connections or from data from dshield's=20 >> daily report. >>=20 >>>>> * People seem to misunderstand this feature: The block as many >>>>> countries as they can click on and think that this makes things=20 =E2=80=9Cmore >>>>> secure=E2=80=9D, >>>>> which of course it doesn=E2=80=99t. >>>>>=20 >>>>>=20 >>>> I think this is education and I suspect that very few of these people >>>> are running servers on their system and need location block. >>>=20 >>> I am personally not here to educate people. I am happy to help out and >>> give my opinion, but I suppose considering that people think as I have >>> outlined above, who am I to educate them? >>>=20 >>=20 >> Sorry I didn't express my reply well, I agree that IPFire users should=20 be=20 >> responsible for their own actions. >=20 > Naturally. But we should also not give them sharp knives when we don=E2=80= =99t=20 expect them to use them properly. >=20 >>> It is just very regretful to me when people are ab-using a feature=20 just >>> to =E2=80=9Cfeel more safe=E2=80=9D. >>>=20 >>>>> * What I personally like as a feature is to apply location filters=20 to >>>>> individual rules. That allows to apply rate limits in a few places=20 and >>>>> easily select some larger groups of potentially less interesting >>>>> traffic. >>>>=20 >>>> I agree and is similar to what I am doing for ports 25 and 587=20 although >>>> with location block. >>>>=20 >>>>=20 >>>>> * Back when this was introduced there was also a lot less use of >>>>> botnets as we know them now. Pretty much all recent DoS attacks I >>>>> remember have been coming from all over the world because systems >>>>> where compromised wherever they were. This renders the filter >>>>> absolutely useless. >>>>=20 >>>> For this, I am currently testing a dynamic block list on my system >>>> which adds an entry to an ipset when an ip address exceeds a=20 threshold >>>> in the previous 24 hours. Currently there are about 7000 entries in >>>> this set which have been detected over the previous month. Not for >>>> everyone because it needs a Whitelist to prevent some 'push' >>>> technologies such as Facebook and to protect from bulk inputs such as >>>> mailing lists. I also blacklist a large number of ip addresses which >>>> are associated with port scanners. >>>>=20 >>>> This would work well as a botnet filter but it needs an understanding >>>> of the blocking process, so it would not be suitable for everyone. >>>=20 >>> We have talked about this on the last team call and came to the >>> conclusion that the whole set of blocking features is a total mess. >>> There are several places where things can be blocked, they all heavily >>> overlap. Not to be going on about Spamhaus again, but the DROP list is >>> in the =E2=80=9CHostile Networks=E2=80=9D feature, there is a special cou= ntry code for >>> it, pretty much every IPS ruleset contains it, the IP blocklist=20 feature >>> supports it. Now there are people throwing this all into DNS as well.=20 I >>> believe this is all getting a little bit out of hand. >>>=20 >>=20 >> I don't use the IPS mainly due to bad experiences when trying to use=20 Snort=20 >> with IPCop. I find it difficult to set up and select the appropriate=20 >> Rulsets for my requirements. I suspect the IPS would be very effective=20 if=20 >> I could get it set up correctly. >=20 > This has massively improved in recent times. And it is FAST! >=20 I'll have another look, but my current set-up is working well, so I'm=20 reluctant to change. >>> I personally would prefer *one* place and we really make this work=20 well. >>>=20 >>> Some people find the IPS has too much overhead - I am beginning to >>> disagree. >>>=20 >>> Some people add all sorts of blacklists that are clearly for >>> experimental/research use and suddenly complain about things not=20 working >>> anymore. It is hard to figure out what is actually blocking traffic=20 when >>> there are about five places to look for. The result often is to=20 disable >>> the IPS, then something else, and after a while all of this is turned >>> off and the firewall is configured to a bare minimum. We don=E2=80=99t wa= nt >>> people to run it like that. >>>=20 >>>>> * Infrastructure is becoming more centralised. Some people attempt=20 to >>>>> block the US and pretty much nothing works any more. Well=E2=80=A6 >>>>>=20 >>>>>=20 >>>> :) >>>>=20 >>>>> Therefore I am happy to rethink this feature - personally I would=20 even >>>>> remove it entirely, but I don=E2=80=99t like to handle the stress about= this >>>>> now. >>>>=20 >>>> No I think you should keep it. Locationblock is very useful, but=20 needs >>>> to be used in a considered way. I could live without it but its >>>> availability makes my life easier. Also being updated on a weekly=20 basis >>>> makes keeping track of abusers easier. >>>=20 >>> We issue a new location database once a day, but the IPFire systems >>> update it only once a week. We deemed that as sufficient as there are >>> not that many changes from one version to another one. >>>=20 >>>>> I think this should log now. I don=E2=80=99t think there is any legitim= ate >>>>> reason to disable firewall logging at all. This is the paper trail >>>>> that you want when debugging anything or doing forensics on an=20 attack. >>>>> Certainly bad flash is not a valid reason for me. >>>>>=20 >>>>>=20 >>>> I have started generating some patches to make 'locationblock=20 logging' >>>> as a selectable feature and could post them here in the next day or=20 so >>>> if of any use. >>>=20 >>> What would this basically do? Log it as a simple rule for all, or do=20 you >>> include the selected country? I think the latter would make more=20 sense. >>>=20 >>=20 >> OK - I have modified location-block.cgi to add an additional checkbox=20 at=20 >> the top of the page titled "Log dropped packets", very similar to=20 >> ipblocklist.cgi. This adds a entry to the /var/ipfire/firewall/ >> locationblock settings file ON or OFF. >>=20 >> In /usr/lib/firewall/rules.pl I add an additional Iptables=20 LOCATIONBLOCK=20 >> log rule with the prefix 'LOCBLOCK-$location '. This is selectable=20 from=20 >> the Log dropped packets entry in the locationblock settings file. >>=20 >> This generates kernel messages such as: >>=20 >> LOCBLOCK-CN IN=3Dppp0 OUT=3Dppp0 MAC=3D SRC=3D114.139.42.71 DST=3D81.187.1= 75.103=20 >> LEN=3D60 TOS=3D0x00 PREC=3D0x00 TTL=3D47 ID=3D34664 DF PROTO=3DTCP SPT=3D4= 6837=20 DPT=3D23=20 >> WINDOW=3D29040 RES=3D0x00 SYN URGP=3D0 >>=20 >> As you can see the prefix contains the country code of the dropped=20 >> packets. >=20 > Cool. So I think coding-wise this is pretty straight forward. >=20 > The more crucial question for me is below=E2=80=A6 >=20 Do have a look at the patches I posted and let me have any comments. If=20 it's not needed and we can add locationblock logging without selection=20 then that's what I have been doing since LB was first implemented. >>> What is the rationale to turn this off? Why don=E2=80=99t we turn this on= for >>> everybody all of the time? >>>=20 >>=20 >> I would be happy with that I was suggesting that we use selectable=20 logging=20 >> to be compatible with earlier IPFire versions. >=20 > What do we have to be compatible with? Just IPFire=E2=80=99s behaviour from= =20 last=20 year? >=20 > I would be happy to break that if we have a good reason, because the=20 world sometimes moves on. What we have released 10 years ago probably=20 doesn=E2=80=99t have good defaults any more. Sometimes we have to revisit tho= se=20 and sometimes we even have to be pushy with some changes if we want some=20 long-standing users to adopt them. IPFire users usually have no reason to=20 do a reinstall and we don=E2=80=99t want to leave the people behind who have = been=20 running IPFire perfectly well for the last 10 years or even longer. So at=20 least I would like to discuss what we push a little harder. >=20 As Adolf pointed out in the Forum,the WIKI stated the rational for not=20 logging was to protect flash memory from excessive use. I don't think that=20 is appropriate now. >>>> In fact I think that IPFire's logging environment, which was=20 inherited >>>> from IPCop needs a re-visit. >>>=20 >>> Yes. But this is probably not a thing I would be looking at in IPFire=20 2. >>>=20 >>> I am also happy to think a little bit more outside of the box and see >>> what we would fundamentally change in the future, but not implement in >>> IPFire 2. >>>=20 >>>> I was pleased to see that a while back there was a discussion of >>>> replacing sylogd with rsyslogd which I think would be a very useful >>>> upgrade and enable kernel messages, like on most current linux=20 systems, >>>> to be logged to alternative log files instead of everything ending up >>>> in /var/log/ messages. >>>=20 >>> The currentl syslog server could do that. The problem is more that we >>> don=E2=80=99t have good tools to view the logs on the web UI. So there is= =20 simply >>> one place where most things go into and we will then filter it out=20 when >>> users use the web UI. This is tremendously slow. >>>=20 >>=20 >> I did look at this a while back and believe that with syslog all=20 kernel=20 >> messages must go to the same log file and cannot be filtered off as=20 can=20 be=20 >> done with rsyslog. I use this facility in rsyslog.d on my server to=20 filter=20 >> off SYN Flood logs to a separate log file with ":msg,contains". I=20 don't=20 >> believe that syslog has this option. >=20 > Different messages could be sent to different =E2=80=9Cfacilities=E2=80=9D = and then we=20 can filter them into individual files. >=20 > I would not want to split firewall hits into different files though. I=20 just wouldn=E2=80=99t mind if firewall hits were no longer in /var/log/messag= es. You need to be careful with the LOG* menus which would need to look in 2=20 or more places for historical logs and so the cgi's would need to be=20 modified. I would however welcome kernel message filtering since it would allow my=20 use of a separate log directory for my auto-dnsbl program to monitor for=20 non-dropped incoming SMTP connections without appearing in 'messages' log=20 file.=20 >=20 >>>> The replacement with rsyslgd could also inhibit the "last message >>>> repeated x times" in syslog and which are ignored by IPFire's Log=20 menu >>>> and causes erroneous results in the 'Count' values. >>>=20 >>> This might be a setting that we can turn off in the current >>> implementation. >>>=20 >>=20 >> I understand that this is hard coded into syslogd and can't be=20 changed.=20 In=20 >> rsyslog it defaults to "off". >>=20 >> https://www.rsyslog.com/doc/configuration/action/ >> rsconf1_repeatedmsgreduction.html >=20 > What impact does this have for you? I never noticed any problem by=20 summarising the logs. I came across this and it illustrates the point well: https://dcid.me/notes/syslog-last-message-repeated-x-times-rant and makes the point that kernel messages can be delayed by several seconds=20 and could affect any DOS filtering that may be in use. The dshield iptables.pl script tries to recover the lost repeated messages=20 but looses the time-stamp.=20 When comparing the dshield logs with IPFire's logs there is invariably a=20 discrepancy with the IPFire's count sometimes considerably less especially=20 with multiple port-scans over a minute or so. Any chance of looking again at rsyslog? IPFire is the only linux system I=20 know of still using syslog. I seem to remember that Adolf had completed all the necessary changes to=20 introduce rsyslog a years or so ago but was not implemented, =20 >=20 >>> We could also think about moving all firewall hits into >>> /var/log/firewall.log or something. That should free up enough in >>> /var/log/messages to make debugging easier. >>>=20 >>=20 >> That would be a sensible option, currently I filter out firewall hits=20 from=20 >> the messages.log with a perl regex.=20 >>=20 >>>>> I would be happy to hear some more opinions on this from the list. >>>>>=20 >>>>> -Michael >>>>=20 >>>> I hope these comments are useful and would be pleased to help where I >>>> can. >>>=20 >>> Of course :) >>>=20 >>> -Michael >>>=20 >>>=20 >>>>=20 >>>> Rob >=20 > -Michael Rob --===============2453050562232315898==--