Re: Logging Locationblock packets

[Michael Tremer <michael.tremer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 21892 bytes --]

Hello

> On 23 Nov 2024, at 15:47, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi Rob,
> 
> On 23/11/2024 15:36, Rob Brewer wrote:
>> On Sat, 23 Nov 2024 10:26:07 +0100, Michael Tremer wrote:
>> 
>>> Hello,
>>> 
>>>> On 21 Nov 2024, at 00:08, Rob Brewer <ipfire-devel(a)grantura.co.uk>
>> wrote:
>>>> Hi Michael and Adolf,
>>>> 
>>>> I'll combine my reply to both of you.
>>>> 
>>>> On Wed, 20 Nov 2024 15:21:28 +0000, Michael Tremer wrote:
>>>> 
>>>>> Hello,
>>>>> 
>>>>>> On 20 Nov 2024, at 13:36, Rob Brewer <ipfire-devel(a)grantura.co.uk>
>>>>>> wrote:
>>>>>> 
>>>>>> On Tue, 19 Nov 2024 17:10:24 +0000, Michael Tremer wrote:
>>>>>> 
>>>>>> Hi Michael, thank for your response, I was beginning to think there
>> was
>>>>>> little interest in this topic.
>>>>> I was traveling and so have a little bit of a back log as usual :)
>>>>> 
>>>>> I can’t speak for the others, though.
>>>>> 
>>>> Not a problem, I should have said that I previously raised this a
>> topic
>> on
>>>> the forum and Adolf suggested that I raised it on the mailing list as
>>>> well.
>>> Yes, I really prefer this here, because the forum is very overwhelming
>> sometimes. There is too many threads, too many people… Mailing lists are
>> great for development.
>> 
>> That's fine, one query though you don't seem to snip older parts of the
>> post, is this your preference?
>> 
>>>>>> See below...
>>>>>> 
>>>>>>> Hello Rob,
>>>>>>> 
>>>>>>> Good to hear from you again.
>>>>>>> 
>>>>>>>> On 9 Nov 2024, at 14:00, Rob Brewer <ipfire-devel(a)grantura.co.uk>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>> As  a contributor to the SANS ISC,  I email my IPTABLES logs hourly
>>>>>>>> to the Dshield database, however the default IPFire IPTABLES logs
>> are
>>>>>>>> incomplete since packets dropped by Locationblock are not logged to
>>>>>>>> syslog.
>>>>>>> This is correct.
>>>>>>> 
>>>>>>>> As a workaround I edit the locationblock function of rules.pl to
>> add
>>>>>>>> a IPTABLES rule to log the dropped packets. This edit does not of
>>>>>>>> course survive any Core-Updates that upgrade rules.pl and the edit
>>>>>>>> needs to be re-applied.
>>>>>>>> 
>>>>>>>> In addition, unless the Locationblock packets are logged to syslog
>>>>>>>> the data displayed in the Firewall logs sections are incomplete as
>>>>>>>> the packets dropped by Location Block are not displayed.
>>>>>>>> 
>>>>>>>> I understand that the original intension of the Location Block
>>>>>>>> feature was to reduce the amount of log messages on installations
>>>>>>>> running on extremely cheap flash storage. I suspect that these
>>>>>>>> installations are now almost zero especially since IPFire no longer
>>>>>>>> supports 32 bit versions.
>>>>>>>> 
>>>>>>>> The obvious solution to this to make Location Block logging
>>>>>>>> selectable as a firewall option and my quick look at making a
>> patches
>>>>>>>> for this would seem to be fairly simple task to rules.pl and and
>>>>>>>> optionsfw.cgi.
>>>>>>>> 
>>>>>>>> I would be happy to provide the necessary patches should this be
>>>>>>>> acceptable.
>>>>>>> I think a couple of things have changed since we introduced this
>>>>>>> feature and we have also learned a couple of lessons here:
>>>>>>> 
>>>>>>> * Yes it is true that this was introduced to keep the logs smaller,
>>>>>>> but not necessarily to avoid using storage, mainly to keep them
>>>>>>> readable by a human. I was always a bit doubtful about this and
>>>>>>> believe that apart from this, there is no more value in a general
>>>>>>> location blocking feature. The firewall will drop those packets
>>>>>>> anyway.
>>>>>>> 
>>>>>>> 
>>>>>> IPFire does a very good job in protecting the internal network from
>>>>>> unwanted probes from the WAN, and I agree that for most users,
>> location
>>>>>> block does not enhance this protection.
>>>>>> 
>>>>>> As I run a mail server, I use locationblock as part of my filtering
>> to
>>>>>> prevent bad actors from accessing my mail server and I think would be
>>>>>> useful for anyone who are running servers on their networks.
>>>>>> 
>>>>>> I don't like blocking traffic, but I have witnessed over the last few
>>>>>> years a substantial rise in abuse from certain countries from where I
>>>>>> would never expect to receive emails and locationblock works very
>> well
>>>>>> for me in filtering out those unwanted connections.
>>>>>> 
>>>>>> I currently have 21 countries blocked including XD Hostile networks.
>>>>>> All of these blocked countries have been seen to support abusive
>>>>>> networks.
>>>>>> 
>>>>>> It would be possible to generate individual rules for these networks
>>>>>> which in some cases I do with Banish, but it is sometimes easier just
>>>>>> to drop the whole country.
>>>>> Easy yes. I am just not sure what we are actually getting out of this.
>>>>> 
>>>>> I don’t like to become political on this list and so I won’t, but I
>> know
>>>>> that some people hold the following beliefs:
>>>>> 
>>>>> * There are “bad” countries out there. In the western world those are
>>>>> Russia, China, and more that fall into this kind of “category”.
>> Nothing
>>>>> is going to stop them to open up an account on AWS and suddenly they
>>>>> have access to resources anywhere in the world and so actually, if
>> those
>>>>> players were up to anything “bad”, they can come out of any IP space
>>>>> they want. There is also Tor which allows this.
>>>>> 
>>>> I have been submitting my iptables logs to dshield for over 20 years
>> and
>>>> in the early days I would be sending a lot less than 1000 a day (this
>> was
>>>> with IPCop of course).
>>>> 
>>>> Over the last few years the number of firewall iptables logs have
>>>> escalated until it had reached dshield's daily limit of 100,000 /day
>> and
>>>> I started looking at ways to limit the logging of some of the 'bad
>> actors'
>>>> firewall probes.
>>> What do you use to send them the logs? The Python script or one of
>> those
>> alternatives?
>> 
>> I use a modified Perl script (iptables.pl) which is about 20 years old was
>> originally written for Smoothwall and Ipcop1. This was part of the Logsend
>> add-on and I modified for IPCop2 about 15 years ago.
>> 
>> Now modified for IPFire and using DMA email instead of the original
>> sendEmail. I've uploaded my version to people.ipfire.org/~helix/dshield.
>> 
>> I had intended to write an install/uninstall script but is in the todo
>> list. Do you think it may be of interest to anyone if I do?
>> 
>>>>> * Looking at actual data, the bullet proof ISPs that annoy us are not
>>>>> that far away. They are hosted in Europe and very prominently so in
>> the
>>>>> Netherlands - a “good” country.
>>>>> 
>>>> I use Banish for this as it can block by ASN and can target bullet
>> proof
>>>> ISPs not only in NL but also in the USA and elsewhere. Using the
>> 'location
>>>> function' the blocked ASN's are updated when the database is updated.
>>>> 
>>>>> This is just an example to explain that we all have our idea of the
>>>>> world in our head, but the Internet doesn’t know these borders.
>>>>> Geopolitics is complicated and I don’t think that this is a tool that
>>>>> helps to fight this kind of problem. I hope this is clear what I am
>>>>> trying to say.
>>>>> 
>>>> Yes my approach is to use multiple tools and Locationblock is just one
>> of
>>>> the tools I am using. Maybe my approach is over the top though.
>>> Probably not massively, but maybe slightly.
>>> 
>>> Wouldn’t it be nicer to have everything in one place instead of having
>> multiple tools that work together in concert?
>> I think that would be a lot of work, would you consider that for IPFire2
>> or 3?
>> 
>>>>> I personally trust in Spamhaus which is feeding data into our “Hostile
>>>>> Networks” blocker and they are looking at the actual traffic that they
>>>>> see from those subnets that they are blocking. They have a clear set
>> of
>>>>> rules and they don’t look just at the country and say “ah yes, the bad
>>>>> guys again”.
>>>>> 
>>>> For emails I also drop packets if listed in a number of DNS blackhole
>>>> lists using Auto-Dnsbl which I adapted for IPFire. See:
>>>> 
>>>> https://people.ipfire.org/~helix/auto-dnsbl/README
>>>> for info.
>>>> 
>>>> I have been using this for about a year and can confirm that it is
>> very
>>>> effective in removing spammers.
>>>> 
>>>>> I have stated this recently in the RPZ discussion that I feel that we
>>>>> are lacking good sources for this kind of data. For blocking anything.
>>>>> Loads of people believe “more is more” and to be fair… once you get to
>>>>> blocking 90% of the internet you probably have blocked 90% of the
>> noise.
>>>>> However this won’t do anything for targeted attacks. I personally
>> would
>>>>> rather have more quality here.
>>>>> 
>>>> I only use a few blocklists with ipblocklist and rely mainly on my
>> locally
>>>> derived  data from failed smtp connections or from data from dshield's
>>>> daily report.
>>>> 
>>>>>>> * People seem to misunderstand this feature: The block as many
>>>>>>> countries as they can click on and think that this makes things
>> “more
>>>>>>> secure”,
>>>>>>> which of course it doesn’t.
>>>>>>> 
>>>>>>> 
>>>>>> I think this is education and I suspect that very few of these people
>>>>>> are running servers on their system and need location block.
>>>>> I am personally not here to educate people. I am happy to help out and
>>>>> give my opinion, but I suppose considering that people think as I have
>>>>> outlined above, who am I to educate them?
>>>>> 
>>>> Sorry I didn't express my reply well, I agree that IPFire users should
>> be
>>>> responsible for their own actions.
>>> Naturally. But we should also not give them sharp knives when we don’t
>> expect them to use them properly.
>>>>> It is just very regretful to me when people are ab-using a feature
>> just
>>>>> to “feel more safe”.
>>>>> 
>>>>>>> * What I personally like as a feature is to apply location filters
>> to
>>>>>>> individual rules. That allows to apply rate limits in a few places
>> and
>>>>>>> easily select some larger groups of potentially less interesting
>>>>>>> traffic.
>>>>>> I agree and is similar to what I am doing for ports 25 and 587
>> although
>>>>>> with location block.
>>>>>> 
>>>>>> 
>>>>>>> * Back when this was introduced there was also a lot less use of
>>>>>>> botnets as we know them now. Pretty much all recent DoS attacks I
>>>>>>> remember have been coming from all over the world because systems
>>>>>>> where compromised wherever they were. This renders the filter
>>>>>>> absolutely useless.
>>>>>> For this, I am currently testing a dynamic block list on my system
>>>>>> which adds an entry to an ipset when an ip address exceeds a
>> threshold
>>>>>> in the previous 24 hours. Currently there are about 7000 entries in
>>>>>> this set which have been detected over the previous month.  Not for
>>>>>> everyone because it needs a Whitelist to prevent some 'push'
>>>>>> technologies such as Facebook and to protect from bulk inputs such as
>>>>>> mailing lists. I also blacklist a large number of ip addresses which
>>>>>> are associated with port scanners.
>>>>>> 
>>>>>> This would work well as a botnet filter but it needs an understanding
>>>>>> of the blocking process, so it would not be suitable for everyone.
>>>>> We have talked about this on the last team call and came to the
>>>>> conclusion that the whole set of blocking features is a total mess.
>>>>> There are several places where things can be blocked, they all heavily
>>>>> overlap. Not to be going on about Spamhaus again, but the DROP list is
>>>>> in the “Hostile Networks” feature, there is a special country code for
>>>>> it, pretty much every IPS ruleset contains it, the IP blocklist
>> feature
>>>>> supports it. Now there are people throwing this all into DNS as well.
>> I
>>>>> believe this is all getting a little bit out of hand.
>>>>> 
>>>> I don't use the IPS mainly due to bad experiences when trying to use
>> Snort
>>>> with IPCop. I find it difficult to set up and select the appropriate
>>>> Rulsets for my requirements. I suspect the IPS would be very effective
>> if
>>>> I could get it set up correctly.
>>> This has massively improved in recent times. And it is FAST!
>>> 
>> I'll have another look, but my current set-up is working well, so I'm
>> reluctant to change.
>> 
>>>>> I personally would prefer *one* place and we really make this work
>> well.
>>>>> Some people find the IPS has too much overhead - I am beginning to
>>>>> disagree.
>>>>> 
>>>>> Some people add all sorts of blacklists that are clearly for
>>>>> experimental/research use and suddenly complain about things not
>> working
>>>>> anymore. It is hard to figure out what is actually blocking traffic
>> when
>>>>> there are about five places to look for. The result often is to
>> disable
>>>>> the IPS, then something else, and after a while all of this is turned
>>>>> off and the firewall is configured to a bare minimum. We don’t want
>>>>> people to run it like that.
>>>>> 
>>>>>>> * Infrastructure is becoming more centralised. Some people attempt
>> to
>>>>>>> block the US and pretty much nothing works any more. Well…
>>>>>>> 
>>>>>>> 
>>>>>> :)
>>>>>> 
>>>>>>> Therefore I am happy to rethink this feature - personally I would
>> even
>>>>>>> remove it entirely, but I don’t like to handle the stress about this
>>>>>>> now.
>>>>>> No I think you should keep it. Locationblock is very useful, but
>> needs
>>>>>> to be used in a considered way. I could live without it but its
>>>>>> availability makes my life easier. Also being updated on a weekly
>> basis
>>>>>> makes keeping track of abusers easier.
>>>>> We issue a new location database once a day, but the IPFire systems
>>>>> update it only once a week. We deemed that as sufficient as there are
>>>>> not that many changes from one version to another one.
>>>>> 
>>>>>>> I think this should log now. I don’t think there is any legitimate
>>>>>>> reason to disable firewall logging at all. This is the paper trail
>>>>>>> that you want when debugging anything or doing forensics on an
>> attack.
>>>>>>> Certainly bad flash is not a valid reason for me.
>>>>>>> 
>>>>>>> 
>>>>>> I have started generating some patches to make 'locationblock
>> logging'
>>>>>> as a selectable feature and could post them here in the next day or
>> so
>>>>>> if of any use.
>>>>> What would this basically do? Log it as a simple rule for all, or do
>> you
>>>>> include the selected country? I think the latter would make more
>> sense.
>>>> OK - I have modified location-block.cgi to add an additional checkbox
>> at
>>>> the top of the page titled "Log dropped packets", very similar to
>>>> ipblocklist.cgi. This adds a entry to the /var/ipfire/firewall/
>>>> locationblock settings file ON or OFF.
>>>> 
>>>> In /usr/lib/firewall/rules.pl I add an additional Iptables
>> LOCATIONBLOCK
>>>> log rule with the prefix 'LOCBLOCK-$location '. This is selectable
>> from
>>>> the Log dropped packets entry in the locationblock settings file.
>>>> 
>>>> This generates kernel messages such as:
>>>> 
>>>> LOCBLOCK-CN IN=ppp0 OUT=ppp0 MAC= SRC=114.139.42.71 DST=81.187.175.103
>>>> LEN=60 TOS=0x00 PREC=0x00 TTL=47 ID=34664 DF PROTO=TCP SPT=46837
>> DPT=23
>>>> WINDOW=29040 RES=0x00 SYN URGP=0
>>>> 
>>>> As you can see the prefix contains the country code of the dropped
>>>> packets.
>>> Cool. So I think coding-wise this is pretty straight forward.
>>> 
>>> The more crucial question for me is below…
>>> 
>> Do have a look at the patches I posted and let me have any comments. If
>> it's not needed and we can add locationblock logging without selection
>> then that's what I have been doing since LB was first implemented.
>> 
>>>>> What is the rationale to turn this off? Why don’t we turn this on for
>>>>> everybody all of the time?
>>>>> 
>>>> I would be happy with that I was suggesting that we use selectable
>> logging
>>>> to be compatible with earlier IPFire versions.
>>> What do we have to be compatible with? Just IPFire’s behaviour from
>> last
>> year?
>>> I would be happy to break that if we have a good reason, because the
>> world sometimes moves on. What we have released 10 years ago probably
>> doesn’t have good defaults any more. Sometimes we have to revisit those
>> and sometimes we even have to be pushy with some changes if we want some
>> long-standing users to adopt them. IPFire users usually have no reason to
>> do a reinstall and we don’t want to leave the people behind who have been
>> running IPFire perfectly well for the last 10 years or even longer. So at
>> least I would like to discuss what we push a little harder.
>> As Adolf pointed out in the Forum,the WIKI stated the rational for not
>> logging was to protect flash memory from excessive use. I don't think that
>> is appropriate now.
>> 
>>>>>> In fact I think that IPFire's logging environment, which was
>> inherited
>>>>>> from IPCop needs a re-visit.
>>>>> Yes. But this is probably not a thing I would be looking at in IPFire
>> 2.
>>>>> I am also happy to think a little bit more outside of the box and see
>>>>> what we would fundamentally change in the future, but not implement in
>>>>> IPFire 2.
>>>>> 
>>>>>> I was pleased to see that a while back there was a discussion of
>>>>>> replacing sylogd with rsyslogd which I think would be a very useful
>>>>>> upgrade and enable kernel messages, like on most current linux
>> systems,
>>>>>> to be logged to alternative log files instead of everything ending up
>>>>>> in /var/log/ messages.
>>>>> The currentl syslog server could do that. The problem is more that we
>>>>> don’t have good tools to view the logs on the web UI. So there is
>> simply
>>>>> one place where most things go into and we will then filter it out
>> when
>>>>> users use the web UI. This is tremendously slow.
>>>>> 
>>>> I did look at this a while back and believe that with syslog all
>> kernel
>>>> messages must go to the same log file and cannot be filtered off as
>> can
>> be
>>>> done with rsyslog. I use this facility in rsyslog.d on my server to
>> filter
>>>> off SYN Flood logs to a separate log file with ":msg,contains". I
>> don't
>>>> believe that syslog has this option.
>>> Different messages could be sent to different “facilities” and then we
>> can filter them into individual files.
>>> I would not want to split firewall hits into different files though. I
>> just wouldn’t mind if firewall hits were no longer in /var/log/messages.
>> 
>> 
>> You need to be careful with the LOG* menus which would need to look in 2
>> or more places for historical logs and so the cgi's would need to be
>> modified.
>> 
>> I would however welcome kernel message filtering since it would allow my
>> use of a separate log directory for my auto-dnsbl program to monitor for
>> non-dropped incoming SMTP connections without appearing in 'messages' log
>> file.
>> 
>> 
>>>>>> The replacement with rsyslgd could also inhibit the "last message
>>>>>> repeated x times" in syslog and which are ignored by IPFire's Log
>> menu
>>>>>> and causes erroneous results in the 'Count' values.
>>>>> This might be a setting that we can turn off in the current
>>>>> implementation.
>>>>> 
>>>> I understand that this is hard coded into syslogd and can't be
>> changed.
>> In
>>>> rsyslog it defaults to "off".
>>>> 
>>>> https://www.rsyslog.com/doc/configuration/action/
>>>> rsconf1_repeatedmsgreduction.html
>>> What impact does this have for you? I never noticed any problem by
>> summarising the logs.
>> 
>> I came across this and it illustrates the point well:
>> 
>> https://dcid.me/notes/syslog-last-message-repeated-x-times-rant
>> 
>> and makes the point that kernel messages can be delayed by several seconds
>> and could affect any DOS filtering that may be in use.
>> 
>> The dshield iptables.pl script tries to recover the lost repeated messages
>> but looses the time-stamp.
>> 
>> When comparing the dshield logs with IPFire's logs there is invariably a
>> discrepancy with the IPFire's count sometimes considerably less especially
>> with multiple port-scans over a minute or so.
>> 
>> Any chance of looking again at rsyslog? IPFire is the only linux system I
>> know of still using syslog.
>> 
>> I seem to remember that Adolf had completed all the necessary changes to
>> introduce rsyslog a years or so ago but was not implemented,
> 
> It wasn't me. It was Erik Kapfer. I have looked at it to have a go at it but there have been a lot of other things that are on my plate that I am still working through and of course sometimes things come along that have to be addressed and jump ahead in the queue.

Story of my life :)

> Regards,
> 
> Adolf.
> 
>>  
>>>>> We could also think about moving all firewall hits into
>>>>> /var/log/firewall.log or something. That should free up enough in
>>>>> /var/log/messages to make debugging easier.
>>>>> 
>>>> That would be a sensible option, currently I filter out firewall hits
>> from
>>>> the messages.log with a perl regex.
>>>> 
>>>>>>> I would be happy to hear some more opinions on this from the list.
>>>>>>> 
>>>>>>> -Michael
>>>>>> I hope these comments are useful and would be pleased to help where I
>>>>>> can.
>>>>> Of course :)
>>>>> 
>>>>> -Michael
>>>>> 
>>>>> 
>>>>>> Rob
>>> -Michael
>> Rob