Re: Logging Locationblock packets

[Adolf Belka <adolf.belka@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 21175 bytes --]

Hi Rob,

On 23/11/2024 15:36, Rob Brewer wrote:
> On Sat, 23 Nov 2024 10:26:07 +0100, Michael Tremer wrote:
>
>> Hello,
>>
>>> On 21 Nov 2024, at 00:08, Rob Brewer <ipfire-devel(a)grantura.co.uk>
> wrote:
>>> Hi Michael and Adolf,
>>>
>>> I'll combine my reply to both of you.
>>>
>>> On Wed, 20 Nov 2024 15:21:28 +0000, Michael Tremer wrote:
>>>
>>>> Hello,
>>>>
>>>>> On 20 Nov 2024, at 13:36, Rob Brewer <ipfire-devel(a)grantura.co.uk>
>>>>> wrote:
>>>>>
>>>>> On Tue, 19 Nov 2024 17:10:24 +0000, Michael Tremer wrote:
>>>>>
>>>>> Hi Michael, thank for your response, I was beginning to think there
> was
>>>>> little interest in this topic.
>>>> I was traveling and so have a little bit of a back log as usual :)
>>>>
>>>> I can’t speak for the others, though.
>>>>
>>> Not a problem, I should have said that I previously raised this a
> topic
> on
>>> the forum and Adolf suggested that I raised it on the mailing list as
>>> well.
>> Yes, I really prefer this here, because the forum is very overwhelming
> sometimes. There is too many threads, too many people… Mailing lists are
> great for development.
>
> That's fine, one query though you don't seem to snip older parts of the
> post, is this your preference?
>
>>>>> See below...
>>>>>
>>>>>> Hello Rob,
>>>>>>
>>>>>> Good to hear from you again.
>>>>>>
>>>>>>> On 9 Nov 2024, at 14:00, Rob Brewer <ipfire-devel(a)grantura.co.uk>
>>>>>>> wrote:
>>>>>>>
>>>>>>> As  a contributor to the SANS ISC,  I email my IPTABLES logs hourly
>>>>>>> to the Dshield database, however the default IPFire IPTABLES logs
> are
>>>>>>> incomplete since packets dropped by Locationblock are not logged to
>>>>>>> syslog.
>>>>>> This is correct.
>>>>>>
>>>>>>> As a workaround I edit the locationblock function of rules.pl to
> add
>>>>>>> a IPTABLES rule to log the dropped packets. This edit does not of
>>>>>>> course survive any Core-Updates that upgrade rules.pl and the edit
>>>>>>> needs to be re-applied.
>>>>>>>
>>>>>>> In addition, unless the Locationblock packets are logged to syslog
>>>>>>> the data displayed in the Firewall logs sections are incomplete as
>>>>>>> the packets dropped by Location Block are not displayed.
>>>>>>>
>>>>>>> I understand that the original intension of the Location Block
>>>>>>> feature was to reduce the amount of log messages on installations
>>>>>>> running on extremely cheap flash storage. I suspect that these
>>>>>>> installations are now almost zero especially since IPFire no longer
>>>>>>> supports 32 bit versions.
>>>>>>>
>>>>>>> The obvious solution to this to make Location Block logging
>>>>>>> selectable as a firewall option and my quick look at making a
> patches
>>>>>>> for this would seem to be fairly simple task to rules.pl and and
>>>>>>> optionsfw.cgi.
>>>>>>>
>>>>>>> I would be happy to provide the necessary patches should this be
>>>>>>> acceptable.
>>>>>> I think a couple of things have changed since we introduced this
>>>>>> feature and we have also learned a couple of lessons here:
>>>>>>
>>>>>> * Yes it is true that this was introduced to keep the logs smaller,
>>>>>> but not necessarily to avoid using storage, mainly to keep them
>>>>>> readable by a human. I was always a bit doubtful about this and
>>>>>> believe that apart from this, there is no more value in a general
>>>>>> location blocking feature. The firewall will drop those packets
>>>>>> anyway.
>>>>>>
>>>>>>
>>>>> IPFire does a very good job in protecting the internal network from
>>>>> unwanted probes from the WAN, and I agree that for most users,
> location
>>>>> block does not enhance this protection.
>>>>>
>>>>> As I run a mail server, I use locationblock as part of my filtering
> to
>>>>> prevent bad actors from accessing my mail server and I think would be
>>>>> useful for anyone who are running servers on their networks.
>>>>>
>>>>> I don't like blocking traffic, but I have witnessed over the last few
>>>>> years a substantial rise in abuse from certain countries from where I
>>>>> would never expect to receive emails and locationblock works very
> well
>>>>> for me in filtering out those unwanted connections.
>>>>>
>>>>> I currently have 21 countries blocked including XD Hostile networks.
>>>>> All of these blocked countries have been seen to support abusive
>>>>> networks.
>>>>>
>>>>> It would be possible to generate individual rules for these networks
>>>>> which in some cases I do with Banish, but it is sometimes easier just
>>>>> to drop the whole country.
>>>> Easy yes. I am just not sure what we are actually getting out of this.
>>>>
>>>> I don’t like to become political on this list and so I won’t, but I
> know
>>>> that some people hold the following beliefs:
>>>>
>>>> * There are “bad” countries out there. In the western world those are
>>>> Russia, China, and more that fall into this kind of “category”.
> Nothing
>>>> is going to stop them to open up an account on AWS and suddenly they
>>>> have access to resources anywhere in the world and so actually, if
> those
>>>> players were up to anything “bad”, they can come out of any IP space
>>>> they want. There is also Tor which allows this.
>>>>
>>> I have been submitting my iptables logs to dshield for over 20 years
> and
>>> in the early days I would be sending a lot less than 1000 a day (this
> was
>>> with IPCop of course).
>>>
>>> Over the last few years the number of firewall iptables logs have
>>> escalated until it had reached dshield's daily limit of 100,000 /day
> and
>>> I started looking at ways to limit the logging of some of the 'bad
> actors'
>>> firewall probes.
>> What do you use to send them the logs? The Python script or one of
> those
> alternatives?
>
> I use a modified Perl script (iptables.pl) which is about 20 years old was
> originally written for Smoothwall and Ipcop1. This was part of the Logsend
> add-on and I modified for IPCop2 about 15 years ago.
>
> Now modified for IPFire and using DMA email instead of the original
> sendEmail. I've uploaded my version to people.ipfire.org/~helix/dshield.
>
> I had intended to write an install/uninstall script but is in the todo
> list. Do you think it may be of interest to anyone if I do?
>
>>>> * Looking at actual data, the bullet proof ISPs that annoy us are not
>>>> that far away. They are hosted in Europe and very prominently so in
> the
>>>> Netherlands - a “good” country.
>>>>
>>> I use Banish for this as it can block by ASN and can target bullet
> proof
>>> ISPs not only in NL but also in the USA and elsewhere. Using the
> 'location
>>> function' the blocked ASN's are updated when the database is updated.
>>>
>>>> This is just an example to explain that we all have our idea of the
>>>> world in our head, but the Internet doesn’t know these borders.
>>>> Geopolitics is complicated and I don’t think that this is a tool that
>>>> helps to fight this kind of problem. I hope this is clear what I am
>>>> trying to say.
>>>>
>>> Yes my approach is to use multiple tools and Locationblock is just one
> of
>>> the tools I am using. Maybe my approach is over the top though.
>> Probably not massively, but maybe slightly.
>>
>> Wouldn’t it be nicer to have everything in one place instead of having
> multiple tools that work together in concert?
> I think that would be a lot of work, would you consider that for IPFire2
> or 3?
>
>>>> I personally trust in Spamhaus which is feeding data into our “Hostile
>>>> Networks” blocker and they are looking at the actual traffic that they
>>>> see from those subnets that they are blocking. They have a clear set
> of
>>>> rules and they don’t look just at the country and say “ah yes, the bad
>>>> guys again”.
>>>>
>>> For emails I also drop packets if listed in a number of DNS blackhole
>>> lists using Auto-Dnsbl which I adapted for IPFire. See:
>>>
>>> https://people.ipfire.org/~helix/auto-dnsbl/README
>>> for info.
>>>
>>> I have been using this for about a year and can confirm that it is
> very
>>> effective in removing spammers.
>>>
>>>> I have stated this recently in the RPZ discussion that I feel that we
>>>> are lacking good sources for this kind of data. For blocking anything.
>>>> Loads of people believe “more is more” and to be fair… once you get to
>>>> blocking 90% of the internet you probably have blocked 90% of the
> noise.
>>>> However this won’t do anything for targeted attacks. I personally
> would
>>>> rather have more quality here.
>>>>
>>> I only use a few blocklists with ipblocklist and rely mainly on my
> locally
>>> derived  data from failed smtp connections or from data from dshield's
>>> daily report.
>>>
>>>>>> * People seem to misunderstand this feature: The block as many
>>>>>> countries as they can click on and think that this makes things
> “more
>>>>>> secure”,
>>>>>> which of course it doesn’t.
>>>>>>
>>>>>>
>>>>> I think this is education and I suspect that very few of these people
>>>>> are running servers on their system and need location block.
>>>> I am personally not here to educate people. I am happy to help out and
>>>> give my opinion, but I suppose considering that people think as I have
>>>> outlined above, who am I to educate them?
>>>>
>>> Sorry I didn't express my reply well, I agree that IPFire users should
> be
>>> responsible for their own actions.
>> Naturally. But we should also not give them sharp knives when we don’t
> expect them to use them properly.
>>>> It is just very regretful to me when people are ab-using a feature
> just
>>>> to “feel more safe”.
>>>>
>>>>>> * What I personally like as a feature is to apply location filters
> to
>>>>>> individual rules. That allows to apply rate limits in a few places
> and
>>>>>> easily select some larger groups of potentially less interesting
>>>>>> traffic.
>>>>> I agree and is similar to what I am doing for ports 25 and 587
> although
>>>>> with location block.
>>>>>
>>>>>
>>>>>> * Back when this was introduced there was also a lot less use of
>>>>>> botnets as we know them now. Pretty much all recent DoS attacks I
>>>>>> remember have been coming from all over the world because systems
>>>>>> where compromised wherever they were. This renders the filter
>>>>>> absolutely useless.
>>>>> For this, I am currently testing a dynamic block list on my system
>>>>> which adds an entry to an ipset when an ip address exceeds a
> threshold
>>>>> in the previous 24 hours. Currently there are about 7000 entries in
>>>>> this set which have been detected over the previous month.  Not for
>>>>> everyone because it needs a Whitelist to prevent some 'push'
>>>>> technologies such as Facebook and to protect from bulk inputs such as
>>>>> mailing lists. I also blacklist a large number of ip addresses which
>>>>> are associated with port scanners.
>>>>>
>>>>> This would work well as a botnet filter but it needs an understanding
>>>>> of the blocking process, so it would not be suitable for everyone.
>>>> We have talked about this on the last team call and came to the
>>>> conclusion that the whole set of blocking features is a total mess.
>>>> There are several places where things can be blocked, they all heavily
>>>> overlap. Not to be going on about Spamhaus again, but the DROP list is
>>>> in the “Hostile Networks” feature, there is a special country code for
>>>> it, pretty much every IPS ruleset contains it, the IP blocklist
> feature
>>>> supports it. Now there are people throwing this all into DNS as well.
> I
>>>> believe this is all getting a little bit out of hand.
>>>>
>>> I don't use the IPS mainly due to bad experiences when trying to use
> Snort
>>> with IPCop. I find it difficult to set up and select the appropriate
>>> Rulsets for my requirements. I suspect the IPS would be very effective
> if
>>> I could get it set up correctly.
>> This has massively improved in recent times. And it is FAST!
>>
> I'll have another look, but my current set-up is working well, so I'm
> reluctant to change.
>
>>>> I personally would prefer *one* place and we really make this work
> well.
>>>> Some people find the IPS has too much overhead - I am beginning to
>>>> disagree.
>>>>
>>>> Some people add all sorts of blacklists that are clearly for
>>>> experimental/research use and suddenly complain about things not
> working
>>>> anymore. It is hard to figure out what is actually blocking traffic
> when
>>>> there are about five places to look for. The result often is to
> disable
>>>> the IPS, then something else, and after a while all of this is turned
>>>> off and the firewall is configured to a bare minimum. We don’t want
>>>> people to run it like that.
>>>>
>>>>>> * Infrastructure is becoming more centralised. Some people attempt
> to
>>>>>> block the US and pretty much nothing works any more. Well…
>>>>>>
>>>>>>
>>>>> :)
>>>>>
>>>>>> Therefore I am happy to rethink this feature - personally I would
> even
>>>>>> remove it entirely, but I don’t like to handle the stress about this
>>>>>> now.
>>>>> No I think you should keep it. Locationblock is very useful, but
> needs
>>>>> to be used in a considered way. I could live without it but its
>>>>> availability makes my life easier. Also being updated on a weekly
> basis
>>>>> makes keeping track of abusers easier.
>>>> We issue a new location database once a day, but the IPFire systems
>>>> update it only once a week. We deemed that as sufficient as there are
>>>> not that many changes from one version to another one.
>>>>
>>>>>> I think this should log now. I don’t think there is any legitimate
>>>>>> reason to disable firewall logging at all. This is the paper trail
>>>>>> that you want when debugging anything or doing forensics on an
> attack.
>>>>>> Certainly bad flash is not a valid reason for me.
>>>>>>
>>>>>>
>>>>> I have started generating some patches to make 'locationblock
> logging'
>>>>> as a selectable feature and could post them here in the next day or
> so
>>>>> if of any use.
>>>> What would this basically do? Log it as a simple rule for all, or do
> you
>>>> include the selected country? I think the latter would make more
> sense.
>>> OK - I have modified location-block.cgi to add an additional checkbox
> at
>>> the top of the page titled "Log dropped packets", very similar to
>>> ipblocklist.cgi. This adds a entry to the /var/ipfire/firewall/
>>> locationblock settings file ON or OFF.
>>>
>>> In /usr/lib/firewall/rules.pl I add an additional Iptables
> LOCATIONBLOCK
>>> log rule with the prefix 'LOCBLOCK-$location '. This is selectable
> from
>>> the Log dropped packets entry in the locationblock settings file.
>>>
>>> This generates kernel messages such as:
>>>
>>> LOCBLOCK-CN IN=ppp0 OUT=ppp0 MAC= SRC=114.139.42.71 DST=81.187.175.103
>>> LEN=60 TOS=0x00 PREC=0x00 TTL=47 ID=34664 DF PROTO=TCP SPT=46837
> DPT=23
>>> WINDOW=29040 RES=0x00 SYN URGP=0
>>>
>>> As you can see the prefix contains the country code of the dropped
>>> packets.
>> Cool. So I think coding-wise this is pretty straight forward.
>>
>> The more crucial question for me is below…
>>
> Do have a look at the patches I posted and let me have any comments. If
> it's not needed and we can add locationblock logging without selection
> then that's what I have been doing since LB was first implemented.
>
>>>> What is the rationale to turn this off? Why don’t we turn this on for
>>>> everybody all of the time?
>>>>
>>> I would be happy with that I was suggesting that we use selectable
> logging
>>> to be compatible with earlier IPFire versions.
>> What do we have to be compatible with? Just IPFire’s behaviour from
> last
> year?
>> I would be happy to break that if we have a good reason, because the
> world sometimes moves on. What we have released 10 years ago probably
> doesn’t have good defaults any more. Sometimes we have to revisit those
> and sometimes we even have to be pushy with some changes if we want some
> long-standing users to adopt them. IPFire users usually have no reason to
> do a reinstall and we don’t want to leave the people behind who have been
> running IPFire perfectly well for the last 10 years or even longer. So at
> least I would like to discuss what we push a little harder.
> As Adolf pointed out in the Forum,the WIKI stated the rational for not
> logging was to protect flash memory from excessive use. I don't think that
> is appropriate now.
>
>>>>> In fact I think that IPFire's logging environment, which was
> inherited
>>>>> from IPCop needs a re-visit.
>>>> Yes. But this is probably not a thing I would be looking at in IPFire
> 2.
>>>> I am also happy to think a little bit more outside of the box and see
>>>> what we would fundamentally change in the future, but not implement in
>>>> IPFire 2.
>>>>
>>>>> I was pleased to see that a while back there was a discussion of
>>>>> replacing sylogd with rsyslogd which I think would be a very useful
>>>>> upgrade and enable kernel messages, like on most current linux
> systems,
>>>>> to be logged to alternative log files instead of everything ending up
>>>>> in /var/log/ messages.
>>>> The currentl syslog server could do that. The problem is more that we
>>>> don’t have good tools to view the logs on the web UI. So there is
> simply
>>>> one place where most things go into and we will then filter it out
> when
>>>> users use the web UI. This is tremendously slow.
>>>>
>>> I did look at this a while back and believe that with syslog all
> kernel
>>> messages must go to the same log file and cannot be filtered off as
> can
> be
>>> done with rsyslog. I use this facility in rsyslog.d on my server to
> filter
>>> off SYN Flood logs to a separate log file with ":msg,contains". I
> don't
>>> believe that syslog has this option.
>> Different messages could be sent to different “facilities” and then we
> can filter them into individual files.
>> I would not want to split firewall hits into different files though. I
> just wouldn’t mind if firewall hits were no longer in /var/log/messages.
>
>
> You need to be careful with the LOG* menus which would need to look in 2
> or more places for historical logs and so the cgi's would need to be
> modified.
>
> I would however welcome kernel message filtering since it would allow my
> use of a separate log directory for my auto-dnsbl program to monitor for
> non-dropped incoming SMTP connections without appearing in 'messages' log
> file.
>
>
>>>>> The replacement with rsyslgd could also inhibit the "last message
>>>>> repeated x times" in syslog and which are ignored by IPFire's Log
> menu
>>>>> and causes erroneous results in the 'Count' values.
>>>> This might be a setting that we can turn off in the current
>>>> implementation.
>>>>
>>> I understand that this is hard coded into syslogd and can't be
> changed.
> In
>>> rsyslog it defaults to "off".
>>>
>>> https://www.rsyslog.com/doc/configuration/action/
>>> rsconf1_repeatedmsgreduction.html
>> What impact does this have for you? I never noticed any problem by
> summarising the logs.
>
> I came across this and it illustrates the point well:
>
> https://dcid.me/notes/syslog-last-message-repeated-x-times-rant
>
> and makes the point that kernel messages can be delayed by several seconds
> and could affect any DOS filtering that may be in use.
>
> The dshield iptables.pl script tries to recover the lost repeated messages
> but looses the time-stamp.
>
> When comparing the dshield logs with IPFire's logs there is invariably a
> discrepancy with the IPFire's count sometimes considerably less especially
> with multiple port-scans over a minute or so.
>
> Any chance of looking again at rsyslog? IPFire is the only linux system I
> know of still using syslog.
>
> I seem to remember that Adolf had completed all the necessary changes to
> introduce rsyslog a years or so ago but was not implemented,

It wasn't me. It was Erik Kapfer. I have looked at it to have a go at it but there have been a lot of other things that are on my plate that I am still working through and of course sometimes things come along that have to be addressed and jump ahead in the queue.

Regards,

Adolf.

>   
>
>>>> We could also think about moving all firewall hits into
>>>> /var/log/firewall.log or something. That should free up enough in
>>>> /var/log/messages to make debugging easier.
>>>>
>>> That would be a sensible option, currently I filter out firewall hits
> from
>>> the messages.log with a perl regex.
>>>
>>>>>> I would be happy to hear some more opinions on this from the list.
>>>>>>
>>>>>> -Michael
>>>>> I hope these comments are useful and would be pleased to help where I
>>>>> can.
>>>> Of course :)
>>>>
>>>> -Michael
>>>>
>>>>
>>>>> Rob
>> -Michael
> Rob