Proposal: Drop iptables logging rate-limit

Proposal: Drop iptables logging rate-limit

[Peter Müller]

[-- Attachment #1: Type: text/plain, Size: 1631 bytes --]

Hello *,

currently, the iptables configuration used in IPFire 2.x does not
log _every_ packet if logging is enabled for whatever reason, but
enforces a rate-limit:

> iptables -A LOG_DROP   -m limit --limit 10/minute -j LOG
(snip taken from /etc/init.d/firewall)

For several reasons, I consider this a bad idea. (Forgive me for
bringing up firewall issues in IPFire 2.x again. :-) )

First, this rate-limit is never mentioned in the firewall WebUI
or our documentation, thus being unintentional for most users
including me.

Second, it makes debugging very hard - I recently spent several
unpleasant days trying to fix a VoIP related network problem,
until I got not every packet dropped by IPFire was actually logged.
Especially for corner cases or non-deterministic issues, this
behaviour makes this more difficult.

Third, it is not compliant. Especially when it comes to post
mortem forensics, firewall logs are important. If you cannot
trust them since there is no way of telling whether a packet
was dropped and not logged, or never seen by the firewall machine,
its best to stop logging anything at all.

I therefore propose to drop iptables logging rate-limit in our
firewall configurations (which goes for IPFire 3.x as well).
Since my systems to not run on problematic hardware (ARM SoCs
with SD cards, crappy flash storage, etc.), I have no idea if
this will cause issues on some systems/platforms.

@All: Thoughts, please. Is anyone aware of potential trouble?

If not, I will send in a patch within this week.

Thanks, and best regards,
Peter Müller
-- 
The road to Hades is easy to travel.
	-- Bion of Borysthenes

Re: Proposal: Drop iptables logging rate-limit

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 2399 bytes --]

Hi,

> On 14 Jul 2019, at 15:56, Peter Müller <peter.mueller(a)ipfire.org> wrote:
> 
> Hello *,
> 
> currently, the iptables configuration used in IPFire 2.x does not
> log _every_ packet if logging is enabled for whatever reason, but
> enforces a rate-limit:
> 
>> iptables -A LOG_DROP   -m limit --limit 10/minute -j LOG
> (snip taken from /etc/init.d/firewall)
> 
> For several reasons, I consider this a bad idea. (Forgive me for
> bringing up firewall issues in IPFire 2.x again. :-) )

IPFire 2 is still being actively maintained, so feel free to do so.

> First, this rate-limit is never mentioned in the firewall WebUI
> or our documentation, thus being unintentional for most users
> including me.
> 
> Second, it makes debugging very hard - I recently spent several
> unpleasant days trying to fix a VoIP related network problem,
> until I got not every packet dropped by IPFire was actually logged.
> Especially for corner cases or non-deterministic issues, this
> behaviour makes this more difficult.
> 
> Third, it is not compliant. Especially when it comes to post
> mortem forensics, firewall logs are important. If you cannot
> trust them since there is no way of telling whether a packet
> was dropped and not logged, or never seen by the firewall machine,
> its best to stop logging anything at all.
> 
> I therefore propose to drop iptables logging rate-limit in our
> firewall configurations (which goes for IPFire 3.x as well).
> Since my systems to not run on problematic hardware (ARM SoCs
> with SD cards, crappy flash storage, etc.), I have no idea if
> this will cause issues on some systems/platforms.
> 
> @All: Thoughts, please. Is anyone aware of potential trouble?

I generally agree with your points. I am not sure where the number is even coming from. We have carried this over from the beginning of IPFire without thinking about it again.

The motivation why this is there is probably that logging is an expensive operation and that you can use this as a DoS vector and fill up somebody’s hard drive just by sending packets. Logs will grow a lot faster after making this change and maybe we want to have a backstop for this?

-Michael

> If not, I will send in a patch within this week.
> 
> Thanks, and best regards,
> Peter Müller
> -- 
> The road to Hades is easy to travel.
> 	-- Bion of Borysthenes

Re: Proposal: Drop iptables logging rate-limit

[Tim FitzGeorge]

[-- Attachment #1: Type: text/plain, Size: 2811 bytes --]

Hi,

On 15/07/2019 11:29, Michael Tremer wrote:
> Hi,
>
>> On 14 Jul 2019, at 15:56, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>
>> Hello *,
>>
>> currently, the iptables configuration used in IPFire 2.x does not
>> log _every_ packet if logging is enabled for whatever reason, but
>> enforces a rate-limit:
>>
>>> iptables -A LOG_DROP   -m limit --limit 10/minute -j LOG
>> (snip taken from /etc/init.d/firewall)
>>
>> For several reasons, I consider this a bad idea. (Forgive me for
>> bringing up firewall issues in IPFire 2.x again. :-) )
> IPFire 2 is still being actively maintained, so feel free to do so.
>
>> First, this rate-limit is never mentioned in the firewall WebUI
>> or our documentation, thus being unintentional for most users
>> including me.
>>
>> Second, it makes debugging very hard - I recently spent several
>> unpleasant days trying to fix a VoIP related network problem,
>> until I got not every packet dropped by IPFire was actually logged.
>> Especially for corner cases or non-deterministic issues, this
>> behaviour makes this more difficult.
>>
>> Third, it is not compliant. Especially when it comes to post
>> mortem forensics, firewall logs are important. If you cannot
>> trust them since there is no way of telling whether a packet
>> was dropped and not logged, or never seen by the firewall machine,
>> its best to stop logging anything at all.
>>
>> I therefore propose to drop iptables logging rate-limit in our
>> firewall configurations (which goes for IPFire 3.x as well).
>> Since my systems to not run on problematic hardware (ARM SoCs
>> with SD cards, crappy flash storage, etc.), I have no idea if
>> this will cause issues on some systems/platforms.
>>
>> @All: Thoughts, please. Is anyone aware of potential trouble?
> I generally agree with your points. I am not sure where the number is even coming from. We have carried this over from the beginning of IPFire without thinking about it again.
>
> The motivation why this is there is probably that logging is an expensive operation and that you can use this as a DoS vector and fill up somebody’s hard drive just by sending packets. Logs will grow a lot faster after making this change and maybe we want to have a backstop for this?
>
> -Michael

I think there probably needs to be some sort of limit to prevent the DoS
scenario, but 10/minute seems very small for modern hardware.  Perhaps
something could be done using hashlimit to prevent any one source IP
causing a DoS while still allowing dropped packets from other sources to
be logged.

Tim


>> If not, I will send in a patch within this week.
>>
>> Thanks, and best regards,
>> Peter Müller
>> -- 
>> The road to Hades is easy to travel.
>> 	-- Bion of Borysthenes
>

[PATCH] firewall: raise log rate limit to 10 packets per second

[Peter Müller]

[-- Attachment #1: Type: text/plain, Size: 3718 bytes --]

Previous setting was to log 10 packets per minute for each
event logging is turned on. This made debugging much harder,
as the limit was rather strict and chances of dropping a
packet without logging it were good.

This patch changes the log rate limit to 10 packets per
second per event, to avoid DoS attacks against the log file.
I plan to drop log rate limit entirely in future changes,
if a better solution for this attack vector is available.

Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org>
Cc: Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk>
Cc: Michael Tremer <michael.tremer(a)ipfire.org>
---
 config/rootfiles/core/135/filelists/files |  1 +
 src/initscripts/system/firewall           | 14 +++++++-------
 2 files changed, 8 insertions(+), 7 deletions(-)

diff --git a/config/rootfiles/core/135/filelists/files b/config/rootfiles/core/135/filelists/files
index d8df9f65b..e5943ddc0 100644
--- a/config/rootfiles/core/135/filelists/files
+++ b/config/rootfiles/core/135/filelists/files
@@ -5,6 +5,7 @@ var/ipfire/langs
 etc/unbound/root.hints
 etc/rc.d/helper/azure-setup
 etc/rc.d/init.d/cloud-init
+etc/rc.d/init.d/firewall
 etc/rc.d/init.d/functions
 etc/rc.d/init.d/networking/red.down/05-remove-dns-forwarders
 etc/rc.d/init.d/partresize
diff --git a/src/initscripts/system/firewall b/src/initscripts/system/firewall
index b3483a744..ec396c708 100644
--- a/src/initscripts/system/firewall
+++ b/src/initscripts/system/firewall
@@ -34,20 +34,20 @@ iptables_init() {
 
 	# Empty LOG_DROP and LOG_REJECT chains
 	iptables -N LOG_DROP
-	iptables -A LOG_DROP   -m limit --limit 10/minute -j LOG
+	iptables -A LOG_DROP   -m limit --limit 10/second -j LOG
 	iptables -A LOG_DROP   -j DROP
 	iptables -N LOG_REJECT
-	iptables -A LOG_REJECT -m limit --limit 10/minute -j LOG
+	iptables -A LOG_REJECT -m limit --limit 10/second -j LOG
 	iptables -A LOG_REJECT -j REJECT
 
 	# This chain will log, then DROPs packets with certain bad combinations
 	# of flags might indicate a port-scan attempt (xmas, null, etc)
 	iptables -N PSCAN
 	if [ "$DROPPORTSCAN" == "on" ]; then
-		iptables -A PSCAN -p tcp  -m limit --limit 10/minute -j LOG --log-prefix "DROP_TCP Scan " -m comment --comment "DROP_TCP PScan"
-		iptables -A PSCAN -p udp  -m limit --limit 10/minute -j LOG --log-prefix "DROP_UDP Scan " -m comment --comment "DROP_UDP PScan"
-		iptables -A PSCAN -p icmp -m limit --limit 10/minute -j LOG --log-prefix "DROP_ICMP Scan " -m comment --comment "DROP_ICMP PScan"
-		iptables -A PSCAN -f      -m limit --limit 10/minute -j LOG --log-prefix "DROP_FRAG Scan " -m comment --comment "DROP_FRAG PScan"
+		iptables -A PSCAN -p tcp  -m limit --limit 10/second -j LOG --log-prefix "DROP_TCP Scan " -m comment --comment "DROP_TCP PScan"
+		iptables -A PSCAN -p udp  -m limit --limit 10/second -j LOG --log-prefix "DROP_UDP Scan " -m comment --comment "DROP_UDP PScan"
+		iptables -A PSCAN -p icmp -m limit --limit 10/second -j LOG --log-prefix "DROP_ICMP Scan " -m comment --comment "DROP_ICMP PScan"
+		iptables -A PSCAN -f      -m limit --limit 10/second -j LOG --log-prefix "DROP_FRAG Scan " -m comment --comment "DROP_FRAG PScan"
 	fi
 	iptables -A PSCAN -j DROP -m comment --comment "DROP_PScan"
 
@@ -55,7 +55,7 @@ iptables_init() {
 	# that's not covered above, may just be a broken windows machine
 	iptables -N NEWNOTSYN
 	if [ "$DROPNEWNOTSYN" == "on" ]; then
-		iptables -A NEWNOTSYN  -m limit --limit 10/minute -j LOG  --log-prefix "DROP_NEWNOTSYN "
+		iptables -A NEWNOTSYN  -m limit --limit 10/second -j LOG  --log-prefix "DROP_NEWNOTSYN "
 	fi
 	iptables -A NEWNOTSYN  -j DROP -m comment --comment "DROP_NEWNOTSYN"
 
-- 
2.16.4

Re: [PATCH] firewall: raise log rate limit to 10 packets per second

[Horace Michael]

[-- Attachment #1: Type: text/plain, Size: 5023 bytes --]

Hello,
Would it be a good ideea to add root.servers in the CUSTOMINPUT and CUSTOMFORWARD chains, acction ACCEPT just in case that somebody uses the GeoIP feature and blocks the countries where these servers resides?

Ex: I just blocked Japan and Sweden in my Internet facing IPFire and rest IPFire boxes that uses it tries to access all ROOT Servers and can't reach the ones from blocked countries.
First line is m.root-servers.net located in Japan and second line is i.root-servers.net located in Sweden:

Jul 29 22:16:27 internet_ipfire kernel: FORWARDFW IN=green0 OUT=ppp0 MAC=*************** SRC=a.b.c.d DST=202.12.27.33 LEN=75 TOS=0x00 PREC=0x00 TTL=63 ID=966 PROTO=UDP SPT=58062 DPT=53 LEN=55
Jul 29 22:16:27 internet_ipfire kernel: FORWARDFW IN=green0 OUT=ppp0 MAC=*************** SRC=a.b.c.d DST=192.36.148.17 LEN=75 TOS=0x00 PREC=0x00 TTL=63 ID=3289 PROTO=UDP SPT=30228 DPT=53 LEN=55

Horace

> Sent: Monday, July 29, 2019 at 11:00 PM
> From: "Peter Müller" <peter.mueller(a)ipfire.org>
> To: "IPFire: Development-List" <development(a)lists.ipfire.org>
> Subject: [PATCH] firewall: raise log rate limit to 10 packets per second
>
> Previous setting was to log 10 packets per minute for each
> event logging is turned on. This made debugging much harder,
> as the limit was rather strict and chances of dropping a
> packet without logging it were good.
> 
> This patch changes the log rate limit to 10 packets per
> second per event, to avoid DoS attacks against the log file.
> I plan to drop log rate limit entirely in future changes,
> if a better solution for this attack vector is available.
> 
> Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org>
> Cc: Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk>
> Cc: Michael Tremer <michael.tremer(a)ipfire.org>
> ---
>  config/rootfiles/core/135/filelists/files |  1 +
>  src/initscripts/system/firewall           | 14 +++++++-------
>  2 files changed, 8 insertions(+), 7 deletions(-)
> 
> diff --git a/config/rootfiles/core/135/filelists/files b/config/rootfiles/core/135/filelists/files
> index d8df9f65b..e5943ddc0 100644
> --- a/config/rootfiles/core/135/filelists/files
> +++ b/config/rootfiles/core/135/filelists/files
> @@ -5,6 +5,7 @@ var/ipfire/langs
>  etc/unbound/root.hints
>  etc/rc.d/helper/azure-setup
>  etc/rc.d/init.d/cloud-init
> +etc/rc.d/init.d/firewall
>  etc/rc.d/init.d/functions
>  etc/rc.d/init.d/networking/red.down/05-remove-dns-forwarders
>  etc/rc.d/init.d/partresize
> diff --git a/src/initscripts/system/firewall b/src/initscripts/system/firewall
> index b3483a744..ec396c708 100644
> --- a/src/initscripts/system/firewall
> +++ b/src/initscripts/system/firewall
> @@ -34,20 +34,20 @@ iptables_init() {
>  
>  	# Empty LOG_DROP and LOG_REJECT chains
>  	iptables -N LOG_DROP
> -	iptables -A LOG_DROP   -m limit --limit 10/minute -j LOG
> +	iptables -A LOG_DROP   -m limit --limit 10/second -j LOG
>  	iptables -A LOG_DROP   -j DROP
>  	iptables -N LOG_REJECT
> -	iptables -A LOG_REJECT -m limit --limit 10/minute -j LOG
> +	iptables -A LOG_REJECT -m limit --limit 10/second -j LOG
>  	iptables -A LOG_REJECT -j REJECT
>  
>  	# This chain will log, then DROPs packets with certain bad combinations
>  	# of flags might indicate a port-scan attempt (xmas, null, etc)
>  	iptables -N PSCAN
>  	if [ "$DROPPORTSCAN" == "on" ]; then
> -		iptables -A PSCAN -p tcp  -m limit --limit 10/minute -j LOG --log-prefix "DROP_TCP Scan " -m comment --comment "DROP_TCP PScan"
> -		iptables -A PSCAN -p udp  -m limit --limit 10/minute -j LOG --log-prefix "DROP_UDP Scan " -m comment --comment "DROP_UDP PScan"
> -		iptables -A PSCAN -p icmp -m limit --limit 10/minute -j LOG --log-prefix "DROP_ICMP Scan " -m comment --comment "DROP_ICMP PScan"
> -		iptables -A PSCAN -f      -m limit --limit 10/minute -j LOG --log-prefix "DROP_FRAG Scan " -m comment --comment "DROP_FRAG PScan"
> +		iptables -A PSCAN -p tcp  -m limit --limit 10/second -j LOG --log-prefix "DROP_TCP Scan " -m comment --comment "DROP_TCP PScan"
> +		iptables -A PSCAN -p udp  -m limit --limit 10/second -j LOG --log-prefix "DROP_UDP Scan " -m comment --comment "DROP_UDP PScan"
> +		iptables -A PSCAN -p icmp -m limit --limit 10/second -j LOG --log-prefix "DROP_ICMP Scan " -m comment --comment "DROP_ICMP PScan"
> +		iptables -A PSCAN -f      -m limit --limit 10/second -j LOG --log-prefix "DROP_FRAG Scan " -m comment --comment "DROP_FRAG PScan"
>  	fi
>  	iptables -A PSCAN -j DROP -m comment --comment "DROP_PScan"
>  
> @@ -55,7 +55,7 @@ iptables_init() {
>  	# that's not covered above, may just be a broken windows machine
>  	iptables -N NEWNOTSYN
>  	if [ "$DROPNEWNOTSYN" == "on" ]; then
> -		iptables -A NEWNOTSYN  -m limit --limit 10/minute -j LOG  --log-prefix "DROP_NEWNOTSYN "
> +		iptables -A NEWNOTSYN  -m limit --limit 10/second -j LOG  --log-prefix "DROP_NEWNOTSYN "
>  	fi
>  	iptables -A NEWNOTSYN  -j DROP -m comment --comment "DROP_NEWNOTSYN"
>  
> -- 
> 2.16.4
> 
>