* [PATCH 0/5] ipblacklist: IP Address Blacklists
@ 2019-11-25 20:13 Tim FitzGeorge
2019-11-25 20:13 ` [PATCH 1/5] ipblacklist: Main script Tim FitzGeorge
` (6 more replies)
0 siblings, 7 replies; 35+ messages in thread
From: Tim FitzGeorge @ 2019-11-25 20:13 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 3243 bytes --]
Implements downloading of IP address blacklists and implementing
them as IPSets. A separate IPSet is used for each blacklist; this
simplifies handling of overlaps between different lists. Traffic
to or from the red0/ppp0 interface is checked against the IPSets.
The check is placed before the IPS check as the IPSet check is
much lighter on CPU use which means that overall CPU use is
reduced.
The available lists are defined in a separate file. A WUI page
allows the desired lists to be enabled and the interval between
checks for updates to be defined. A minimum update check interval
is defined for each blacklist in the definition file.
Optionally, an automatically updating blacklist can be enabled.
This adds addresses to an IPSet if the rate of packets dropped by
the default red0/ppp0 input policy exceeds a user defined threshold.
The addresses are kept in the IPSet until a user defined period
without packets from the blocked address has passed.
Tim FitzGeorge (5):
ipblacklist: Main script
ipblacklist: WUI and language file
ipblacklist: Ancillary files
ipblacklist: Modifications to system
ipblacklist: Build infrastructure
config/backup/backup.pl | 1 +
config/backup/include | 2 +
config/firewall/firewall-policy | 5 +
config/ipblacklist/sources | 151 +++
config/logwatch/ipblacklist | 103 ++
config/logwatch/ipblacklist.conf | 34 +
config/menu/50-firewall.menu | 5 +
config/rootfiles/common/aarch64/stage2 | 1 +
config/rootfiles/common/configroot | 2 +
config/rootfiles/common/ipblacklist-sources | 1 +
config/rootfiles/common/logwatch | 2 +
config/rootfiles/common/misc-progs | 2 +
config/rootfiles/common/stage2 | 1 +
config/rootfiles/common/web-user-interface | 1 +
config/rootfiles/common/x86_64/stage2 | 1 +
html/cgi-bin/ipblacklist.cgi | 725 +++++++++++++
html/cgi-bin/logs.cgi/log.dat | 2 +
langs/en/cgi-bin/en.pl | 31 +
lfs/configroot | 4 +-
lfs/ipblacklist-sources | 53 +
lfs/logwatch | 2 +
make.sh | 11 +-
src/initscripts/system/firewall | 20 +
src/misc-progs/Makefile | 2 +-
src/misc-progs/getipsetstat.c | 28 +
src/misc-progs/ipblacklistctrl.c | 52 +
src/scripts/ipblacklist | 1558 +++++++++++++++++++++++++++
27 files changed, 2792 insertions(+), 8 deletions(-)
create mode 100644 config/ipblacklist/sources
create mode 100644 config/logwatch/ipblacklist
create mode 100644 config/logwatch/ipblacklist.conf
create mode 100644 config/rootfiles/common/ipblacklist-sources
create mode 100644 html/cgi-bin/ipblacklist.cgi
create mode 100644 lfs/ipblacklist-sources
create mode 100644 src/misc-progs/getipsetstat.c
create mode 100644 src/misc-progs/ipblacklistctrl.c
create mode 100755 src/scripts/ipblacklist
--
2.16.4
^ permalink raw reply [flat|nested] 35+ messages in thread
* [PATCH 1/5] ipblacklist: Main script
2019-11-25 20:13 [PATCH 0/5] ipblacklist: IP Address Blacklists Tim FitzGeorge
@ 2019-11-25 20:13 ` Tim FitzGeorge
2019-11-25 20:13 ` [PATCH 2/5] ipblacklist: WUI and language file Tim FitzGeorge
` (5 subsequent siblings)
6 siblings, 0 replies; 35+ messages in thread
From: Tim FitzGeorge @ 2019-11-25 20:13 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 47086 bytes --]
Responsible for downloading blacklists and creating/modifying IPSets
Does all work involving creating, deleting and chaging IPTables and
IPSets.
Signed-off-by: Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk>
---
src/scripts/ipblacklist | 1558 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 1558 insertions(+)
create mode 100755 src/scripts/ipblacklist
diff --git a/src/scripts/ipblacklist b/src/scripts/ipblacklist
new file mode 100755
index 000000000..b3f8048d9
--- /dev/null
+++ b/src/scripts/ipblacklist
@@ -0,0 +1,1558 @@
+#! /usr/bin/perl
+
+############################################################################
+# #
+# IP Address blocklists for IPFire #
+# #
+# This is free software; you can redistribute it and/or modify #
+# it under the terms of the GNU General Public License as published by #
+# the Free Software Foundation; either version 3 of the License, or #
+# (at your option) any later version. #
+# #
+# This is distributed in the hope that it will be useful, #
+# but WITHOUT ANY WARRANTY; without even the implied warranty of #
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the #
+# GNU General Public License for more details. #
+# #
+# You should have received a copy of the GNU General Public License #
+# along with IPFire; if not, write to the Free Software #
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA #
+# #
+# Copyright (C) 2018 - 2019 The IPFire team #
+# #
+############################################################################
+# #
+# This script use a file containing blacklist details in #
+# /var/ipfire/ipblacklist/sources as well as #
+# /var/ipfire/ipblacklistsettings containing an enable/disable flag for #
+# each source. #
+# #
+# Two IPTables chains are used, IPBLACKLISTREDIN and IPBLACKLISTREDOUT, #
+# which are inserted into the main INPUT, OUTPUT and FORWARD chains. #
+# #
+# For each blacklist that is loaded, a chain is created to optionally log #
+# and then to drop matching packets. An IPSet is created containing the #
+# addresses or networks blocked by the blacklist, and then rules are added #
+# to the IPBLACKLISTREDIN and IPBLACKLISTREDOUT chains to jump to this #
+# chain if appropriate packet list matches in the set. #
+# #
+# When checking for updates, the modification time is read for each source #
+# and if necessary the list is downloaded. The downloaded list is #
+# compared to the existing IPSet contents and entries created or deleted #
+# as necessary. #
+# #
+############################################################################
+
+use strict;
+use warnings;
+
+use Carp;
+use Sys::Syslog qw(:standard :macros);
+use HTTP::Request;
+use LWP::UserAgent;
+
+require "/var/ipfire/general-functions.pl";
+
+############################################################################
+# Configuration variables
+#
+# These variables give the locations of various files used by this script
+############################################################################
+
+my $settingsdir = "/var/ipfire/ipblacklist";
+my $savedir = "/var/lib/ipblacklist";
+my $tmpdir = "/var/tmp";
+
+my $settings = "$settingsdir/settings";
+my $sources = "$settingsdir/sources";
+my $checked = "$settingsdir/checked";
+my $modified = "$settingsdir/modified";
+my $iptables_list = "/var/tmp/iptables.txt";
+my $getipstat = "/usr/local/bin/getipstat";
+my $iptables = "/sbin/iptables";
+my $ipset = "/usr/sbin/ipset";
+my $fcrontab = "/usr/bin/fcrontab";
+my $lockfile = "/var/run/ipblacklist.pid";
+my $proxy_settings = "${General::swroot}/proxy/settings";
+my $red_setting = "/var/ipfire/red/iface";
+my $detailed_log = "$tmpdir/ipblacklist_log.txt";
+my $autoblacklist = 'AUTOBLACKLIST';
+
+my %parsers = ( 'text-with-hash-comments' => \&parse_text_with_hash_comments,
+ 'text-with-semicolon-comments' => \&parse_text_with_semicolon_comments,
+ 'dshield' => \&parse_dshield );
+
+############################################################################
+# Default settings
+# Should be overwritten by reading settings files
+############################################################################
+
+my %sources = ( );
+
+my %settings = ( 'DEBUG' => 0,
+ 'LOGGING' => 'on',
+ 'RATE' => 24,
+ 'ENABLE' => 'off' );
+
+my %proxy_settings = ( 'UPSTREAM_PROXY' => '' ); # No Proxy in use
+
+############################################################################
+# Function prototypes
+############################################################################
+
+sub abort( $ );
+sub autoblacklist_update();
+sub autoblacklist_clear();
+sub create_autoblacklist();
+sub create_list( $ );
+sub create_ipset( $$$ );
+sub debug( $$ );
+sub delete_autoblacklist();
+sub delete_list( $ );
+sub disable_logging();
+sub disable_updates();
+sub do_delete();
+sub do_start();
+sub do_stop();
+sub do_update();
+sub download_list( $$$ );
+sub download_check_header_time( $$$ );
+sub download_wget( $$$ );
+sub enable_logging();
+sub enable_updates();
+sub get_ipsets();
+sub iptables( $ );
+sub ipset( $ );
+sub stop_ipset();
+sub is_connected();
+sub log_message( $$ );
+sub parse_dshield( $ );
+sub parse_text_with_hash_comments( $ );
+sub parse_text_with_semicolon_comments( $ );
+sub read_ipset( $$$ );
+sub update_list( $$$ );
+
+############################################################################
+# Variables
+############################################################################
+
+my %chains; # The Blacklist IPSets already loaded
+my %old_blacklist; # Already blocked IP Addresses and/or networks
+ # downloaded for current blacklist
+my $update_status = 0; # Set to 1 to update status file
+my $ipset_running = 0; # Set to 1 if IPSet process is running
+my %status; # Status information
+my %checked; # Time blacklists last changed
+my %modified; # Time blacklists last modified
+my $red_iface; # Name of red interface
+my $hours = 3600; # One hour in seconds
+my $margin = 600; # Allowance for run time etc
+my $count = 30; # Maximum time to wait for another instance (300s)
+my @wget_status = ( 'Success', 'Error', 'Parse Error', 'File I/O Error',
+ 'Network Error', 'SSL Verification Error',
+ 'Authentication Error', 'Protocol Error', 'Server Error' );
+
+
+############################################################################
+# Synchronise runs
+############################################################################
+
+# This script can be triggered either by cron or the WUI. If another
+# instance is running, wait for it to finish.
+
+while (-r $lockfile and $count > 0)
+{
+ open LOCKFILE, '<', $lockfile or die "Can't open lockfile";
+ my $pid = <LOCKFILE>;
+ close LOCKFILE;
+
+ chomp $pid;
+
+ last unless (-e "/proc/$pid");
+
+ sleep 10;
+ $count--;
+}
+
+# Create pid file before starting main processing
+
+open LOCKFILE, '>', '/var/run/ipblacklist.pid' or die "Can't open PID file: $!";
+print LOCKFILE "$$\n";
+close LOCKFILE;
+
+############################################################################
+# Set up for update
+############################################################################
+
+mkdir $settingsdir unless (-d $settingsdir);
+
+# Connect to the system log
+
+openlog( "ipblacklist", "nofatal", LOG_USER);
+log_message LOG_INFO, "Starting IP Blacklist processing";
+
+# Read settings
+
+General::readhash( $settings, \%settings ) if (-e $settings);
+General::readhash( $checked, \%checked ) if (-e $checked);
+General::readhash( $modified, \%modified ) if (-e $modified);
+General::readhash( $proxy_settings, \%proxy_settings ) if (-e $proxy_settings);
+
+if (-r $sources)
+{
+ debug 1, "Reading sources file";
+
+ eval qx|/bin/cat $sources|;
+}
+
+# Find out the red interface name
+
+if (-r $red_setting)
+{
+ open IN, '<', $red_setting or die "Can't open red interface name file: $!";
+
+ $red_iface = <IN>;
+ chomp $red_iface;
+
+ close IN;
+}
+
+if (@ARGV)
+{
+ foreach my $cmd (@ARGV)
+ {
+ if ('update' =~ m/^$cmd/i)
+ {
+ # Called hourly when enabled and on setting changes.
+ # Update the blacklists.
+
+ if ($settings{'ENABLE'} eq 'on')
+ {
+ do_update();
+ }
+ }
+ elsif ('start' =~ m/^$cmd/i)
+ {
+ # Called during system startup.
+ # Restore saved blacklists.
+ # Don't do an update since that takes too long.
+
+ do_start() if ($settings{'ENABLE'} eq 'on');
+ }
+ elsif ('stop' =~ m/^$cmd/i)
+ {
+ # Called when shutting down.
+ # Delete IPSets and IPTables chains
+
+ do_stop();
+ }
+ elsif ('restore' =~ m/^$cmd/i)
+ {
+ # Called after restoring backup.
+ # Delete IPSets and IPTables chains, then re-create them with the new
+ # (restored) settings and make sure updates are enabled.
+
+ do_stop();
+
+ if ($settings{'ENABLE'} eq 'on')
+ {
+ do_start();
+ enable_updates();
+ }
+ else
+ {
+ disable_updates();
+ }
+ }
+ elsif ('log-on' =~ m/^$cmd/i)
+ {
+ # Called from WUI.
+ # Create entries in IPTables chains to log dropped packets.
+
+ if ($settings{'ENABLE'} eq 'on')
+ {
+ enable_logging();
+ }
+ }
+ elsif ('log-off' =~ m/^$cmd/i)
+ {
+ # Called from WUI
+ # Delete entries in IPTables chains to log dropped packets.
+
+ disable_logging();
+ }
+ elsif ('enable' =~ m/^$cmd/i)
+ {
+ # Called from WUI to enable blacklists
+ # Do an update and then enable automatic updates
+
+ if ($settings{'ENABLE'} eq 'on')
+ {
+ do_update();
+ enable_updates();
+ }
+ }
+ elsif ('disable' =~ m/^$cmd/i)
+ {
+ # Called from WUI to disable blacklists.
+ # Disable updates, delete IPSets, IPTables chains and save files.
+
+ disable_updates();
+ do_delete();
+ }
+ elsif ('autoblacklist-update' =~ m/^$cmd/i)
+ {
+ # Updates AUTOBLACKLIST options
+
+ autoblacklist_update();
+ }
+ elsif ('autoblacklist-clear' =~ m/^$cmd/i)
+ {
+ # Clears AUTOBLACKLIST contents
+
+ autoblacklist_clear();
+ }
+ else
+ {
+ print "Usage: $0 [update|start|stop|restart|log-on|log-off|enable|disable|autoblacklist-update|autoblacklist-clear]\n";
+ }
+ }
+}
+elsif ($settings{'ENABLE'} eq 'on')
+{
+ do_update();
+}
+
+stop_ipset();
+
+if ($update_status)
+{
+ debug 1, "Writing updated status file";
+
+ General::writehash( $checked, \%checked );
+ General::writehash( $modified, \%modified );
+}
+
+# Remove the pid file
+
+unlink $lockfile;
+
+log_message LOG_INFO, "Finished IP Blacklist processing";
+closelog();
+
+
+#------------------------------------------------------------------------------
+# sub do_stop
+#
+# Deletes all the IPTables chains and the IPSets
+#------------------------------------------------------------------------------
+
+sub do_stop()
+{
+ get_ipsets();
+
+ log_message LOG_NOTICE, "Stopping IP Blacklists";
+
+ foreach my $list ( $autoblacklist, sort keys %sources )
+ {
+ if (exists $chains{$list})
+ {
+ delete_list( $list );
+ }
+ }
+}
+
+
+#------------------------------------------------------------------------------
+# sub do_start
+#
+# Recreates the IPTables chains and the IPSets from the saved values
+#------------------------------------------------------------------------------
+
+sub do_start()
+{
+ log_message LOG_NOTICE, "Starting IP Blacklists";
+
+ foreach my $list ( sort keys %sources )
+ {
+ if (-e "$savedir/$list.conf")
+ {
+ log_message LOG_INFO, "Restoring blacklist $list";
+ system( "$ipset restore -f $savedir/$list.conf" );
+
+ create_list( $list );
+ }
+ }
+
+ if ($settings{$autoblacklist} eq 'on')
+ {
+ create_autoblacklist();
+ }
+}
+
+
+#------------------------------------------------------------------------------
+# sub do_delete
+#
+# Deletes the IPTables chains, the IPSets and the saved values.
+#------------------------------------------------------------------------------
+
+sub do_delete()
+{
+ # Get the list of current ipsets
+
+ get_ipsets();
+
+ log_message LOG_NOTICE, "Deleting IP Blacklists";
+
+ foreach my $source ( sort keys %sources )
+ {
+ if (exists $chains{$source})
+ {
+ delete_list( $source );
+ }
+
+ if (-e "$savedir/$source.conf")
+ {
+ unlink "$savedir/$source.conf";
+ }
+ }
+
+ if ($settings{$autoblacklist} eq 'on')
+ {
+ delete_autoblacklist();
+ }
+}
+
+
+#------------------------------------------------------------------------------
+# sub do_update
+#
+# Updates all the blacklists.
+# Creates or deletes the blacklist firewall rules as necessary and checks for
+# updates to the blacklists.
+#------------------------------------------------------------------------------
+
+sub do_update()
+{
+ return unless (is_connected());
+
+ my $type = 'hash:ip';
+
+ # Get the list of current ipsets
+
+ get_ipsets();
+
+ # Check sources
+
+ debug 1, "Checking blacklist sources";
+
+ foreach my $list ( sort keys %sources )
+ {
+ my @new_blacklist = ();
+ my $name = $sources{$list}{'name'};
+ my $rate = $sources{$list}{'rate'};
+ my $last_checked = $checked{$list} || 0;
+ my $enabled = 0;
+
+ if (exists $modified{$list})
+ {
+ # Limit the check rate to the minimum defined in the WUI, unless we're
+ # creating the list
+
+ $rate = $settings{'RATE'} if ($settings{'RATE'} > $rate);
+ }
+
+ if (exists $settings{$list})
+ {
+ $enabled = $settings{$list} eq 'on';
+ }
+
+ debug 1, "Checking blacklist source: $name";
+
+ if ($enabled)
+ {
+ # Has enough time passed since the last time we checked the list?
+
+ if (($last_checked + $rate * $hours) < (time() + $margin))
+ {
+ download_list( $list, \@new_blacklist, \$type );
+
+ next unless (@new_blacklist);
+
+ if (not exists $chains{$list})
+ {
+ # Doesn't currently exist: Create it.
+
+ create_ipset( $list, $type, scalar @new_blacklist );
+ create_list( $list );
+ }
+
+ update_list( $list, \@new_blacklist, $type );
+ }
+ }
+ elsif (exists $chains{$list})
+ {
+ # Exists, but not enabled: Delete it.
+
+ delete_list( $list );
+
+ # Delete the save file
+ # Don't delete the checked time from the status, in case the list is
+ # re-enabled quickly - don't want to exceed maximum allowed download
+ # rate.
+
+ unlink "$savedir/$list.conf" if (-e "$savedir/$list.conf");
+
+ delete $modified{$list} if (exists $modified{$list});
+ $update_status = 1;
+ }
+ }
+
+ # Check for any deleted lists
+
+ foreach my $list (keys %sources)
+ {
+ if (not exists $sources{$list})
+ {
+ delete_list( $list );
+
+ # Delete the save file
+
+ unlink "$savedir/$list.conf" if (-e "$savedir/$list.conf");
+
+ # Delete from the status
+
+ delete $modified{$list} if (exists $modified{$list});
+ delete $checked{$list} if (exists $checked{$list});
+ $update_status = 1;
+ }
+ }
+
+ if ($settings{$autoblacklist} eq 'on')
+ {
+ create_autoblacklist() if (not exists $chains{$autoblacklist});
+ }
+ else
+ {
+ delete_autoblacklist() if (exists $chains{$autoblacklist});
+ }
+
+ log_message LOG_INFO, "Completed IP Blacklist update";
+}
+
+
+#------------------------------------------------------------------------------
+# sub autoblacklist_update()
+#
+# Updates the settings for the AUTOBLACKLIST
+#------------------------------------------------------------------------------
+
+sub autoblacklist_update()
+{
+ # Get the list of current ipsets
+
+ get_ipsets();
+
+ # Delete the existing AUTOBLACKLIST, if it currently exists.
+
+ delete_autoblacklist() if (exists $chains{$autoblacklist});
+
+ # Re-create the AUTOBLACKLIST with the correct parameters.
+
+ create_autoblacklist() if ($settings{$autoblacklist} eq 'on');
+}
+
+
+#------------------------------------------------------------------------------
+# sub autoblacklist_clear()
+#
+# Clears the contents of the AUTOBLACKLIST
+#------------------------------------------------------------------------------
+
+sub autoblacklist_clear()
+{
+ log_message LOG_INFO, "Flush Automatic blacklist";
+ ipset( "flush $autoblacklist" );
+}
+
+
+#------------------------------------------------------------------------------
+# sub is_connected()
+#
+# Checks that the system is connected to the internet.
+#
+# This looks for a file created by IPFire when connected to the internet
+#------------------------------------------------------------------------------
+
+sub is_connected()
+{
+ return (-e "${General::swroot}/red/active");
+}
+
+
+#------------------------------------------------------------------------------
+# sub create_list( list )
+#
+# Creates a new IPTables chain for a blacklist source.
+# The set must be created before calling this function.
+#
+# Parameters:
+# list The name of the blacklist
+#------------------------------------------------------------------------------
+
+sub create_list( $ )
+{
+ my ($list) = @_;
+
+ log_message LOG_INFO, "Create IPTables chains for blacklist $list";
+
+ # Create new chain in filter table
+
+ iptables( " -N ${list}_BLOCK" ) == 0 or
+ ( abort "Could not create IPTables chain ${list}_BLOCK", return );
+
+ # Add the logging and drop rules
+
+ if ($settings{'LOGGING'} eq 'on')
+ {
+ iptables( "-A ${list}_BLOCK -j LOG -m limit --limit 10/second --log-prefix 'DROP_$list'" ) == 0 or
+ ( abort "Could not create IPTables chain $list LOG rule", return );
+ }
+
+ iptables( "-A ${list}_BLOCK -j DROP" ) == 0 or
+ ( abort "Could not create IPTables chain $list drop rule", return );
+
+ # Add the rules to check against the set
+
+ iptables( "-A IPBLACKLISTREDIN -p ALL -m set --match-set $list src -j ${list}_BLOCK" );
+ iptables( "-A IPBLACKLISTREDOUT -p ALL -m set --match-set $list dst -j ${list}_BLOCK" );
+}
+
+
+#------------------------------------------------------------------------------
+# sub create_autoblacklist()
+#
+# Creates a new IPTables chain for the AUTOBLACKLIST. This also creates the
+# IPSet with the correct timeout.
+#------------------------------------------------------------------------------
+
+sub create_autoblacklist()
+{
+ return unless ($red_iface); # Can't add rule to policy unless this is set
+
+ # Create the set for the AUTOBLACKLIST
+
+ ipset( "create $autoblacklist hash:ip timeout $settings{BLOCK_PERIOD}" );
+
+ # Create new chain in filter table
+
+ create_list( $autoblacklist );
+
+ # For the AUTOBLACKLIST there are extra rules to reset the timeout on the
+ # blockled addresses
+
+ iptables( "-I ${autoblacklist}_BLOCK -m set --match-set $autoblacklist src -j SET --add-set $autoblacklist src --exist" );
+ iptables( "-I ${autoblacklist}_BLOCK -m set --match-set $autoblacklist dst -j SET --add-set $autoblacklist dst --exist" );
+
+ # For the AUTOBLACKLIST there is an extra rule to add an entry to the list
+ # of blocked addresses. This is added to the input policy chain.
+
+ iptables( "-I POLICYIN 1 -i $red_iface -m hashlimit --hashlimit-mode srcip --hashlimit-above $settings{BLOCK_THRESHOLD}/hour --hashlimit-name $autoblacklist -j SET --add-set $autoblacklist src" );
+}
+
+
+#------------------------------------------------------------------------------
+# sub delete_list( $list )
+#
+# Deletes an IPTables chain when a blacklist source is disabled. Also flushes
+# and destroys the IPSet.
+#
+# Parameters:
+# list The name of the blacklist
+#------------------------------------------------------------------------------
+
+sub delete_list( $ )
+{
+ my ($list) = @_;
+
+ log_message LOG_INFO, "Delete IPTables chains for blacklist $list";
+
+ # Remove the blacklist chains from the main INPUT and OUTPUT chains
+
+ iptables( "-D IPBLACKLISTREDIN -p ALL -m set --match-set $list src -j ${list}_BLOCK" ) == 0 or
+ log_message LOG_ERR, "Could not remove IPSet $list from IPBLACKLISTREDIN chain";
+
+ iptables( "-D IPBLACKLISTREDOUT -p ALL -m set --match-set $list dst -j ${list}_BLOCK" ) == 0 or
+ log_message LOG_ERR, "Could not remove IPSet $list from IPBLACKLISTREDOUT chain";
+
+ # Flush and delete the chain
+
+ iptables( "-F ${list}_BLOCK" ) == 0 or
+ log_message LOG_ERR, "Could not flush IPTables chain ${list}_BLOCK";
+
+ iptables( "-X ${list}_BLOCK" ) == 0 or
+ log_message LOG_ERR, "Could not delete IPTables chain ${list}_BLOCK";
+
+ # Flush and delete the set
+
+ ipset( "flush $list" );
+
+ ipset( "destroy $list" );
+}
+
+
+#------------------------------------------------------------------------------
+# sub delete_autoblacklist()
+#
+# Deletes the autoblacklist IPTables chain when it is disabled. Also flushes
+# and destroys the IPSet.
+#------------------------------------------------------------------------------
+
+sub delete_autoblacklist()
+{
+ # For the AUTOBLACKLIST there is an extra rule to remove
+
+ unless ($red_iface)
+ {
+ iptables( "-D POLICYIN -i $red_iface -m hashlimit --hashlimit-mode srcip --hashlimit-above $settings{BLOCK_THRESHOLD}/hour --hashlimit-name $autoblacklist -j SET --add-set $autoblacklist src" );
+ }
+
+ # Now do a normal delete
+
+ delete_list( $autoblacklist );
+}
+
+
+#------------------------------------------------------------------------------
+# sub download_list( chain, ref_list, ref_type )
+#
+# Updates the IP Addresses for a blacklist. Depending on the blacklist one of
+# two methods are used:
+#
+# - For some lists the header is downloaded and the modification date checked.
+# If newer than the existing list, the update is downloaded.
+# - For other lists this is not supported,so the whole file has to be
+# downloaded regardless.
+#
+# Once downloaded the list is parsed to get the IP addresses and/or networks.
+#
+# Parameters:
+# list The name of the blacklist
+# ref_list A reference to an array to store the downloaded blacklist
+# ref_type A reference to store the type of the blacklist
+#------------------------------------------------------------------------------
+
+sub download_list( $$$ )
+{
+ my ($list, $new_blacklist, $type) = @_;
+
+ $checked{$list} = time();
+ $update_status = 1;
+
+ # Check the parser for the blacklist
+
+ if (not exists $parsers{ $sources{$list}{'parser'} })
+ {
+ log_message LOG_ERR, "Can't find parser $sources{$list}{'parser'} for $list blacklist";
+ return;
+ }
+
+ if ($sources{$list}{'method'} eq 'check-header-time')
+ {
+ download_check_header_time( $list, $new_blacklist, $type );
+ }
+ else
+ {
+ download_wget( $list, $new_blacklist, $type );
+ }
+}
+
+
+#------------------------------------------------------------------------------
+# sub download_check_header_time( chain, ref_list, ref_type )
+#
+# Updates the IP Addresses for a blacklist. The header is downloaded and the
+# modification date checked. If newer than the existing list, the update is
+# downloaded.
+#
+# Once downloaded the list is parsed to get the IP addresses and/or networks.
+#
+# Parameters:
+# list The name of the blacklist
+# ref_list A reference to an array to store the downloaded blacklist
+# ref_type A reference to store the type of the blacklist
+#------------------------------------------------------------------------------
+
+sub download_check_header_time( $$$ )
+{
+ my ($list, $new_blacklist, $type) = @_;
+ my $found_ip = 0;
+ my $found_net = 0;
+
+ # Get the parser for the blacklist
+
+ my $parser = $parsers{ $sources{$list}{'parser'} };
+
+ log_message LOG_INFO, "Checking modification time for blacklist $list update with LWP";
+
+ # Create a user agent for downloading the blacklist
+ # Limit the download size for safety (10 MiB)
+
+ my $ua = LWP::UserAgent->new( max_size => 10485760 );
+
+ # Get the Proxy settings
+
+ if ($proxy_settings{'UPSTREAM_PROXY'})
+ {
+ if ($proxy_settings{'UPSTREAM_USER'})
+ {
+ $ua->proxy("http" => "http://$proxy_settings{'UPSTREAM_USER'}:$proxy_settings{'UPSTREAM_PASSWORD'}\@$proxy_settings{'UPSTREAM_PROXY'}/");
+ $ua->proxy("https" => "http://$proxy_settings{'UPSTREAM_USER'}:$proxy_settings{'UPSTREAM_PASSWORD'}\@$proxy_settings{'UPSTREAM_PROXY'}/");
+ }
+ else
+ {
+ $ua->proxy("http" => "http://$proxy_settings{'UPSTREAM_PROXY'}/");
+ $ua->proxy("https" => "http://$proxy_settings{'UPSTREAM_PROXY'}/");
+ }
+ }
+
+ # Get the blacklist modification time from the internet
+
+ my $request = HTTP::Request->new( HEAD => $sources{$list}{'url'} );
+
+ my $response = $ua->request( $request );
+
+ if (not $response->is_success)
+ {
+ log_message LOG_WARNING, "Failed to download $list header $sources{$list}{'url'}: ". $response->status_line;
+
+ return;
+ }
+
+ # Has the blacklist been modified since we last read it?
+
+ if (exists $modified{$list} and $modified{$list} >= $response->last_modified)
+ {
+ # We've already got this version of the blacklist
+
+ debug 1, "Blacklist $list not modified";
+ return;
+ }
+
+ debug 1, "Blacklist $list Modification times: old " . $modified{$list} . ", new " . $response->last_modified if (exists $modified{$list});
+ log_message LOG_INFO, "Downloading blacklist $list with LWP";
+
+ # Download the blacklist
+
+ $request = HTTP::Request->new( GET => $sources{$list}{'url'} );
+ $response = $ua->request($request);
+
+ if (not $response->is_success)
+ {
+ log_message LOG_WARNING, "Failed to download $list blacklist $sources{$list}{'url'}: ". $response->status_line;
+
+ return;
+ }
+
+ $modified{$list} = $response->last_modified;
+
+ foreach my $line (split /[\r\n]+/, $response->content)
+ {
+ chomp $line;
+
+ my $address = &$parser( $line );
+
+ next unless ($address and $address =~ m/\d+\.\d+\.\d+\.\d+/);
+
+ if ($address =~ m|/\d+|)
+ {
+ $found_net = 1;
+ }
+ else
+ {
+ $found_ip = 1;
+ }
+
+ push @{ $new_blacklist }, $address;
+ }
+
+ if ($found_net and $found_ip)
+ {
+ # Convert mixed address and network set to all network
+
+ foreach my $address (@{ $new_blacklist })
+ {
+ $address .= '/32' unless ($address =~ m|/\d+|);
+ }
+
+ $found_ip = 0;
+ }
+
+ $$type = $found_net ? 'hash:net' : 'hash:ip';
+}
+
+
+#------------------------------------------------------------------------------
+# sub download_wget( chain, ref_list, ref_type )
+#
+# Updates the IP Addresses for a blacklist. The whole file is download with
+# wget and then the modification time compared with the stored modification
+# time. If the update is newer then the downloaded list is parsed.
+#
+# Once downloaded the list is parsed to get the IP addresses and/or networks.
+#
+# Parameters:
+# list The name of the blacklist
+# ref_list A reference to an array to store the downloaded blacklist
+# ref_type A reference to store the type of the blacklist
+#------------------------------------------------------------------------------
+
+sub download_wget( $$$ )
+{
+ my ($list, $new_blacklist, $type) = @_;
+ my $wget_proxy = '';
+ my $found_ip = 0;
+ my $found_net = 0;
+
+ my $parser = $parsers{ $sources{$list}{'parser'} };
+
+ log_message LOG_INFO, "Downloading blacklist $list update with wget";
+
+ # Get the Proxy settings
+
+ if ($proxy_settings{'UPSTREAM_PROXY'})
+ {
+ if ($proxy_settings{'UPSTREAM_USER'})
+ {
+ $wget_proxy = "--proxy=on --proxy-user=$proxy_settings{'UPSTREAM_USER'} --proxy-passwd=$proxy_settings{'UPSTREAM_PASSWORD'} -e http_proxy=http://$proxy_settings{'UPSTREAM_PROXY'}/";
+ }
+ else
+ {
+ $wget_proxy = "--proxy=on -e http_proxy=http://$proxy_settings{'UPSTREAM_PROXY'}/";
+ }
+ }
+
+ my $retv = system( "wget $wget_proxy --no-show-progress -o $detailed_log -O $tmpdir/ipblacklist_$list $sources{$list}{'url'}" );
+
+ if ($retv != 0)
+ {
+ my $error = $wget_status[ $retv/256 ];
+ log_message LOG_WARNING, "Failed to download $list blacklist $sources{$list}{'url'}: $error";
+ return;
+ }
+
+ my @file_info = stat( "$tmpdir/ipblacklist_$list" );
+
+ if (exists $modified{$list} and $modified{$list} >= $file_info[9])
+ {
+ # We've already got this version of the blocklist
+
+ debug 1, "Blacklist $list not modified";
+ unlink "$tmpdir/ipblacklist_$list";
+ return;
+ }
+
+ open LIST, '<', "$tmpdir/ipblacklist_$list" or (abort "Can't open downloaded blacklist for $list: $!", return);
+
+ $modified{$list} = $file_info[9];
+
+ foreach my $line (<LIST>)
+ {
+ chomp $line;
+
+ my $address = &$parser( $line );
+
+ next unless ($address);
+ next unless ($address =~ m|\d+\.\d+\.\d+\.\d+|);
+
+ if ($address =~ m|/\d+|)
+ {
+ $found_net = 1;
+ }
+ else
+ {
+ $found_ip = 1;
+ }
+
+ push @{ $new_blacklist }, $address;
+ }
+
+ close LIST;
+
+ unlink "$tmpdir/ipblacklist_$list";
+
+ if ($found_net and $found_ip)
+ {
+ # Convert mixed address and network set to all network
+
+ foreach my $address (@{ $new_blacklist })
+ {
+ $address .= '/32' unless ($address =~ m|/\d+|);
+ }
+
+ $found_ip = 0;
+ }
+
+ $$type = $found_net ? 'hash:net' : 'hash:ip';
+}
+
+
+#------------------------------------------------------------------------------
+# sub read_ipset( list, old, type )
+#
+# Reads the existing contents of the set
+#
+# Parameters:
+# chain The name of the blacklist
+# old Reference to array to contain blacklist
+# type Reference to type
+#------------------------------------------------------------------------------
+
+sub read_ipset( $$$ )
+{
+ my ($list, $old, $type) = @_;
+ my $found_net = 0;
+ my $found_ip = 0;
+
+ debug 2, "Reading existing ipset for blacklist $list";
+
+ foreach my $line (qx/$ipset list $list/)
+ {
+ next unless ($line =~ m|(\d+\.\d+\.\d+\.\d+(?:/\d+)?)|);
+
+ my $address = $1;
+
+ if (($address =~ m|/\d+$|) and ($address !~ m|/32$|))
+ {
+ $found_net = 1;
+ }
+ else
+ {
+ $found_ip = 1;
+ $address =~ s|/32$||;
+ }
+
+ $$old{$address} = 1;
+ }
+
+ if ($found_ip and $found_net)
+ {
+ # Convert mixed address and network set to all network
+
+ my @ads_list = keys %{ $old };
+
+ foreach my $address (@ads_list)
+ {
+ unless ($address =~ m|/\d+|)
+ {
+ delete $$old{$address};
+ $$old{"$address/32"} = 1;
+ }
+ }
+
+ $found_ip = 0;
+ }
+
+ $$type = $found_net ? 'hash:net' : 'hash:ip';
+}
+
+
+#------------------------------------------------------------------------------
+# sub update_list( chain, new, new_type )
+#
+# Updates the IP Addresses for a blacklist
+#
+# The new list is compared to the existing list and new entries added or old
+# entries deleted as necessary.
+#
+# Parameters:
+# list The name of the blacklist
+# new Reference to array of new blacklist entries
+# new_type The type of the updated list (hash:ip or hash:net)
+#------------------------------------------------------------------------------
+
+sub update_list( $$$ )
+{
+ my ($list, $new, $new_type) = @_;
+ my %old;
+ my $old_type;
+ my $changes = 0;
+
+ debug 2, "Checking for $list blacklist update from $sources{$list}{'url'}";
+
+ log_message LOG_INFO, "Updating $list blacklist";
+
+ read_ipset( $list, \%old, \$old_type );
+
+ # Check the IPSet type hasn't changed
+
+ if ($new_type ne $old_type)
+ {
+ # Change the IPSet type. This requires removing references to it first.
+ # We could delete and then create the chain, but doing it like this keeps
+ # the statistics.
+
+ log_message LOG_NOTICE, "Blacklist $list changed type from $old_type to $new_type";
+
+ # Remove the IPSet from the IPTables chains
+
+ iptables( "-D 'IPBLACKLISTREDIN' -p ALL -m set --match-set $list src -j ${list}_BLOCK" ) == 0 or
+ log_message LOG_ERR, "Could not remove ${list} from IPBLACKLISTREDIN chain";
+
+ iptables( "-D 'IPBLACKLISTREDOUT' -p ALL -m set --match-set $list dst -j ${list}_BLOCK" ) == 0 or
+ log_message LOG_ERR, "Could not remove ${list} from IPBLACKLISTREDOUT chain";
+
+ # Flush and delete the old set
+
+ ipset( "flush $list" );
+ ipset( "destroy $list" );
+
+ %old = ();
+
+ # Create the new ipset
+
+ create_ipset( $list, $new_type, scalar @{ $new } );
+
+ # Add the rules to check against the set
+
+ iptables( "-A 'IPBLACKLISTREDIN' -p ALL -m set --match-set $list src -j ${list}_BLOCK" ) == 0 or
+ log_message LOG_ERR, "Could not add IPSet $list to IPBLACKLISTREDIN chain";
+
+ iptables( "-A 'IPBLACKLISTREDOUT' -p ALL -m set --match-set $list dst -j ${list}_BLOCK" ) == 0 or
+ log_message LOG_ERR, "Could not add IPSet $list to IPBLACKLISTREDOUT chain";
+ }
+
+ # Process the blacklist
+
+ foreach my $address ( @{ $new } )
+ {
+ # We've got an address. Add to IPSet if it's new
+
+ if (exists $old{$address})
+ {
+ delete $old{$address}; # Not new
+ }
+ else
+ {
+ ipset( "add $list $address -exist" );
+
+ $changes++;
+ }
+
+ debug 3, "Add net $address to blacklist $list";
+ }
+
+ # Delete old entries that aren't needed any more
+
+ debug 2, "Removing deleted rules from IPTables chain for blacklist $list";
+
+ foreach my $address ( keys %old )
+ {
+ ipset( "del $list $address" );
+
+ $changes++;
+
+ debug 3, "Delete old net $address from blacklist $list";
+ }
+
+
+ log_message LOG_INFO, "Finished updating $list blacklist with $changes changes";
+
+ # Save the blacklist for the next reboot
+
+ mkdir "$savedir" unless (-d "$savedir" );
+
+ ipset( "save $list -file $savedir/$list.conf" );
+
+ stop_ipset();
+}
+
+
+#------------------------------------------------------------------------------
+# sub get_ipsets( )
+#
+# Gets a list of the current IPSets
+#------------------------------------------------------------------------------
+
+sub get_ipsets( )
+{
+ debug 1, "Reading list of existing ipsets";
+
+ my @sets = qx($ipset -n list);
+
+ # Parse the tables
+
+ foreach my $line (@sets)
+ {
+ chomp $line;
+
+ next unless ($line);
+
+ $chains{$line} = 1;
+ }
+}
+
+
+#------------------------------------------------------------------------------
+# sub create_ipset( name, type, size )
+#
+# Creates a new IPSet. The current and maximum size of the set are determined
+# by taking the next power of two greater than the numer of entries, subject to
+# a minimum size. This allows for future expansion.
+#
+# Parameters:
+# name The name of the blacklist
+# type The type of the blacklist (hash:ip or hash:net)
+# size The number of entries in the lsit
+#------------------------------------------------------------------------------
+
+sub create_ipset( $$$ )
+{
+ my ($name, $type, $size) = @_;
+
+ my $hashsize = 1;
+ $hashsize <<= 1 while ($hashsize < $size);
+ my $maxsize = ($hashsize < 16384) ? 32768 : $hashsize * 2;
+
+ # Create the new ipset
+ ipset( "create $name $type hashsize $hashsize maxelem $maxsize" );
+ stop_ipset(); # Need to do this to action the IPSet commands
+}
+
+
+#------------------------------------------------------------------------------
+# sub enable_logging()
+#
+# Enable logging of packets dropped by IP Blacklist rules.
+#------------------------------------------------------------------------------
+
+sub enable_logging()
+{
+ get_ipsets();
+
+ log_message LOG_NOTICE, "Enabling IP Blacklist logging";
+
+ foreach my $list ( sort keys %sources )
+ {
+ if (exists $chains{$list})
+ {
+ iptables( "-I ${list}_BLOCK 1 -j LOG -m limit --limit 10/second --log-prefix 'DROP_$list'" );
+ }
+ }
+}
+
+
+#------------------------------------------------------------------------------
+# sub disable_logging()
+#
+# Disable logging of packets dropped by IP Blacklist rules.
+#------------------------------------------------------------------------------
+
+sub disable_logging()
+{
+ get_ipsets();
+
+ log_message LOG_NOTICE, "Disabling IP Blacklist logging";
+
+ foreach my $list ( sort keys %sources )
+ {
+ if (exists $chains{$list})
+ {
+ iptables( "-D ${list}_BLOCK -j LOG -m limit --limit 10/second --log-prefix 'DROP_$list'" );
+ }
+ }
+}
+
+
+#------------------------------------------------------------------------------
+# sub enable_updates()
+#
+# Adds a command to the fcrontab to run the update hourly.
+# The update is executed at an offset from the hour so that all the users don't
+# try to download the updates at exactly the same time - the blacklists are
+# provided free, so it's good manners to spread the load on the servers.
+#------------------------------------------------------------------------------
+
+sub enable_updates()
+{
+ my @lines = qx/$fcrontab -l/;
+ my $found = 0;
+
+ # Check for an existing fcrontab entry.
+
+ foreach my $line (@lines)
+ {
+ if ($line =~ m|/usr/local/bin/ipblacklist|)
+ {
+ return if ($line !~ m/^#/); # Already enabled
+
+ # Found - uncomment the line
+
+ $line =~ s/^#+//;
+ $found = 1;
+ log_message LOG_INFO, "Enable IP Address Blacklist update in crontab";
+ last;
+ }
+ }
+
+ if (not $found)
+ {
+ # Add a new entry
+
+ my $start = int( rand(50) ) + 5;
+
+ push @lines, "\n";
+ push @lines, "# IP Blacklist update\n";
+ push @lines, "\%hourly,nice(1),random,serial $start /usr/local/bin/ipblacklist\n";
+ log_message LOG_INFO, "Add IP Address Blacklist update to crontab";
+ }
+
+ open FCRONTAB, "| $fcrontab -" or (abort "Can't open pipe to write fcrontab: $!", return);
+ print FCRONTAB @lines;
+ close FCRONTAB;
+}
+
+
+#------------------------------------------------------------------------------
+# sub disable_updates()
+#
+# Comments out the entry in the fcrontab that runs the updates.
+#------------------------------------------------------------------------------
+
+sub disable_updates()
+{
+ my @lines = qx/$fcrontab -l/;
+ my $found = 0;
+
+ foreach my $line (@lines)
+ {
+ if ($line =~ m|/usr/local/bin/ipblacklist|)
+ {
+ return if ($line =~ m/^#/); # Already disabled
+
+ # Found - comment the line
+
+ $line =~ s/^#*/#/;
+ $found = 1;
+ log_message LOG_INFO, "Disable IP Address Blacklist updates";
+ last;
+ }
+ }
+
+ return unless ($found); # Don't update crontab unnecessarily
+
+ open FCRONTAB, "| $fcrontab -" or (abort "Can't open pipe to write fcrontab: $!", return);
+ print FCRONTAB @lines;
+ close FCRONTAB;
+}
+
+
+#------------------------------------------------------------------------------
+# sub parse_text_with_hash_comments( line )
+#
+# Parses an input line removing comments.
+#
+# Parameters:
+# line The line to parse
+#
+# Returns:
+# Either an IP Address or a null string
+#------------------------------------------------------------------------------
+
+sub parse_text_with_hash_comments( $ )
+{
+ my ($line) = @_;
+
+ return "" if ($line =~ m/^\s*#/);
+
+ $line =~ s/#.*$//;
+
+ $line =~ m|(\d+\.\d+\.\d+\.\d+(?:/\d+)?)|;
+
+ return $1;
+}
+
+
+#------------------------------------------------------------------------------
+# sub parse_text_with_semicolon_comments( line )
+#
+# Parses an input line removing comments.
+#
+# Parameters:
+# line The line to parse
+#
+# Returns:
+# Either and IP Address or a null string
+#------------------------------------------------------------------------------
+
+sub parse_text_with_semicolon_comments( $ )
+{
+ my ($line) = @_;
+
+ return "" if ($line =~ m/^\s*;/);
+
+ $line =~ s/;.*$//;
+
+ $line =~ m|(\d+\.\d+\.\d+\.\d+(?:/\d+)?)|;
+
+ return $1;
+}
+
+
+#------------------------------------------------------------------------------
+# sub parse_dshield( line )
+#
+# Parses an input line removing comments.
+#
+# The format is:
+# Start Addrs End Addrs Netmask Nb Attacks Network Name Country email
+# We're only interested in the start address and netmask.
+#
+# Parameters:
+# line The line to parse
+#
+# Returns:
+# Either and IP Address or a null string
+#------------------------------------------------------------------------------
+
+sub parse_dshield( $ )
+{
+ my ($line) = @_;
+
+ return "" if ($line =~ m/^\s*#/);
+
+ $line =~ s/#.*$//;
+
+ $line =~ m|(\d+\.\d+\.\d+\.\d+(?:/\d+)?)\s+\d+\.\d+\.\d+\.\d+(?:/\d+)?\s+(\d+)|;
+
+ return unless ($1);
+ return "$1/32" unless ($2);
+
+ return "$1/$2";
+}
+
+
+#------------------------------------------------------------------------------
+# sub iptables( cmd )
+#
+# Executes an IPTables command, waiting for the lock to ensure only one change
+# is made at a time.
+#
+# Parameters:
+# cmd The command to execute
+#
+# Returns:
+# Status of command
+#------------------------------------------------------------------------------
+
+sub iptables( $ )
+{
+ my ($cmd) = @_;
+
+ return system( "$iptables $cmd" );
+}
+
+
+#------------------------------------------------------------------------------
+# sub ipset( cmd )
+#
+# Executes an IPSet command. The command is piped to a sub-process running
+# ipset, rather than exected separately. This saves the overhead of starting a
+# new process for each command. The sub-process is started if it's not already
+# running.
+#
+# Parameters:
+# cmd The command to execute
+#------------------------------------------------------------------------------
+
+sub ipset( $ )
+{
+ my ($cmd) = @_;
+
+ if (not $ipset_running)
+ {
+ local $SIG{PIPE} = 'IGNORE';
+ open IPSET, "|-", $ipset, "restore" or die "Can't start ipset: $!";
+ $ipset_running = 1;
+ }
+
+ print IPSET "$cmd\n";
+}
+
+
+#------------------------------------------------------------------------------
+# sub stop_ipset( )
+#
+# Stops the ipset sub-process.
+#------------------------------------------------------------------------------
+
+sub stop_ipset( )
+{
+ if ($ipset_running)
+ {
+ close IPSET or abort "ipset process died: $! $?";
+ $ipset_running = 0;
+ }
+}
+
+
+#------------------------------------------------------------------------------
+# sub abort( message, parameters... )
+#
+# Aborts the current activity, printing out an error message.
+#
+# Parameters:
+# message Message to be printed
+#------------------------------------------------------------------------------
+
+sub abort( $ )
+{
+ my ($message) = @_;
+
+ log_message( LOG_ERR, $message );
+ carp $message;
+}
+
+
+#------------------------------------------------------------------------------
+# sub log_message( level, message )
+#
+# Logs a message. If the script is run from a terminal messages are also
+# output on STDOUT.
+#
+# Parameters:
+# level Severity of message
+# message Message to be logged
+#------------------------------------------------------------------------------
+
+sub log_message( $$ )
+{
+ my ($level, $message) = @_;
+
+ print "($level) $message\n" if (-t STDIN);
+ syslog( $level, $message );
+}
+
+
+#------------------------------------------------------------------------------
+# sub debug( level, message )
+#
+# Optionally logs a debug message. If the script is run from a terminal, level
+# 1 debug messages are output regardless of the debug setting.
+#
+# Parameters:
+# level Debug level
+# message Message to be logged
+#------------------------------------------------------------------------------
+
+sub debug( $$ )
+{
+ my ($level, $message) = @_;
+
+ if (($level <= $settings{'DEBUG'}) or
+ ($level == 1 and -t STDIN))
+ {
+ log_message LOG_DEBUG, $message;
+ }
+}
--
2.16.4
^ permalink raw reply [flat|nested] 35+ messages in thread
* [PATCH 2/5] ipblacklist: WUI and language file
2019-11-25 20:13 [PATCH 0/5] ipblacklist: IP Address Blacklists Tim FitzGeorge
2019-11-25 20:13 ` [PATCH 1/5] ipblacklist: Main script Tim FitzGeorge
@ 2019-11-25 20:13 ` Tim FitzGeorge
2019-11-25 20:13 ` [PATCH 3/5] ipblacklist: Ancillary files Tim FitzGeorge
` (4 subsequent siblings)
6 siblings, 0 replies; 35+ messages in thread
From: Tim FitzGeorge @ 2019-11-25 20:13 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 25491 bytes --]
Main WUI page and update to system logs page
Signed-off-by: Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk>
---
| 5 +
html/cgi-bin/ipblacklist.cgi | 725 ++++++++++++++++++++++++++++++++++++++++++
html/cgi-bin/logs.cgi/log.dat | 2 +
langs/en/cgi-bin/en.pl | 31 ++
4 files changed, 763 insertions(+)
create mode 100644 html/cgi-bin/ipblacklist.cgi
--git a/config/menu/50-firewall.menu b/config/menu/50-firewall.menu
index 5ec1f67fc..cd82bfaa3 100644
--- a/config/menu/50-firewall.menu
+++ b/config/menu/50-firewall.menu
@@ -21,6 +21,11 @@
'title' => "$Lang::tr{'intrusion detection system'}",
'enabled' => 1,
};
+ $subfirewall->{'45.ipblacklist'} = {'caption' => $Lang::tr{'ipblacklist'},
+ 'uri' => '/cgi-bin/ipblacklist.cgi',
+ 'title' => "$Lang::tr{'ipblacklist'}",
+ 'enabled' => 1,
+ };
$subfirewall->{'50.p2p'} = {
'caption' => $Lang::tr{'p2p block'},
'uri' => '/cgi-bin/p2p-block.cgi',
diff --git a/html/cgi-bin/ipblacklist.cgi b/html/cgi-bin/ipblacklist.cgi
new file mode 100644
index 000000000..b2ccf7b3f
--- /dev/null
+++ b/html/cgi-bin/ipblacklist.cgi
@@ -0,0 +1,725 @@
+#!/usr/bin/perl
+
+###############################################################################
+# #
+# IPFire.org - A linux based firewall #
+# #
+# This program is free software: you can redistribute it and/or modify #
+# it under the terms of the GNU General Public License as published by #
+# the Free Software Foundation, either version 3 of the License, or #
+# (at your option) any later version. #
+# #
+# This program is distributed in the hope that it will be useful, #
+# but WITHOUT ANY WARRANTY; without even the implied warranty of #
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the #
+# GNU General Public License for more details. #
+# #
+# You should have received a copy of the GNU General Public License #
+# along with this program. If not, see <http://www.gnu.org/licenses/>. #
+# #
+# Copyright (C) 2018 - 2019 The IPFire Team #
+# #
+###############################################################################
+
+use strict;
+use CGI qw/:standard/;
+#enable only the following on debugging purpose
+#use warnings;
+#use CGI::Carp 'fatalsToBrowser';
+use Sort::Naturally;
+use Socket;
+
+require '/var/ipfire/general-functions.pl';
+require "${General::swroot}/lang.pl";
+require "${General::swroot}/header.pl";
+
+###############################################################################
+# Initialize variables and hashes
+###############################################################################
+
+my $settings = "${General::swroot}/ipblacklist/settings";
+my $modified = "${General::swroot}/ipblacklist/modified";
+my $sources = "${General::swroot}/ipblacklist/sources";
+my $getipstat = '/usr/local/bin/getipstat';
+my $getipsetstat = '/usr/local/bin/getipsetstat';
+my $control = '/usr/local/bin/ipblacklistctrl';
+my $lockfile = '/var/run/ipblacklist.pid';
+my $autoblacklist = 'AUTOBLACKLIST';
+my %cgiparams = ('ACTION' => '', 'AUTOACTION' => '');
+my $errormessage = '';
+my $updating = 0;
+my %mainsettings;
+my %color;
+my %modified;
+my %sources;
+my %stats;
+my %autoblock_addresses;
+
+my %settings = ( 'DEBUG' => 0,
+ 'LOGGING' => 'on',
+ 'RATE' => 24,
+ 'ENABLE' => 'off',
+ 'BLOCK_THRESHOLD' => 10,
+ 'BLOCK_PERIOD' => 3600,
+ $autoblacklist => 'off' );
+
+# Read all parameters for site
+Header::getcgihash( \%cgiparams);
+General::readhash( "${General::swroot}/main/settings", \%mainsettings );
+General::readhash( "/srv/web/ipfire/html/themes/".$mainsettings{'THEME'}."/include/colors.txt", \%color );
+General::readhash( $settings, \%settings ) if (-r $settings);
+General::readhash( $modified, \%modified) if (-r $modified);
+eval qx|/bin/cat $sources| if (-r $sources);
+
+# Show Headers
+
+Header::showhttpheaders();
+
+# Process actions
+
+if ($cgiparams{'ACTION'} eq "$Lang::tr{'save'}")
+{
+ #Save Button on configsite
+
+ my %new_settings = ( 'ENABLE' => 'off',
+ 'RATE' => 24,
+ 'LOGGING' => 'off',
+ 'DEBUG' => 0,
+ 'BLOCK_THRESHOLD' => $settings{'BLOCK_THRESHOLD'} || 10,
+ 'BLOCK_PERIOD' => $settings{'BLOCK_PERIOD'} || 3600,
+ $autoblacklist => $settings{$autoblacklist} );
+
+ $errormessage .= "$Lang::tr{'ipblacklist invalid check rate'}<br>" if (($cgiparams{'RATE'} !~ m/^\d+$/) or
+ ($cgiparams{'RATE'} < 1) or
+ ($cgiparams{'RATE'} > 1000));
+
+ $new_settings{'RATE'} = $cgiparams{'RATE'};
+ delete $cgiparams{'RATE'};
+
+ foreach my $item ('LOGGING', 'ENABLE', keys %sources)
+ {
+ $new_settings{$item} = (exists $cgiparams{$item}) ? 'on' : 'off';
+
+ $updating = 1 if (not exists $settings{$item} or $new_settings{$item} ne $settings{$item});
+ }
+
+ # Check for redundant blacklists being enabled
+
+ foreach my $list (keys %sources)
+ {
+ if (exists $settings{$list} and
+ $settings{$list} eq 'on' and
+ exists $sources{$list}{'override'} and
+ $settings{$sources{$list}{'override'}} eq 'on')
+ {
+ $settings{$sources{$list}{'override'}} = 'off';
+
+ $updating = 1;
+ $errormessage .= "$Lang::tr{'ipblacklist disable pre'} $sources{$list}{'override'} " .
+ "$Lang::tr{'ipblacklist disable mid'} $list $Lang::tr{'ipblacklist disable post'}<br>\n";
+ }
+ }
+
+ if ($settings{'LOGGING'} ne $new_settings{'LOGGING'})
+ {
+ if ($new_settings{'LOGGING'} eq 'on')
+ {
+ system( "$control log-on" );
+ }
+ else
+ {
+ system( "$control log-off" );
+ }
+ }
+
+ if ($settings{'ENABLE'} ne $new_settings{'ENABLE'})
+ {
+ if ($new_settings{'ENABLE'} eq 'on')
+ {
+ system( "$control enable" );
+ }
+ else
+ {
+ $settings{$autoblacklist} = 'off';
+ system( "$control disable" );
+ }
+
+ $updating = 1;
+ }
+
+ %settings = %new_settings;
+
+ if ($errormessage)
+ {
+ $updating = 0;
+ }
+ else
+ {
+ General::writehash($settings, \%new_settings);
+
+ if ($updating)
+ {
+ system( "$control update &" );
+ get_ipset_stats();
+ show_running();
+ exit 0;
+ }
+ }
+}
+elsif ($cgiparams{'AUTOACTION'} eq "$Lang::tr{'save'}")
+{
+ $updating = 1 if ($settings{$autoblacklist} eq 'on' and not exists $cgiparams{$autoblacklist});
+ $updating = 1 if ($settings{$autoblacklist} eq 'off' and exists $cgiparams{$autoblacklist});
+
+ $settings{$autoblacklist} = (exists $cgiparams{$autoblacklist}) ? 'on' : 'off';
+ $settings{'BLOCK_THRESHOLD'} = $cgiparams{'BLOCK_THRESHOLD'};
+ $settings{'BLOCK_PERIOD'} = $cgiparams{'BLOCK_PERIOD'} ;
+
+ if (($cgiparams{'BLOCK_THRESHOLD'} !~ m/^\d+$/) or
+ ($cgiparams{'BLOCK_THRESHOLD'} < 1) or
+ ($cgiparams{'BLOCK_THRESHOLD'} > 1000000))
+ {
+ $errormessage .= "$Lang::tr{'ipblacklist invalid threshold'}: $cgiparams{'BLOCK_THRESHOLD'}<br>";
+ }
+
+ if (($cgiparams{'BLOCK_PERIOD'} !~ m/^\d+$/) or
+ ($cgiparams{'BLOCK_PERIOD'} < 1) or
+ ($cgiparams{'BLOCK_PERIOD'} > 86400))
+ {
+ $errormessage .= "$Lang::tr{'ipblacklist invalid block time'}: $cgiparams{'BLOCK_PERIOD'}<br>";
+ }
+
+ if ($errormessage)
+ {
+ $updating = 0;
+ }
+ else
+ {
+ General::writehash($settings, \%settings);
+ system( "$control autoblacklist-update" ) if ($updating);
+ }
+}
+elsif ($cgiparams{'AUTOACTION'} eq "$Lang::tr{'unblock all'}")
+{
+ system( "$control autoblacklist-clear" );
+}
+
+get_ipset_stats();
+
+if (is_running())
+{
+ show_running();
+ exit 0;
+}
+
+# Get blacklist statistics
+
+get_iptables_stats();
+
+# Show site
+
+Header::openpage($Lang::tr{'ipblacklist'}, 1, '');
+Header::openbigbox('100%', 'left');
+error();
+
+showstatus() if ($settings{ENABLE} eq 'on');
+
+configsite();
+
+# End of page
+
+Header::closebigbox();
+Header::closepage();
+
+exit 0;
+
+
+#------------------------------------------------------------------------------
+# sub configsite()
+#
+# Displays configuration
+#------------------------------------------------------------------------------
+
+sub configsite
+{
+ # Find preselections
+
+ my $enable = 'checked';
+ Header::openbox('100%', 'left', $Lang::tr{'settings'});
+
+ #### JAVA SCRIPT ####
+
+ print<<END;
+<script>
+ \$(document).ready(function()
+ {
+ // Show/Hide elements when ENABLE checkbox is checked.
+ if (\$("#ENABLE").attr("checked"))
+ {
+ \$(".sources").show();
+ }
+ else
+ {
+ \$(".sources").hide();
+ }
+
+ // Toggle Source list elements when "ENABLE" checkbox is clicked
+ \$("#ENABLE").change(function()
+ {
+ \$(".sources").toggle();
+ });
+ });
+</script>
+END
+
+ ##### JAVA SCRIPT END ####
+
+ # Enable checkbox
+
+ $enable = ($settings{'ENABLE'} eq 'on') ? ' checked' : '';
+
+ print<<END;
+ <form method='post' action='$ENV{'SCRIPT_NAME'}'>
+ <table style='width:100%' border='0'>
+ <tr>
+ <td style='width:24em'>$Lang::tr{'ipblacklist use ipblacklists'}</td>
+ <td><input type='checkbox' name='ENABLE' id='ENABLE'$enable></td>
+ </tr>
+ </table><br>
+
+END
+
+ # The following are only displayed if the blacklists are enabled
+
+ $enable = ($settings{'LOGGING'} eq 'on') ? ' checked' : '';
+
+ print <<END;
+<div class='sources'>
+ <table style='width:100%' border='0'>
+ <tr>
+ <td style='width:24em'>$Lang::tr{'ipblacklist log'}</td>
+ <td><input type='checkbox' name="LOGGING" id="LOGGING"$enable></td>
+ <td style='width:24em'>$Lang::tr{'ipblacklist check rate'}</td>
+ <td>
+ <input type='number' name='RATE' min='1' max='1000' maxlength='7' pattern='\\d+' value='$settings{'RATE'}'>
+ </td>
+ </tr>
+ </table>
+ <br><br>
+ <h2>$Lang::tr{'ipblacklist blacklist settings'}</h2>
+ <table width='100%' cellspacing='1' class='tbl'>
+ <tr>
+ <th align='left'>$Lang::tr{'ipblacklist id'}</th>
+ <th align='left'>$Lang::tr{'ipblacklist name'}</th>
+ <th align='center'>$Lang::tr{'ipblacklist safe'}</th>
+ <th align='center'>$Lang::tr{'ipblacklist enable'}</th>
+ </tr>
+END
+
+ # Iterate through the list of sources
+
+ my $lines = 0;
+
+ foreach my $list (sort keys %sources)
+ {
+ my $name = escapeHTML( $sources{$list}{'name'} );
+ my $safe = $Lang::tr{$sources{$list}{safe}};
+ $enable = '';
+ my $col = ($lines++ % 2) ? "bgcolor='$color{'color20'}'" : "bgcolor='$color{'color22'}'";
+
+ $enable = ' checked' if (exists $settings{$list} and $settings{$list} eq 'on');
+
+ print <<END;
+ <tr $col>
+ <td>
+END
+
+ if ($sources{$list}{info})
+ {
+ print "<a href='$sources{$list}{info}' target='_blank'>$list</a>\n";
+ }
+ else
+ {
+ print "$list\n";
+ }
+
+ print <<END;
+ </td>
+ <td>$name</td>
+ <td align='center'>$safe</td>
+ <td align='center'><input type='checkbox' name="$list" id="$list"$enable></td>
+ </tr>\n
+END
+ }
+
+ # The save button at the bottom of the table
+
+ print <<END;
+ </table>
+ <p>$Lang::tr{'ipblacklist safe note'}</p>
+ </div>
+ <table style='width:100%;'>
+ <tr>
+ <td colspan='3' display:inline align='right'><input type='submit' name='ACTION' value='$Lang::tr{'save'}'></td>
+ </tr>
+ </table>
+ <div class='sources'>
+ <br>
+END
+
+ $enable = $settings{$autoblacklist} eq 'on' ? ' checked' : '';
+
+ print <<END;
+ <br><br>
+ <h2>$Lang::tr{'ipblacklist auto list'}</h2>
+ <table style='width:100%' border='0'>
+ <tr>
+ <td>
+ $Lang::tr{'ipblacklist autoblacklist enable'}
+ </td>
+ <td>
+ <input type='checkbox' name="$autoblacklist" id="$autoblacklist"$enable>
+ </td>
+ <td>
+
+ </td>
+ </tr>
+ <tr>
+ <td>
+ $Lang::tr{'ipblacklist autoblacklist threshold'}
+ </td>
+ <td>
+ <input type='number' name='BLOCK_THRESHOLD' min='1' max='1000000' maxlength='7' pattern='\\d+' value='$settings{BLOCK_THRESHOLD}'>
+ </input>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ $Lang::tr{'ipblacklist autoblacklist block time'}
+ </td>
+ <td>
+ <input type='number' name='BLOCK_PERIOD' min='1' max='86400' maxlength='7' pattern='\\d+' value='$settings{BLOCK_PERIOD}'>
+ </input>
+ </td>
+ </tr>
+ </table>
+ <table style='width:100%;'>
+ <tr>
+ <td colspan='3' display:inline align='right'><input type='submit' name='AUTOACTION' value='$Lang::tr{'save'}'></td>
+ </tr>
+ </table>
+ </form>
+ </div>
+END
+
+ Header::closebox();
+}
+
+
+#------------------------------------------------------------------------------
+# sub showstatus()
+#
+# Displays current blacklist status
+#------------------------------------------------------------------------------
+
+sub showstatus
+{
+ Header::openbox('100%', 'center', $Lang::tr{'status'});
+
+ print <<END;
+ <table width='100%' cellspacing='1'>
+ <tr>
+ <th align='left'>$Lang::tr{'ipblacklist id'}</th>
+ <th align='right'>$Lang::tr{'ipblacklist entries'}</th>
+ <th align='right'>$Lang::tr{'ipblacklist pkts in'}</th>
+ <th align='right'>$Lang::tr{'ipblacklist bytes in'}</th>
+ <th align='right'>$Lang::tr{'ipblacklist pkts out'}</th>
+ <th align='right'>$Lang::tr{'ipblacklist bytes out'}</th>
+ <th align='center'>$Lang::tr{'ipblacklist updated'}</th>
+ </tr>
+END
+
+ # Iterate through the list of sources
+
+ foreach my $list ($autoblacklist, sort keys %sources)
+ {
+ next unless ($settings{$list} eq 'on');
+
+ my $size = ' ';
+ my $pkts_in = ' ';
+ my $bytes_in = ' ';
+ my $pkts_out = ' ';
+ my $bytes_out = ' ';
+ my $updated = ' ';
+
+ if (exists $stats{$list})
+ {
+ ($pkts_in, $bytes_in) = @{ $stats{$list}{IPBLACKLISTREDIN} } if (exists $stats{$list}{IPBLACKLISTREDIN});
+ ($pkts_out, $bytes_out) = @{ $stats{$list}{IPBLACKLISTREDOUT} } if (exists $stats{$list}{IPBLACKLISTREDOUT});
+ $size = $stats{$list}{size} if (exists $stats{$list}{size});
+ }
+
+ if (exists $modified{$list} and $modified{$list} > 0)
+ {
+ $updated = localtime( $modified{$list} );
+ }
+
+ print <<END;
+ <tr>
+ <td>$list</td>
+ <td align='right'>$size</td>
+ <td align='right'>$pkts_in</td>
+ <td align='right'>$bytes_in</td>
+ <td align='right'>$pkts_out</td>
+ <td align='right'>$bytes_out</td>
+ <td align='center'>$updated</td>
+ </tr>\n
+END
+
+ }
+
+ print <<END;
+ </table>
+END
+
+ if ($settings{$autoblacklist} eq 'on')
+ {
+ print <<END;
+ <br><br>
+ <h2>$Lang::tr{'ipblacklist auto list'}</h2>
+ <table width='60%' cellspacing='1'>
+ <tr>
+ <th align='left'>$Lang::tr{'ip address'}</th>
+ <th align='center'>$Lang::tr{'ipblacklist block time remaining'}</th>
+ </tr>
+END
+
+ foreach my $address (nsort keys %autoblock_addresses )
+ {
+ print "<tr><td>$address</td><td align='center'>$autoblock_addresses{$address}</td></tr>\n";
+ }
+
+ print <<END;
+ </table>
+ <table width='60%' border='0'>
+ <form method='post' action='$ENV{'SCRIPT_NAME'}'>
+ <tr>
+ <td align='right' width='15%'><input type='submit' name='ACTION' value='$Lang::tr{'unblock all'}'></td>
+ </tr>
+ </form>
+ </table>
+END
+ }
+
+ Header::closebox();
+}
+
+
+#------------------------------------------------------------------------------
+# sub get_ipset_stats()
+#
+# Gets the number of entries in each IPSet.
+#------------------------------------------------------------------------------
+
+sub get_ipset_stats
+{
+ my $name;
+
+ system( $getipsetstat );
+
+ # Get the number of entries in each IP set
+
+ if (-r '/var/tmp/ipsets.txt')
+ {
+ open STATS, '<', '/var/tmp/ipsets.txt' or die "Can't open IP Sets stats file: $!";
+
+ foreach my $line (<STATS>)
+ {
+ if ($line =~ m/Name: (\w+)/)
+ {
+ $name = $1;
+ next;
+ }
+
+ if ($line =~ m/Number of entries: (\d+)/)
+ {
+ $stats{$name}{'size'} = $1;
+ }
+ }
+
+ close STATS;
+
+ unlink( '/var/tmp/ipsets.txt' );
+ }
+
+ # Get the IP addresses in the autoblacklist
+
+ if (-r '/var/tmp/autoblacklist.txt')
+ {
+ open HASHTABLE, '<', '/var/tmp/autoblacklist.txt' or die "Can't open autoblacklist address file: $!";
+
+ # Iterate through the blocked addresses
+
+ foreach my $line (<HASHTABLE>)
+ {
+ next unless ($line =~ m/(\d+\.\d+\.\d+\.\d+) timeout (\d+)/);
+
+ $autoblock_addresses{$1} = format_time( $2 );
+ }
+
+ close HASHTABLE;
+
+ unlink( '/var/tmp/autoblacklist.txt' );
+ }
+
+}
+
+
+#------------------------------------------------------------------------------
+# sub get_iptables_stats()
+#
+# Gets information on the number of packets and bytes rejected by each
+# blacklist
+#------------------------------------------------------------------------------
+
+sub get_iptables_stats
+{
+ system( $getipstat );
+ unlink( '/var/tmp/iptablesmangle.txt' );
+ unlink( '/var/tmp/iptablesnat.txt' );
+
+ return unless (-r '/var/tmp/iptables.txt');
+
+ open STATS, '<', '/var/tmp/iptables.txt' or die "Can't open IP Tables stats file: $!";
+
+ my $table = 'Unknown';
+
+ foreach my $line (<STATS>)
+ {
+ if ($line =~ m/^Chain (\w+)/)
+ {
+ $table = $1;
+ next;
+ }
+
+ next unless ($line =~ m/_BLOCK/);
+
+ my ($pkts, $bytes, $chain) = $line =~ m/^\s*(\d+\w?)\s+(\d+\w?)\s+(\w+)_BLOCK/;
+ $stats{$chain}{$table} = [ $pkts, $bytes ];
+ }
+
+ close STATS;
+
+ unlink( '/var/tmp/iptables.txt' );
+}
+
+
+#------------------------------------------------------------------------------
+# sub is_running()
+#
+# Checks to see if the main script is running
+#------------------------------------------------------------------------------
+
+sub is_running
+{
+ return 0 unless (-r $lockfile);
+
+ open LOCKFILE, '<', $lockfile or die "Can't open lockfile";
+ my $pid = <LOCKFILE>;
+ close LOCKFILE;
+
+ chomp $pid;
+
+ return (-e "/proc/$pid");
+}
+
+
+#------------------------------------------------------------------------------
+# sub show_running
+#
+# Displayed when update is running
+#------------------------------------------------------------------------------
+
+sub show_running
+{
+ # Open site
+
+ Header::openpage( $Lang::tr{'ipblacklist'}, 1, '<meta http-equiv="refresh" content="1;url=/cgi-bin/ipblacklist.cgi">' );
+ Header::openbigbox( '100%', 'center' );
+ error();
+ Header::openbox( 'Working', 'center', "$Lang::tr{'ipblacklist working'}" );
+
+ print <<END;
+ <table width='100%'>
+ <tr>
+ <td align='center'>
+ <img src='/images/indicator.gif' alt='$Lang::tr{'aktiv'}'>
+ <td>
+ </tr>
+ </table>
+ <br>
+ <table cellspacing='1' align='center'>
+ <tr><th>$Lang::tr{'ipblacklist id'}</th><th>$Lang::tr{'ipblacklist entries'}</th></tr>
+END
+
+ foreach my $name (keys %sources)
+ {
+ $stats{$name}{'size'} = ' ' if (not exists ($stats{$name}) and
+ exists $settings{$name} and
+ $settings{$name} eq 'on');
+ }
+
+ foreach my $name (sort keys %stats)
+ {
+ print "<tr><td>$name</td><td align='right'>$stats{$name}{'size'}</td></tr>\n" if (exists $stats{$name}{'size'});
+ }
+
+ print <<END;
+ </table>
+END
+
+ Header::closebox();
+
+ Header::closebigbox();
+ Header::closepage();
+}
+
+
+#------------------------------------------------------------------------------
+# sub error()
+#
+# Shows error messages
+#------------------------------------------------------------------------------
+
+sub error
+{
+ if ($errormessage)
+ {
+ Header::openbox('100%', 'left', $Lang::tr{'error messages'});
+ print "<class name='base'>$errormessage\n";
+ print " </class>\n";
+ Header::closebox();
+ }
+}
+
+
+#------------------------------------------------------------------------------
+# sub format_time( seconds )
+#
+# Converts time in seconds to HH:MM:SS
+#------------------------------------------------------------------------------
+
+sub format_time($) {
+ my $time = shift;
+
+ my $seconds = $time % 60;
+ my $minutes = $time / 60;
+
+ my $hours = 0;
+ if ($minutes >= 60) {
+ $hours = $minutes / 60;
+ $minutes %= 60;
+ }
+
+ return sprintf("%3d:%02d:%02d", $hours, $minutes, $seconds);
+}
diff --git a/html/cgi-bin/logs.cgi/log.dat b/html/cgi-bin/logs.cgi/log.dat
index 8ca32d675..ba1a482c8 100644
--- a/html/cgi-bin/logs.cgi/log.dat
+++ b/html/cgi-bin/logs.cgi/log.dat
@@ -59,6 +59,7 @@ my %sections = (
'dhcp' => '(dhcpd: )',
'dma' => '(dma: |dma\[.*\]: |postfix/\w*\[\d*\]: )',
'guardian' => '(guardian\[.*\]: )',
+ 'ipblacklist' => '(ipblacklist: )',
'ipfire' => '(ipfire: )',
'ipsec' => '(ipsec_[\w_]+: |pluto\[.*\]: |charon: |vpnwatch: )',
'kernel' => '(kernel: (?!DROP_))',
@@ -87,6 +88,7 @@ my %trsections = (
'dhcp' => "$Lang::tr{'dhcp server'}",
'dma' => 'Mail',
'guardian' => "$Lang::tr{'guardian'}",
+ 'ipblacklist' => "$Lang::tr{'ipblacklist'}",
'ipfire' => 'IPFire',
'ipsec' => 'IPSec',
'kernel' => "$Lang::tr{'kernel'}",
diff --git a/langs/en/cgi-bin/en.pl b/langs/en/cgi-bin/en.pl
index b40ef9390..5acf3678c 100644
--- a/langs/en/cgi-bin/en.pl
+++ b/langs/en/cgi-bin/en.pl
@@ -1518,6 +1518,37 @@
'ip alias changed' => 'External IP alias changed',
'ip alias removed' => 'External IP alias removed',
'ip info' => 'IP Information',
+'ipblacklist auto list' => 'Automatic blacklist',
+'ipblacklist autoblacklist enable' => 'Enable automatically updating local blacklist',
+'ipblacklist autoblacklist threshold' => 'Threshold (packets/hour)',
+'ipblacklist autoblacklist block time' => 'Block time (seconds)',
+'ipblacklist blacklist settings' => 'Blacklist settings',
+'ipblacklist bytes in' => 'bytes in',
+'ipblacklist bytes out' => 'bytes out',
+'ipblacklist check rate' => 'Update check rate (hours)',
+'ipblacklist day' => 'day',
+'ipblacklist disable pre' => 'Disabling',
+'ipblacklist disable mid' => 'because it is included in',
+'ipblacklist disable port' => '',
+'ipblacklist enable' => 'Enable',
+'ipblacklist entries' => 'Entries',
+'ipblacklist hour' => 'hour',
+'ipblacklist id' => 'Blacklist',
+'ipblacklist invalid block time' => 'Invalid automatic blacklist block time',
+'ipblacklist invalid check rate' => 'Invalid update check rate',
+'ipblacklist invalid threshold' => 'Invalid automatic blacklist threshold',
+'ipblacklist log' => 'Log dropped packets',
+'ipblacklist name' => 'Name',
+'ipblacklist pkts in' => 'pkts in',
+'ipblacklist pkts out' => 'pkts out',
+'ipblacklist safe note' => 'Note: safe blacklists block addresses that only generate malicious traffic and therefore will not block any wanted sites.',
+'ipblacklist safe' => 'Safe',
+'ipblacklist sixhour' => 'six hours',
+'ipblacklist updated' => 'Last updated',
+'ipblacklist use ipblacklists' => 'Enable IP Blacklists',
+'ipblacklist week' => 'week',
+'ipblacklist working' => 'Updating IP address blacklists...',
+'ipblacklist' => 'IP Address Blacklists',
'ipfire has now rebooted' => 'IPFire is rebooting now.',
'ipfire has now shutdown' => 'IPFire is shutting down now.',
'ipfire side' => 'IPFire side:',
--
2.16.4
^ permalink raw reply [flat|nested] 35+ messages in thread
* [PATCH 3/5] ipblacklist: Ancillary files
2019-11-25 20:13 [PATCH 0/5] ipblacklist: IP Address Blacklists Tim FitzGeorge
2019-11-25 20:13 ` [PATCH 1/5] ipblacklist: Main script Tim FitzGeorge
2019-11-25 20:13 ` [PATCH 2/5] ipblacklist: WUI and language file Tim FitzGeorge
@ 2019-11-25 20:13 ` Tim FitzGeorge
2019-11-25 20:13 ` [PATCH 4/5] ipblacklist: Modifications to system Tim FitzGeorge
` (3 subsequent siblings)
6 siblings, 0 replies; 35+ messages in thread
From: Tim FitzGeorge @ 2019-11-25 20:13 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 15594 bytes --]
getipsetstat Gets information on IPSETs for WUI
ipblacklistctrl Allows WUI to call main script as root
sources List of blacklists used by main script and WUI
Signed-off-by: Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk>
---
config/ipblacklist/sources | 151 +++++++++++++++++++++++++++++++++++++++
src/misc-progs/getipsetstat.c | 28 ++++++++
src/misc-progs/ipblacklistctrl.c | 52 ++++++++++++++
3 files changed, 231 insertions(+)
create mode 100644 config/ipblacklist/sources
create mode 100644 src/misc-progs/getipsetstat.c
create mode 100644 src/misc-progs/ipblacklistctrl.c
diff --git a/config/ipblacklist/sources b/config/ipblacklist/sources
new file mode 100644
index 000000000..ab991e12a
--- /dev/null
+++ b/config/ipblacklist/sources
@@ -0,0 +1,151 @@
+############################################################################
+# #
+# IP Address blacklists for IPFire #
+# #
+# This file contains a list of blacklist sources that will replace the one #
+# internal to the updated if it is found at /var/ipfire/blacklist/sources. #
+# The intention is to provide a common source of information for both the #
+# updater and WUI. #
+# #
+# The chains created in the packet filter will be named by the top level #
+# key and this will also be used in the log message to identify the reason #
+# for the dropped packet. #
+# #
+# The fields are: #
+# #
+# name The blacklist's full name #
+# url URL of the file containing the list #
+# info URL giving information about the source #
+# parser The parser function used to extract IP addresses from the #
+# downloaded list #
+# method Method used to download updates. #
+# rate Minimum number of hours between checks for updates #
+# safe 'yes' if the list is unlikely to contain addresses that can be #
+# used for legitimate traffic, or 'no' otherwise #
+# disable Name of another list to disable if this one is enabled. Used #
+# when the other list is a subset of this one. #
+# #
+# The info and safe fields are purely for documentation. #
+# #
+# Note that the Emerging Threats blacklist is a composite list containing #
+# addresses from some of the other lists. It is unnecessary to enable #
+# this list if the other lists are enabled. #
+# #
+############################################################################
+
+%sources = ( 'EMERGING_FWRULE' => { 'name' => 'Emerging Threats Blocklist',
+ 'url' => 'https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt',
+ 'info' => 'https://doc.emergingthreats.net/bin/view/Main/EmergingFirewallRules',
+ 'parser' => 'text-with-hash-comments',
+ 'method' => 'check-header-time',
+ 'rate' => 1,
+ 'safe' => 'no' },
+ 'EMERGING_COMPROMISED' => { 'name' => 'Emerging Threats Compromised IPs',
+ 'url' => 'https://rules.emergingthreats.net/blockrules/compromised-ips.txt',
+ 'info' => 'https://doc.emergingthreats.net/bin/view/Main/CompromisedHost',
+ 'parser' => 'text-with-hash-comments',
+ 'method' => 'check-header-time',
+ 'rate' => 1,
+ 'safe' => 'no' },
+ 'SPAMHAUS_DROP' => { 'name' => "Spamhaus Don't Route or Peer List",
+ 'url' => 'https://www.spamhaus.org/drop/drop.txt',
+ 'info' => 'https://www.spamhaus.org/drop/',
+ 'parser' => 'text-with-semicolon-comments',
+ 'method' => 'check-header-time',
+ 'rate' => 12,
+ 'safe' => 'yes' },
+ 'SPAMHAUS_EDROP' => { 'name' => "Spamhaus Extended Don't Route or Peer List",
+ 'url' => 'https://www.spamhaus.org/drop/edrop.txt',
+ 'info' => 'https://www.spamhaus.org/drop/',
+ 'parser' => 'text-with-semicolon-comments',
+ 'method' => 'check-header-time',
+ 'rate' => 1,
+ 'safe' => 'no' },
+ 'DSHIELD' => { 'name' => 'Dshield.org Recommended Block List',
+ 'url' => 'https://www.dshield.org/block.txt',
+ 'info' => 'https://dshield.org/',
+ 'parser' => 'dshield',
+ 'method' => 'check-header-time',
+ 'rate' => 2,
+ 'safe' => 'no' },
+ 'FEODO_IP' => { 'name' => 'Feodo Trojan IP Blocklist',
+ 'url' => 'https://feodotracker.abuse.ch/downloads/ipblocklist.txt',
+ 'info' => 'https://feodotracker.abuse.ch/blocklist',
+ 'parser' => 'text-with-hash-comments',
+ 'method' => 'check-header-time',
+ 'rate' => 1,
+ 'safe' => 'no' },
+ 'FEODO_AGGRESIVE' => { 'name' => 'Feodo Trojan IP Blocklist (Aggresive)',
+ 'url' => 'https://feodotracker.abuse.ch/downloads/ipblocklist_aggressive.txt',
+ 'info' => 'https://feodotracker.abuse.ch/blocklist',
+ 'parser' => 'text-with-hash-comments',
+ 'method' => 'check-header-time',
+ 'rate' => 1,
+ 'safe' => 'no',
+ 'disable' => 'FEODO_IP' },
+ 'ABUSE_CH' => { 'name' => 'Abuse.ch Ransomware C&C Blocklist',
+ 'url' => 'https://ransomwaretracker.abuse.ch/downloads/RW_IPBL.txt',
+ 'info' => 'https://ransomwaretracker.abuse.ch/blocklist/',
+ 'parser' => 'text-with-hash-comments',
+ 'method' => 'check-header-time',
+ 'rate' => 1,
+ 'safe' => 'no' },
+ 'CIARMY' => { 'name' => 'The CINS Army List',
+ 'url' => 'https://cinsscore.com/list/ci-badguys.txt',
+ 'info' => 'https://cinsscore.com/#list',
+ 'parser' => 'text-with-hash-comments',
+ 'method' => 'check-header-time',
+ 'rate' => 1,
+ 'safe' => 'no' },
+ 'TOR_ALL' => { 'name' => 'Known TOR Nodes',
+ 'url' => 'https://www.dan.me.uk/torlist',
+ 'info' => 'https://www.dan.me.uk/tornodes',
+ 'parser' => 'text-with-hash-comments',
+ 'method' => 'wget',
+ 'rate' => 1,
+ 'safe' => 'no',
+ 'disable' => 'TOR_EXIT' },
+ 'TOR_EXIT' => { 'name' => 'Known TOR Exit Nodes',
+ 'url' => 'https://www.dan.me.uk/torlist/?exit',
+ 'info' => 'https://www.dan.me.uk/tornodes',
+ 'parser' => 'text-with-hash-comments',
+ 'method' => 'wget',
+ 'rate' => 1,
+ 'safe' => 'no' },
+ 'TALOS_MALICIOUS' => { 'name' => 'Talos Malicious hosts list',
+ 'url' => 'https://www.talosintelligence.com/documents/ip-blacklist',
+ 'info' => 'https://www.talosintelligence.com/reputation',
+ 'parser' => 'text-with-hash-comments',
+ 'method' => 'wget',
+ 'rate' => 24,
+ 'safe' => 'no' },
+ 'ALIENVAULT' => { 'name' => 'AlienVault IP Reputation database',
+ 'url' => 'https://reputation.alienvault.com/reputation.generic',
+ 'info' => 'https://www.alienvault.com/resource-center/videos/what-is-ip-domain-reputation',
+ 'parser' => 'text-with-hash-comments',
+ 'method' => 'check-header-time',
+ 'rate' => 1,
+ 'safe' => 'no' },
+ 'BOGON' => { 'name' => 'Bogus address list (Martian)',
+ 'url' => 'https://www.team-cymru.org/Services/Bogons/bogon-bn-agg.txt',
+ 'info' => 'https://www.team-cymru.com/bogon-reference.html',
+ 'parser' => 'text-with-hash-comments',
+ 'method' => 'check-header-time',
+ 'rate' => 24,
+ 'safe' => 'yes' },
+ 'BOGON_FULL' => { 'name' => 'Full Bogus Address List',
+ 'url' => 'https://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt',
+ 'info' => 'https://www.team-cymru.com/bogon-reference.html',
+ 'parser' => 'text-with-hash-comments',
+ 'method' => 'check-header-time',
+ 'rate' => 24,
+ 'safe' => 'yes',
+ 'disable' => 'BOGON' },
+ 'SHODAN' => { 'name' => 'ISC Shodan scanner blacklist',
+ 'url' => 'https://isc.sans.edu/api/threatlist/shodan?tab',
+ 'info' => 'https://isc.sans.edu',
+ 'parser' => 'text-with-hash-comments',
+ 'method' => 'wget',
+ 'rate' => 24,
+ 'safe' => 'no' }
+ );
diff --git a/src/misc-progs/getipsetstat.c b/src/misc-progs/getipsetstat.c
new file mode 100644
index 000000000..aee79542a
--- /dev/null
+++ b/src/misc-progs/getipsetstat.c
@@ -0,0 +1,28 @@
+/* IPFire helper program - GetIPSetStat
+ *
+ * Get the list from IPSET LIST
+ *
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+#include <stdlib.h>
+#include <sys/types.h>
+#include <fcntl.h>
+#include "setuid.h"
+
+
+int main(void)
+{
+ if (!(initsetuid()))
+ exit(1);
+
+ safe_system("/usr/sbin/ipset list -t -f /var/tmp/ipsets.txt");
+ safe_system("chown nobody:nobody /var/tmp/ipsets.txt");
+
+ safe_system("/usr/sbin/ipset list AUTOBLACKLIST -q -f /var/tmp/autoblacklist.txt");
+ safe_system("chown -f nobody:nobody /var/tmp/autoblacklist.txt");
+
+ return 0;
+}
diff --git a/src/misc-progs/ipblacklistctrl.c b/src/misc-progs/ipblacklistctrl.c
new file mode 100644
index 000000000..506fa2f46
--- /dev/null
+++ b/src/misc-progs/ipblacklistctrl.c
@@ -0,0 +1,52 @@
+/* This file is part of the IPFire Firewall.
+ *
+ * This program is distributed under the terms of the GNU General Public
+ * Licence. See the file COPYING for details.
+ *
+ */
+
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+#include <sys/types.h>
+#include <fcntl.h>
+#include "setuid.h"
+
+int main(int argc, char *argv[]) {
+
+ if (!(initsetuid()))
+ exit(1);
+
+ if (argc < 2) {
+ fprintf(stderr, "\nNo argument given.\n"
+ "ipblacklistctrl (update|restore|log-on|log-off|"
+ "enable|disable|autoblacklist-update|autoblacklist-clear)\n\n");
+ exit(1);
+ }
+
+ if (strcmp(argv[1], "update") == 0) {
+ safe_system("/usr/local/bin/ipblacklist update >/dev/null 2>&1 &");
+ } else if (strcmp(argv[1], "restore") == 0) {
+ safe_system("/usr/local/bin/ipblacklist restore >/dev/null 2>&1 &");
+ } else if (strcmp(argv[1], "log-on") == 0) {
+ safe_system("/usr/local/bin/ipblacklist log-on >/dev/null 2>&1 &");
+ } else if (strcmp(argv[1], "log-off") == 0) {
+ safe_system("/usr/local/bin/ipblacklist log-off >/dev/null 2>&1 &");
+ } else if (strcmp(argv[1], "enable") == 0) {
+ safe_system("/usr/local/bin/ipblacklist enable >/dev/null 2>&1 &");
+ } else if (strcmp(argv[1], "disable") == 0) {
+ safe_system("/usr/local/bin/ipblacklist disable >/dev/null 2>&1 &");
+ } else if (strcmp(argv[1], "autoblacklist-update") == 0) {
+ safe_system("/usr/local/bin/ipblacklist autoblacklist-update >/dev/null 2>&1 &");
+ } else if (strcmp(argv[1], "autoblacklist-clear") == 0) {
+ safe_system("/usr/local/bin/ipblacklist autoblacklist-clear >/dev/null 2>&1 &");
+ } else {
+ fprintf(stderr, "\nBad argument given.\n"
+ "ipblacklistctrl (update|restore|log-on|log-off|"
+ "enable|disable|autoblacklist-update|autoblacklist-clear)\n\n");
+ exit(1);
+ }
+
+ return 0;
+}
--
2.16.4
^ permalink raw reply [flat|nested] 35+ messages in thread
* [PATCH 4/5] ipblacklist: Modifications to system
2019-11-25 20:13 [PATCH 0/5] ipblacklist: IP Address Blacklists Tim FitzGeorge
` (2 preceding siblings ...)
2019-11-25 20:13 ` [PATCH 3/5] ipblacklist: Ancillary files Tim FitzGeorge
@ 2019-11-25 20:13 ` Tim FitzGeorge
2019-11-25 20:13 ` [PATCH 5/5] ipblacklist: Build infrastructure Tim FitzGeorge
` (2 subsequent siblings)
6 siblings, 0 replies; 35+ messages in thread
From: Tim FitzGeorge @ 2019-11-25 20:13 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 9361 bytes --]
backup.pl Restart when restoring backup
ipblacklist ) Adds ipblacklist stats and errors to
ipblacklist.conf ) daily log summary
include Add blacklists and settings to backups
firewall-policy ) Add main IPTables used to invoke
firewall ) IPSets
Signed-off-by: Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk>
---
config/backup/backup.pl | 1 +
config/backup/include | 2 +
config/firewall/firewall-policy | 5 ++
config/logwatch/ipblacklist | 103 +++++++++++++++++++++++++++++++++++++++
config/logwatch/ipblacklist.conf | 34 +++++++++++++
src/initscripts/system/firewall | 20 ++++++++
6 files changed, 165 insertions(+)
create mode 100644 config/logwatch/ipblacklist
create mode 100644 config/logwatch/ipblacklist.conf
diff --git a/config/backup/backup.pl b/config/backup/backup.pl
index b1dd1d297..17b797c20 100644
--- a/config/backup/backup.pl
+++ b/config/backup/backup.pl
@@ -130,6 +130,7 @@ restore_backup() {
# Reload firewall
firewallctrl
+ /usr/local/bin/ipblacklistctrl restore
# Convert old OpenVPN CCD files (CN change, Core Update 75)
convert-ovpn
diff --git a/config/backup/include b/config/backup/include
index 1190eda81..78ff926f7 100644
--- a/config/backup/include
+++ b/config/backup/include
@@ -38,6 +38,7 @@
/var/ipfire/ethernet/wireless
/var/ipfire/firewall
/var/ipfire/fwhosts
+/var/ipfire/ipblacklist/modified
/var/ipfire/main/*
/var/ipfire/ovpn
/var/ipfire/ovpn/collectd.vpn
@@ -52,6 +53,7 @@
/var/ipfire/time/
/var/ipfire/urlfilter
/var/ipfire/vpn
+/var/lib/ipblacklist
/var/lib/suricata
/var/log/ip-acct/*
/var/log/rrd/*
diff --git a/config/firewall/firewall-policy b/config/firewall/firewall-policy
index 21165e933..1198d120f 100755
--- a/config/firewall/firewall-policy
+++ b/config/firewall/firewall-policy
@@ -22,6 +22,7 @@
eval $(/usr/local/bin/readhash /var/ipfire/ethernet/settings)
eval $(/usr/local/bin/readhash /var/ipfire/firewall/settings)
eval $(/usr/local/bin/readhash /var/ipfire/optionsfw/settings)
+eval $(/usr/local/bin/readhash /var/ipfire/ipblacklist/settings)
function iptables() {
/sbin/iptables --wait "$@"
@@ -97,6 +98,10 @@ case "${HAVE_OPENVPN},${POLICY}" in
;;
esac
+if [ "${AUTOBLACKLIST}" = "on" ]; then
+ iptables -A POLICYIN -i ${IFACE} -m hashlimit --hashlimit-mode srcip --hashlimit-above ${BLOCK_THRESHOLD}/hour --hashlimit-name AUTOBLACKLIST -j SET --add-set AUTOBLACKLIST src
+fi
+
case "${FWPOLICY2}" in
REJECT)
if [ "${DROPINPUT}" = "on" ]; then
diff --git a/config/logwatch/ipblacklist b/config/logwatch/ipblacklist
new file mode 100644
index 000000000..0fadc6250
--- /dev/null
+++ b/config/logwatch/ipblacklist
@@ -0,0 +1,103 @@
+###########################################################################
+# ipblacklist script for Logwatch
+# Analyzes the IPFire IP Blacklist log
+#
+#########################################################################
+
+########################################################
+## Copyright (c) 2008 Lars Skj�rlund
+## Covered under the included MIT/X-Consortium License:
+## http://www.opensource.org/licenses/mit-license.php
+## All modifications and contributions by other persons to
+## this script are assumed to have been donated to the
+## Logwatch project and thus assume the above copyright
+## and licensing terms. If you want to make contributions
+## under your own copyright or a different license this
+## must be explicitly stated in the contribution and the
+## Logwatch project reserves the right to not accept such
+## contributions. If you have made significant
+## contributions to this script and want to claim
+## copyright please contact logwatch-devel(a)lists.sourceforge.net.
+#########################################################
+
+#########################################################################
+# Files - all shown with default paths:
+#
+# /usr/share/logwatch/default.conf/logfiles/messages.conf
+# /usr/share/logwatch/dist.conf/services/blacklist.conf
+# /usr/share/logwatch/scripts/services/ipblacklist (this file)
+#
+# ... and of course
+#
+# /var/log/messages
+#########################################################################
+
+use Logwatch ':dates';
+
+my $Detail = $ENV{'LOGWATCH_DETAIL_LEVEL'};
+
+my $SearchDate;
+
+my %Updates;
+my %Errors;
+
+$SearchDate = TimeFilter("%b %e");
+
+while (defined(my $ThisLine = <STDIN>))
+{
+ next unless ($ThisLine =~ m/^\s*\w+\s+\w+\s+(..:..:..) .* ipblacklist: (.*)/);
+
+ my $text = $2;
+
+ if ($text =~ m/Finished updating (\w+) blacklist with (\d+) changes/)
+ {
+ $Updates{$1}{updates}++;
+ $Updates{$1}{changes} += $2;
+ }
+ elsif ($text !~ m/Starting IP Blacklists/ and
+ $text !~ m/Starting IP Blacklist processing/ and
+ $text !~ m/Updating \w+ blacklist/ and
+ $text !~ m/Stopping IP Blacklists/ and
+ $text !~ m/Deleting IP Blacklists/ and
+ $text !~ m/Completed IP Blacklist update/ and
+ $text !~ m/Finished IP Blacklist processing/ and
+ $text !~ m/Blacklist \w+ Modification times/ and
+ $text !~ m/Create IPTables chains for blacklist/ and
+ $text !~ m/Delete IPTables chains for blacklist/ and
+ $text !~ m/Checking modification time for blacklist/ and
+ $text !~ m/Restoring blacklist / and
+ $text !~ m/Downloading blacklist/ )
+ {
+ $Errors{$text}++;
+ }
+}
+
+#####################################################################
+
+if (keys %Updates)
+{
+ print "\nThe following block lists were updated:\n";
+ foreach my $Lists (sort keys %Updates)
+ {
+ print " $Lists: $Updates{$Lists}{updates} Time(s) - $Updates{$Lists}{changes} change(s)\n";
+ }
+}
+
+if (keys %Errors)
+{
+ print "\nThe following errors were detected:\n";
+
+ foreach my $Text (keys %Errors)
+ {
+ print " $Text: $Errors{$Text} Time(s)\n";
+ }
+}
+
+exit(0);
+
+# vi: shiftwidth=3 tabstop=3 syntax=perl et
+# Local Variables:
+# mode: perl
+# perl-indent-level: 3
+# indent-tabs-mode: nil
+# End:
diff --git a/config/logwatch/ipblacklist.conf b/config/logwatch/ipblacklist.conf
new file mode 100644
index 000000000..ed0ecc5f1
--- /dev/null
+++ b/config/logwatch/ipblacklist.conf
@@ -0,0 +1,34 @@
+#########################################################################
+# ids-update script for Logwatch
+# Analyzes the IPFire IP Blacklist update log
+#
+# Version: 1.0.0
+# Initial release
+#
+#########################################################################
+
+#########################################################################
+# This script is subject to the same copyright as Logwatch itself
+#########################################################################
+
+#########################################################################
+# Files - all shown with default paths:
+#
+# /usr/share/logwatch/default.conf/logfiles/messages.conf
+# /usr/share/logwatch/dist.conf/services/blacklist.conf (this file)
+# /usr/share/logwatch/scripts/services/blacklist
+#
+# ... and of course
+#
+# /var/log/messages
+#########################################################################
+
+
+Title = "IP Blacklist"
+
+# Which logfile group...
+LogFile = messages
+
+*applystddate
+
+# vi: shiftwidth=3 tabstop=3 et
diff --git a/src/initscripts/system/firewall b/src/initscripts/system/firewall
index ec396c708..a3596cd0e 100644
--- a/src/initscripts/system/firewall
+++ b/src/initscripts/system/firewall
@@ -180,6 +180,16 @@ iptables_init() {
iptables -A FORWARD -j P2PBLOCK
iptables -A OUTPUT -j P2PBLOCK
+ # IP Address Blacklist chains
+ iptables -N IPBLACKLISTIN
+ iptables -N IPBLACKLISTOUT
+ iptables -N IPBLACKLISTREDIN
+ iptables -N IPBLACKLISTREDOUT
+ iptables -A INPUT ! -p icmp -j IPBLACKLISTIN
+ iptables -A FORWARD ! -p icmp -j IPBLACKLISTIN
+ iptables -A FORWARD ! -p icmp -j IPBLACKLISTOUT
+ iptables -A OUTPUT ! -p icmp -j IPBLACKLISTOUT
+
# Guardian (IPS) chains
iptables -N GUARDIAN
iptables -A INPUT -j GUARDIAN
@@ -382,6 +392,9 @@ iptables_init() {
# run captivectrl
/usr/local/bin/captivectrl
+ # run IP Blacklist start
+ /usr/local/bin/ipblacklist start
+
# POLICY CHAIN
iptables -N POLICYIN
iptables -A INPUT -j POLICYIN
@@ -407,6 +420,8 @@ iptables_red_up() {
iptables -F REDINPUT
iptables -F REDFORWARD
iptables -t nat -F REDNAT
+ iptables -F IPBLACKLISTIN
+ iptables -F IPBLACKLISTOUT
# PPPoE / PPTP Device
if [ "$IFACE" != "" ]; then
@@ -464,6 +479,10 @@ iptables_red_up() {
iptables -t nat -A REDNAT -s "${network}" -o "${IFACE}" -j RETURN
done
+ # IP Address Blacklists
+ iptables -A IPBLACKLISTIN -i $IFACE -j IPBLACKLISTREDIN
+ iptables -A IPBLACKLISTOUT -o $IFACE -j IPBLACKLISTREDOUT
+
# Masquerade everything else
iptables -t nat -A REDNAT -o $IFACE -j MASQUERADE
fi
@@ -504,6 +523,7 @@ case "$1" in
evaluate_retval
;;
restart)
+ /usr/local/bin/ipblacklist stop
$0 start
;;
*)
--
2.16.4
^ permalink raw reply [flat|nested] 35+ messages in thread
* [PATCH 5/5] ipblacklist: Build infrastructure
2019-11-25 20:13 [PATCH 0/5] ipblacklist: IP Address Blacklists Tim FitzGeorge
` (3 preceding siblings ...)
2019-11-25 20:13 ` [PATCH 4/5] ipblacklist: Modifications to system Tim FitzGeorge
@ 2019-11-25 20:13 ` Tim FitzGeorge
2019-11-25 21:09 ` [PATCH 0/5] ipblacklist: IP Address Blacklists Peter Müller
2019-11-28 12:03 ` Michael Tremer
6 siblings, 0 replies; 35+ messages in thread
From: Tim FitzGeorge @ 2019-11-25 20:13 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 12254 bytes --]
Signed-off-by: Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk>
---
config/rootfiles/common/aarch64/stage2 | 1 +
config/rootfiles/common/configroot | 2 ++
config/rootfiles/common/ipblacklist-sources | 1 +
config/rootfiles/common/logwatch | 2 ++
config/rootfiles/common/misc-progs | 2 ++
config/rootfiles/common/stage2 | 1 +
config/rootfiles/common/web-user-interface | 1 +
config/rootfiles/common/x86_64/stage2 | 1 +
lfs/configroot | 4 +--
lfs/ipblacklist-sources | 53 +++++++++++++++++++++++++++++
lfs/logwatch | 2 ++
make.sh | 11 +++---
src/misc-progs/Makefile | 2 +-
13 files changed, 75 insertions(+), 8 deletions(-)
create mode 100644 config/rootfiles/common/ipblacklist-sources
create mode 100644 lfs/ipblacklist-sources
diff --git a/config/rootfiles/common/aarch64/stage2 b/config/rootfiles/common/aarch64/stage2
index 366ab2bb0..5a598e3b1 100644
--- a/config/rootfiles/common/aarch64/stage2
+++ b/config/rootfiles/common/aarch64/stage2
@@ -93,6 +93,7 @@ usr/local/bin/connscheduler
usr/local/bin/consort.sh
usr/local/bin/convert-ovpn
usr/local/bin/hddshutdown
+usr/local/bin/ipblacklist
usr/local/bin/ipsec-interfaces
usr/local/bin/makegraphs
usr/local/bin/qosd
diff --git a/config/rootfiles/common/configroot b/config/rootfiles/common/configroot
index 56b0257bc..2f0e2440a 100644
--- a/config/rootfiles/common/configroot
+++ b/config/rootfiles/common/configroot
@@ -81,6 +81,8 @@ var/ipfire/geoip-functions.pl
var/ipfire/graphs.pl
var/ipfire/header.pl
var/ipfire/ids-functions.pl
+var/ipfire/ipblacklist
+#var/ipfire/ipblacklist/settings
var/ipfire/isdn
#var/ipfire/isdn/settings
var/ipfire/key
diff --git a/config/rootfiles/common/ipblacklist-sources b/config/rootfiles/common/ipblacklist-sources
new file mode 100644
index 000000000..7f54b1bbf
--- /dev/null
+++ b/config/rootfiles/common/ipblacklist-sources
@@ -0,0 +1 @@
+var/ipfire/ipblacklist/sources
diff --git a/config/rootfiles/common/logwatch b/config/rootfiles/common/logwatch
index c47fb4199..8b4810d97 100644
--- a/config/rootfiles/common/logwatch
+++ b/config/rootfiles/common/logwatch
@@ -192,6 +192,7 @@ usr/share/logwatch/default.conf/services/zz-sys.conf
usr/share/logwatch/dist.conf/logfiles
usr/share/logwatch/dist.conf/services
usr/share/logwatch/dist.conf/services/dialup.conf
+usr/share/logwatch/dist.conf/services/ipblacklist.conf
#usr/share/logwatch/lib
usr/share/logwatch/lib/Logwatch.pm
#usr/share/logwatch/scripts
@@ -256,6 +257,7 @@ usr/share/logwatch/scripts/services/http
usr/share/logwatch/scripts/services/imapd
#usr/share/logwatch/scripts/services/in.qpopper
usr/share/logwatch/scripts/services/init
+usr/share/logwatch/scripts/services/ipblacklist
usr/share/logwatch/scripts/services/ipop3d
usr/share/logwatch/scripts/services/iptables
usr/share/logwatch/scripts/services/kernel
diff --git a/config/rootfiles/common/misc-progs b/config/rootfiles/common/misc-progs
index c48a474b2..d17f3dd80 100644
--- a/config/rootfiles/common/misc-progs
+++ b/config/rootfiles/common/misc-progs
@@ -10,8 +10,10 @@ usr/local/bin/extrahdctrl
usr/local/bin/fireinfoctrl
usr/local/bin/firewallctrl
usr/local/bin/getconntracktable
+usr/local/bin/getipsetstat
usr/local/bin/getipstat
#usr/local/bin/iowrap
+usr/local/bin/ipblacklistctrl
usr/local/bin/ipfirereboot
usr/local/bin/ipsecctrl
usr/local/bin/launch-ether-wake
diff --git a/config/rootfiles/common/stage2 b/config/rootfiles/common/stage2
index d9068415b..a558050a7 100644
--- a/config/rootfiles/common/stage2
+++ b/config/rootfiles/common/stage2
@@ -92,6 +92,7 @@ usr/local/bin/connscheduler
usr/local/bin/consort.sh
usr/local/bin/convert-ovpn
usr/local/bin/hddshutdown
+usr/local/bin/ipblacklist
usr/local/bin/ipsec-interfaces
usr/local/bin/makegraphs
usr/local/bin/qosd
diff --git a/config/rootfiles/common/web-user-interface b/config/rootfiles/common/web-user-interface
index a88dd8770..da4fcde77 100644
--- a/config/rootfiles/common/web-user-interface
+++ b/config/rootfiles/common/web-user-interface
@@ -35,6 +35,7 @@ srv/web/ipfire/cgi-bin/hardwaregraphs.cgi
srv/web/ipfire/cgi-bin/hosts.cgi
srv/web/ipfire/cgi-bin/ids.cgi
srv/web/ipfire/cgi-bin/index.cgi
+srv/web/ipfire/cgi-bin/ipblacklist.cgi
srv/web/ipfire/cgi-bin/ipinfo.cgi
srv/web/ipfire/cgi-bin/iptables.cgi
srv/web/ipfire/cgi-bin/logs.cgi
diff --git a/config/rootfiles/common/x86_64/stage2 b/config/rootfiles/common/x86_64/stage2
index d90e3d70a..9c9b6c756 100644
--- a/config/rootfiles/common/x86_64/stage2
+++ b/config/rootfiles/common/x86_64/stage2
@@ -94,6 +94,7 @@ usr/local/bin/connscheduler
usr/local/bin/consort.sh
usr/local/bin/convert-ovpn
usr/local/bin/hddshutdown
+usr/local/bin/ipblacklist
usr/local/bin/ipsec-interfaces
usr/local/bin/makegraphs
usr/local/bin/qosd
diff --git a/lfs/configroot b/lfs/configroot
index 227d09239..4a4c919de 100644
--- a/lfs/configroot
+++ b/lfs/configroot
@@ -51,7 +51,7 @@ $(TARGET) :
# Create all directories
for i in addon-lang auth backup ca captive certs connscheduler crls ddns dhcp dhcpc dns dnsforward \
- ethernet extrahd/bin fwlogs fwhosts firewall isdn key langs logging mac main \
+ ethernet extrahd/bin fwlogs fwhosts firewall ipblacklist isdn key langs logging mac main \
menu.d modem optionsfw \
ovpn patches pakfire portfw ppp private proxy/advanced/cre \
proxy/calamaris/bin qos/bin red remote sensors suricata time \
@@ -65,7 +65,7 @@ $(TARGET) :
captive/settings captive/agb.txt captive/clients captive/voucher_out certs/index.txt certs/index.txt.attr ddns/config ddns/settings ddns/ipcache dhcp/settings \
dhcp/fixleases dhcp/advoptions dhcp/dhcpd.conf.local dns/settings dnsforward/config ethernet/aliases ethernet/settings ethernet/known_nics ethernet/scanned_nics \
ethernet/wireless extrahd/scan extrahd/devices extrahd/partitions extrahd/settings firewall/settings firewall/config firewall/geoipblock firewall/input firewall/outgoing \
- fwhosts/customnetworks fwhosts/customhosts fwhosts/customgroups fwhosts/customservicegrp fwhosts/customgeoipgrp fwlogs/ipsettings fwlogs/portsettings \
+ fwhosts/customnetworks fwhosts/customhosts fwhosts/customgroups fwhosts/customservicegrp fwhosts/customgeoipgrp fwlogs/ipsettings fwlogs/portsettings ipblacklist/settings \
isdn/settings mac/settings main/hosts main/routing main/security main/settings optionsfw/settings \
ovpn/ccd.conf ovpn/ccdroute ovpn/ccdroute2 pakfire/settings portfw/config ppp/settings-1 ppp/settings-2 ppp/settings-3 ppp/settings-4 \
ppp/settings-5 ppp/settings proxy/settings proxy/squid.conf proxy/advanced/settings proxy/advanced/cre/enable remote/settings qos/settings qos/classes qos/subclasses qos/level7config qos/portconfig \
diff --git a/lfs/ipblacklist-sources b/lfs/ipblacklist-sources
new file mode 100644
index 000000000..c9431285d
--- /dev/null
+++ b/lfs/ipblacklist-sources
@@ -0,0 +1,53 @@
+###############################################################################
+# #
+# IPFire.org - A linux based firewall #
+# Copyright (C) 2007 Michael Tremer & Christian Schmidt #
+# #
+# This program is free software: you can redistribute it and/or modify #
+# it under the terms of the GNU General Public License as published by #
+# the Free Software Foundation, either version 3 of the License, or #
+# (at your option) any later version. #
+# #
+# This program is distributed in the hope that it will be useful, #
+# but WITHOUT ANY WARRANTY; without even the implied warranty of #
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the #
+# GNU General Public License for more details. #
+# #
+# You should have received a copy of the GNU General Public License #
+# along with this program. If not, see <http://www.gnu.org/licenses/>. #
+# #
+###############################################################################
+
+###############################################################################
+# Definitions
+###############################################################################
+
+include Config
+
+VER = ipfire
+
+THISAPP = ipblacklist-sources
+TARGET = $(DIR_INFO)/$(THISAPP)
+
+###############################################################################
+# Top-level Rules
+###############################################################################
+
+install : $(TARGET)
+
+check :
+
+download :
+
+md5 :
+
+###############################################################################
+# Installation Details
+###############################################################################
+
+$(TARGET) :
+ @$(PREBUILD)
+ mkdir -p /var/ipfire/ipblacklist
+ install -v -m 0644 $(DIR_SRC)/config/ipblacklist/sources /var/ipfire/ipblacklist
+
+ @$(POSTBUILD)
diff --git a/lfs/logwatch b/lfs/logwatch
index eb576717c..368a6b6bf 100644
--- a/lfs/logwatch
+++ b/lfs/logwatch
@@ -93,6 +93,8 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
# done
cp -f $(DIR_SRC)/config/logwatch/dialup /usr/share/logwatch/scripts/services/dialup
cp -f $(DIR_SRC)/config/logwatch/dialup.conf /usr/share/logwatch/dist.conf/services/dialup.conf
+ cp -f $(DIR_SRC)/config/logwatch/ipblacklist /usr/share/logwatch/scripts/services/ipblacklist
+ cp -f $(DIR_SRC)/config/logwatch/ipblacklist.conf /usr/share/logwatch/dist.conf/services/ipblacklist.conf
-mkdir -p /var/cache/logwatch
chmod -v 777 /var/cache/logwatch
diff --git a/make.sh b/make.sh
index 771c5ff89..207ca331b 100755
--- a/make.sh
+++ b/make.sh
@@ -1631,6 +1631,7 @@ buildipfire() {
lfsmake2 tshark
lfsmake2 geoip-generator
lfsmake2 speedtest-cli
+ lfsmake2 ipblacklist-sources
}
buildinstaller() {
@@ -1648,7 +1649,7 @@ buildpackages() {
export LOGFILE
echo "... see detailed log in _build.*.log files" >> $LOGFILE
-
+
# Generating list of packages used
print_line "Generating packages list from logs"
rm -f $BASEDIR/doc/packages-list
@@ -1663,7 +1664,7 @@ buildpackages() {
rm -f $BASEDIR/doc/packages-list
# packages-list.txt is ready to be displayed for wiki page
print_status DONE
-
+
# Update changelog
cd $BASEDIR
[ -z $GIT_TAG ] || LAST_TAG=$GIT_TAG
@@ -1738,7 +1739,7 @@ while [ $# -gt 0 ]; do
done
# See what we're supposed to do
-case "$1" in
+case "$1" in
build)
START_TIME=$(now)
@@ -1777,7 +1778,7 @@ build)
print_build_stage "Building packages"
buildpackages
-
+
print_build_stage "Checking Logfiles for new Files"
cd $BASEDIR
@@ -1842,7 +1843,7 @@ downloadsrc)
FINISHED=0
cd $BASEDIR/lfs
for c in `seq $MAX_RETRIES`; do
- if (( FINISHED==1 )); then
+ if (( FINISHED==1 )); then
break
fi
FINISHED=1
diff --git a/src/misc-progs/Makefile b/src/misc-progs/Makefile
index bea54e773..60b3965e0 100644
--- a/src/misc-progs/Makefile
+++ b/src/misc-progs/Makefile
@@ -32,7 +32,7 @@ SUID_PROGS = squidctrl sshctrl ipfirereboot \
smartctrl clamavctrl addonctrl pakfire mpfirectrl wlanapctrl \
setaliases urlfilterctrl updxlratorctrl fireinfoctrl rebuildroutes \
getconntracktable wirelessclient torctrl ddnsctrl unboundctrl \
- captivectrl
+ captivectrl ipblacklistctrl getipsetstat
SUID_UPDX = updxsetperms
OBJS = $(patsubst %,%.o,$(PROGS) $(SUID_PROGS))
--
2.16.4
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 0/5] ipblacklist: IP Address Blacklists
2019-11-25 20:13 [PATCH 0/5] ipblacklist: IP Address Blacklists Tim FitzGeorge
` (4 preceding siblings ...)
2019-11-25 20:13 ` [PATCH 5/5] ipblacklist: Build infrastructure Tim FitzGeorge
@ 2019-11-25 21:09 ` Peter Müller
2019-11-27 21:34 ` Tim FitzGeorge
2019-11-28 12:03 ` Michael Tremer
6 siblings, 1 reply; 35+ messages in thread
From: Peter Müller @ 2019-11-25 21:09 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 4238 bytes --]
Hello Tim,
thank you very much for providing this patchset. After having read
through it, I consider this being an extremely helpful addition to
IPFire, especially when it comes to dropping bogon and unallocated
prefixes - which can be safely enabled by default as far as I am concerned.
Unfortunately, I am currently busy, so please give me a few days for
having a closer look at all these patches. :-)
Just a footnote: The DShield/ISC SANS feed is also available at
http://feeds.dshield.org/block.txt , which seems to be the preferred
location for fetching the file (I believe https://www.dshield.org/block.txt
exists for compatibility reasons).
blocklist.de (https://lists.blocklist.de/lists/all.txt) might be another
interesting data feed; I will hand in patches for it as soon yours
were accepted.
Many thanks again, and best regards,
Peter Müller
> Implements downloading of IP address blacklists and implementing
> them as IPSets. A separate IPSet is used for each blacklist; this
> simplifies handling of overlaps between different lists. Traffic
> to or from the red0/ppp0 interface is checked against the IPSets.
> The check is placed before the IPS check as the IPSet check is
> much lighter on CPU use which means that overall CPU use is
> reduced.
>
> The available lists are defined in a separate file. A WUI page
> allows the desired lists to be enabled and the interval between
> checks for updates to be defined. A minimum update check interval
> is defined for each blacklist in the definition file.
>
> Optionally, an automatically updating blacklist can be enabled.
> This adds addresses to an IPSet if the rate of packets dropped by
> the default red0/ppp0 input policy exceeds a user defined threshold.
> The addresses are kept in the IPSet until a user defined period
> without packets from the blocked address has passed.
>
> Tim FitzGeorge (5):
> ipblacklist: Main script
> ipblacklist: WUI and language file
> ipblacklist: Ancillary files
> ipblacklist: Modifications to system
> ipblacklist: Build infrastructure
>
> config/backup/backup.pl | 1 +
> config/backup/include | 2 +
> config/firewall/firewall-policy | 5 +
> config/ipblacklist/sources | 151 +++
> config/logwatch/ipblacklist | 103 ++
> config/logwatch/ipblacklist.conf | 34 +
> config/menu/50-firewall.menu | 5 +
> config/rootfiles/common/aarch64/stage2 | 1 +
> config/rootfiles/common/configroot | 2 +
> config/rootfiles/common/ipblacklist-sources | 1 +
> config/rootfiles/common/logwatch | 2 +
> config/rootfiles/common/misc-progs | 2 +
> config/rootfiles/common/stage2 | 1 +
> config/rootfiles/common/web-user-interface | 1 +
> config/rootfiles/common/x86_64/stage2 | 1 +
> html/cgi-bin/ipblacklist.cgi | 725 +++++++++++++
> html/cgi-bin/logs.cgi/log.dat | 2 +
> langs/en/cgi-bin/en.pl | 31 +
> lfs/configroot | 4 +-
> lfs/ipblacklist-sources | 53 +
> lfs/logwatch | 2 +
> make.sh | 11 +-
> src/initscripts/system/firewall | 20 +
> src/misc-progs/Makefile | 2 +-
> src/misc-progs/getipsetstat.c | 28 +
> src/misc-progs/ipblacklistctrl.c | 52 +
> src/scripts/ipblacklist | 1558 +++++++++++++++++++++++++++
> 27 files changed, 2792 insertions(+), 8 deletions(-)
> create mode 100644 config/ipblacklist/sources
> create mode 100644 config/logwatch/ipblacklist
> create mode 100644 config/logwatch/ipblacklist.conf
> create mode 100644 config/rootfiles/common/ipblacklist-sources
> create mode 100644 html/cgi-bin/ipblacklist.cgi
> create mode 100644 lfs/ipblacklist-sources
> create mode 100644 src/misc-progs/getipsetstat.c
> create mode 100644 src/misc-progs/ipblacklistctrl.c
> create mode 100755 src/scripts/ipblacklist
>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 0/5] ipblacklist: IP Address Blacklists
2019-11-25 21:09 ` [PATCH 0/5] ipblacklist: IP Address Blacklists Peter Müller
@ 2019-11-27 21:34 ` Tim FitzGeorge
0 siblings, 0 replies; 35+ messages in thread
From: Tim FitzGeorge @ 2019-11-27 21:34 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 4710 bytes --]
Hello Peter,
I've updated the sources file with the corrected DShield url and added
the blocklist.de list. I've also fixed a minor bug preventing
unblocking addresses in the automatic blacklist. Since they're minor
changes I'll wait before submitting updated patches in case there's
anything else you spot.
Tim
On 25/11/2019 21:09, Peter Müller wrote:
> Hello Tim,
>
> thank you very much for providing this patchset. After having read
> through it, I consider this being an extremely helpful addition to
> IPFire, especially when it comes to dropping bogon and unallocated
> prefixes - which can be safely enabled by default as far as I am concerned.
>
> Unfortunately, I am currently busy, so please give me a few days for
> having a closer look at all these patches. :-)
>
> Just a footnote: The DShield/ISC SANS feed is also available at
> http://feeds.dshield.org/block.txt , which seems to be the preferred
> location for fetching the file (I believe https://www.dshield.org/block.txt
> exists for compatibility reasons).
>
> blocklist.de (https://lists.blocklist.de/lists/all.txt) might be another
> interesting data feed; I will hand in patches for it as soon yours
> were accepted.
>
> Many thanks again, and best regards,
> Peter Müller
>
>> Implements downloading of IP address blacklists and implementing
>> them as IPSets. A separate IPSet is used for each blacklist; this
>> simplifies handling of overlaps between different lists. Traffic
>> to or from the red0/ppp0 interface is checked against the IPSets.
>> The check is placed before the IPS check as the IPSet check is
>> much lighter on CPU use which means that overall CPU use is
>> reduced.
>>
>> The available lists are defined in a separate file. A WUI page
>> allows the desired lists to be enabled and the interval between
>> checks for updates to be defined. A minimum update check interval
>> is defined for each blacklist in the definition file.
>>
>> Optionally, an automatically updating blacklist can be enabled.
>> This adds addresses to an IPSet if the rate of packets dropped by
>> the default red0/ppp0 input policy exceeds a user defined threshold.
>> The addresses are kept in the IPSet until a user defined period
>> without packets from the blocked address has passed.
>>
>> Tim FitzGeorge (5):
>> ipblacklist: Main script
>> ipblacklist: WUI and language file
>> ipblacklist: Ancillary files
>> ipblacklist: Modifications to system
>> ipblacklist: Build infrastructure
>>
>> config/backup/backup.pl | 1 +
>> config/backup/include | 2 +
>> config/firewall/firewall-policy | 5 +
>> config/ipblacklist/sources | 151 +++
>> config/logwatch/ipblacklist | 103 ++
>> config/logwatch/ipblacklist.conf | 34 +
>> config/menu/50-firewall.menu | 5 +
>> config/rootfiles/common/aarch64/stage2 | 1 +
>> config/rootfiles/common/configroot | 2 +
>> config/rootfiles/common/ipblacklist-sources | 1 +
>> config/rootfiles/common/logwatch | 2 +
>> config/rootfiles/common/misc-progs | 2 +
>> config/rootfiles/common/stage2 | 1 +
>> config/rootfiles/common/web-user-interface | 1 +
>> config/rootfiles/common/x86_64/stage2 | 1 +
>> html/cgi-bin/ipblacklist.cgi | 725 +++++++++++++
>> html/cgi-bin/logs.cgi/log.dat | 2 +
>> langs/en/cgi-bin/en.pl | 31 +
>> lfs/configroot | 4 +-
>> lfs/ipblacklist-sources | 53 +
>> lfs/logwatch | 2 +
>> make.sh | 11 +-
>> src/initscripts/system/firewall | 20 +
>> src/misc-progs/Makefile | 2 +-
>> src/misc-progs/getipsetstat.c | 28 +
>> src/misc-progs/ipblacklistctrl.c | 52 +
>> src/scripts/ipblacklist | 1558 +++++++++++++++++++++++++++
>> 27 files changed, 2792 insertions(+), 8 deletions(-)
>> create mode 100644 config/ipblacklist/sources
>> create mode 100644 config/logwatch/ipblacklist
>> create mode 100644 config/logwatch/ipblacklist.conf
>> create mode 100644 config/rootfiles/common/ipblacklist-sources
>> create mode 100644 html/cgi-bin/ipblacklist.cgi
>> create mode 100644 lfs/ipblacklist-sources
>> create mode 100644 src/misc-progs/getipsetstat.c
>> create mode 100644 src/misc-progs/ipblacklistctrl.c
>> create mode 100755 src/scripts/ipblacklist
>>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 0/5] ipblacklist: IP Address Blacklists
2019-11-25 20:13 [PATCH 0/5] ipblacklist: IP Address Blacklists Tim FitzGeorge
` (5 preceding siblings ...)
2019-11-25 21:09 ` [PATCH 0/5] ipblacklist: IP Address Blacklists Peter Müller
@ 2019-11-28 12:03 ` Michael Tremer
2019-11-28 21:39 ` Peter Müller
6 siblings, 1 reply; 35+ messages in thread
From: Michael Tremer @ 2019-11-28 12:03 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 4947 bytes --]
Hello Tim,
Thank you for sending in this patchset.
I think as well that this functionality would be a great addition to the IPS that we have right now and we have talked about this on numerous occasions - including implementation details.
Now, you have done the whole thing. Well done.
Before we dive into the code, can I ask a couple of high-level questions?
Peter is always bringing up that downloading blacklists isn’t a good idea. It has actually become one of the biggest obstacles in our conversations and I am surprised he didn’t bring it up again :)
The automatic blacklist feature. What is the objective here? We saw value in having traffic even from malicious sources passed to the IPS so that it will examine it and log it. The idea was to have a better picture of the threats instead of just silencing them. Not sure what is best in the end.
I am unsure how users will deal with this and turn on “all the lists”(TM) and suddenly things do not work any more. How are they meant to figure out a good threshold? Should we not make that decision for them instead?
About the implementation: Your code is very very clean as always. There are a couple of things that I would like to see changed around how those iptables rules are being inserted into the existing chains. You are adding something into the POLICYIN chain with surgical precision which might break when the chain is being modified. This is potentially all minor stuff and can be fixed in minutes.
Well done!
-Michael
> On 25 Nov 2019, at 20:13, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>
> Implements downloading of IP address blacklists and implementing
> them as IPSets. A separate IPSet is used for each blacklist; this
> simplifies handling of overlaps between different lists. Traffic
> to or from the red0/ppp0 interface is checked against the IPSets.
> The check is placed before the IPS check as the IPSet check is
> much lighter on CPU use which means that overall CPU use is
> reduced.
>
> The available lists are defined in a separate file. A WUI page
> allows the desired lists to be enabled and the interval between
> checks for updates to be defined. A minimum update check interval
> is defined for each blacklist in the definition file.
>
> Optionally, an automatically updating blacklist can be enabled.
> This adds addresses to an IPSet if the rate of packets dropped by
> the default red0/ppp0 input policy exceeds a user defined threshold.
> The addresses are kept in the IPSet until a user defined period
> without packets from the blocked address has passed.
>
> Tim FitzGeorge (5):
> ipblacklist: Main script
> ipblacklist: WUI and language file
> ipblacklist: Ancillary files
> ipblacklist: Modifications to system
> ipblacklist: Build infrastructure
>
> config/backup/backup.pl | 1 +
> config/backup/include | 2 +
> config/firewall/firewall-policy | 5 +
> config/ipblacklist/sources | 151 +++
> config/logwatch/ipblacklist | 103 ++
> config/logwatch/ipblacklist.conf | 34 +
> config/menu/50-firewall.menu | 5 +
> config/rootfiles/common/aarch64/stage2 | 1 +
> config/rootfiles/common/configroot | 2 +
> config/rootfiles/common/ipblacklist-sources | 1 +
> config/rootfiles/common/logwatch | 2 +
> config/rootfiles/common/misc-progs | 2 +
> config/rootfiles/common/stage2 | 1 +
> config/rootfiles/common/web-user-interface | 1 +
> config/rootfiles/common/x86_64/stage2 | 1 +
> html/cgi-bin/ipblacklist.cgi | 725 +++++++++++++
> html/cgi-bin/logs.cgi/log.dat | 2 +
> langs/en/cgi-bin/en.pl | 31 +
> lfs/configroot | 4 +-
> lfs/ipblacklist-sources | 53 +
> lfs/logwatch | 2 +
> make.sh | 11 +-
> src/initscripts/system/firewall | 20 +
> src/misc-progs/Makefile | 2 +-
> src/misc-progs/getipsetstat.c | 28 +
> src/misc-progs/ipblacklistctrl.c | 52 +
> src/scripts/ipblacklist | 1558 +++++++++++++++++++++++++++
> 27 files changed, 2792 insertions(+), 8 deletions(-)
> create mode 100644 config/ipblacklist/sources
> create mode 100644 config/logwatch/ipblacklist
> create mode 100644 config/logwatch/ipblacklist.conf
> create mode 100644 config/rootfiles/common/ipblacklist-sources
> create mode 100644 html/cgi-bin/ipblacklist.cgi
> create mode 100644 lfs/ipblacklist-sources
> create mode 100644 src/misc-progs/getipsetstat.c
> create mode 100644 src/misc-progs/ipblacklistctrl.c
> create mode 100755 src/scripts/ipblacklist
>
> --
> 2.16.4
>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 0/5] ipblacklist: IP Address Blacklists
2019-11-28 12:03 ` Michael Tremer
@ 2019-11-28 21:39 ` Peter Müller
2019-11-29 23:25 ` Tim FitzGeorge
2019-12-02 11:06 ` Michael Tremer
0 siblings, 2 replies; 35+ messages in thread
From: Peter Müller @ 2019-11-28 21:39 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 7424 bytes --]
Hello Tim, hello Michael, hello *,
first, I'd like to apologise for the late response. I used to
be more active on this mailing list than I am today, but hopefully
this will be a temporary situation only.
Michael is right, I did not have time to mention my concerns about
fetching the blacklists from external sources. In my opinion, this
causes privacy issues as we disclose the public IP addresses of
all IPFire installations using a blacklist to its provider/vendor.
In my humble opinion, we can rely on a diverse and robust mirror
infrastructure for our Core Updates and packages, so why not use them
for other data such as blacklists or upcoming libloc as well?
We could periodically update the blacklists on our main mirror (and
wait for the network to sync it), make sure it is signed and write
a small downloader that fetches, validates and installs them.
@All: Thoughts on this?
Talking about the preference of packet filter and IPS, I prefer to
use the latter as well as it gains more insight in what kind of malicious
traffic tried to pass a firewall machine. On systems with low resources,
this might be problematic and removing load from the IPS can be preferred
(make this configurable?!), on others, people might want to have both
results.
Regarding the GeoIP block feature, this has always been my strongest
point against it: Neither were dropped packets logged (hope this is
still true, as I have not tried for quite a while), nor is more fine
grained filtering possible - e.g. by limiting this to certain services -,
nor are IPS results available for packets dropped.
We should _always_ log dropped packets (no matter which feature drops them),
and leave it to the user to decide whether traffic from/to blacklisted
IP addresses should traverse the IPS or not.
Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default.
I would love to see the bogon ruleset here, too (think about 8chan successor
hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause
interference if RED does not have a public IP address assigned.
Regarding the code quality, I agree with Michael. Thanks again for providing
this. :-) As soon as we agree on answers to the questions raised meanwhile, I
will add my "Reviewed-by"-tag.
Thanks, and best regards,
Peter Müller
> Hello Tim,
>
> Thank you for sending in this patchset.
>
> I think as well that this functionality would be a great addition to the IPS that we have right now and we have talked about this on numerous occasions - including implementation details.
>
> Now, you have done the whole thing. Well done.
>
> Before we dive into the code, can I ask a couple of high-level questions?
>
> Peter is always bringing up that downloading blacklists isn’t a good idea. It has actually become one of the biggest obstacles in our conversations and I am surprised he didn’t bring it up again :)
>
> The automatic blacklist feature. What is the objective here? We saw value in having traffic even from malicious sources passed to the IPS so that it will examine it and log it. The idea was to have a better picture of the threats instead of just silencing them. Not sure what is best in the end.
>
> I am unsure how users will deal with this and turn on “all the lists”(TM) and suddenly things do not work any more. How are they meant to figure out a good threshold? Should we not make that decision for them instead?
>
> About the implementation: Your code is very very clean as always. There are a couple of things that I would like to see changed around how those iptables rules are being inserted into the existing chains. You are adding something into the POLICYIN chain with surgical precision which might break when the chain is being modified. This is potentially all minor stuff and can be fixed in minutes.
>
> Well done!
>
> -Michael
>
>> On 25 Nov 2019, at 20:13, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>
>> Implements downloading of IP address blacklists and implementing
>> them as IPSets. A separate IPSet is used for each blacklist; this
>> simplifies handling of overlaps between different lists. Traffic
>> to or from the red0/ppp0 interface is checked against the IPSets.
>> The check is placed before the IPS check as the IPSet check is
>> much lighter on CPU use which means that overall CPU use is
>> reduced.
>>
>> The available lists are defined in a separate file. A WUI page
>> allows the desired lists to be enabled and the interval between
>> checks for updates to be defined. A minimum update check interval
>> is defined for each blacklist in the definition file.
>>
>> Optionally, an automatically updating blacklist can be enabled.
>> This adds addresses to an IPSet if the rate of packets dropped by
>> the default red0/ppp0 input policy exceeds a user defined threshold.
>> The addresses are kept in the IPSet until a user defined period
>> without packets from the blocked address has passed.
>>
>> Tim FitzGeorge (5):
>> ipblacklist: Main script
>> ipblacklist: WUI and language file
>> ipblacklist: Ancillary files
>> ipblacklist: Modifications to system
>> ipblacklist: Build infrastructure
>>
>> config/backup/backup.pl | 1 +
>> config/backup/include | 2 +
>> config/firewall/firewall-policy | 5 +
>> config/ipblacklist/sources | 151 +++
>> config/logwatch/ipblacklist | 103 ++
>> config/logwatch/ipblacklist.conf | 34 +
>> config/menu/50-firewall.menu | 5 +
>> config/rootfiles/common/aarch64/stage2 | 1 +
>> config/rootfiles/common/configroot | 2 +
>> config/rootfiles/common/ipblacklist-sources | 1 +
>> config/rootfiles/common/logwatch | 2 +
>> config/rootfiles/common/misc-progs | 2 +
>> config/rootfiles/common/stage2 | 1 +
>> config/rootfiles/common/web-user-interface | 1 +
>> config/rootfiles/common/x86_64/stage2 | 1 +
>> html/cgi-bin/ipblacklist.cgi | 725 +++++++++++++
>> html/cgi-bin/logs.cgi/log.dat | 2 +
>> langs/en/cgi-bin/en.pl | 31 +
>> lfs/configroot | 4 +-
>> lfs/ipblacklist-sources | 53 +
>> lfs/logwatch | 2 +
>> make.sh | 11 +-
>> src/initscripts/system/firewall | 20 +
>> src/misc-progs/Makefile | 2 +-
>> src/misc-progs/getipsetstat.c | 28 +
>> src/misc-progs/ipblacklistctrl.c | 52 +
>> src/scripts/ipblacklist | 1558 +++++++++++++++++++++++++++
>> 27 files changed, 2792 insertions(+), 8 deletions(-)
>> create mode 100644 config/ipblacklist/sources
>> create mode 100644 config/logwatch/ipblacklist
>> create mode 100644 config/logwatch/ipblacklist.conf
>> create mode 100644 config/rootfiles/common/ipblacklist-sources
>> create mode 100644 html/cgi-bin/ipblacklist.cgi
>> create mode 100644 lfs/ipblacklist-sources
>> create mode 100644 src/misc-progs/getipsetstat.c
>> create mode 100644 src/misc-progs/ipblacklistctrl.c
>> create mode 100755 src/scripts/ipblacklist
>>
>> --
>> 2.16.4
>>
>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 0/5] ipblacklist: IP Address Blacklists
2019-11-28 21:39 ` Peter Müller
@ 2019-11-29 23:25 ` Tim FitzGeorge
2019-12-02 11:17 ` Michael Tremer
2019-12-02 11:06 ` Michael Tremer
1 sibling, 1 reply; 35+ messages in thread
From: Tim FitzGeorge @ 2019-11-29 23:25 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 11648 bytes --]
Hello Peter and Michael,
My responses to both of your emails are below:
On 28/11/2019 21:39, Peter Müller wrote:
> Hello Tim, hello Michael, hello *,
>
> first, I'd like to apologise for the late response. I used to
> be more active on this mailing list than I am today, but hopefully
> this will be a temporary situation only.
>
> Michael is right, I did not have time to mention my concerns about
> fetching the blacklists from external sources. In my opinion, this
> causes privacy issues as we disclose the public IP addresses of
> all IPFire installations using a blacklist to its provider/vendor.
>
> In my humble opinion, we can rely on a diverse and robust mirror
> infrastructure for our Core Updates and packages, so why not use them
> for other data such as blacklists or upcoming libloc as well?
>
> We could periodically update the blacklists on our main mirror (and
> wait for the network to sync it), make sure it is signed and write
> a small downloader that fetches, validates and installs them.
>
> @All: Thoughts on this?
I think there are a number of points here.
Firstly, from the point of a third party using IPFire, is this really
solving the privacy disclosure problem? There's no way round disclosing
your public IP address to someone you're downloading from; all this does
is change who that information is being disclosed to. For the user
there's no way of knowing whether the source is more or less protective
of the user's privacy than the blacklist provider. Indeed it won't be
possible to know who the lists are being downloaded from until the
download starts.
Secondly, latency; some of the lists are updated every 5 minutes. While
I've limited the maximum check rate to hourly, will the updates
propagate quickly enough. For reference on my main system the 24
updates on the CIARMY list made 143 498 changes (additions or
deletions). I've seem it do over 200 000.
Third, bandwidth; while the downloads are fairly small (ALIENVAULT is
the largest at a few MB), there are going to be a lot of them. How will
this affect the willingness of people to mirror IPFire?
>
> Talking about the preference of packet filter and IPS, I prefer to
> use the latter as well as it gains more insight in what kind of malicious
> traffic tried to pass a firewall machine. On systems with low resources,
> this might be problematic and removing load from the IPS can be preferred
> (make this configurable?!), on others, people might want to have both
> results.
>
You're only going to get one result for a packet whichever way round the
IP blacklist and IPS are since whichever comes first will drop the
packet before it reaches the second (well it would be possible to put
the IP blacklist first and get it to log and mark packets which are then
dropped after the IPS, but I think that's getting a little complicated.
In addition I've seen the messages about the trouble marking was
causing in the QoS).
I think it's a 50/50 choice as to which is more valuable first; it's
probably going to differ from packet to packet. For me the possibility
of reducing the IPS load means I prefer putting the IP blacklist first
It should be fairly easy to add the choice of where to put the IP
blacklist. I think it'll have to be in the main firewall script, so
it'll require a firewall restart, but it's not something that'll be
changed often.
> Regarding the GeoIP block feature, this has always been my strongest
> point against it: Neither were dropped packets logged (hope this is
> still true, as I have not tried for quite a while), nor is more fine
> grained filtering possible - e.g. by limiting this to certain services -,
> nor are IPS results available for packets dropped.
>
> We should _always_ log dropped packets (no matter which feature drops them),
> and leave it to the user to decide whether traffic from/to blacklisted
> IP addresses should traverse the IPS or not.
I agree. I added the possibility to turn off logging because of user
requests for the earlier version I made available on GitHub, but logging
is enabled by default.
>
> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default.
> I would love to see the bogon ruleset here, too (think about 8chan successor
> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause
> interference if RED does not have a public IP address assigned.
I can add a field to the options file that controls whether a list is
enabled by default.
>
> Regarding the code quality, I agree with Michael. Thanks again for providing
> this. :-) As soon as we agree on answers to the questions raised meanwhile, I
> will add my "Reviewed-by"-tag.
>
> Thanks, and best regards,
> Peter Müller
>
>
>> Hello Tim,
>>
>> Thank you for sending in this patchset.
>>
>> I think as well that this functionality would be a great addition to the IPS that we have right now and we have talked about this on numerous occasions - including implementation details.
>>
>> Now, you have done the whole thing. Well done.
>>
>> Before we dive into the code, can I ask a couple of high-level questions?
>>
>> Peter is always bringing up that downloading blacklists isn’t a good idea. It has actually become one of the biggest obstacles in our conversations and I am surprised he didn’t bring it up again :)
>>
>> The automatic blacklist feature. What is the objective here? We saw value in having traffic even from malicious sources passed to the IPS so that it will examine it and log it. The idea was to have a better picture of the threats instead of just silencing them. Not sure what is best in the end.
It's intended to provide a quick local response to portscans based on
packets which are dropped due to the red input policy.
>>
>> I am unsure how users will deal with this and turn on “all the lists”(TM) and suddenly things do not work any more. How are they meant to figure out a good threshold? Should we not make that decision for them instead?
Indeed. I've seen the messages from people who have turned on all the
IPS rules. In addition I know that at least one person who used the
version on GitHub turned on all the lists, even the ones where one is a
subset of another list (I added code to prevent this). I don't believe
that it caused major problems since, unlike the IPS, it doesn't have
rules that can block what for some people is normal activity. It's
still a bad idea. I think it makes sense to deliver it with a default
configuration, which might hint that it's not meant to have all the
lists enabled.
I'll obviously cover this in the Wiki page(s), but that doesn't mean
people will read it...
>>
>> About the implementation: Your code is very very clean as always. There are a couple of things that I would like to see changed around how those iptables rules are being inserted into the existing chains. You are adding something into the POLICYIN chain with surgical precision which might break when the chain is being modified. This is potentially all minor stuff and can be fixed in minutes.
This is one of the areas I was unsure about. I've looked at the
existing code, but I obviously don't have the detailed knowledge of how
the different parts of the system interact when setting up the firewall.
The rule added to POLICYIN is the one that adds addresses to the
autoblacklist; it's not all that critical where it goes in POLICYIN, but
the ideal place is just before the LOG. Normally the code in
firewall-policy will put it in the right place, but I think I know how
to place it better in ipblacklist.
(Aside - I write onboard software for spacecraft and we always try to
bear in mind that it could be use that has to modify it in a hurry after
working on something else for ten years. It does tend to encourage
being tidy).
>>
>> Well done!
>>
>> -Michael
If you let me know what you think, I can make some modifications and
then send a V2.
Tim
>>
>>> On 25 Nov 2019, at 20:13, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>
>>> Implements downloading of IP address blacklists and implementing
>>> them as IPSets. A separate IPSet is used for each blacklist; this
>>> simplifies handling of overlaps between different lists. Traffic
>>> to or from the red0/ppp0 interface is checked against the IPSets.
>>> The check is placed before the IPS check as the IPSet check is
>>> much lighter on CPU use which means that overall CPU use is
>>> reduced.
>>>
>>> The available lists are defined in a separate file. A WUI page
>>> allows the desired lists to be enabled and the interval between
>>> checks for updates to be defined. A minimum update check interval
>>> is defined for each blacklist in the definition file.
>>>
>>> Optionally, an automatically updating blacklist can be enabled.
>>> This adds addresses to an IPSet if the rate of packets dropped by
>>> the default red0/ppp0 input policy exceeds a user defined threshold.
>>> The addresses are kept in the IPSet until a user defined period
>>> without packets from the blocked address has passed.
>>>
>>> Tim FitzGeorge (5):
>>> ipblacklist: Main script
>>> ipblacklist: WUI and language file
>>> ipblacklist: Ancillary files
>>> ipblacklist: Modifications to system
>>> ipblacklist: Build infrastructure
>>>
>>> config/backup/backup.pl | 1 +
>>> config/backup/include | 2 +
>>> config/firewall/firewall-policy | 5 +
>>> config/ipblacklist/sources | 151 +++
>>> config/logwatch/ipblacklist | 103 ++
>>> config/logwatch/ipblacklist.conf | 34 +
>>> config/menu/50-firewall.menu | 5 +
>>> config/rootfiles/common/aarch64/stage2 | 1 +
>>> config/rootfiles/common/configroot | 2 +
>>> config/rootfiles/common/ipblacklist-sources | 1 +
>>> config/rootfiles/common/logwatch | 2 +
>>> config/rootfiles/common/misc-progs | 2 +
>>> config/rootfiles/common/stage2 | 1 +
>>> config/rootfiles/common/web-user-interface | 1 +
>>> config/rootfiles/common/x86_64/stage2 | 1 +
>>> html/cgi-bin/ipblacklist.cgi | 725 +++++++++++++
>>> html/cgi-bin/logs.cgi/log.dat | 2 +
>>> langs/en/cgi-bin/en.pl | 31 +
>>> lfs/configroot | 4 +-
>>> lfs/ipblacklist-sources | 53 +
>>> lfs/logwatch | 2 +
>>> make.sh | 11 +-
>>> src/initscripts/system/firewall | 20 +
>>> src/misc-progs/Makefile | 2 +-
>>> src/misc-progs/getipsetstat.c | 28 +
>>> src/misc-progs/ipblacklistctrl.c | 52 +
>>> src/scripts/ipblacklist | 1558 +++++++++++++++++++++++++++
>>> 27 files changed, 2792 insertions(+), 8 deletions(-)
>>> create mode 100644 config/ipblacklist/sources
>>> create mode 100644 config/logwatch/ipblacklist
>>> create mode 100644 config/logwatch/ipblacklist.conf
>>> create mode 100644 config/rootfiles/common/ipblacklist-sources
>>> create mode 100644 html/cgi-bin/ipblacklist.cgi
>>> create mode 100644 lfs/ipblacklist-sources
>>> create mode 100644 src/misc-progs/getipsetstat.c
>>> create mode 100644 src/misc-progs/ipblacklistctrl.c
>>> create mode 100755 src/scripts/ipblacklist
>>>
>>> --
>>> 2.16.4
>>>
>>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 0/5] ipblacklist: IP Address Blacklists
2019-11-28 21:39 ` Peter Müller
2019-11-29 23:25 ` Tim FitzGeorge
@ 2019-12-02 11:06 ` Michael Tremer
1 sibling, 0 replies; 35+ messages in thread
From: Michael Tremer @ 2019-12-02 11:06 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 9353 bytes --]
Hi,
> On 28 Nov 2019, at 21:39, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>
> Hello Tim, hello Michael, hello *,
>
> first, I'd like to apologise for the late response. I used to
> be more active on this mailing list than I am today, but hopefully
> this will be a temporary situation only.
>
> Michael is right, I did not have time to mention my concerns about
> fetching the blacklists from external sources. In my opinion, this
> causes privacy issues as we disclose the public IP addresses of
> all IPFire installations using a blacklist to its provider/vendor.
>
> In my humble opinion, we can rely on a diverse and robust mirror
> infrastructure for our Core Updates and packages, so why not use them
> for other data such as blacklists or upcoming libloc as well?
>
> We could periodically update the blacklists on our main mirror (and
> wait for the network to sync it), make sure it is signed and write
> a small downloader that fetches, validates and installs them.
>
> @All: Thoughts on this?
Well, this causes us new problems:
* Does the licence of those lists allow us to redistribute them?
* They are usually updated very often (within minutes). We ask our mirror servers to sync at least every 12 hours but not more often than every 6. In practice everyone seems to be doing it once an hour. This will be way too slow and we will distribute a list that might be a day old.
* Bandwidth
* If we sign them, how do we validate that we have seen the correct list? This has to be automated and I do not see a way how we can make this bullet-proof.
In my personal opinion: This all dies with the first reason. One list that has a strict licence will be enough.
> Talking about the preference of packet filter and IPS, I prefer to
> use the latter as well as it gains more insight in what kind of malicious
> traffic tried to pass a firewall machine. On systems with low resources,
> this might be problematic and removing load from the IPS can be preferred
> (make this configurable?!), on others, people might want to have both
> results.
True. But that relies on the IPS having a ruleset that would catch this threat.
Traffic from Spamhaus DROP should be dropped regardless.
I think we can say that the blacklists will always drop when an IP address is on the list. The IPS won’t. So is there not a slight advantage to security if the blacklists come first?
> Regarding the GeoIP block feature, this has always been my strongest
> point against it: Neither were dropped packets logged (hope this is
> still true, as I have not tried for quite a while), nor is more fine
> grained filtering possible - e.g. by limiting this to certain services -,
> nor are IPS results available for packets dropped.
The GeoIP feature was designed to prevent noise in the firewall logs and drop those packets silently.
It works as designed. You want something different which could be added with a single checkbox.
> We should _always_ log dropped packets (no matter which feature drops them),
> and leave it to the user to decide whether traffic from/to blacklisted
> IP addresses should traverse the IPS or not.
>
> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default.
> I would love to see the bogon ruleset here, too (think about 8chan successor
> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause
> interference if RED does not have a public IP address assigned.
Yes, I was planning to have this enabled by default in IPFire 3.
However, there must be an obvious switch to allow RFC1918.
We should distinguish between bogons from public IP address space and RFC1918 and similar address space.
Best,
-Michael
> Regarding the code quality, I agree with Michael. Thanks again for providing
> this. :-) As soon as we agree on answers to the questions raised meanwhile, I
> will add my "Reviewed-by"-tag.
You can do that right now, but I suppose we might have a couple of iterations of this patchset and I would suggest doing it at the end.
But please review every time.
>
> Thanks, and best regards,
> Peter Müller
>
>
>> Hello Tim,
>>
>> Thank you for sending in this patchset.
>>
>> I think as well that this functionality would be a great addition to the IPS that we have right now and we have talked about this on numerous occasions - including implementation details.
>>
>> Now, you have done the whole thing. Well done.
>>
>> Before we dive into the code, can I ask a couple of high-level questions?
>>
>> Peter is always bringing up that downloading blacklists isn’t a good idea. It has actually become one of the biggest obstacles in our conversations and I am surprised he didn’t bring it up again :)
>>
>> The automatic blacklist feature. What is the objective here? We saw value in having traffic even from malicious sources passed to the IPS so that it will examine it and log it. The idea was to have a better picture of the threats instead of just silencing them. Not sure what is best in the end.
>>
>> I am unsure how users will deal with this and turn on “all the lists”(TM) and suddenly things do not work any more. How are they meant to figure out a good threshold? Should we not make that decision for them instead?
>>
>> About the implementation: Your code is very very clean as always. There are a couple of things that I would like to see changed around how those iptables rules are being inserted into the existing chains. You are adding something into the POLICYIN chain with surgical precision which might break when the chain is being modified. This is potentially all minor stuff and can be fixed in minutes.
>>
>> Well done!
>>
>> -Michael
>>
>>> On 25 Nov 2019, at 20:13, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>
>>> Implements downloading of IP address blacklists and implementing
>>> them as IPSets. A separate IPSet is used for each blacklist; this
>>> simplifies handling of overlaps between different lists. Traffic
>>> to or from the red0/ppp0 interface is checked against the IPSets.
>>> The check is placed before the IPS check as the IPSet check is
>>> much lighter on CPU use which means that overall CPU use is
>>> reduced.
>>>
>>> The available lists are defined in a separate file. A WUI page
>>> allows the desired lists to be enabled and the interval between
>>> checks for updates to be defined. A minimum update check interval
>>> is defined for each blacklist in the definition file.
>>>
>>> Optionally, an automatically updating blacklist can be enabled.
>>> This adds addresses to an IPSet if the rate of packets dropped by
>>> the default red0/ppp0 input policy exceeds a user defined threshold.
>>> The addresses are kept in the IPSet until a user defined period
>>> without packets from the blocked address has passed.
>>>
>>> Tim FitzGeorge (5):
>>> ipblacklist: Main script
>>> ipblacklist: WUI and language file
>>> ipblacklist: Ancillary files
>>> ipblacklist: Modifications to system
>>> ipblacklist: Build infrastructure
>>>
>>> config/backup/backup.pl | 1 +
>>> config/backup/include | 2 +
>>> config/firewall/firewall-policy | 5 +
>>> config/ipblacklist/sources | 151 +++
>>> config/logwatch/ipblacklist | 103 ++
>>> config/logwatch/ipblacklist.conf | 34 +
>>> config/menu/50-firewall.menu | 5 +
>>> config/rootfiles/common/aarch64/stage2 | 1 +
>>> config/rootfiles/common/configroot | 2 +
>>> config/rootfiles/common/ipblacklist-sources | 1 +
>>> config/rootfiles/common/logwatch | 2 +
>>> config/rootfiles/common/misc-progs | 2 +
>>> config/rootfiles/common/stage2 | 1 +
>>> config/rootfiles/common/web-user-interface | 1 +
>>> config/rootfiles/common/x86_64/stage2 | 1 +
>>> html/cgi-bin/ipblacklist.cgi | 725 +++++++++++++
>>> html/cgi-bin/logs.cgi/log.dat | 2 +
>>> langs/en/cgi-bin/en.pl | 31 +
>>> lfs/configroot | 4 +-
>>> lfs/ipblacklist-sources | 53 +
>>> lfs/logwatch | 2 +
>>> make.sh | 11 +-
>>> src/initscripts/system/firewall | 20 +
>>> src/misc-progs/Makefile | 2 +-
>>> src/misc-progs/getipsetstat.c | 28 +
>>> src/misc-progs/ipblacklistctrl.c | 52 +
>>> src/scripts/ipblacklist | 1558 +++++++++++++++++++++++++++
>>> 27 files changed, 2792 insertions(+), 8 deletions(-)
>>> create mode 100644 config/ipblacklist/sources
>>> create mode 100644 config/logwatch/ipblacklist
>>> create mode 100644 config/logwatch/ipblacklist.conf
>>> create mode 100644 config/rootfiles/common/ipblacklist-sources
>>> create mode 100644 html/cgi-bin/ipblacklist.cgi
>>> create mode 100644 lfs/ipblacklist-sources
>>> create mode 100644 src/misc-progs/getipsetstat.c
>>> create mode 100644 src/misc-progs/ipblacklistctrl.c
>>> create mode 100755 src/scripts/ipblacklist
>>>
>>> --
>>> 2.16.4
>>>
>>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 0/5] ipblacklist: IP Address Blacklists
2019-11-29 23:25 ` Tim FitzGeorge
@ 2019-12-02 11:17 ` Michael Tremer
2019-12-04 17:05 ` Peter Müller
0 siblings, 1 reply; 35+ messages in thread
From: Michael Tremer @ 2019-12-02 11:17 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 13577 bytes --]
Hi,
> On 29 Nov 2019, at 23:25, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>
> Hello Peter and Michael,
>
> My responses to both of your emails are below:
>
> On 28/11/2019 21:39, Peter Müller wrote:
>> Hello Tim, hello Michael, hello *,
>>
>> first, I'd like to apologise for the late response. I used to
>> be more active on this mailing list than I am today, but hopefully
>> this will be a temporary situation only.
>>
>> Michael is right, I did not have time to mention my concerns about
>> fetching the blacklists from external sources. In my opinion, this
>> causes privacy issues as we disclose the public IP addresses of
>> all IPFire installations using a blacklist to its provider/vendor.
>>
>> In my humble opinion, we can rely on a diverse and robust mirror
>> infrastructure for our Core Updates and packages, so why not use them
>> for other data such as blacklists or upcoming libloc as well?
>>
>> We could periodically update the blacklists on our main mirror (and
>> wait for the network to sync it), make sure it is signed and write
>> a small downloader that fetches, validates and installs them.
>>
>> @All: Thoughts on this?
>
> I think there are a number of points here.
>
> Firstly, from the point of a third party using IPFire, is this really
> solving the privacy disclosure problem? There's no way round disclosing
> your public IP address to someone you're downloading from; all this does
> is change who that information is being disclosed to. For the user
> there's no way of knowing whether the source is more or less protective
> of the user's privacy than the blacklist provider. Indeed it won't be
> possible to know who the lists are being downloaded from until the
> download starts.
There is a way: Tor. But that is a totally different story.
The point is rather that a forget list can be sent instead of the “real” one.
> Secondly, latency; some of the lists are updated every 5 minutes. While
> I've limited the maximum check rate to hourly, will the updates
> propagate quickly enough. For reference on my main system the 24
> updates on the CIARMY list made 143 498 changes (additions or
> deletions). I've seem it do over 200 000.
How did you come up with the hour? Will it be retried more often if the download was not successful?
> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is
> the largest at a few MB), there are going to be a lot of them. How will
> this affect the willingness of people to mirror IPFire?
>
>>
>> Talking about the preference of packet filter and IPS, I prefer to
>> use the latter as well as it gains more insight in what kind of malicious
>> traffic tried to pass a firewall machine. On systems with low resources,
>> this might be problematic and removing load from the IPS can be preferred
>> (make this configurable?!), on others, people might want to have both
>> results.
>>
> You're only going to get one result for a packet whichever way round the
> IP blacklist and IPS are since whichever comes first will drop the
> packet before it reaches the second (well it would be possible to put
> the IP blacklist first and get it to log and mark packets which are then
> dropped after the IPS, but I think that's getting a little complicated.
> In addition I've seen the messages about the trouble marking was
> causing in the QoS).
>
> I think it's a 50/50 choice as to which is more valuable first; it's
> probably going to differ from packet to packet. For me the possibility
> of reducing the IPS load means I prefer putting the IP blacklist first
>
> It should be fairly easy to add the choice of where to put the IP
> blacklist. I think it'll have to be in the main firewall script, so
> it'll require a firewall restart, but it's not something that'll be
> changed often.
I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier.
See my comments in my previous email about this.
I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads.
We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily.
I do not even think it makes sense to swap the order in the outgoing direction.
>> Regarding the GeoIP block feature, this has always been my strongest
>> point against it: Neither were dropped packets logged (hope this is
>> still true, as I have not tried for quite a while), nor is more fine
>> grained filtering possible - e.g. by limiting this to certain services -,
>> nor are IPS results available for packets dropped.
>>
>> We should _always_ log dropped packets (no matter which feature drops them),
>> and leave it to the user to decide whether traffic from/to blacklisted
>> IP addresses should traverse the IPS or not.
>
> I agree. I added the possibility to turn off logging because of user
> requests for the earlier version I made available on GitHub, but logging
> is enabled by default.
What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.
>>
>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default.
>> I would love to see the bogon ruleset here, too (think about 8chan successor
>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause
>> interference if RED does not have a public IP address assigned.
>
> I can add a field to the options file that controls whether a list is
> enabled by default.
To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat.
-Michael
>
>>
>> Regarding the code quality, I agree with Michael. Thanks again for providing
>> this. :-) As soon as we agree on answers to the questions raised meanwhile, I
>> will add my "Reviewed-by"-tag.
>>
>> Thanks, and best regards,
>> Peter Müller
>>
>>
>>> Hello Tim,
>>>
>>> Thank you for sending in this patchset.
>>>
>>> I think as well that this functionality would be a great addition to the IPS that we have right now and we have talked about this on numerous occasions - including implementation details.
>>>
>>> Now, you have done the whole thing. Well done.
>>>
>>> Before we dive into the code, can I ask a couple of high-level questions?
>>>
>>> Peter is always bringing up that downloading blacklists isn’t a good idea. It has actually become one of the biggest obstacles in our conversations and I am surprised he didn’t bring it up again :)
>>>
>>> The automatic blacklist feature. What is the objective here? We saw value in having traffic even from malicious sources passed to the IPS so that it will examine it and log it. The idea was to have a better picture of the threats instead of just silencing them. Not sure what is best in the end.
>
> It's intended to provide a quick local response to portscans based on
> packets which are dropped due to the red input policy.
>
>>>
>>> I am unsure how users will deal with this and turn on “all the lists”(TM) and suddenly things do not work any more. How are they meant to figure out a good threshold? Should we not make that decision for them instead?
>
> Indeed. I've seen the messages from people who have turned on all the
> IPS rules. In addition I know that at least one person who used the
> version on GitHub turned on all the lists, even the ones where one is a
> subset of another list (I added code to prevent this). I don't believe
> that it caused major problems since, unlike the IPS, it doesn't have
> rules that can block what for some people is normal activity. It's
> still a bad idea. I think it makes sense to deliver it with a default
> configuration, which might hint that it's not meant to have all the
> lists enabled.
>
> I'll obviously cover this in the Wiki page(s), but that doesn't mean
> people will read it...
>>>
>>> About the implementation: Your code is very very clean as always. There are a couple of things that I would like to see changed around how those iptables rules are being inserted into the existing chains. You are adding something into the POLICYIN chain with surgical precision which might break when the chain is being modified. This is potentially all minor stuff and can be fixed in minutes.
>
> This is one of the areas I was unsure about. I've looked at the
> existing code, but I obviously don't have the detailed knowledge of how
> the different parts of the system interact when setting up the firewall.
>
> The rule added to POLICYIN is the one that adds addresses to the
> autoblacklist; it's not all that critical where it goes in POLICYIN, but
> the ideal place is just before the LOG. Normally the code in
> firewall-policy will put it in the right place, but I think I know how
> to place it better in ipblacklist.
>
> (Aside - I write onboard software for spacecraft and we always try to
> bear in mind that it could be use that has to modify it in a hurry after
> working on something else for ten years. It does tend to encourage
> being tidy).
>
>>>
>>> Well done!
>>>
>>> -Michael
>
> If you let me know what you think, I can make some modifications and
> then send a V2.
>
> Tim
>
>>>
>>>> On 25 Nov 2019, at 20:13, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>>
>>>> Implements downloading of IP address blacklists and implementing
>>>> them as IPSets. A separate IPSet is used for each blacklist; this
>>>> simplifies handling of overlaps between different lists. Traffic
>>>> to or from the red0/ppp0 interface is checked against the IPSets.
>>>> The check is placed before the IPS check as the IPSet check is
>>>> much lighter on CPU use which means that overall CPU use is
>>>> reduced.
>>>>
>>>> The available lists are defined in a separate file. A WUI page
>>>> allows the desired lists to be enabled and the interval between
>>>> checks for updates to be defined. A minimum update check interval
>>>> is defined for each blacklist in the definition file.
>>>>
>>>> Optionally, an automatically updating blacklist can be enabled.
>>>> This adds addresses to an IPSet if the rate of packets dropped by
>>>> the default red0/ppp0 input policy exceeds a user defined threshold.
>>>> The addresses are kept in the IPSet until a user defined period
>>>> without packets from the blocked address has passed.
>>>>
>>>> Tim FitzGeorge (5):
>>>> ipblacklist: Main script
>>>> ipblacklist: WUI and language file
>>>> ipblacklist: Ancillary files
>>>> ipblacklist: Modifications to system
>>>> ipblacklist: Build infrastructure
>>>>
>>>> config/backup/backup.pl | 1 +
>>>> config/backup/include | 2 +
>>>> config/firewall/firewall-policy | 5 +
>>>> config/ipblacklist/sources | 151 +++
>>>> config/logwatch/ipblacklist | 103 ++
>>>> config/logwatch/ipblacklist.conf | 34 +
>>>> config/menu/50-firewall.menu | 5 +
>>>> config/rootfiles/common/aarch64/stage2 | 1 +
>>>> config/rootfiles/common/configroot | 2 +
>>>> config/rootfiles/common/ipblacklist-sources | 1 +
>>>> config/rootfiles/common/logwatch | 2 +
>>>> config/rootfiles/common/misc-progs | 2 +
>>>> config/rootfiles/common/stage2 | 1 +
>>>> config/rootfiles/common/web-user-interface | 1 +
>>>> config/rootfiles/common/x86_64/stage2 | 1 +
>>>> html/cgi-bin/ipblacklist.cgi | 725 +++++++++++++
>>>> html/cgi-bin/logs.cgi/log.dat | 2 +
>>>> langs/en/cgi-bin/en.pl | 31 +
>>>> lfs/configroot | 4 +-
>>>> lfs/ipblacklist-sources | 53 +
>>>> lfs/logwatch | 2 +
>>>> make.sh | 11 +-
>>>> src/initscripts/system/firewall | 20 +
>>>> src/misc-progs/Makefile | 2 +-
>>>> src/misc-progs/getipsetstat.c | 28 +
>>>> src/misc-progs/ipblacklistctrl.c | 52 +
>>>> src/scripts/ipblacklist | 1558 +++++++++++++++++++++++++++
>>>> 27 files changed, 2792 insertions(+), 8 deletions(-)
>>>> create mode 100644 config/ipblacklist/sources
>>>> create mode 100644 config/logwatch/ipblacklist
>>>> create mode 100644 config/logwatch/ipblacklist.conf
>>>> create mode 100644 config/rootfiles/common/ipblacklist-sources
>>>> create mode 100644 html/cgi-bin/ipblacklist.cgi
>>>> create mode 100644 lfs/ipblacklist-sources
>>>> create mode 100644 src/misc-progs/getipsetstat.c
>>>> create mode 100644 src/misc-progs/ipblacklistctrl.c
>>>> create mode 100755 src/scripts/ipblacklist
>>>>
>>>> --
>>>> 2.16.4
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 0/5] ipblacklist: IP Address Blacklists
2019-12-02 11:17 ` Michael Tremer
@ 2019-12-04 17:05 ` Peter Müller
2019-12-05 22:25 ` Michael Tremer
0 siblings, 1 reply; 35+ messages in thread
From: Peter Müller @ 2019-12-04 17:05 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 6811 bytes --]
Hello Tim, hello Michael,
please see my responses inline...
>>>
>>> We could periodically update the blacklists on our main mirror (and
>>> wait for the network to sync it), make sure it is signed and write
>>> a small downloader that fetches, validates and installs them.
>>>
>>> @All: Thoughts on this?
>>
>> I think there are a number of points here.
>>
>> Firstly, from the point of a third party using IPFire, is this really
>> solving the privacy disclosure problem? There's no way round disclosing
>> your public IP address to someone you're downloading from; all this does
>> is change who that information is being disclosed to. For the user
>> there's no way of knowing whether the source is more or less protective
>> of the user's privacy than the blacklist provider. Indeed it won't be
>> possible to know who the lists are being downloaded from until the
>> download starts.
>
> There is a way: Tor. But that is a totally different story.
Well, I see a third option on this: Use the mirror infrastructure we already
have. Every IPFire installation discloses its public IP address to one of these
servers sooner or later, so we do not disclose additional data if the blacklists
were fetched from these.
Needless to say, Tor (hidden services) would be better, but that is a different
story indeed. :-)
>
> The point is rather that a forget list can be sent instead of the “real” one.
I did not get this. Forget? Forged?? ???
>
>> Secondly, latency; some of the lists are updated every 5 minutes. While
>> I've limited the maximum check rate to hourly, will the updates
>> propagate quickly enough. For reference on my main system the 24
>> updates on the CIARMY list made 143 498 changes (additions or
>> deletions). I've seem it do over 200 000.
Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not
document a recommended update interval anywhere, so we can only guess.
Personally, more static lists seem to be preferable for packet filtering. Highly
dynamic ones such as CIArmy should be done via DNSBL queries or something similar
- do we really want to have that list here?
>
> How did you come up with the hour? Will it be retried more often if the download was not successful?
One hour is the most common interval indeed, but adding some random time might
be useful in order to reduce load on the servers providing a blacklist.
>
>> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is
>> the largest at a few MB), there are going to be a lot of them. How will
>> this affect the willingness of people to mirror IPFire?
I do not consider this being a problem as we do not generate that much traffic
to them. Of course, that depends on the update interval again.
>>
>>>
>>> Talking about the preference of packet filter and IPS, I prefer to
>>> use the latter as well as it gains more insight in what kind of malicious
>>> traffic tried to pass a firewall machine. On systems with low resources,
>>> this might be problematic and removing load from the IPS can be preferred
>>> (make this configurable?!), on others, people might want to have both
>>> results.
>>>
>> You're only going to get one result for a packet whichever way round the
>> IP blacklist and IPS are since whichever comes first will drop the
>> packet before it reaches the second (well it would be possible to put
>> the IP blacklist first and get it to log and mark packets which are then
>> dropped after the IPS, but I think that's getting a little complicated.
>> In addition I've seen the messages about the trouble marking was
>> causing in the QoS).
>>
>> I think it's a 50/50 choice as to which is more valuable first; it's
>> probably going to differ from packet to packet. For me the possibility
>> of reducing the IPS load means I prefer putting the IP blacklist first
>>
>> It should be fairly easy to add the choice of where to put the IP
>> blacklist. I think it'll have to be in the main firewall script, so
>> it'll require a firewall restart, but it's not something that'll be
>> changed often.
>
> I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier.
Agreed.
>
> I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads.
>
> We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily.
So dropping packets from blacklisted IP addresses/networks before IPS is it, then.
>
> I do not even think it makes sense to swap the order in the outgoing direction.
Me too.
>
> What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this.
Based on my personal experience, firewall hits observed on a single machine exposed
to the internet are interesting, but the overall situation across multiple machines
is even more interesting. Very quickly, you'll end on something like a centralised
logging server and custom statistical analysis here...
>
>>>
>>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default.
>>> I would love to see the bogon ruleset here, too (think about 8chan successor
>>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause
>>> interference if RED does not have a public IP address assigned.
>>
>> I can add a field to the options file that controls whether a list is
>> enabled by default.
Thank you. :-)
>
> To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat.
This is my only objection against this patchset. Now, what can we do about it?
One possibility is to apply the patchset now and implement a custom download
source thing later on, or do that before releasing Core Update 139 (or which version
the patchset will be to) after we agreed on something.
If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not
seem to be problematic, as they are hosted on 3rd party servers, too).
Thanks, and best regards,
Peter Müller
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 0/5] ipblacklist: IP Address Blacklists
2019-12-04 17:05 ` Peter Müller
@ 2019-12-05 22:25 ` Michael Tremer
2019-12-08 20:50 ` Tim FitzGeorge
0 siblings, 1 reply; 35+ messages in thread
From: Michael Tremer @ 2019-12-05 22:25 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 8244 bytes --]
Hello,
> On 4 Dec 2019, at 17:05, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>
> Hello Tim, hello Michael,
>
> please see my responses inline...
>>>>
>>>> We could periodically update the blacklists on our main mirror (and
>>>> wait for the network to sync it), make sure it is signed and write
>>>> a small downloader that fetches, validates and installs them.
>>>>
>>>> @All: Thoughts on this?
>>>
>>> I think there are a number of points here.
>>>
>>> Firstly, from the point of a third party using IPFire, is this really
>>> solving the privacy disclosure problem? There's no way round disclosing
>>> your public IP address to someone you're downloading from; all this does
>>> is change who that information is being disclosed to. For the user
>>> there's no way of knowing whether the source is more or less protective
>>> of the user's privacy than the blacklist provider. Indeed it won't be
>>> possible to know who the lists are being downloaded from until the
>>> download starts.
>>
>> There is a way: Tor. But that is a totally different story.
> Well, I see a third option on this: Use the mirror infrastructure we already
> have. Every IPFire installation discloses its public IP address to one of these
> servers sooner or later, so we do not disclose additional data if the blacklists
> were fetched from these.
>
> Needless to say, Tor (hidden services) would be better, but that is a different
> story indeed. :-)
>>
>> The point is rather that a forget list can be sent instead of the “real” one.
> I did not get this. Forget? Forged?? ???
Yes, I meant to write forged, but auto-correct didn’t let me.
>>> Secondly, latency; some of the lists are updated every 5 minutes. While
>>> I've limited the maximum check rate to hourly, will the updates
>>> propagate quickly enough. For reference on my main system the 24
>>> updates on the CIARMY list made 143 498 changes (additions or
>>> deletions). I've seem it do over 200 000.
> Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not
> document a recommended update interval anywhere, so we can only guess.
>
> Personally, more static lists seem to be preferable for packet filtering. Highly
> dynamic ones such as CIArmy should be done via DNSBL queries or something similar
> - do we really want to have that list here?
It is not really an option to implement a DNSBL into a packet filter, but I get your point.
>> How did you come up with the hour? Will it be retried more often if the download was not successful?
> One hour is the most common interval indeed, but adding some random time might
> be useful in order to reduce load on the servers providing a blacklist.
Yes, definitely. Otherwise we will shoot down our mirrors.
>>> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is
>>> the largest at a few MB), there are going to be a lot of them. How will
>>> this affect the willingness of people to mirror IPFire?
> I do not consider this being a problem as we do not generate that much traffic
> to them. Of course, that depends on the update interval again.
That depends on your point of view.
I do not have a problem with this at all in my data center, but there are plenty of people with a volume-based LTE plan or simply a 128 kBit/s connection. It will take a longer time to download the lists for them. We need to mind that.
>>>
>>>>
>>>> Talking about the preference of packet filter and IPS, I prefer to
>>>> use the latter as well as it gains more insight in what kind of malicious
>>>> traffic tried to pass a firewall machine. On systems with low resources,
>>>> this might be problematic and removing load from the IPS can be preferred
>>>> (make this configurable?!), on others, people might want to have both
>>>> results.
>>>>
>>> You're only going to get one result for a packet whichever way round the
>>> IP blacklist and IPS are since whichever comes first will drop the
>>> packet before it reaches the second (well it would be possible to put
>>> the IP blacklist first and get it to log and mark packets which are then
>>> dropped after the IPS, but I think that's getting a little complicated.
>>> In addition I've seen the messages about the trouble marking was
>>> causing in the QoS).
>>>
>>> I think it's a 50/50 choice as to which is more valuable first; it's
>>> probably going to differ from packet to packet. For me the possibility
>>> of reducing the IPS load means I prefer putting the IP blacklist first
>>>
>>> It should be fairly easy to add the choice of where to put the IP
>>> blacklist. I think it'll have to be in the main firewall script, so
>>> it'll require a firewall restart, but it's not something that'll be
>>> changed often.
>>
>> I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier.
> Agreed.
>
>>
>> I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads.
>>
>> We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily.
> So dropping packets from blacklisted IP addresses/networks before IPS is it, then.
>>
>> I do not even think it makes sense to swap the order in the outgoing direction.
> Me too.
>
>>
>> What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this.
> Based on my personal experience, firewall hits observed on a single machine exposed
> to the internet are interesting, but the overall situation across multiple machines
> is even more interesting. Very quickly, you'll end on something like a centralised
> logging server and custom statistical analysis here...
Probably a project for IPFire 4.0 :)
>>
>>>>
>>>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default.
>>>> I would love to see the bogon ruleset here, too (think about 8chan successor
>>>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause
>>>> interference if RED does not have a public IP address assigned.
>>>
>>> I can add a field to the options file that controls whether a list is
>>> enabled by default.
> Thank you. :-)
>>
>> To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat.
> This is my only objection against this patchset. Now, what can we do about it?
> One possibility is to apply the patchset now and implement a custom download
> source thing later on, or do that before releasing Core Update 139 (or which version
> the patchset will be to) after we agreed on something.
I do not see this being merged for 139. But that is not important. We need to get it right first and then release it.
As far as I know, nobody has tested this, yet.
I have huge concerns about the automatic blacklist. @Peter: What is your opinion on this?
> If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not
> seem to be problematic, as they are hosted on 3rd party servers, too).
One of them will be, sooner or later. And one is enough I suppose.
I do not really want to overthink this - we didn’t do this with clamav for example either. But it probably is a decision to be made by the user what they want to enable and we should not enable anything by default. So no data will be leaked as long as the user does not consent.
-Michael
>
> Thanks, and best regards,
> Peter Müller
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 0/5] ipblacklist: IP Address Blacklists
2019-12-05 22:25 ` Michael Tremer
@ 2019-12-08 20:50 ` Tim FitzGeorge
2019-12-13 23:11 ` Michael Tremer
0 siblings, 1 reply; 35+ messages in thread
From: Tim FitzGeorge @ 2019-12-08 20:50 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 14430 bytes --]
Hello,
It's my turn to apologise for being slow to respond - I've had a busy
week, but I should have plenty of time over the next couple of weeks.
I've made most of the comments inline, however I think Michael had a
question (which I can't find now) about what happens if someone enables
all the lists. One thing which would perhaps make this less likely is
that the WUI tags the available lists with whether they're safe or not,
with a footnote that safe means that the list only blocks malicious
traffic. This won't guarantee that a user won't still try to enable all
the lists, but it should make them realise that they should think first.
I have considered replacing this tag with a risk high/medium/low and
maybe adding a category (invalid/application/scanner/C&C or something
like that), but that may provide too much information and dissuade them
from actually following the links to checkout what the list actually does.
Tim
On 05/12/2019 22:25, Michael Tremer wrote:
> Hello,
>
>> On 4 Dec 2019, at 17:05, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>
>> Hello Tim, hello Michael,
>>
>> please see my responses inline...
>>>>>
>>>>> We could periodically update the blacklists on our main mirror (and
>>>>> wait for the network to sync it), make sure it is signed and write
>>>>> a small downloader that fetches, validates and installs them.
>>>>>
>>>>> @All: Thoughts on this?
>>>>
>>>> I think there are a number of points here.
>>>>
>>>> Firstly, from the point of a third party using IPFire, is this really
>>>> solving the privacy disclosure problem? There's no way round disclosing
>>>> your public IP address to someone you're downloading from; all this does
>>>> is change who that information is being disclosed to. For the user
>>>> there's no way of knowing whether the source is more or less protective
>>>> of the user's privacy than the blacklist provider. Indeed it won't be
>>>> possible to know who the lists are being downloaded from until the
>>>> download starts.
>>>
>>> There is a way: Tor. But that is a totally different story.
>> Well, I see a third option on this: Use the mirror infrastructure we already
>> have. Every IPFire installation discloses its public IP address to one of these
>> servers sooner or later, so we do not disclose additional data if the blacklists
>> were fetched from these.
>>
>> Needless to say, Tor (hidden services) would be better, but that is a different
>> story indeed. :-)
>>>
>>> The point is rather that a forget list can be sent instead of the “real” one.
>> I did not get this. Forget? Forged?? ???
>
> Yes, I meant to write forged, but auto-correct didn’t let me.
>
>>>> Secondly, latency; some of the lists are updated every 5 minutes. While
>>>> I've limited the maximum check rate to hourly, will the updates
>>>> propagate quickly enough. For reference on my main system the 24
>>>> updates on the CIARMY list made 143 498 changes (additions or
>>>> deletions). I've seem it do over 200 000.
>> Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not
>> document a recommended update interval anywhere, so we can only guess.
>>
>> Personally, more static lists seem to be preferable for packet filtering. Highly
>> dynamic ones such as CIArmy should be done via DNSBL queries or something similar
>> - do we really want to have that list here?
>
> It is not really an option to implement a DNSBL into a packet filter, but I get your point.
One of the 'selling points' for an IP address blacklist is that it can
respond quickly to new threats - or rather new attackers. While a new
IDS/IPS rule needs time to analyse the threat, generate a rule and check
it, it's easy to add an address to a list. So, I think the CIArmy list
is potentially useful for protecting home systems etc. with budget
hardware, but I would be very careful about using it for a protecting a
general access website.
>
>>> How did you come up with the hour? Will it be retried more often if the download was not successful?
>> One hour is the most common interval indeed, but adding some random time might
>> be useful in order to reduce load on the servers providing a blacklist.
>
> Yes, definitely. Otherwise we will shoot down our mirrors.
When I implemented that section of code, specifying the minimum check
period in hours seemed to provide a convenient way of allowing a check
period covering a wide range, with an hour as the fastest and a week as
the slowest. I didn't looked at the CIArmy list until much later. Most
of the lists don't change nearly as much, but the CIArmy list is
described as one that deliberately responds quickly.
>From my production system, for yesterday:
The following block lists were updated:
BLOCKLIST_DE: 24 Time(s) - 9341 change(s)
BOGON_FULL: 1 Time(s) - 10 change(s)
CIARMY: 24 Time(s) - 159134 change(s)
DSHIELD: 7 Time(s) - 18 change(s)
EMERGING_FWRULE: 1 Time(s) - 50 change(s)
FEODO_AGGRESIVE: 24 Time(s) - 13 change(s)
SHODAN: 1 Time(s) - 0 change(s)
SPAMHAUS_DROP: 1 Time(s) - 0 change(s)
TOR_EXIT: 24 Time(s) - 162 change(s)
and my test system:
The following block lists were updated:
ALIENVAULT: 19 Time(s) - 5331 change(s)
EMERGING_COMPROMISED: 1 Time(s) - 26 change(s)
TALOS_MALICIOUS: 1 Time(s) - 36 change(s)
That covers most of the lists. From the WUI, since 1 Dec:
Blacklist Entries pkts bytes Last updated
in in
AUTOBLACKLIST 0 731 51144 Sun Dec 8 18:40:02 2019
BLOCKLIST_DE 28020 857 46735 Sun Dec 8 18:40:02 2019
BOGON_FULL 214 5255 189K Sun Dec 8 16:50:04 2019
CIARMY 15000 19774 976K Sun Dec 8 18:04:01 2019
DSHIELD 20 7992 321K Sun Dec 8 16:45:13 2019
EMERGING_FWRULE 1647 197 8383 Fri Dec 6 05:29:07 2019
FEODO_AGGRESIVE 7169 0 0 Sun Dec 8 18:50:07 2019
SHODAN 32 34 1530 Sun Dec 8 17:54:09 2019
SPAMHAUS_DROP 823 0 0 Fri Dec 6 18:22:35 2019
SPAMHAUS_EDROP 111 82 16433 Thu Dec 5 17:27:09 2019
TOR_EXIT 1055 0 0 Sun Dec 8 18:31:02 2019
(I've left out the pkts/bytes out fields which were all 0)
Note that where possible I do a HEAD request first and then only
download the list if the modification time has changed since the last
check. For dynamically generated lists this isn't possible.
If the download isn't successful it just gives up and waits for the next
attempt (apart from the usual retries in the library). I probably
should to change that so that it only applies the per list minimum
update period in this case (specified in the sources file) rather than
the user specified value as well.
I already use a time offset on the downloads - when it's started from
boot, backup restore or WUI enable, it checks to see if it's installed
in the fcrontab, and if not adds itself at a randomly generated offset
in the hour.
>
>>>> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is
>>>> the largest at a few MB), there are going to be a lot of them. How will
>>>> this affect the willingness of people to mirror IPFire?
>> I do not consider this being a problem as we do not generate that much traffic
>> to them. Of course, that depends on the update interval again.
>
> That depends on your point of view.
>
> I do not have a problem with this at all in my data center, but there are plenty of people with a volume-based LTE plan or simply a 128 kBit/s connection. It will take a longer time to download the lists for them. We need to mind that.
>
>>>>
>>>>>
>>>>> Talking about the preference of packet filter and IPS, I prefer to
>>>>> use the latter as well as it gains more insight in what kind of malicious
>>>>> traffic tried to pass a firewall machine. On systems with low resources,
>>>>> this might be problematic and removing load from the IPS can be preferred
>>>>> (make this configurable?!), on others, people might want to have both
>>>>> results.
>>>>>
>>>> You're only going to get one result for a packet whichever way round the
>>>> IP blacklist and IPS are since whichever comes first will drop the
>>>> packet before it reaches the second (well it would be possible to put
>>>> the IP blacklist first and get it to log and mark packets which are then
>>>> dropped after the IPS, but I think that's getting a little complicated.
>>>> In addition I've seen the messages about the trouble marking was
>>>> causing in the QoS).
>>>>
>>>> I think it's a 50/50 choice as to which is more valuable first; it's
>>>> probably going to differ from packet to packet. For me the possibility
>>>> of reducing the IPS load means I prefer putting the IP blacklist first
>>>>
>>>> It should be fairly easy to add the choice of where to put the IP
>>>> blacklist. I think it'll have to be in the main firewall script, so
>>>> it'll require a firewall restart, but it's not something that'll be
>>>> changed often.
>>>
>>> I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier.
>> Agreed.
>>
>>>
>>> I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads.
>>>
>>> We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily.
>> So dropping packets from blacklisted IP addresses/networks before IPS is it, then.
>>>
>>> I do not even think it makes sense to swap the order in the outgoing direction.
>> Me too.
>>
>>>
>>> What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this.
>> Based on my personal experience, firewall hits observed on a single machine exposed
>> to the internet are interesting, but the overall situation across multiple machines
>> is even more interesting. Very quickly, you'll end on something like a centralised
>> logging server and custom statistical analysis here...
>
> Probably a project for IPFire 4.0 :)
>
Or use one of the existing services, like the DSHIELD client
https://dshield.org/howto.html (subject to privacy concerns again).
>>>
>>>>>
>>>>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default.
>>>>> I would love to see the bogon ruleset here, too (think about 8chan successor
>>>>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause
>>>>> interference if RED does not have a public IP address assigned.
>>>>
>>>> I can add a field to the options file that controls whether a list is
>>>> enabled by default.
>> Thank you. :-)
>>>
>>> To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat.
>> This is my only objection against this patchset. Now, what can we do about it?
>> One possibility is to apply the patchset now and implement a custom download
>> source thing later on, or do that before releasing Core Update 139 (or which version
>> the patchset will be to) after we agreed on something.
>
> I do not see this being merged for 139. But that is not important. We need to get it right first and then release it.
>
> As far as I know, nobody has tested this, yet.
There are a number of people who have been running an earlier version
which I shared on GitHub. There were a few early issues, but it seems
to be OK now.
https://forum.ipfire.org/viewtopic.php?f=27&t=21845
This version wasn't integrated into IPFire, so (for example) it inserted
itself into the INPUT IPTables chain rather than having it's chains
created as part of the firewall start-up script.
>
> I have huge concerns about the automatic blacklist. @Peter: What is your opinion on this?
>
While I implemented it, I'm aware of its potential to cause problems,
which is why it has to be separately enabled. It's not caused me any
issues at the default settings (blocks at over 10 packets per hour until
1 hour has passed without seeing packets from the address), but I've not
used it on a site with publicly announced services. If I was going to
use it on a web site I would want to, at the very minimum, drop the
block period drastically.
On the other hand, it's good at responding quickly. Usually I see only
1-2% of blocks from the automatic list:
Reason Count % First Last
CIARMY 2416 45 Dec 6 00:00 Dec 6 23:59
DSHIELD 1353 25 Dec 6 00:00 Dec 6 23:59
INPUT 1294 24 Dec 6 00:00 Dec 6 23:59
AUTOBLACKLIST 122 2 Dec 6 00:20 Dec 6 16:28
BLOCKLIST_DE 89 2 Dec 6 00:20 Dec 6 23:46
and sometimes none at all, but one one occasion it blocked over 8000
packets. Again I'm aware this is for a home system, which is rather
different than from a Web server.
>> If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not
>> seem to be problematic, as they are hosted on 3rd party servers, too).
>
> One of them will be, sooner or later. And one is enough I suppose.
DShield (https://dshield.org/api/#threatfeeds) and firehol
(http://iplists.firehol.org/) seem to host copies of most of the lists
as well.
>
> I do not really want to overthink this - we didn’t do this with clamav for example either. But it probably is a decision to be made by the user what they want to enable and we should not enable anything by default. So no data will be leaked as long as the user does not consent.
>
> -Michael
>
>>
>> Thanks, and best regards,
>> Peter Müller
>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 0/5] ipblacklist: IP Address Blacklists
2019-12-08 20:50 ` Tim FitzGeorge
@ 2019-12-13 23:11 ` Michael Tremer
0 siblings, 0 replies; 35+ messages in thread
From: Michael Tremer @ 2019-12-13 23:11 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 17902 bytes --]
Hi,
Again my apologies for my late reply. Busy busy weeks.
> On 8 Dec 2019, at 20:50, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>
> Hello,
>
> It's my turn to apologise for being slow to respond - I've had a busy
> week, but I should have plenty of time over the next couple of weeks.
No worries. Turns out we all do :)
> I've made most of the comments inline, however I think Michael had a
> question (which I can't find now) about what happens if someone enables
> all the lists. One thing which would perhaps make this less likely is
> that the WUI tags the available lists with whether they're safe or not,
> with a footnote that safe means that the list only blocks malicious
> traffic. This won't guarantee that a user won't still try to enable all
> the lists, but it should make them realise that they should think first.
We had a couple of features going slightly wrong or being “misunderstood” by some users. People still seem to panic when they see “local recursor”. To this day I do not know why.
We cannot make everything idiot-proof. And when some user if of that category, they probably should shutdown their IPFire box, educate themselves and then come back again. So I do not want to limit people, but make things as easy as possible.
If someone enables all the lists, good luck with passing packets :)
> I have considered replacing this tag with a risk high/medium/low and
> maybe adding a category (invalid/application/scanner/C&C or something
> like that), but that may provide too much information and dissuade them
> from actually following the links to checkout what the list actually does.
Can we have a screenshot of the GUI right now? I didn’t run the code, yet.
We should document the lists like we do it with the rulesets of the IPS. People might ignore this, but that is on them.
>
> Tim
>
>
> On 05/12/2019 22:25, Michael Tremer wrote:
>> Hello,
>>
>>> On 4 Dec 2019, at 17:05, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>>
>>> Hello Tim, hello Michael,
>>>
>>> please see my responses inline...
>>>>>>
>>>>>> We could periodically update the blacklists on our main mirror (and
>>>>>> wait for the network to sync it), make sure it is signed and write
>>>>>> a small downloader that fetches, validates and installs them.
>>>>>>
>>>>>> @All: Thoughts on this?
>>>>>
>>>>> I think there are a number of points here.
>>>>>
>>>>> Firstly, from the point of a third party using IPFire, is this really
>>>>> solving the privacy disclosure problem? There's no way round disclosing
>>>>> your public IP address to someone you're downloading from; all this does
>>>>> is change who that information is being disclosed to. For the user
>>>>> there's no way of knowing whether the source is more or less protective
>>>>> of the user's privacy than the blacklist provider. Indeed it won't be
>>>>> possible to know who the lists are being downloaded from until the
>>>>> download starts.
>>>>
>>>> There is a way: Tor. But that is a totally different story.
>>> Well, I see a third option on this: Use the mirror infrastructure we already
>>> have. Every IPFire installation discloses its public IP address to one of these
>>> servers sooner or later, so we do not disclose additional data if the blacklists
>>> were fetched from these.
>>>
>>> Needless to say, Tor (hidden services) would be better, but that is a different
>>> story indeed. :-)
>>>>
>>>> The point is rather that a forget list can be sent instead of the “real” one.
>>> I did not get this. Forget? Forged?? ???
>>
>> Yes, I meant to write forged, but auto-correct didn’t let me.
>>
>>>>> Secondly, latency; some of the lists are updated every 5 minutes. While
>>>>> I've limited the maximum check rate to hourly, will the updates
>>>>> propagate quickly enough. For reference on my main system the 24
>>>>> updates on the CIARMY list made 143 498 changes (additions or
>>>>> deletions). I've seem it do over 200 000.
>>> Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not
>>> document a recommended update interval anywhere, so we can only guess.
>>>
>>> Personally, more static lists seem to be preferable for packet filtering. Highly
>>> dynamic ones such as CIArmy should be done via DNSBL queries or something similar
>>> - do we really want to have that list here?
>>
>> It is not really an option to implement a DNSBL into a packet filter, but I get your point.
>
> One of the 'selling points' for an IP address blacklist is that it can
> respond quickly to new threats - or rather new attackers. While a new
> IDS/IPS rule needs time to analyse the threat, generate a rule and check
> it, it's easy to add an address to a list. So, I think the CIArmy list
> is potentially useful for protecting home systems etc. with budget
> hardware, but I would be very careful about using it for a protecting a
> general access website.
If they are very volatile, we should honour that and update them often, too.
It probably is more about false-positives being removed very quickly instead of adding threats very quickly. The average IPFire user is probably not under threats like these to need to react very quickly.
So I do not see much value in adding those lists and then updating once a day. Does it have to be every 5 min? No. I would suggest 15 which should be good enough for everyone.
Other lists should of course not be updated every 15 minutes when not needed.
Running every 15 minutes would allow us to retry downloading lists that are on an hourly schedule if the download failed.
>>
>>>> How did you come up with the hour? Will it be retried more often if the download was not successful?
>>> One hour is the most common interval indeed, but adding some random time might
>>> be useful in order to reduce load on the servers providing a blacklist.
>>
>> Yes, definitely. Otherwise we will shoot down our mirrors.
>
> When I implemented that section of code, specifying the minimum check
> period in hours seemed to provide a convenient way of allowing a check
> period covering a wide range, with an hour as the fastest and a week as
> the slowest. I didn't looked at the CIArmy list until much later. Most
> of the lists don't change nearly as much, but the CIArmy list is
> described as one that deliberately responds quickly.
>
> From my production system, for yesterday:
>
> The following block lists were updated:
> BLOCKLIST_DE: 24 Time(s) - 9341 change(s)
> BOGON_FULL: 1 Time(s) - 10 change(s)
> CIARMY: 24 Time(s) - 159134 change(s)
> DSHIELD: 7 Time(s) - 18 change(s)
> EMERGING_FWRULE: 1 Time(s) - 50 change(s)
> FEODO_AGGRESIVE: 24 Time(s) - 13 change(s)
> SHODAN: 1 Time(s) - 0 change(s)
> SPAMHAUS_DROP: 1 Time(s) - 0 change(s)
> TOR_EXIT: 24 Time(s) - 162 change(s)
Very interesting statistics.
>
> and my test system:
>
> The following block lists were updated:
> ALIENVAULT: 19 Time(s) - 5331 change(s)
> EMERGING_COMPROMISED: 1 Time(s) - 26 change(s)
> TALOS_MALICIOUS: 1 Time(s) - 36 change(s)
>
> That covers most of the lists. From the WUI, since 1 Dec:
>
> Blacklist Entries pkts bytes Last updated
> in in
> AUTOBLACKLIST 0 731 51144 Sun Dec 8 18:40:02 2019
> BLOCKLIST_DE 28020 857 46735 Sun Dec 8 18:40:02 2019
> BOGON_FULL 214 5255 189K Sun Dec 8 16:50:04 2019
> CIARMY 15000 19774 976K Sun Dec 8 18:04:01 2019
> DSHIELD 20 7992 321K Sun Dec 8 16:45:13 2019
> EMERGING_FWRULE 1647 197 8383 Fri Dec 6 05:29:07 2019
> FEODO_AGGRESIVE 7169 0 0 Sun Dec 8 18:50:07 2019
> SHODAN 32 34 1530 Sun Dec 8 17:54:09 2019
> SPAMHAUS_DROP 823 0 0 Fri Dec 6 18:22:35 2019
> SPAMHAUS_EDROP 111 82 16433 Thu Dec 5 17:27:09 2019
> TOR_EXIT 1055 0 0 Sun Dec 8 18:31:02 2019
This as well.
Those are more packets than I would have expected.
>
> (I've left out the pkts/bytes out fields which were all 0)
>
> Note that where possible I do a HEAD request first and then only
> download the list if the modification time has changed since the last
> check. For dynamically generated lists this isn't possible.
You won’t need a HEAD request for it. You can include it in the GET request.
Have a look at location downloader where I use that. The server will respond with 304 and not send any payload.
https://git.ipfire.org/?p=location/libloc.git;a=blob;f=src/python/location-downloader.in;h=1b5932d5822e03737d32b2a27816815b2f7e74dd;hb=HEAD#l151
>
> If the download isn't successful it just gives up and waits for the next
> attempt (apart from the usual retries in the library). I probably
> should to change that so that it only applies the per list minimum
> update period in this case (specified in the sources file) rather than
> the user specified value as well.
I think it is not the worst if an update fails. It might just happen every once in a while.
So I would suggest to just re-run the script more often and when the mtime of the file is older than the threshold, a download is attempted. You can use that timestamp for the GET request.
> I already use a time offset on the downloads - when it's started from
> boot, backup restore or WUI enable, it checks to see if it's installed
> in the fcrontab, and if not adds itself at a randomly generated offset
> in the hour.
That should potentially go to red.up, or if we can settle on 15 minutes, I would consider that often enough to quickly update all lists after a reboot.
>
>>
>>>>> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is
>>>>> the largest at a few MB), there are going to be a lot of them. How will
>>>>> this affect the willingness of people to mirror IPFire?
>>> I do not consider this being a problem as we do not generate that much traffic
>>> to them. Of course, that depends on the update interval again.
>>
>> That depends on your point of view.
>>
>> I do not have a problem with this at all in my data center, but there are plenty of people with a volume-based LTE plan or simply a 128 kBit/s connection. It will take a longer time to download the lists for them. We need to mind that.
>>
>>>>>
>>>>>>
>>>>>> Talking about the preference of packet filter and IPS, I prefer to
>>>>>> use the latter as well as it gains more insight in what kind of malicious
>>>>>> traffic tried to pass a firewall machine. On systems with low resources,
>>>>>> this might be problematic and removing load from the IPS can be preferred
>>>>>> (make this configurable?!), on others, people might want to have both
>>>>>> results.
>>>>>>
>>>>> You're only going to get one result for a packet whichever way round the
>>>>> IP blacklist and IPS are since whichever comes first will drop the
>>>>> packet before it reaches the second (well it would be possible to put
>>>>> the IP blacklist first and get it to log and mark packets which are then
>>>>> dropped after the IPS, but I think that's getting a little complicated.
>>>>> In addition I've seen the messages about the trouble marking was
>>>>> causing in the QoS).
>>>>>
>>>>> I think it's a 50/50 choice as to which is more valuable first; it's
>>>>> probably going to differ from packet to packet. For me the possibility
>>>>> of reducing the IPS load means I prefer putting the IP blacklist first
>>>>>
>>>>> It should be fairly easy to add the choice of where to put the IP
>>>>> blacklist. I think it'll have to be in the main firewall script, so
>>>>> it'll require a firewall restart, but it's not something that'll be
>>>>> changed often.
>>>>
>>>> I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier.
>>> Agreed.
>>>
>>>>
>>>> I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads.
>>>>
>>>> We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily.
>>> So dropping packets from blacklisted IP addresses/networks before IPS is it, then.
>>>>
>>>> I do not even think it makes sense to swap the order in the outgoing direction.
>>> Me too.
>>>
>>>>
>>>> What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this.
>>> Based on my personal experience, firewall hits observed on a single machine exposed
>>> to the internet are interesting, but the overall situation across multiple machines
>>> is even more interesting. Very quickly, you'll end on something like a centralised
>>> logging server and custom statistical analysis here...
>>
>> Probably a project for IPFire 4.0 :)
>>
> Or use one of the existing services, like the DSHIELD client
> https://dshield.org/howto.html (subject to privacy concerns again).
>>>>
>>>>>>
>>>>>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default.
>>>>>> I would love to see the bogon ruleset here, too (think about 8chan successor
>>>>>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause
>>>>>> interference if RED does not have a public IP address assigned.
>>>>>
>>>>> I can add a field to the options file that controls whether a list is
>>>>> enabled by default.
>>> Thank you. :-)
>>>>
>>>> To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat.
>>> This is my only objection against this patchset. Now, what can we do about it?
>>> One possibility is to apply the patchset now and implement a custom download
>>> source thing later on, or do that before releasing Core Update 139 (or which version
>>> the patchset will be to) after we agreed on something.
>>
>> I do not see this being merged for 139. But that is not important. We need to get it right first and then release it.
>>
>> As far as I know, nobody has tested this, yet.
>
> There are a number of people who have been running an earlier version
> which I shared on GitHub. There were a few early issues, but it seems
> to be OK now.
>
> https://forum.ipfire.org/viewtopic.php?f=27&t=21845
>
> This version wasn't integrated into IPFire, so (for example) it inserted
> itself into the INPUT IPTables chain rather than having it's chains
> created as part of the firewall start-up script.
>
>>
>> I have huge concerns about the automatic blacklist. @Peter: What is your opinion on this?
>>
>
> While I implemented it, I'm aware of its potential to cause problems,
> which is why it has to be separately enabled. It's not caused me any
> issues at the default settings (blocks at over 10 packets per hour until
> 1 hour has passed without seeing packets from the address), but I've not
> used it on a site with publicly announced services. If I was going to
> use it on a web site I would want to, at the very minimum, drop the
> block period drastically.
I suppose this is entirely unusable on an IPFire box in a data center that hosts things. Let’s say our rack in Hanover.
You will have hits from some broken IP stacks and people might just end up on this without doing anything wrong.
You can denial-of-service easily as well and I suppose without aggregation of data from many many systems, it does not make sense to instantly block addresses. And even then you probably would block an entire subnet.
So I am not sure how I should feel about it. I do not think that it adds anything because the packets would have been blocked anyways.
>
> On the other hand, it's good at responding quickly. Usually I see only
> 1-2% of blocks from the automatic list:
>
> Reason Count % First Last
> CIARMY 2416 45 Dec 6 00:00 Dec 6 23:59
> DSHIELD 1353 25 Dec 6 00:00 Dec 6 23:59
> INPUT 1294 24 Dec 6 00:00 Dec 6 23:59
> AUTOBLACKLIST 122 2 Dec 6 00:20 Dec 6 16:28
> BLOCKLIST_DE 89 2 Dec 6 00:20 Dec 6 23:46
>
> and sometimes none at all, but one one occasion it blocked over 8000
> packets. Again I'm aware this is for a home system, which is rather
> different than from a Web server.
>
>>> If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not
>>> seem to be problematic, as they are hosted on 3rd party servers, too).
>>
>> One of them will be, sooner or later. And one is enough I suppose.
>
> DShield (https://dshield.org/api/#threatfeeds) and firehol
> (http://iplists.firehol.org/) seem to host copies of most of the lists
> as well.
>>
>> I do not really want to overthink this - we didn’t do this with clamav for example either. But it probably is a decision to be made by the user what they want to enable and we should not enable anything by default. So no data will be leaked as long as the user does not consent.
>>
>> -Michael
>>
>>>
>>> Thanks, and best regards,
>>> Peter Müller
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 0/5] ipblacklist: IP Address Blacklists
2020-01-30 20:24 ` Tim FitzGeorge
@ 2020-01-30 21:26 ` Michael Tremer
0 siblings, 0 replies; 35+ messages in thread
From: Michael Tremer @ 2020-01-30 21:26 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 43944 bytes --]
Hi,
> On 30 Jan 2020, at 20:24, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>
> Hi,
>
> On 30/01/2020 12:54, Michael Tremer wrote:
>> Hi,
>>
>>> On 29 Jan 2020, at 20:40, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>
>>> Hi,
>>>
>>> On 28/01/2020 17:14, Michael Tremer wrote:
>>>> Hi,
>>>>
>>>> Apologies for my late reply again. I marked this email in my inbox and then it scrolled down because of loads of new stuff coming in. Please feel free to ping me if I do not respond in time.
>>>
>>> That's OK. It took me a little longer than expected to check the patches.
>>>>
>>>>> On 24 Jan 2020, at 19:40, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>>>
>>>>> I'm now nearly ready to submit the updated patches, but before then I'd
>>>>> like to check that you're happy with some of my design decisions.
>>>>
>>>> Cool.
>>>>
>>>>> _IPTables_Chains_
>>>>>
>>>>> When setting up the firewall (in iptables_init()), four new chains are
>>>>> created, IPBLACKLISTIN, IPBLACKLISTOUT, IPBLACKLISTREDIN and
>>>>> IPBLACKLISTREDOUT. The first two are inserted into the INPUT, OUTPUT
>>>>> and FORWARD chains, before the IPS chains. The main script is called to
>>>>> restore the previously saved blacklists, creating the rules in the
>>>>> latter two chains.
>>>>
>>>> For readability you can simply call them BLACKLIST*. The IP is a given.
>>>
>>> Good point. I'll make the change.
>>>
>>>>
>>>>> In iptables_red_up(), rules are created to forward traffic to/from the
>>>>> red interface from IPBLACKLISTIN/IPBLACKLISTOUT to
>>>>> IPBLACKLISTREDIN/IPBLACKLISTREDOUT.
>>>>> I did consider putting the blacklist rules into the
>>>>> IPBLACKLISTIN/IPBLACKLISTOUT, but this couldn't be done until red comes
>>>>> up and would require much more processing at this point - not a problem
>>>>> if your red interface comes up and stays up, but if you've got one that
>>>>> disconnects on no traffic you probably don't want the extra overhead.
>>>>> The way I've done it also has the side effects of 1) the main script
>>>>> doesn't have to worry about what the red interface is, and 2) if you're
>>>>> using multiple blacklists, the check for the red interface is only done
>>>>> once.
>>>>
>>>> You would only want to check the interface once and then jump into the chain. Can we not realise it that way?
>>>
>>> I'm not sure what you're suggesting.
>>> I insert BLACKLISTIN into INPUT & FORWARD and BLACKLISTOUT into OUTPUT &
>>> FORWARD. These receive all non-ICMP traffic. On red up traffic from
>>> the red interface is sent from BLACKLISTIN to BLACKLISTREDIN and traffic
>>> to the red interface is sent from BLACKLISTOUT to BLACKLISTREDOUT.
>>> The rules to check against the IPSets are put into the *RED* chain,
>>> checking against the source or destination addresses as appropriate.
>>
>> This sounds very complicated to me.
>>
>> Why can we not check this in BLACKLISTIN/OUT?
>
> Unfortunately at that point in the setup the red interface in
> /var/ipfire/red/iface is not always set, so it's not possible to check
> the interface when creating the rules to jump to BLACKLISTIN/OUT.
Yes, I agree with the delayed initialisation of that chain.
But where is the difference between the two chains here?
You would add rules like this:
iptables -A BLACKLISTIN -i $RED -m set --set some-blacklist src -j REJECT
Those can go into BLACKLISTIN. There is no point in jumping into BLACKLISTREDIN first and then having this rule:
iptables -A BLACKLISTREDIN -m set --set some-blacklist src -j REJECT
> It would be possible to delay creating the rules to check against the
> IPSets until red comes up and then check the interface in the individual
> blacklist rules, but that duplicates the interface check. The current
> arrangement is the best I've come up with.
The interface check is cheap. It is an integer. I suppose jumping to another chain is worse, but not by much.
>
>>
>>>>
>>>>> _Log_WUI_Pages_
>>>>>
>>>>> The two Log WUI pages are edited from existing pages. I've resisted the
>>>>> temptation to reformat them, which means that they can be diff'ed with
>>>>> the corresponding pages.
>>>>
>>>> Loads of the UI code is messy and deliberately not touched any more. I guess this is fine.
>>>
>>> Yes, it's not pretty, but it is pragmatic.
>>>>
>>>>> _Update_
>>>>>
>>>>> The update is fairly simple - it calculates whether the check period is
>>>>> passed, tries a download specifying the If-Modified-Since and parses the
>>>>> returned file. It then compares the new list with the existing list,
>>>>> adding new addresses/networks and deleting those that have been removed.
>>>>> There's a check to handle a list changing from a list of addresses to a
>>>>> list of networks, in which case it changes the type of the IPSet - this
>>>>> should be a vary rare event, but could happen for a composite list.
>>>>
>>>> Why do you need the diff? Has that been in the first patchset?
>>>
>>> It was, yes.
>>>
>>> Originally I did it for performance. It took over a minute to process
>>> the ALIENVAULT list (over 80 000 entries); just calling ipset with
>>> changes reduced this to a few seconds. I've since modified the code to
>>> pipe commands to a process running 'ipset restore', which has
>>> drastically speed up the processing, but it's still a little bit quicker
>>> to just do changes.
>>>
>>> The second reason is that it doesn't feel right to flush the old list,
>>> thus removing its protection while the new one is loaded, especially if
>>> the new list is the same as the old list (not all sources implement
>>> IP-Modified-Since). The actual period of reduced protection may be
>>> short, but I prefer to avoid it.
>>
>> Okay. I was hoping that the “ipset restore” update would flush and add everything again in one large atomic operation.
>
> It would be possible to send a 'flush' command to 'ipset restore', but
> the blacklists still have to be read and then the entries turned into
> the correct 'add' commands. There's also some additional processing to
> handle the hash type changing (between hash:ip and hash:net) and also
> for the size growing too large for the size of the IPSet. By the time
> this is all done, I don't think that there's any advantage to be gained
> in doing a flush followed by a bulk load of everything.
>>
>>>>
>>>> Just reloading the whole ipset should be easier.
>>>>
>>>>> _Enabling/Disabling_Blacklists_
>>>>>
>>>>> For changes to the existing firewall rules, the firewall is shut down
>>>>> and restarted from scratch. The blacklist script does this on the fly,
>>>>> since the possibilities are simpler than all the options available for
>>>>> the firewall and also the necessary code for doing updates means that
>>>>> adding this code is fairly simple. This includes turning logging on and
>>>>> off.
>>>>
>>>> What do you mean by that? /etc/init.d/firewall restart?
>>>>
>>>> That is something we cannot do because IPsec and some other services create temporary rules that would be lost.
>>>
>>> No. I'm actually trying to say the opposite (badly). The existing
>>> IPFire code only implements some firewall changes by shutting down and
>>> restarting the firewall, but all the IP Blacklist functions
>>> (enable/disable/update lists, enable/disable logging) can be done
>>> without this - the necessary rules can be added to/deleted from the
>>> BLACKLISTRED* chains without affecting any of the rest of the firewall
>>> system.
>>
>> Ah cool. That is how it needs to be.
>>
>> -Michael
>>
>>> Tim
>>
>> P.S. Is there actually a reason why you do not have a shell account on our servers and host your changed on our git server? I am not sure if I ever offered you one, but it would help me a lot to have a look at the code in a git repo :)
>
> You haven't previously offered me one (or I can't remember it anyway).
> I think it would make sense given the moderately large amount of code.
Okay, I will contact you privately about this. Sorry for not doing this earlier.
Best,
-Michael
>
> Tim
>>
>>>
>>>>
>>>>> _Logging_
>>>>>
>>>>> The main scripts logs to syslog as ipblacklist, but if you run it from
>>>>> the command line it'll also output to the terminal.
>>>>> There's also some debug code which can be turned on by setting a value
>>>>> in the settings file - it's turned on at the lowest level automatically
>>>>> if you run from the command line.
>>>>
>>>> Yes that is fine.
>>>>
>>>> Best,
>>>> -Michael
>>>>
>>>>> Tim
>>>>>
>>>>> On 06/01/2020 11:27, Michael Tremer wrote:
>>>>>> Hello all,
>>>>>>
>>>>>>> On 4 Jan 2020, at 19:02, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I've now updated the code - screenshots are attached.
>>>>>>>
>>>>>>> IP_Address_Blacklists.png
>>>>>>>
>>>>>>> This is the settings page. I don't think there's anything unexpected here.
>>>>>>
>>>>>> Yes, that looks a lot like the other IPFire pages :)
>>>>>>
>>>>>>> IP_Address_Blacklist_Logs.png
>>>>>>>
>>>>>>> The log page, replacing the status information that was previously at
>>>>>>> the top of the settings page.
>>>>>>
>>>>>> Looks great!
>>>>>>
>>>>>>> I've sorted the lists alphabetically, rather than by the number of hits
>>>>>>> since the worst threat is not necessary the largest number of blocked
>>>>>>> packets (which could implied by sorting it by number).
>>>>>>
>>>>>> Yes, that is indeed a very good idea.
>>>>>>
>>>>>> You could add a % sign in the percentage column, so it cannot be confused so easily with the other column.
>>>>>>
>>>>>>> Firewall_log_blacklist.png
>>>>>>>
>>>>>>> Details page accessed from the log. Almost identical to the other
>>>>>>> similar pages.
>>>>>>
>>>>>> *thumbs up*
>>>>>>
>>>>>>> Log_Summary.png
>>>>>>>
>>>>>>> This is an extract from the Log Summary page produced by logwatch.
>>>>>>>
>>>>>>> I still need to do some tidying, remove unused language strings etc.,
>>>>>>> and then update and test the patches, so it'll be some time before I'm
>>>>>>> ready to submit updated patches.
>>>>>>
>>>>>> Feel free to send some RFC to the list. It might be slightly noisy, but it is easier to review changes bit after bit.
>>>>>>
>>>>>> Great great work!
>>>>>>
>>>>>> -Michael
>>>>>>
>>>>>>>
>>>>>>> Tim
>>>>>>>
>>>>>>> On 28/12/2019 20:59, Tim FitzGeorge wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I'll code and test what we've agreed on. In the meantime I'm going to
>>>>>>>> reply on the sub-thread started by Tom Rymes as to what we call the list
>>>>>>>> categories, so that we can keep that discussion separate from further
>>>>>>>> functionality etc. discussion.
>>>>>>>>
>>>>>>>> Tim
>>>>>>>>
>>>>>>>> On 21/12/2019 18:34, Tim FitzGeorge wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I've got my systems checking for downloads at 15 minute intervals and
>>>>>>>>> using If-Modified-Since. This all seems to be working OK.
>>>>>>>>>
>>>>>>>>> Responses to comments below:
>>>>>>>>>
>>>>>>>>> On 18/12/2019 12:07, Michael Tremer wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>>> On 17 Dec 2019, at 18:21, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I think we've agreed on the following:
>>>>>>>>>>>
>>>>>>>>>>> - Checks to be made every 15 minutes, subject to per-list minimum rate.
>>>>>>>>>>> - Modify download to use If-Modified-Since rather than separate HEAD
>>>>>>>>>>> request.
>>>>>>>>>>> - Remove automatic blacklist.
>>>>>>>>>>
>>>>>>>>>> I thought someone wants to counter my argument. This is not a dictatorship :)
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> other comments are below.
>>>>>>>>>>>
>>>>>>>>>>> Tim
>>>>>>>>>>>
>>>>>>>>>>> On 16/12/2019 22:20, Michael Tremer wrote:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>>> On 16 Dec 2019, at 20:06, Tim FitzGeorge <lists(a)tfitzgeorge.me.uk> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've attached the current GUI screenshot.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for that.
>>>>>>>>>>>>
>>>>>>>>>>>> I have a couple of suggestions/concerns about it:
>>>>>>>>>>>>
>>>>>>>>>>>> a) I am not sure what the value is about the top box that is called “Status”. It is basically a summary of the iptables output, but will it help the user? A blacklist with more or fewer entries is not better or worse, blocking more packets isn’t better than blocking fewer. It is about the quality of the blocks.
>>>>>>>>>>>
>>>>>>>>>>> I agree that the size of the lists is not all that useful, but the
>>>>>>>>>>> information about the number of blocked packets can be an indication of
>>>>>>>>>>> compromise. It depends on the type of the list, but a list that targets
>>>>>>>>>>> C&C channels (e.g. the FEODO lists) should not show any input packets
>>>>>>>>>>> being blocked, unless one of the computers being protected is infected.
>>>>>>>>>>> Apart from the BOGON lists, none of the lists should show any output
>>>>>>>>>>> packets being blocked; again it's a potential indication of infection if
>>>>>>>>>>> otherwise.
>>>>>>>>>>
>>>>>>>>>> The Shodan list will definitely show some activity unless you are behind CG NAT.
>>>>>>>>>>
>>>>>>>>>> Should we not have this rather in the logging section?
>>>>>>>>>>
>>>>>>>>>> And these counters here will be reset when the firewall is being reloaded. Should this data therefore not be collected from the logs so that you can go back in time? That would be very important to identify any start of compromise.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> That makes sense. I'll remove the status from this page and create a
>>>>>>>>> log page for the information. If anyone actually wants the iptables
>>>>>>>>> information it's available in from Firewall > iptables anyway.
>>>>>>>>>
>>>>>>>>> Do you think it makes more sense to show packets blocked in/out or
>>>>>>>>> packets blocked per source interface? The latter would of (limited)
>>>>>>>>> help in tracking down potential compromise.
>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> b) I would prefer to move the settings box to the top. If you like you can show the size of the blacklists there and when they have been updated, too.
>>>>>>>>>>>
>>>>>>>>>>> I put the status at the top since it's usually what I want to see when I
>>>>>>>>>>> go to the page. Under normal circumstances I don't change the settings,
>>>>>>>>>>> so I'd rather not have to scroll past them to get to the information I'm
>>>>>>>>>>> actually interested in. The update times are just useful to show that
>>>>>>>>>>> the lists are actually being updated. The size is interesting to me,
>>>>>>>>>>> but as you say it probably won't help the average user, especially if
>>>>>>>>>>> you don't know which lists block individual addresses and which block
>>>>>>>>>>> nets (BOGON, DSHIELD and EMERGING_COMPROMISED).
>>>>>>>>>>
>>>>>>>>>> See my comment on the order above. Splitting the page might make sense.
>>>>>>>>>>
>>>>>>>>>> When you show size, does that mean lines in the blacklists? So it could be networks or single IP addresses? Does it not make sense to count IP addresses (i.e. 256 for a /24 network and so on) and put it in the table?
>>>>>>>>> It's lines in the blacklist.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I think I would only be interested in the size for two things:
>>>>>>>>>>
>>>>>>>>>> a) Is there any chance for overblocking? i.e. does the list have 2 billion entries which would be half the public IPv4 address space or so. I cannot really come up with a good example here.
>>>>>>>>>>
>>>>>>>>>> Or
>>>>>>>>>>
>>>>>>>>>> b) What is the performance impact/memory consumption of the list?
>>>>>>>>>
>>>>>>>>> It would be interesting to show the type (single address or net), number
>>>>>>>>> of entries and number of addresses; the first two affect performance,
>>>>>>>>> the second memory use and the latter the number of blocked addresses,
>>>>>>>>> but that's really only going to be of interest to a very small number of
>>>>>>>>> people, who can always do 'ipset list -t' to get the information. So
>>>>>>>>> I'll remove the size column.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> I originally had the status and settings in one list, but I decided that
>>>>>>>>>>> it was getting a bit messy and difficult to follow. I've attached a
>>>>>>>>>>> screenshot of a quickly hacked page. (Note that the status area only
>>>>>>>>>>> appears once the blacklists have been enabled.)
>>>>>>>>>>
>>>>>>>>>> You are right. This is getting quite tight there.
>>>>>>>>>>
>>>>>>>>>> You could write packets in/out into on column like 2k/1023, but see my concerns about this above.
>>>>>>>>>
>>>>>>>>> Actually I've come to the conclusion that the number of bytes blocked
>>>>>>>>> isn't all that important; it's sufficient to show the number of packets.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
>>>>>>>>>>>
>>>>>>>>>>> I agree that it's it's an over simplistic summary of the lists. I note
>>>>>>>>>>> that Tom Rymes has asked to keep it, but I think its only value is if it
>>>>>>>>>>> suggests to users that things are more complicated than they appear and
>>>>>>>>>>> therefore prompts them to read the Wiki.
>>>>>>>>>>
>>>>>>>>>> I will reply to that separately.
>>>>>>>>>
>>>>>>>>> Looking at that reply, I think it makes sense to include a category
>>>>>>>>> column in both the settings and the log summary page - the former to
>>>>>>>>> help selection of which lists to use, and the latter as a help when
>>>>>>>>> deciding whether the information could indicate compromise or not.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> d) I do not understand what the “update check rate” means. I suppose we should show the update interval of the lists in the table and just refresh them according to that.
>>>>>>>>>>>
>>>>>>>>>>> It's the minimum check period for updates. The system checks for
>>>>>>>>>>> updates at either the rate specific to the list or this rate, whichever
>>>>>>>>>>> is slower. This is a hangover from the early days, copied from the IPS
>>>>>>>>>>> updates. I suppose it could have a value for some people who may wish
>>>>>>>>>>> to lower the check rate, but I'm not sure that it has sufficient value
>>>>>>>>>>> to be worth keeping.
>>>>>>>>>>
>>>>>>>>>> No, I would not let the user decide when and how often to update the lists.
>>>>>>>>>
>>>>>>>>> OK. I'll remove it.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Generally there is this argument internally how often some databases should be updated. People with slow internet connections or volume based data plans will find this rather expensive to update those lists too often.
>>>>>>>>>>
>>>>>>>>>> If we ever want to go ahead with that in IPFire 2, there should be a global switch for a “low data mode” which then delays updating these things.
>>>>>>>>>
>>>>>>>>> That makes sense. Maybe also the ability to set when downloads take
>>>>>>>>> place - but as you say that's for the future.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> e) Do we want to keep the automatic blacklists now? I do not think it will actually improve the security of IPFire.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I'll remove them. I think they do have a use, in that they can detect
>>>>>>>>>>> someone doing a port scan and block them before they find an open port,
>>>>>>>>>>> but as you say there is also the possibility of a DOS.
>>>>>>>>>>
>>>>>>>>>> How would this block a DOS?
>>>>>>>>>
>>>>>>>>> Sorry, I didn't make myself clear - there's the possibility that someone
>>>>>>>>> could cause a DOS with the automatic list by faking the source IP
>>>>>>>>> address and sending a few packets on a random port.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -Michael
>>>>>>>>>
>>>>>>>>> Tim
>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'll update the code based on our discussions, and submit an updated set
>>>>>>>>>>>>> of patches - I imagine there will have to be at least one more iteration.
>>>>>>>>>>>>
>>>>>>>>>>>> Let’s wait until we have come to decisions :)
>>>>>>>>>>>>
>>>>>>>>>>>> -Michael
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Tim
>>>>>>>>>>>>>
>>>>>>>>>>>>> (No additional comments below)
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 13/12/2019 23:11, Michael Tremer wrote:
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Again my apologies for my late reply. Busy busy weeks.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 8 Dec 2019, at 20:50, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It's my turn to apologise for being slow to respond - I've had a busy
>>>>>>>>>>>>>>> week, but I should have plenty of time over the next couple of weeks.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> No worries. Turns out we all do :)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I've made most of the comments inline, however I think Michael had a
>>>>>>>>>>>>>>> question (which I can't find now) about what happens if someone enables
>>>>>>>>>>>>>>> all the lists. One thing which would perhaps make this less likely is
>>>>>>>>>>>>>>> that the WUI tags the available lists with whether they're safe or not,
>>>>>>>>>>>>>>> with a footnote that safe means that the list only blocks malicious
>>>>>>>>>>>>>>> traffic. This won't guarantee that a user won't still try to enable all
>>>>>>>>>>>>>>> the lists, but it should make them realise that they should think first.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We had a couple of features going slightly wrong or being “misunderstood” by some users. People still seem to panic when they see “local recursor”. To this day I do not know why.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We cannot make everything idiot-proof. And when some user if of that category, they probably should shutdown their IPFire box, educate themselves and then come back again. So I do not want to limit people, but make things as easy as possible.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If someone enables all the lists, good luck with passing packets :)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have considered replacing this tag with a risk high/medium/low and
>>>>>>>>>>>>>>> maybe adding a category (invalid/application/scanner/C&C or something
>>>>>>>>>>>>>>> like that), but that may provide too much information and dissuade them
>>>>>>>>>>>>>>> from actually following the links to checkout what the list actually does.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can we have a screenshot of the GUI right now? I didn’t run the code, yet.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We should document the lists like we do it with the rulesets of the IPS. People might ignore this, but that is on them.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Tim
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 05/12/2019 22:25, Michael Tremer wrote:
>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 4 Dec 2019, at 17:05, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hello Tim, hello Michael,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> please see my responses inline...
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> We could periodically update the blacklists on our main mirror (and
>>>>>>>>>>>>>>>>>>>> wait for the network to sync it), make sure it is signed and write
>>>>>>>>>>>>>>>>>>>> a small downloader that fetches, validates and installs them.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> @All: Thoughts on this?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I think there are a number of points here.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Firstly, from the point of a third party using IPFire, is this really
>>>>>>>>>>>>>>>>>>> solving the privacy disclosure problem? There's no way round disclosing
>>>>>>>>>>>>>>>>>>> your public IP address to someone you're downloading from; all this does
>>>>>>>>>>>>>>>>>>> is change who that information is being disclosed to. For the user
>>>>>>>>>>>>>>>>>>> there's no way of knowing whether the source is more or less protective
>>>>>>>>>>>>>>>>>>> of the user's privacy than the blacklist provider. Indeed it won't be
>>>>>>>>>>>>>>>>>>> possible to know who the lists are being downloaded from until the
>>>>>>>>>>>>>>>>>>> download starts.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> There is a way: Tor. But that is a totally different story.
>>>>>>>>>>>>>>>>> Well, I see a third option on this: Use the mirror infrastructure we already
>>>>>>>>>>>>>>>>> have. Every IPFire installation discloses its public IP address to one of these
>>>>>>>>>>>>>>>>> servers sooner or later, so we do not disclose additional data if the blacklists
>>>>>>>>>>>>>>>>> were fetched from these.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Needless to say, Tor (hidden services) would be better, but that is a different
>>>>>>>>>>>>>>>>> story indeed. :-)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The point is rather that a forget list can be sent instead of the “real” one.
>>>>>>>>>>>>>>>>> I did not get this. Forget? Forged?? ???
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yes, I meant to write forged, but auto-correct didn’t let me.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Secondly, latency; some of the lists are updated every 5 minutes. While
>>>>>>>>>>>>>>>>>>> I've limited the maximum check rate to hourly, will the updates
>>>>>>>>>>>>>>>>>>> propagate quickly enough. For reference on my main system the 24
>>>>>>>>>>>>>>>>>>> updates on the CIARMY list made 143 498 changes (additions or
>>>>>>>>>>>>>>>>>>> deletions). I've seem it do over 200 000.
>>>>>>>>>>>>>>>>> Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not
>>>>>>>>>>>>>>>>> document a recommended update interval anywhere, so we can only guess.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Personally, more static lists seem to be preferable for packet filtering. Highly
>>>>>>>>>>>>>>>>> dynamic ones such as CIArmy should be done via DNSBL queries or something similar
>>>>>>>>>>>>>>>>> - do we really want to have that list here?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It is not really an option to implement a DNSBL into a packet filter, but I get your point.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> One of the 'selling points' for an IP address blacklist is that it can
>>>>>>>>>>>>>>> respond quickly to new threats - or rather new attackers. While a new
>>>>>>>>>>>>>>> IDS/IPS rule needs time to analyse the threat, generate a rule and check
>>>>>>>>>>>>>>> it, it's easy to add an address to a list. So, I think the CIArmy list
>>>>>>>>>>>>>>> is potentially useful for protecting home systems etc. with budget
>>>>>>>>>>>>>>> hardware, but I would be very careful about using it for a protecting a
>>>>>>>>>>>>>>> general access website.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If they are very volatile, we should honour that and update them often, too.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It probably is more about false-positives being removed very quickly instead of adding threats very quickly. The average IPFire user is probably not under threats like these to need to react very quickly.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So I do not see much value in adding those lists and then updating once a day. Does it have to be every 5 min? No. I would suggest 15 which should be good enough for everyone.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Other lists should of course not be updated every 15 minutes when not needed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Running every 15 minutes would allow us to retry downloading lists that are on an hourly schedule if the download failed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> How did you come up with the hour? Will it be retried more often if the download was not successful?
>>>>>>>>>>>>>>>>> One hour is the most common interval indeed, but adding some random time might
>>>>>>>>>>>>>>>>> be useful in order to reduce load on the servers providing a blacklist.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yes, definitely. Otherwise we will shoot down our mirrors.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> When I implemented that section of code, specifying the minimum check
>>>>>>>>>>>>>>> period in hours seemed to provide a convenient way of allowing a check
>>>>>>>>>>>>>>> period covering a wide range, with an hour as the fastest and a week as
>>>>>>>>>>>>>>> the slowest. I didn't looked at the CIArmy list until much later. Most
>>>>>>>>>>>>>>> of the lists don't change nearly as much, but the CIArmy list is
>>>>>>>>>>>>>>> described as one that deliberately responds quickly.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> From my production system, for yesterday:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The following block lists were updated:
>>>>>>>>>>>>>>> BLOCKLIST_DE: 24 Time(s) - 9341 change(s)
>>>>>>>>>>>>>>> BOGON_FULL: 1 Time(s) - 10 change(s)
>>>>>>>>>>>>>>> CIARMY: 24 Time(s) - 159134 change(s)
>>>>>>>>>>>>>>> DSHIELD: 7 Time(s) - 18 change(s)
>>>>>>>>>>>>>>> EMERGING_FWRULE: 1 Time(s) - 50 change(s)
>>>>>>>>>>>>>>> FEODO_AGGRESIVE: 24 Time(s) - 13 change(s)
>>>>>>>>>>>>>>> SHODAN: 1 Time(s) - 0 change(s)
>>>>>>>>>>>>>>> SPAMHAUS_DROP: 1 Time(s) - 0 change(s)
>>>>>>>>>>>>>>> TOR_EXIT: 24 Time(s) - 162 change(s)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Very interesting statistics.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> and my test system:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The following block lists were updated:
>>>>>>>>>>>>>>> ALIENVAULT: 19 Time(s) - 5331 change(s)
>>>>>>>>>>>>>>> EMERGING_COMPROMISED: 1 Time(s) - 26 change(s)
>>>>>>>>>>>>>>> TALOS_MALICIOUS: 1 Time(s) - 36 change(s)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> That covers most of the lists. From the WUI, since 1 Dec:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Blacklist Entries pkts bytes Last updated
>>>>>>>>>>>>>>> in in
>>>>>>>>>>>>>>> AUTOBLACKLIST 0 731 51144 Sun Dec 8 18:40:02 2019
>>>>>>>>>>>>>>> BLOCKLIST_DE 28020 857 46735 Sun Dec 8 18:40:02 2019
>>>>>>>>>>>>>>> BOGON_FULL 214 5255 189K Sun Dec 8 16:50:04 2019
>>>>>>>>>>>>>>> CIARMY 15000 19774 976K Sun Dec 8 18:04:01 2019
>>>>>>>>>>>>>>> DSHIELD 20 7992 321K Sun Dec 8 16:45:13 2019
>>>>>>>>>>>>>>> EMERGING_FWRULE 1647 197 8383 Fri Dec 6 05:29:07 2019
>>>>>>>>>>>>>>> FEODO_AGGRESIVE 7169 0 0 Sun Dec 8 18:50:07 2019
>>>>>>>>>>>>>>> SHODAN 32 34 1530 Sun Dec 8 17:54:09 2019
>>>>>>>>>>>>>>> SPAMHAUS_DROP 823 0 0 Fri Dec 6 18:22:35 2019
>>>>>>>>>>>>>>> SPAMHAUS_EDROP 111 82 16433 Thu Dec 5 17:27:09 2019
>>>>>>>>>>>>>>> TOR_EXIT 1055 0 0 Sun Dec 8 18:31:02 2019
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This as well.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Those are more packets than I would have expected.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> (I've left out the pkts/bytes out fields which were all 0)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Note that where possible I do a HEAD request first and then only
>>>>>>>>>>>>>>> download the list if the modification time has changed since the last
>>>>>>>>>>>>>>> check. For dynamically generated lists this isn't possible.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You won’t need a HEAD request for it. You can include it in the GET request.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Have a look at location downloader where I use that. The server will respond with 304 and not send any payload.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://git.ipfire.org/?p=location/libloc.git;a=blob;f=src/python/location-downloader.in;h=1b5932d5822e03737d32b2a27816815b2f7e74dd;hb=HEAD#l151
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If the download isn't successful it just gives up and waits for the next
>>>>>>>>>>>>>>> attempt (apart from the usual retries in the library). I probably
>>>>>>>>>>>>>>> should to change that so that it only applies the per list minimum
>>>>>>>>>>>>>>> update period in this case (specified in the sources file) rather than
>>>>>>>>>>>>>>> the user specified value as well.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think it is not the worst if an update fails. It might just happen every once in a while.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So I would suggest to just re-run the script more often and when the mtime of the file is older than the threshold, a download is attempted. You can use that timestamp for the GET request.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I already use a time offset on the downloads - when it's started from
>>>>>>>>>>>>>>> boot, backup restore or WUI enable, it checks to see if it's installed
>>>>>>>>>>>>>>> in the fcrontab, and if not adds itself at a randomly generated offset
>>>>>>>>>>>>>>> in the hour.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> That should potentially go to red.up, or if we can settle on 15 minutes, I would consider that often enough to quickly update all lists after a reboot.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is
>>>>>>>>>>>>>>>>>>> the largest at a few MB), there are going to be a lot of them. How will
>>>>>>>>>>>>>>>>>>> this affect the willingness of people to mirror IPFire?
>>>>>>>>>>>>>>>>> I do not consider this being a problem as we do not generate that much traffic
>>>>>>>>>>>>>>>>> to them. Of course, that depends on the update interval again.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> That depends on your point of view.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I do not have a problem with this at all in my data center, but there are plenty of people with a volume-based LTE plan or simply a 128 kBit/s connection. It will take a longer time to download the lists for them. We need to mind that.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Talking about the preference of packet filter and IPS, I prefer to
>>>>>>>>>>>>>>>>>>>> use the latter as well as it gains more insight in what kind of malicious
>>>>>>>>>>>>>>>>>>>> traffic tried to pass a firewall machine. On systems with low resources,
>>>>>>>>>>>>>>>>>>>> this might be problematic and removing load from the IPS can be preferred
>>>>>>>>>>>>>>>>>>>> (make this configurable?!), on others, people might want to have both
>>>>>>>>>>>>>>>>>>>> results.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> You're only going to get one result for a packet whichever way round the
>>>>>>>>>>>>>>>>>>> IP blacklist and IPS are since whichever comes first will drop the
>>>>>>>>>>>>>>>>>>> packet before it reaches the second (well it would be possible to put
>>>>>>>>>>>>>>>>>>> the IP blacklist first and get it to log and mark packets which are then
>>>>>>>>>>>>>>>>>>> dropped after the IPS, but I think that's getting a little complicated.
>>>>>>>>>>>>>>>>>>> In addition I've seen the messages about the trouble marking was
>>>>>>>>>>>>>>>>>>> causing in the QoS).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I think it's a 50/50 choice as to which is more valuable first; it's
>>>>>>>>>>>>>>>>>>> probably going to differ from packet to packet. For me the possibility
>>>>>>>>>>>>>>>>>>> of reducing the IPS load means I prefer putting the IP blacklist first
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> It should be fairly easy to add the choice of where to put the IP
>>>>>>>>>>>>>>>>>>> blacklist. I think it'll have to be in the main firewall script, so
>>>>>>>>>>>>>>>>>>> it'll require a firewall restart, but it's not something that'll be
>>>>>>>>>>>>>>>>>>> changed often.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier.
>>>>>>>>>>>>>>>>> Agreed.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily.
>>>>>>>>>>>>>>>>> So dropping packets from blacklisted IP addresses/networks before IPS is it, then.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I do not even think it makes sense to swap the order in the outgoing direction.
>>>>>>>>>>>>>>>>> Me too.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this.
>>>>>>>>>>>>>>>>> Based on my personal experience, firewall hits observed on a single machine exposed
>>>>>>>>>>>>>>>>> to the internet are interesting, but the overall situation across multiple machines
>>>>>>>>>>>>>>>>> is even more interesting. Very quickly, you'll end on something like a centralised
>>>>>>>>>>>>>>>>> logging server and custom statistical analysis here...
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Probably a project for IPFire 4.0 :)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Or use one of the existing services, like the DSHIELD client
>>>>>>>>>>>>>>> https://dshield.org/howto.html (subject to privacy concerns again).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default.
>>>>>>>>>>>>>>>>>>>> I would love to see the bogon ruleset here, too (think about 8chan successor
>>>>>>>>>>>>>>>>>>>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause
>>>>>>>>>>>>>>>>>>>> interference if RED does not have a public IP address assigned.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I can add a field to the options file that controls whether a list is
>>>>>>>>>>>>>>>>>>> enabled by default.
>>>>>>>>>>>>>>>>> Thank you. :-)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat.
>>>>>>>>>>>>>>>>> This is my only objection against this patchset. Now, what can we do about it?
>>>>>>>>>>>>>>>>> One possibility is to apply the patchset now and implement a custom download
>>>>>>>>>>>>>>>>> source thing later on, or do that before releasing Core Update 139 (or which version
>>>>>>>>>>>>>>>>> the patchset will be to) after we agreed on something.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I do not see this being merged for 139. But that is not important. We need to get it right first and then release it.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> As far as I know, nobody has tested this, yet.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> There are a number of people who have been running an earlier version
>>>>>>>>>>>>>>> which I shared on GitHub. There were a few early issues, but it seems
>>>>>>>>>>>>>>> to be OK now.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> https://forum.ipfire.org/viewtopic.php?f=27&t=21845
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This version wasn't integrated into IPFire, so (for example) it inserted
>>>>>>>>>>>>>>> itself into the INPUT IPTables chain rather than having it's chains
>>>>>>>>>>>>>>> created as part of the firewall start-up script.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I have huge concerns about the automatic blacklist. @Peter: What is your opinion on this?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> While I implemented it, I'm aware of its potential to cause problems,
>>>>>>>>>>>>>>> which is why it has to be separately enabled. It's not caused me any
>>>>>>>>>>>>>>> issues at the default settings (blocks at over 10 packets per hour until
>>>>>>>>>>>>>>> 1 hour has passed without seeing packets from the address), but I've not
>>>>>>>>>>>>>>> used it on a site with publicly announced services. If I was going to
>>>>>>>>>>>>>>> use it on a web site I would want to, at the very minimum, drop the
>>>>>>>>>>>>>>> block period drastically.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I suppose this is entirely unusable on an IPFire box in a data center that hosts things. Let’s say our rack in Hanover.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You will have hits from some broken IP stacks and people might just end up on this without doing anything wrong.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You can denial-of-service easily as well and I suppose without aggregation of data from many many systems, it does not make sense to instantly block addresses. And even then you probably would block an entire subnet.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So I am not sure how I should feel about it. I do not think that it adds anything because the packets would have been blocked anyways.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On the other hand, it's good at responding quickly. Usually I see only
>>>>>>>>>>>>>>> 1-2% of blocks from the automatic list:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Reason Count % First Last
>>>>>>>>>>>>>>> CIARMY 2416 45 Dec 6 00:00 Dec 6 23:59
>>>>>>>>>>>>>>> DSHIELD 1353 25 Dec 6 00:00 Dec 6 23:59
>>>>>>>>>>>>>>> INPUT 1294 24 Dec 6 00:00 Dec 6 23:59
>>>>>>>>>>>>>>> AUTOBLACKLIST 122 2 Dec 6 00:20 Dec 6 16:28
>>>>>>>>>>>>>>> BLOCKLIST_DE 89 2 Dec 6 00:20 Dec 6 23:46
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> and sometimes none at all, but one one occasion it blocked over 8000
>>>>>>>>>>>>>>> packets. Again I'm aware this is for a home system, which is rather
>>>>>>>>>>>>>>> different than from a Web server.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not
>>>>>>>>>>>>>>>>> seem to be problematic, as they are hosted on 3rd party servers, too).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> One of them will be, sooner or later. And one is enough I suppose.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> DShield (https://dshield.org/api/#threatfeeds) and firehol
>>>>>>>>>>>>>>> (http://iplists.firehol.org/) seem to host copies of most of the lists
>>>>>>>>>>>>>>> as well.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I do not really want to overthink this - we didn’t do this with clamav for example either. But it probably is a decision to be made by the user what they want to enable and we should not enable anything by default. So no data will be leaked as long as the user does not consent.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks, and best regards,
>>>>>>>>>>>>>>>>> Peter Müller
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> <Screenshot_2019-12-16_IP_Address_Blacklists.png>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> <Screenshot_2019-12-17_IP_Address_Blacklists.png>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> <IP_Address_Blacklists.png><IP_Address_Blacklist_Logs.png><Firewall_log_blacklist.png><Log_Summary.png>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 0/5] ipblacklist: IP Address Blacklists
2020-01-30 12:54 ` Michael Tremer
@ 2020-01-30 20:24 ` Tim FitzGeorge
2020-01-30 21:26 ` Michael Tremer
0 siblings, 1 reply; 35+ messages in thread
From: Tim FitzGeorge @ 2020-01-30 20:24 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 42151 bytes --]
Hi,
On 30/01/2020 12:54, Michael Tremer wrote:
> Hi,
>
>> On 29 Jan 2020, at 20:40, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>
>> Hi,
>>
>> On 28/01/2020 17:14, Michael Tremer wrote:
>>> Hi,
>>>
>>> Apologies for my late reply again. I marked this email in my inbox and then it scrolled down because of loads of new stuff coming in. Please feel free to ping me if I do not respond in time.
>>
>> That's OK. It took me a little longer than expected to check the patches.
>>>
>>>> On 24 Jan 2020, at 19:40, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>>
>>>> I'm now nearly ready to submit the updated patches, but before then I'd
>>>> like to check that you're happy with some of my design decisions.
>>>
>>> Cool.
>>>
>>>> _IPTables_Chains_
>>>>
>>>> When setting up the firewall (in iptables_init()), four new chains are
>>>> created, IPBLACKLISTIN, IPBLACKLISTOUT, IPBLACKLISTREDIN and
>>>> IPBLACKLISTREDOUT. The first two are inserted into the INPUT, OUTPUT
>>>> and FORWARD chains, before the IPS chains. The main script is called to
>>>> restore the previously saved blacklists, creating the rules in the
>>>> latter two chains.
>>>
>>> For readability you can simply call them BLACKLIST*. The IP is a given.
>>
>> Good point. I'll make the change.
>>
>>>
>>>> In iptables_red_up(), rules are created to forward traffic to/from the
>>>> red interface from IPBLACKLISTIN/IPBLACKLISTOUT to
>>>> IPBLACKLISTREDIN/IPBLACKLISTREDOUT.
>>>> I did consider putting the blacklist rules into the
>>>> IPBLACKLISTIN/IPBLACKLISTOUT, but this couldn't be done until red comes
>>>> up and would require much more processing at this point - not a problem
>>>> if your red interface comes up and stays up, but if you've got one that
>>>> disconnects on no traffic you probably don't want the extra overhead.
>>>> The way I've done it also has the side effects of 1) the main script
>>>> doesn't have to worry about what the red interface is, and 2) if you're
>>>> using multiple blacklists, the check for the red interface is only done
>>>> once.
>>>
>>> You would only want to check the interface once and then jump into the chain. Can we not realise it that way?
>>
>> I'm not sure what you're suggesting.
>> I insert BLACKLISTIN into INPUT & FORWARD and BLACKLISTOUT into OUTPUT &
>> FORWARD. These receive all non-ICMP traffic. On red up traffic from
>> the red interface is sent from BLACKLISTIN to BLACKLISTREDIN and traffic
>> to the red interface is sent from BLACKLISTOUT to BLACKLISTREDOUT.
>> The rules to check against the IPSets are put into the *RED* chain,
>> checking against the source or destination addresses as appropriate.
>
> This sounds very complicated to me.
>
> Why can we not check this in BLACKLISTIN/OUT?
Unfortunately at that point in the setup the red interface in
/var/ipfire/red/iface is not always set, so it's not possible to check
the interface when creating the rules to jump to BLACKLISTIN/OUT.
It would be possible to delay creating the rules to check against the
IPSets until red comes up and then check the interface in the individual
blacklist rules, but that duplicates the interface check. The current
arrangement is the best I've come up with.
>
>>>
>>>> _Log_WUI_Pages_
>>>>
>>>> The two Log WUI pages are edited from existing pages. I've resisted the
>>>> temptation to reformat them, which means that they can be diff'ed with
>>>> the corresponding pages.
>>>
>>> Loads of the UI code is messy and deliberately not touched any more. I guess this is fine.
>>
>> Yes, it's not pretty, but it is pragmatic.
>>>
>>>> _Update_
>>>>
>>>> The update is fairly simple - it calculates whether the check period is
>>>> passed, tries a download specifying the If-Modified-Since and parses the
>>>> returned file. It then compares the new list with the existing list,
>>>> adding new addresses/networks and deleting those that have been removed.
>>>> There's a check to handle a list changing from a list of addresses to a
>>>> list of networks, in which case it changes the type of the IPSet - this
>>>> should be a vary rare event, but could happen for a composite list.
>>>
>>> Why do you need the diff? Has that been in the first patchset?
>>
>> It was, yes.
>>
>> Originally I did it for performance. It took over a minute to process
>> the ALIENVAULT list (over 80 000 entries); just calling ipset with
>> changes reduced this to a few seconds. I've since modified the code to
>> pipe commands to a process running 'ipset restore', which has
>> drastically speed up the processing, but it's still a little bit quicker
>> to just do changes.
>>
>> The second reason is that it doesn't feel right to flush the old list,
>> thus removing its protection while the new one is loaded, especially if
>> the new list is the same as the old list (not all sources implement
>> IP-Modified-Since). The actual period of reduced protection may be
>> short, but I prefer to avoid it.
>
> Okay. I was hoping that the “ipset restore” update would flush and add everything again in one large atomic operation.
It would be possible to send a 'flush' command to 'ipset restore', but
the blacklists still have to be read and then the entries turned into
the correct 'add' commands. There's also some additional processing to
handle the hash type changing (between hash:ip and hash:net) and also
for the size growing too large for the size of the IPSet. By the time
this is all done, I don't think that there's any advantage to be gained
in doing a flush followed by a bulk load of everything.
>
>>>
>>> Just reloading the whole ipset should be easier.
>>>
>>>> _Enabling/Disabling_Blacklists_
>>>>
>>>> For changes to the existing firewall rules, the firewall is shut down
>>>> and restarted from scratch. The blacklist script does this on the fly,
>>>> since the possibilities are simpler than all the options available for
>>>> the firewall and also the necessary code for doing updates means that
>>>> adding this code is fairly simple. This includes turning logging on and
>>>> off.
>>>
>>> What do you mean by that? /etc/init.d/firewall restart?
>>>
>>> That is something we cannot do because IPsec and some other services create temporary rules that would be lost.
>>
>> No. I'm actually trying to say the opposite (badly). The existing
>> IPFire code only implements some firewall changes by shutting down and
>> restarting the firewall, but all the IP Blacklist functions
>> (enable/disable/update lists, enable/disable logging) can be done
>> without this - the necessary rules can be added to/deleted from the
>> BLACKLISTRED* chains without affecting any of the rest of the firewall
>> system.
>
> Ah cool. That is how it needs to be.
>
> -Michael
>
>> Tim
>
> P.S. Is there actually a reason why you do not have a shell account on our servers and host your changed on our git server? I am not sure if I ever offered you one, but it would help me a lot to have a look at the code in a git repo :)
You haven't previously offered me one (or I can't remember it anyway).
I think it would make sense given the moderately large amount of code.
Tim
>
>>
>>>
>>>> _Logging_
>>>>
>>>> The main scripts logs to syslog as ipblacklist, but if you run it from
>>>> the command line it'll also output to the terminal.
>>>> There's also some debug code which can be turned on by setting a value
>>>> in the settings file - it's turned on at the lowest level automatically
>>>> if you run from the command line.
>>>
>>> Yes that is fine.
>>>
>>> Best,
>>> -Michael
>>>
>>>> Tim
>>>>
>>>> On 06/01/2020 11:27, Michael Tremer wrote:
>>>>> Hello all,
>>>>>
>>>>>> On 4 Jan 2020, at 19:02, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I've now updated the code - screenshots are attached.
>>>>>>
>>>>>> IP_Address_Blacklists.png
>>>>>>
>>>>>> This is the settings page. I don't think there's anything unexpected here.
>>>>>
>>>>> Yes, that looks a lot like the other IPFire pages :)
>>>>>
>>>>>> IP_Address_Blacklist_Logs.png
>>>>>>
>>>>>> The log page, replacing the status information that was previously at
>>>>>> the top of the settings page.
>>>>>
>>>>> Looks great!
>>>>>
>>>>>> I've sorted the lists alphabetically, rather than by the number of hits
>>>>>> since the worst threat is not necessary the largest number of blocked
>>>>>> packets (which could implied by sorting it by number).
>>>>>
>>>>> Yes, that is indeed a very good idea.
>>>>>
>>>>> You could add a % sign in the percentage column, so it cannot be confused so easily with the other column.
>>>>>
>>>>>> Firewall_log_blacklist.png
>>>>>>
>>>>>> Details page accessed from the log. Almost identical to the other
>>>>>> similar pages.
>>>>>
>>>>> *thumbs up*
>>>>>
>>>>>> Log_Summary.png
>>>>>>
>>>>>> This is an extract from the Log Summary page produced by logwatch.
>>>>>>
>>>>>> I still need to do some tidying, remove unused language strings etc.,
>>>>>> and then update and test the patches, so it'll be some time before I'm
>>>>>> ready to submit updated patches.
>>>>>
>>>>> Feel free to send some RFC to the list. It might be slightly noisy, but it is easier to review changes bit after bit.
>>>>>
>>>>> Great great work!
>>>>>
>>>>> -Michael
>>>>>
>>>>>>
>>>>>> Tim
>>>>>>
>>>>>> On 28/12/2019 20:59, Tim FitzGeorge wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I'll code and test what we've agreed on. In the meantime I'm going to
>>>>>>> reply on the sub-thread started by Tom Rymes as to what we call the list
>>>>>>> categories, so that we can keep that discussion separate from further
>>>>>>> functionality etc. discussion.
>>>>>>>
>>>>>>> Tim
>>>>>>>
>>>>>>> On 21/12/2019 18:34, Tim FitzGeorge wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I've got my systems checking for downloads at 15 minute intervals and
>>>>>>>> using If-Modified-Since. This all seems to be working OK.
>>>>>>>>
>>>>>>>> Responses to comments below:
>>>>>>>>
>>>>>>>> On 18/12/2019 12:07, Michael Tremer wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>>> On 17 Dec 2019, at 18:21, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I think we've agreed on the following:
>>>>>>>>>>
>>>>>>>>>> - Checks to be made every 15 minutes, subject to per-list minimum rate.
>>>>>>>>>> - Modify download to use If-Modified-Since rather than separate HEAD
>>>>>>>>>> request.
>>>>>>>>>> - Remove automatic blacklist.
>>>>>>>>>
>>>>>>>>> I thought someone wants to counter my argument. This is not a dictatorship :)
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> other comments are below.
>>>>>>>>>>
>>>>>>>>>> Tim
>>>>>>>>>>
>>>>>>>>>> On 16/12/2019 22:20, Michael Tremer wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>>> On 16 Dec 2019, at 20:06, Tim FitzGeorge <lists(a)tfitzgeorge.me.uk> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I've attached the current GUI screenshot.
>>>>>>>>>>>
>>>>>>>>>>> Thanks for that.
>>>>>>>>>>>
>>>>>>>>>>> I have a couple of suggestions/concerns about it:
>>>>>>>>>>>
>>>>>>>>>>> a) I am not sure what the value is about the top box that is called “Status”. It is basically a summary of the iptables output, but will it help the user? A blacklist with more or fewer entries is not better or worse, blocking more packets isn’t better than blocking fewer. It is about the quality of the blocks.
>>>>>>>>>>
>>>>>>>>>> I agree that the size of the lists is not all that useful, but the
>>>>>>>>>> information about the number of blocked packets can be an indication of
>>>>>>>>>> compromise. It depends on the type of the list, but a list that targets
>>>>>>>>>> C&C channels (e.g. the FEODO lists) should not show any input packets
>>>>>>>>>> being blocked, unless one of the computers being protected is infected.
>>>>>>>>>> Apart from the BOGON lists, none of the lists should show any output
>>>>>>>>>> packets being blocked; again it's a potential indication of infection if
>>>>>>>>>> otherwise.
>>>>>>>>>
>>>>>>>>> The Shodan list will definitely show some activity unless you are behind CG NAT.
>>>>>>>>>
>>>>>>>>> Should we not have this rather in the logging section?
>>>>>>>>>
>>>>>>>>> And these counters here will be reset when the firewall is being reloaded. Should this data therefore not be collected from the logs so that you can go back in time? That would be very important to identify any start of compromise.
>>>>>>>>>
>>>>>>>>
>>>>>>>> That makes sense. I'll remove the status from this page and create a
>>>>>>>> log page for the information. If anyone actually wants the iptables
>>>>>>>> information it's available in from Firewall > iptables anyway.
>>>>>>>>
>>>>>>>> Do you think it makes more sense to show packets blocked in/out or
>>>>>>>> packets blocked per source interface? The latter would of (limited)
>>>>>>>> help in tracking down potential compromise.
>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> b) I would prefer to move the settings box to the top. If you like you can show the size of the blacklists there and when they have been updated, too.
>>>>>>>>>>
>>>>>>>>>> I put the status at the top since it's usually what I want to see when I
>>>>>>>>>> go to the page. Under normal circumstances I don't change the settings,
>>>>>>>>>> so I'd rather not have to scroll past them to get to the information I'm
>>>>>>>>>> actually interested in. The update times are just useful to show that
>>>>>>>>>> the lists are actually being updated. The size is interesting to me,
>>>>>>>>>> but as you say it probably won't help the average user, especially if
>>>>>>>>>> you don't know which lists block individual addresses and which block
>>>>>>>>>> nets (BOGON, DSHIELD and EMERGING_COMPROMISED).
>>>>>>>>>
>>>>>>>>> See my comment on the order above. Splitting the page might make sense.
>>>>>>>>>
>>>>>>>>> When you show size, does that mean lines in the blacklists? So it could be networks or single IP addresses? Does it not make sense to count IP addresses (i.e. 256 for a /24 network and so on) and put it in the table?
>>>>>>>> It's lines in the blacklist.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I think I would only be interested in the size for two things:
>>>>>>>>>
>>>>>>>>> a) Is there any chance for overblocking? i.e. does the list have 2 billion entries which would be half the public IPv4 address space or so. I cannot really come up with a good example here.
>>>>>>>>>
>>>>>>>>> Or
>>>>>>>>>
>>>>>>>>> b) What is the performance impact/memory consumption of the list?
>>>>>>>>
>>>>>>>> It would be interesting to show the type (single address or net), number
>>>>>>>> of entries and number of addresses; the first two affect performance,
>>>>>>>> the second memory use and the latter the number of blocked addresses,
>>>>>>>> but that's really only going to be of interest to a very small number of
>>>>>>>> people, who can always do 'ipset list -t' to get the information. So
>>>>>>>> I'll remove the size column.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>> I originally had the status and settings in one list, but I decided that
>>>>>>>>>> it was getting a bit messy and difficult to follow. I've attached a
>>>>>>>>>> screenshot of a quickly hacked page. (Note that the status area only
>>>>>>>>>> appears once the blacklists have been enabled.)
>>>>>>>>>
>>>>>>>>> You are right. This is getting quite tight there.
>>>>>>>>>
>>>>>>>>> You could write packets in/out into on column like 2k/1023, but see my concerns about this above.
>>>>>>>>
>>>>>>>> Actually I've come to the conclusion that the number of bytes blocked
>>>>>>>> isn't all that important; it's sufficient to show the number of packets.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
>>>>>>>>>>
>>>>>>>>>> I agree that it's it's an over simplistic summary of the lists. I note
>>>>>>>>>> that Tom Rymes has asked to keep it, but I think its only value is if it
>>>>>>>>>> suggests to users that things are more complicated than they appear and
>>>>>>>>>> therefore prompts them to read the Wiki.
>>>>>>>>>
>>>>>>>>> I will reply to that separately.
>>>>>>>>
>>>>>>>> Looking at that reply, I think it makes sense to include a category
>>>>>>>> column in both the settings and the log summary page - the former to
>>>>>>>> help selection of which lists to use, and the latter as a help when
>>>>>>>> deciding whether the information could indicate compromise or not.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> d) I do not understand what the “update check rate” means. I suppose we should show the update interval of the lists in the table and just refresh them according to that.
>>>>>>>>>>
>>>>>>>>>> It's the minimum check period for updates. The system checks for
>>>>>>>>>> updates at either the rate specific to the list or this rate, whichever
>>>>>>>>>> is slower. This is a hangover from the early days, copied from the IPS
>>>>>>>>>> updates. I suppose it could have a value for some people who may wish
>>>>>>>>>> to lower the check rate, but I'm not sure that it has sufficient value
>>>>>>>>>> to be worth keeping.
>>>>>>>>>
>>>>>>>>> No, I would not let the user decide when and how often to update the lists.
>>>>>>>>
>>>>>>>> OK. I'll remove it.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Generally there is this argument internally how often some databases should be updated. People with slow internet connections or volume based data plans will find this rather expensive to update those lists too often.
>>>>>>>>>
>>>>>>>>> If we ever want to go ahead with that in IPFire 2, there should be a global switch for a “low data mode” which then delays updating these things.
>>>>>>>>
>>>>>>>> That makes sense. Maybe also the ability to set when downloads take
>>>>>>>> place - but as you say that's for the future.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> e) Do we want to keep the automatic blacklists now? I do not think it will actually improve the security of IPFire.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I'll remove them. I think they do have a use, in that they can detect
>>>>>>>>>> someone doing a port scan and block them before they find an open port,
>>>>>>>>>> but as you say there is also the possibility of a DOS.
>>>>>>>>>
>>>>>>>>> How would this block a DOS?
>>>>>>>>
>>>>>>>> Sorry, I didn't make myself clear - there's the possibility that someone
>>>>>>>> could cause a DOS with the automatic list by faking the source IP
>>>>>>>> address and sending a few packets on a random port.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> -Michael
>>>>>>>>
>>>>>>>> Tim
>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I'll update the code based on our discussions, and submit an updated set
>>>>>>>>>>>> of patches - I imagine there will have to be at least one more iteration.
>>>>>>>>>>>
>>>>>>>>>>> Let’s wait until we have come to decisions :)
>>>>>>>>>>>
>>>>>>>>>>> -Michael
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Tim
>>>>>>>>>>>>
>>>>>>>>>>>> (No additional comments below)
>>>>>>>>>>>>
>>>>>>>>>>>> On 13/12/2019 23:11, Michael Tremer wrote:
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Again my apologies for my late reply. Busy busy weeks.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 8 Dec 2019, at 20:50, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It's my turn to apologise for being slow to respond - I've had a busy
>>>>>>>>>>>>>> week, but I should have plenty of time over the next couple of weeks.
>>>>>>>>>>>>>
>>>>>>>>>>>>> No worries. Turns out we all do :)
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've made most of the comments inline, however I think Michael had a
>>>>>>>>>>>>>> question (which I can't find now) about what happens if someone enables
>>>>>>>>>>>>>> all the lists. One thing which would perhaps make this less likely is
>>>>>>>>>>>>>> that the WUI tags the available lists with whether they're safe or not,
>>>>>>>>>>>>>> with a footnote that safe means that the list only blocks malicious
>>>>>>>>>>>>>> traffic. This won't guarantee that a user won't still try to enable all
>>>>>>>>>>>>>> the lists, but it should make them realise that they should think first.
>>>>>>>>>>>>>
>>>>>>>>>>>>> We had a couple of features going slightly wrong or being “misunderstood” by some users. People still seem to panic when they see “local recursor”. To this day I do not know why.
>>>>>>>>>>>>>
>>>>>>>>>>>>> We cannot make everything idiot-proof. And when some user if of that category, they probably should shutdown their IPFire box, educate themselves and then come back again. So I do not want to limit people, but make things as easy as possible.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If someone enables all the lists, good luck with passing packets :)
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have considered replacing this tag with a risk high/medium/low and
>>>>>>>>>>>>>> maybe adding a category (invalid/application/scanner/C&C or something
>>>>>>>>>>>>>> like that), but that may provide too much information and dissuade them
>>>>>>>>>>>>>> from actually following the links to checkout what the list actually does.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Can we have a screenshot of the GUI right now? I didn’t run the code, yet.
>>>>>>>>>>>>>
>>>>>>>>>>>>> We should document the lists like we do it with the rulesets of the IPS. People might ignore this, but that is on them.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Tim
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 05/12/2019 22:25, Michael Tremer wrote:
>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 4 Dec 2019, at 17:05, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hello Tim, hello Michael,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> please see my responses inline...
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> We could periodically update the blacklists on our main mirror (and
>>>>>>>>>>>>>>>>>>> wait for the network to sync it), make sure it is signed and write
>>>>>>>>>>>>>>>>>>> a small downloader that fetches, validates and installs them.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> @All: Thoughts on this?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I think there are a number of points here.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Firstly, from the point of a third party using IPFire, is this really
>>>>>>>>>>>>>>>>>> solving the privacy disclosure problem? There's no way round disclosing
>>>>>>>>>>>>>>>>>> your public IP address to someone you're downloading from; all this does
>>>>>>>>>>>>>>>>>> is change who that information is being disclosed to. For the user
>>>>>>>>>>>>>>>>>> there's no way of knowing whether the source is more or less protective
>>>>>>>>>>>>>>>>>> of the user's privacy than the blacklist provider. Indeed it won't be
>>>>>>>>>>>>>>>>>> possible to know who the lists are being downloaded from until the
>>>>>>>>>>>>>>>>>> download starts.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> There is a way: Tor. But that is a totally different story.
>>>>>>>>>>>>>>>> Well, I see a third option on this: Use the mirror infrastructure we already
>>>>>>>>>>>>>>>> have. Every IPFire installation discloses its public IP address to one of these
>>>>>>>>>>>>>>>> servers sooner or later, so we do not disclose additional data if the blacklists
>>>>>>>>>>>>>>>> were fetched from these.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Needless to say, Tor (hidden services) would be better, but that is a different
>>>>>>>>>>>>>>>> story indeed. :-)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The point is rather that a forget list can be sent instead of the “real” one.
>>>>>>>>>>>>>>>> I did not get this. Forget? Forged?? ???
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yes, I meant to write forged, but auto-correct didn’t let me.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Secondly, latency; some of the lists are updated every 5 minutes. While
>>>>>>>>>>>>>>>>>> I've limited the maximum check rate to hourly, will the updates
>>>>>>>>>>>>>>>>>> propagate quickly enough. For reference on my main system the 24
>>>>>>>>>>>>>>>>>> updates on the CIARMY list made 143 498 changes (additions or
>>>>>>>>>>>>>>>>>> deletions). I've seem it do over 200 000.
>>>>>>>>>>>>>>>> Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not
>>>>>>>>>>>>>>>> document a recommended update interval anywhere, so we can only guess.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Personally, more static lists seem to be preferable for packet filtering. Highly
>>>>>>>>>>>>>>>> dynamic ones such as CIArmy should be done via DNSBL queries or something similar
>>>>>>>>>>>>>>>> - do we really want to have that list here?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It is not really an option to implement a DNSBL into a packet filter, but I get your point.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> One of the 'selling points' for an IP address blacklist is that it can
>>>>>>>>>>>>>> respond quickly to new threats - or rather new attackers. While a new
>>>>>>>>>>>>>> IDS/IPS rule needs time to analyse the threat, generate a rule and check
>>>>>>>>>>>>>> it, it's easy to add an address to a list. So, I think the CIArmy list
>>>>>>>>>>>>>> is potentially useful for protecting home systems etc. with budget
>>>>>>>>>>>>>> hardware, but I would be very careful about using it for a protecting a
>>>>>>>>>>>>>> general access website.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If they are very volatile, we should honour that and update them often, too.
>>>>>>>>>>>>>
>>>>>>>>>>>>> It probably is more about false-positives being removed very quickly instead of adding threats very quickly. The average IPFire user is probably not under threats like these to need to react very quickly.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So I do not see much value in adding those lists and then updating once a day. Does it have to be every 5 min? No. I would suggest 15 which should be good enough for everyone.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Other lists should of course not be updated every 15 minutes when not needed.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Running every 15 minutes would allow us to retry downloading lists that are on an hourly schedule if the download failed.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> How did you come up with the hour? Will it be retried more often if the download was not successful?
>>>>>>>>>>>>>>>> One hour is the most common interval indeed, but adding some random time might
>>>>>>>>>>>>>>>> be useful in order to reduce load on the servers providing a blacklist.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yes, definitely. Otherwise we will shoot down our mirrors.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> When I implemented that section of code, specifying the minimum check
>>>>>>>>>>>>>> period in hours seemed to provide a convenient way of allowing a check
>>>>>>>>>>>>>> period covering a wide range, with an hour as the fastest and a week as
>>>>>>>>>>>>>> the slowest. I didn't looked at the CIArmy list until much later. Most
>>>>>>>>>>>>>> of the lists don't change nearly as much, but the CIArmy list is
>>>>>>>>>>>>>> described as one that deliberately responds quickly.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> From my production system, for yesterday:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The following block lists were updated:
>>>>>>>>>>>>>> BLOCKLIST_DE: 24 Time(s) - 9341 change(s)
>>>>>>>>>>>>>> BOGON_FULL: 1 Time(s) - 10 change(s)
>>>>>>>>>>>>>> CIARMY: 24 Time(s) - 159134 change(s)
>>>>>>>>>>>>>> DSHIELD: 7 Time(s) - 18 change(s)
>>>>>>>>>>>>>> EMERGING_FWRULE: 1 Time(s) - 50 change(s)
>>>>>>>>>>>>>> FEODO_AGGRESIVE: 24 Time(s) - 13 change(s)
>>>>>>>>>>>>>> SHODAN: 1 Time(s) - 0 change(s)
>>>>>>>>>>>>>> SPAMHAUS_DROP: 1 Time(s) - 0 change(s)
>>>>>>>>>>>>>> TOR_EXIT: 24 Time(s) - 162 change(s)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Very interesting statistics.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> and my test system:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The following block lists were updated:
>>>>>>>>>>>>>> ALIENVAULT: 19 Time(s) - 5331 change(s)
>>>>>>>>>>>>>> EMERGING_COMPROMISED: 1 Time(s) - 26 change(s)
>>>>>>>>>>>>>> TALOS_MALICIOUS: 1 Time(s) - 36 change(s)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> That covers most of the lists. From the WUI, since 1 Dec:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Blacklist Entries pkts bytes Last updated
>>>>>>>>>>>>>> in in
>>>>>>>>>>>>>> AUTOBLACKLIST 0 731 51144 Sun Dec 8 18:40:02 2019
>>>>>>>>>>>>>> BLOCKLIST_DE 28020 857 46735 Sun Dec 8 18:40:02 2019
>>>>>>>>>>>>>> BOGON_FULL 214 5255 189K Sun Dec 8 16:50:04 2019
>>>>>>>>>>>>>> CIARMY 15000 19774 976K Sun Dec 8 18:04:01 2019
>>>>>>>>>>>>>> DSHIELD 20 7992 321K Sun Dec 8 16:45:13 2019
>>>>>>>>>>>>>> EMERGING_FWRULE 1647 197 8383 Fri Dec 6 05:29:07 2019
>>>>>>>>>>>>>> FEODO_AGGRESIVE 7169 0 0 Sun Dec 8 18:50:07 2019
>>>>>>>>>>>>>> SHODAN 32 34 1530 Sun Dec 8 17:54:09 2019
>>>>>>>>>>>>>> SPAMHAUS_DROP 823 0 0 Fri Dec 6 18:22:35 2019
>>>>>>>>>>>>>> SPAMHAUS_EDROP 111 82 16433 Thu Dec 5 17:27:09 2019
>>>>>>>>>>>>>> TOR_EXIT 1055 0 0 Sun Dec 8 18:31:02 2019
>>>>>>>>>>>>>
>>>>>>>>>>>>> This as well.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Those are more packets than I would have expected.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (I've left out the pkts/bytes out fields which were all 0)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Note that where possible I do a HEAD request first and then only
>>>>>>>>>>>>>> download the list if the modification time has changed since the last
>>>>>>>>>>>>>> check. For dynamically generated lists this isn't possible.
>>>>>>>>>>>>>
>>>>>>>>>>>>> You won’t need a HEAD request for it. You can include it in the GET request.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Have a look at location downloader where I use that. The server will respond with 304 and not send any payload.
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://git.ipfire.org/?p=location/libloc.git;a=blob;f=src/python/location-downloader.in;h=1b5932d5822e03737d32b2a27816815b2f7e74dd;hb=HEAD#l151
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If the download isn't successful it just gives up and waits for the next
>>>>>>>>>>>>>> attempt (apart from the usual retries in the library). I probably
>>>>>>>>>>>>>> should to change that so that it only applies the per list minimum
>>>>>>>>>>>>>> update period in this case (specified in the sources file) rather than
>>>>>>>>>>>>>> the user specified value as well.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think it is not the worst if an update fails. It might just happen every once in a while.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So I would suggest to just re-run the script more often and when the mtime of the file is older than the threshold, a download is attempted. You can use that timestamp for the GET request.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I already use a time offset on the downloads - when it's started from
>>>>>>>>>>>>>> boot, backup restore or WUI enable, it checks to see if it's installed
>>>>>>>>>>>>>> in the fcrontab, and if not adds itself at a randomly generated offset
>>>>>>>>>>>>>> in the hour.
>>>>>>>>>>>>>
>>>>>>>>>>>>> That should potentially go to red.up, or if we can settle on 15 minutes, I would consider that often enough to quickly update all lists after a reboot.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is
>>>>>>>>>>>>>>>>>> the largest at a few MB), there are going to be a lot of them. How will
>>>>>>>>>>>>>>>>>> this affect the willingness of people to mirror IPFire?
>>>>>>>>>>>>>>>> I do not consider this being a problem as we do not generate that much traffic
>>>>>>>>>>>>>>>> to them. Of course, that depends on the update interval again.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> That depends on your point of view.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I do not have a problem with this at all in my data center, but there are plenty of people with a volume-based LTE plan or simply a 128 kBit/s connection. It will take a longer time to download the lists for them. We need to mind that.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Talking about the preference of packet filter and IPS, I prefer to
>>>>>>>>>>>>>>>>>>> use the latter as well as it gains more insight in what kind of malicious
>>>>>>>>>>>>>>>>>>> traffic tried to pass a firewall machine. On systems with low resources,
>>>>>>>>>>>>>>>>>>> this might be problematic and removing load from the IPS can be preferred
>>>>>>>>>>>>>>>>>>> (make this configurable?!), on others, people might want to have both
>>>>>>>>>>>>>>>>>>> results.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> You're only going to get one result for a packet whichever way round the
>>>>>>>>>>>>>>>>>> IP blacklist and IPS are since whichever comes first will drop the
>>>>>>>>>>>>>>>>>> packet before it reaches the second (well it would be possible to put
>>>>>>>>>>>>>>>>>> the IP blacklist first and get it to log and mark packets which are then
>>>>>>>>>>>>>>>>>> dropped after the IPS, but I think that's getting a little complicated.
>>>>>>>>>>>>>>>>>> In addition I've seen the messages about the trouble marking was
>>>>>>>>>>>>>>>>>> causing in the QoS).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I think it's a 50/50 choice as to which is more valuable first; it's
>>>>>>>>>>>>>>>>>> probably going to differ from packet to packet. For me the possibility
>>>>>>>>>>>>>>>>>> of reducing the IPS load means I prefer putting the IP blacklist first
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> It should be fairly easy to add the choice of where to put the IP
>>>>>>>>>>>>>>>>>> blacklist. I think it'll have to be in the main firewall script, so
>>>>>>>>>>>>>>>>>> it'll require a firewall restart, but it's not something that'll be
>>>>>>>>>>>>>>>>>> changed often.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier.
>>>>>>>>>>>>>>>> Agreed.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily.
>>>>>>>>>>>>>>>> So dropping packets from blacklisted IP addresses/networks before IPS is it, then.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I do not even think it makes sense to swap the order in the outgoing direction.
>>>>>>>>>>>>>>>> Me too.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this.
>>>>>>>>>>>>>>>> Based on my personal experience, firewall hits observed on a single machine exposed
>>>>>>>>>>>>>>>> to the internet are interesting, but the overall situation across multiple machines
>>>>>>>>>>>>>>>> is even more interesting. Very quickly, you'll end on something like a centralised
>>>>>>>>>>>>>>>> logging server and custom statistical analysis here...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Probably a project for IPFire 4.0 :)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Or use one of the existing services, like the DSHIELD client
>>>>>>>>>>>>>> https://dshield.org/howto.html (subject to privacy concerns again).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default.
>>>>>>>>>>>>>>>>>>> I would love to see the bogon ruleset here, too (think about 8chan successor
>>>>>>>>>>>>>>>>>>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause
>>>>>>>>>>>>>>>>>>> interference if RED does not have a public IP address assigned.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I can add a field to the options file that controls whether a list is
>>>>>>>>>>>>>>>>>> enabled by default.
>>>>>>>>>>>>>>>> Thank you. :-)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat.
>>>>>>>>>>>>>>>> This is my only objection against this patchset. Now, what can we do about it?
>>>>>>>>>>>>>>>> One possibility is to apply the patchset now and implement a custom download
>>>>>>>>>>>>>>>> source thing later on, or do that before releasing Core Update 139 (or which version
>>>>>>>>>>>>>>>> the patchset will be to) after we agreed on something.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I do not see this being merged for 139. But that is not important. We need to get it right first and then release it.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> As far as I know, nobody has tested this, yet.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> There are a number of people who have been running an earlier version
>>>>>>>>>>>>>> which I shared on GitHub. There were a few early issues, but it seems
>>>>>>>>>>>>>> to be OK now.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://forum.ipfire.org/viewtopic.php?f=27&t=21845
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This version wasn't integrated into IPFire, so (for example) it inserted
>>>>>>>>>>>>>> itself into the INPUT IPTables chain rather than having it's chains
>>>>>>>>>>>>>> created as part of the firewall start-up script.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have huge concerns about the automatic blacklist. @Peter: What is your opinion on this?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> While I implemented it, I'm aware of its potential to cause problems,
>>>>>>>>>>>>>> which is why it has to be separately enabled. It's not caused me any
>>>>>>>>>>>>>> issues at the default settings (blocks at over 10 packets per hour until
>>>>>>>>>>>>>> 1 hour has passed without seeing packets from the address), but I've not
>>>>>>>>>>>>>> used it on a site with publicly announced services. If I was going to
>>>>>>>>>>>>>> use it on a web site I would want to, at the very minimum, drop the
>>>>>>>>>>>>>> block period drastically.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I suppose this is entirely unusable on an IPFire box in a data center that hosts things. Let’s say our rack in Hanover.
>>>>>>>>>>>>>
>>>>>>>>>>>>> You will have hits from some broken IP stacks and people might just end up on this without doing anything wrong.
>>>>>>>>>>>>>
>>>>>>>>>>>>> You can denial-of-service easily as well and I suppose without aggregation of data from many many systems, it does not make sense to instantly block addresses. And even then you probably would block an entire subnet.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So I am not sure how I should feel about it. I do not think that it adds anything because the packets would have been blocked anyways.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On the other hand, it's good at responding quickly. Usually I see only
>>>>>>>>>>>>>> 1-2% of blocks from the automatic list:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Reason Count % First Last
>>>>>>>>>>>>>> CIARMY 2416 45 Dec 6 00:00 Dec 6 23:59
>>>>>>>>>>>>>> DSHIELD 1353 25 Dec 6 00:00 Dec 6 23:59
>>>>>>>>>>>>>> INPUT 1294 24 Dec 6 00:00 Dec 6 23:59
>>>>>>>>>>>>>> AUTOBLACKLIST 122 2 Dec 6 00:20 Dec 6 16:28
>>>>>>>>>>>>>> BLOCKLIST_DE 89 2 Dec 6 00:20 Dec 6 23:46
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> and sometimes none at all, but one one occasion it blocked over 8000
>>>>>>>>>>>>>> packets. Again I'm aware this is for a home system, which is rather
>>>>>>>>>>>>>> different than from a Web server.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not
>>>>>>>>>>>>>>>> seem to be problematic, as they are hosted on 3rd party servers, too).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> One of them will be, sooner or later. And one is enough I suppose.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> DShield (https://dshield.org/api/#threatfeeds) and firehol
>>>>>>>>>>>>>> (http://iplists.firehol.org/) seem to host copies of most of the lists
>>>>>>>>>>>>>> as well.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I do not really want to overthink this - we didn’t do this with clamav for example either. But it probably is a decision to be made by the user what they want to enable and we should not enable anything by default. So no data will be leaked as long as the user does not consent.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks, and best regards,
>>>>>>>>>>>>>>>> Peter Müller
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> <Screenshot_2019-12-16_IP_Address_Blacklists.png>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> <Screenshot_2019-12-17_IP_Address_Blacklists.png>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> <IP_Address_Blacklists.png><IP_Address_Blacklist_Logs.png><Firewall_log_blacklist.png><Log_Summary.png>
>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 0/5] ipblacklist: IP Address Blacklists
2020-01-29 20:40 ` Tim FitzGeorge
@ 2020-01-30 12:54 ` Michael Tremer
2020-01-30 20:24 ` Tim FitzGeorge
0 siblings, 1 reply; 35+ messages in thread
From: Michael Tremer @ 2020-01-30 12:54 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 40469 bytes --]
Hi,
> On 29 Jan 2020, at 20:40, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>
> Hi,
>
> On 28/01/2020 17:14, Michael Tremer wrote:
>> Hi,
>>
>> Apologies for my late reply again. I marked this email in my inbox and then it scrolled down because of loads of new stuff coming in. Please feel free to ping me if I do not respond in time.
>
> That's OK. It took me a little longer than expected to check the patches.
>>
>>> On 24 Jan 2020, at 19:40, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>
>>> I'm now nearly ready to submit the updated patches, but before then I'd
>>> like to check that you're happy with some of my design decisions.
>>
>> Cool.
>>
>>> _IPTables_Chains_
>>>
>>> When setting up the firewall (in iptables_init()), four new chains are
>>> created, IPBLACKLISTIN, IPBLACKLISTOUT, IPBLACKLISTREDIN and
>>> IPBLACKLISTREDOUT. The first two are inserted into the INPUT, OUTPUT
>>> and FORWARD chains, before the IPS chains. The main script is called to
>>> restore the previously saved blacklists, creating the rules in the
>>> latter two chains.
>>
>> For readability you can simply call them BLACKLIST*. The IP is a given.
>
> Good point. I'll make the change.
>
>>
>>> In iptables_red_up(), rules are created to forward traffic to/from the
>>> red interface from IPBLACKLISTIN/IPBLACKLISTOUT to
>>> IPBLACKLISTREDIN/IPBLACKLISTREDOUT.
>>> I did consider putting the blacklist rules into the
>>> IPBLACKLISTIN/IPBLACKLISTOUT, but this couldn't be done until red comes
>>> up and would require much more processing at this point - not a problem
>>> if your red interface comes up and stays up, but if you've got one that
>>> disconnects on no traffic you probably don't want the extra overhead.
>>> The way I've done it also has the side effects of 1) the main script
>>> doesn't have to worry about what the red interface is, and 2) if you're
>>> using multiple blacklists, the check for the red interface is only done
>>> once.
>>
>> You would only want to check the interface once and then jump into the chain. Can we not realise it that way?
>
> I'm not sure what you're suggesting.
> I insert BLACKLISTIN into INPUT & FORWARD and BLACKLISTOUT into OUTPUT &
> FORWARD. These receive all non-ICMP traffic. On red up traffic from
> the red interface is sent from BLACKLISTIN to BLACKLISTREDIN and traffic
> to the red interface is sent from BLACKLISTOUT to BLACKLISTREDOUT.
> The rules to check against the IPSets are put into the *RED* chain,
> checking against the source or destination addresses as appropriate.
This sounds very complicated to me.
Why can we not check this in BLACKLISTIN/OUT?
>>
>>> _Log_WUI_Pages_
>>>
>>> The two Log WUI pages are edited from existing pages. I've resisted the
>>> temptation to reformat them, which means that they can be diff'ed with
>>> the corresponding pages.
>>
>> Loads of the UI code is messy and deliberately not touched any more. I guess this is fine.
>
> Yes, it's not pretty, but it is pragmatic.
>>
>>> _Update_
>>>
>>> The update is fairly simple - it calculates whether the check period is
>>> passed, tries a download specifying the If-Modified-Since and parses the
>>> returned file. It then compares the new list with the existing list,
>>> adding new addresses/networks and deleting those that have been removed.
>>> There's a check to handle a list changing from a list of addresses to a
>>> list of networks, in which case it changes the type of the IPSet - this
>>> should be a vary rare event, but could happen for a composite list.
>>
>> Why do you need the diff? Has that been in the first patchset?
>
> It was, yes.
>
> Originally I did it for performance. It took over a minute to process
> the ALIENVAULT list (over 80 000 entries); just calling ipset with
> changes reduced this to a few seconds. I've since modified the code to
> pipe commands to a process running 'ipset restore', which has
> drastically speed up the processing, but it's still a little bit quicker
> to just do changes.
>
> The second reason is that it doesn't feel right to flush the old list,
> thus removing its protection while the new one is loaded, especially if
> the new list is the same as the old list (not all sources implement
> IP-Modified-Since). The actual period of reduced protection may be
> short, but I prefer to avoid it.
Okay. I was hoping that the “ipset restore” update would flush and add everything again in one large atomic operation.
>>
>> Just reloading the whole ipset should be easier.
>>
>>> _Enabling/Disabling_Blacklists_
>>>
>>> For changes to the existing firewall rules, the firewall is shut down
>>> and restarted from scratch. The blacklist script does this on the fly,
>>> since the possibilities are simpler than all the options available for
>>> the firewall and also the necessary code for doing updates means that
>>> adding this code is fairly simple. This includes turning logging on and
>>> off.
>>
>> What do you mean by that? /etc/init.d/firewall restart?
>>
>> That is something we cannot do because IPsec and some other services create temporary rules that would be lost.
>
> No. I'm actually trying to say the opposite (badly). The existing
> IPFire code only implements some firewall changes by shutting down and
> restarting the firewall, but all the IP Blacklist functions
> (enable/disable/update lists, enable/disable logging) can be done
> without this - the necessary rules can be added to/deleted from the
> BLACKLISTRED* chains without affecting any of the rest of the firewall
> system.
Ah cool. That is how it needs to be.
-Michael
> Tim
P.S. Is there actually a reason why you do not have a shell account on our servers and host your changed on our git server? I am not sure if I ever offered you one, but it would help me a lot to have a look at the code in a git repo :)
>
>>
>>> _Logging_
>>>
>>> The main scripts logs to syslog as ipblacklist, but if you run it from
>>> the command line it'll also output to the terminal.
>>> There's also some debug code which can be turned on by setting a value
>>> in the settings file - it's turned on at the lowest level automatically
>>> if you run from the command line.
>>
>> Yes that is fine.
>>
>> Best,
>> -Michael
>>
>>> Tim
>>>
>>> On 06/01/2020 11:27, Michael Tremer wrote:
>>>> Hello all,
>>>>
>>>>> On 4 Jan 2020, at 19:02, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> I've now updated the code - screenshots are attached.
>>>>>
>>>>> IP_Address_Blacklists.png
>>>>>
>>>>> This is the settings page. I don't think there's anything unexpected here.
>>>>
>>>> Yes, that looks a lot like the other IPFire pages :)
>>>>
>>>>> IP_Address_Blacklist_Logs.png
>>>>>
>>>>> The log page, replacing the status information that was previously at
>>>>> the top of the settings page.
>>>>
>>>> Looks great!
>>>>
>>>>> I've sorted the lists alphabetically, rather than by the number of hits
>>>>> since the worst threat is not necessary the largest number of blocked
>>>>> packets (which could implied by sorting it by number).
>>>>
>>>> Yes, that is indeed a very good idea.
>>>>
>>>> You could add a % sign in the percentage column, so it cannot be confused so easily with the other column.
>>>>
>>>>> Firewall_log_blacklist.png
>>>>>
>>>>> Details page accessed from the log. Almost identical to the other
>>>>> similar pages.
>>>>
>>>> *thumbs up*
>>>>
>>>>> Log_Summary.png
>>>>>
>>>>> This is an extract from the Log Summary page produced by logwatch.
>>>>>
>>>>> I still need to do some tidying, remove unused language strings etc.,
>>>>> and then update and test the patches, so it'll be some time before I'm
>>>>> ready to submit updated patches.
>>>>
>>>> Feel free to send some RFC to the list. It might be slightly noisy, but it is easier to review changes bit after bit.
>>>>
>>>> Great great work!
>>>>
>>>> -Michael
>>>>
>>>>>
>>>>> Tim
>>>>>
>>>>> On 28/12/2019 20:59, Tim FitzGeorge wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I'll code and test what we've agreed on. In the meantime I'm going to
>>>>>> reply on the sub-thread started by Tom Rymes as to what we call the list
>>>>>> categories, so that we can keep that discussion separate from further
>>>>>> functionality etc. discussion.
>>>>>>
>>>>>> Tim
>>>>>>
>>>>>> On 21/12/2019 18:34, Tim FitzGeorge wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I've got my systems checking for downloads at 15 minute intervals and
>>>>>>> using If-Modified-Since. This all seems to be working OK.
>>>>>>>
>>>>>>> Responses to comments below:
>>>>>>>
>>>>>>> On 18/12/2019 12:07, Michael Tremer wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>>> On 17 Dec 2019, at 18:21, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I think we've agreed on the following:
>>>>>>>>>
>>>>>>>>> - Checks to be made every 15 minutes, subject to per-list minimum rate.
>>>>>>>>> - Modify download to use If-Modified-Since rather than separate HEAD
>>>>>>>>> request.
>>>>>>>>> - Remove automatic blacklist.
>>>>>>>>
>>>>>>>> I thought someone wants to counter my argument. This is not a dictatorship :)
>>>>>>>>
>>>>>>>>>
>>>>>>>>> other comments are below.
>>>>>>>>>
>>>>>>>>> Tim
>>>>>>>>>
>>>>>>>>> On 16/12/2019 22:20, Michael Tremer wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>>> On 16 Dec 2019, at 20:06, Tim FitzGeorge <lists(a)tfitzgeorge.me.uk> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I've attached the current GUI screenshot.
>>>>>>>>>>
>>>>>>>>>> Thanks for that.
>>>>>>>>>>
>>>>>>>>>> I have a couple of suggestions/concerns about it:
>>>>>>>>>>
>>>>>>>>>> a) I am not sure what the value is about the top box that is called “Status”. It is basically a summary of the iptables output, but will it help the user? A blacklist with more or fewer entries is not better or worse, blocking more packets isn’t better than blocking fewer. It is about the quality of the blocks.
>>>>>>>>>
>>>>>>>>> I agree that the size of the lists is not all that useful, but the
>>>>>>>>> information about the number of blocked packets can be an indication of
>>>>>>>>> compromise. It depends on the type of the list, but a list that targets
>>>>>>>>> C&C channels (e.g. the FEODO lists) should not show any input packets
>>>>>>>>> being blocked, unless one of the computers being protected is infected.
>>>>>>>>> Apart from the BOGON lists, none of the lists should show any output
>>>>>>>>> packets being blocked; again it's a potential indication of infection if
>>>>>>>>> otherwise.
>>>>>>>>
>>>>>>>> The Shodan list will definitely show some activity unless you are behind CG NAT.
>>>>>>>>
>>>>>>>> Should we not have this rather in the logging section?
>>>>>>>>
>>>>>>>> And these counters here will be reset when the firewall is being reloaded. Should this data therefore not be collected from the logs so that you can go back in time? That would be very important to identify any start of compromise.
>>>>>>>>
>>>>>>>
>>>>>>> That makes sense. I'll remove the status from this page and create a
>>>>>>> log page for the information. If anyone actually wants the iptables
>>>>>>> information it's available in from Firewall > iptables anyway.
>>>>>>>
>>>>>>> Do you think it makes more sense to show packets blocked in/out or
>>>>>>> packets blocked per source interface? The latter would of (limited)
>>>>>>> help in tracking down potential compromise.
>>>>>>>
>>>>>>>>>>
>>>>>>>>>> b) I would prefer to move the settings box to the top. If you like you can show the size of the blacklists there and when they have been updated, too.
>>>>>>>>>
>>>>>>>>> I put the status at the top since it's usually what I want to see when I
>>>>>>>>> go to the page. Under normal circumstances I don't change the settings,
>>>>>>>>> so I'd rather not have to scroll past them to get to the information I'm
>>>>>>>>> actually interested in. The update times are just useful to show that
>>>>>>>>> the lists are actually being updated. The size is interesting to me,
>>>>>>>>> but as you say it probably won't help the average user, especially if
>>>>>>>>> you don't know which lists block individual addresses and which block
>>>>>>>>> nets (BOGON, DSHIELD and EMERGING_COMPROMISED).
>>>>>>>>
>>>>>>>> See my comment on the order above. Splitting the page might make sense.
>>>>>>>>
>>>>>>>> When you show size, does that mean lines in the blacklists? So it could be networks or single IP addresses? Does it not make sense to count IP addresses (i.e. 256 for a /24 network and so on) and put it in the table?
>>>>>>> It's lines in the blacklist.
>>>>>>>
>>>>>>>>
>>>>>>>> I think I would only be interested in the size for two things:
>>>>>>>>
>>>>>>>> a) Is there any chance for overblocking? i.e. does the list have 2 billion entries which would be half the public IPv4 address space or so. I cannot really come up with a good example here.
>>>>>>>>
>>>>>>>> Or
>>>>>>>>
>>>>>>>> b) What is the performance impact/memory consumption of the list?
>>>>>>>
>>>>>>> It would be interesting to show the type (single address or net), number
>>>>>>> of entries and number of addresses; the first two affect performance,
>>>>>>> the second memory use and the latter the number of blocked addresses,
>>>>>>> but that's really only going to be of interest to a very small number of
>>>>>>> people, who can always do 'ipset list -t' to get the information. So
>>>>>>> I'll remove the size column.
>>>>>>>
>>>>>>>>
>>>>>>>>> I originally had the status and settings in one list, but I decided that
>>>>>>>>> it was getting a bit messy and difficult to follow. I've attached a
>>>>>>>>> screenshot of a quickly hacked page. (Note that the status area only
>>>>>>>>> appears once the blacklists have been enabled.)
>>>>>>>>
>>>>>>>> You are right. This is getting quite tight there.
>>>>>>>>
>>>>>>>> You could write packets in/out into on column like 2k/1023, but see my concerns about this above.
>>>>>>>
>>>>>>> Actually I've come to the conclusion that the number of bytes blocked
>>>>>>> isn't all that important; it's sufficient to show the number of packets.
>>>>>>>
>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
>>>>>>>>>
>>>>>>>>> I agree that it's it's an over simplistic summary of the lists. I note
>>>>>>>>> that Tom Rymes has asked to keep it, but I think its only value is if it
>>>>>>>>> suggests to users that things are more complicated than they appear and
>>>>>>>>> therefore prompts them to read the Wiki.
>>>>>>>>
>>>>>>>> I will reply to that separately.
>>>>>>>
>>>>>>> Looking at that reply, I think it makes sense to include a category
>>>>>>> column in both the settings and the log summary page - the former to
>>>>>>> help selection of which lists to use, and the latter as a help when
>>>>>>> deciding whether the information could indicate compromise or not.
>>>>>>>
>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> d) I do not understand what the “update check rate” means. I suppose we should show the update interval of the lists in the table and just refresh them according to that.
>>>>>>>>>
>>>>>>>>> It's the minimum check period for updates. The system checks for
>>>>>>>>> updates at either the rate specific to the list or this rate, whichever
>>>>>>>>> is slower. This is a hangover from the early days, copied from the IPS
>>>>>>>>> updates. I suppose it could have a value for some people who may wish
>>>>>>>>> to lower the check rate, but I'm not sure that it has sufficient value
>>>>>>>>> to be worth keeping.
>>>>>>>>
>>>>>>>> No, I would not let the user decide when and how often to update the lists.
>>>>>>>
>>>>>>> OK. I'll remove it.
>>>>>>>
>>>>>>>>
>>>>>>>> Generally there is this argument internally how often some databases should be updated. People with slow internet connections or volume based data plans will find this rather expensive to update those lists too often.
>>>>>>>>
>>>>>>>> If we ever want to go ahead with that in IPFire 2, there should be a global switch for a “low data mode” which then delays updating these things.
>>>>>>>
>>>>>>> That makes sense. Maybe also the ability to set when downloads take
>>>>>>> place - but as you say that's for the future.
>>>>>>>
>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> e) Do we want to keep the automatic blacklists now? I do not think it will actually improve the security of IPFire.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'll remove them. I think they do have a use, in that they can detect
>>>>>>>>> someone doing a port scan and block them before they find an open port,
>>>>>>>>> but as you say there is also the possibility of a DOS.
>>>>>>>>
>>>>>>>> How would this block a DOS?
>>>>>>>
>>>>>>> Sorry, I didn't make myself clear - there's the possibility that someone
>>>>>>> could cause a DOS with the automatic list by faking the source IP
>>>>>>> address and sending a few packets on a random port.
>>>>>>>
>>>>>>>>
>>>>>>>> -Michael
>>>>>>>
>>>>>>> Tim
>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I'll update the code based on our discussions, and submit an updated set
>>>>>>>>>>> of patches - I imagine there will have to be at least one more iteration.
>>>>>>>>>>
>>>>>>>>>> Let’s wait until we have come to decisions :)
>>>>>>>>>>
>>>>>>>>>> -Michael
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Tim
>>>>>>>>>>>
>>>>>>>>>>> (No additional comments below)
>>>>>>>>>>>
>>>>>>>>>>> On 13/12/2019 23:11, Michael Tremer wrote:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> Again my apologies for my late reply. Busy busy weeks.
>>>>>>>>>>>>
>>>>>>>>>>>>> On 8 Dec 2019, at 20:50, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>
>>>>>>>>>>>>> It's my turn to apologise for being slow to respond - I've had a busy
>>>>>>>>>>>>> week, but I should have plenty of time over the next couple of weeks.
>>>>>>>>>>>>
>>>>>>>>>>>> No worries. Turns out we all do :)
>>>>>>>>>>>>
>>>>>>>>>>>>> I've made most of the comments inline, however I think Michael had a
>>>>>>>>>>>>> question (which I can't find now) about what happens if someone enables
>>>>>>>>>>>>> all the lists. One thing which would perhaps make this less likely is
>>>>>>>>>>>>> that the WUI tags the available lists with whether they're safe or not,
>>>>>>>>>>>>> with a footnote that safe means that the list only blocks malicious
>>>>>>>>>>>>> traffic. This won't guarantee that a user won't still try to enable all
>>>>>>>>>>>>> the lists, but it should make them realise that they should think first.
>>>>>>>>>>>>
>>>>>>>>>>>> We had a couple of features going slightly wrong or being “misunderstood” by some users. People still seem to panic when they see “local recursor”. To this day I do not know why.
>>>>>>>>>>>>
>>>>>>>>>>>> We cannot make everything idiot-proof. And when some user if of that category, they probably should shutdown their IPFire box, educate themselves and then come back again. So I do not want to limit people, but make things as easy as possible.
>>>>>>>>>>>>
>>>>>>>>>>>> If someone enables all the lists, good luck with passing packets :)
>>>>>>>>>>>>
>>>>>>>>>>>>> I have considered replacing this tag with a risk high/medium/low and
>>>>>>>>>>>>> maybe adding a category (invalid/application/scanner/C&C or something
>>>>>>>>>>>>> like that), but that may provide too much information and dissuade them
>>>>>>>>>>>>> from actually following the links to checkout what the list actually does.
>>>>>>>>>>>>
>>>>>>>>>>>> Can we have a screenshot of the GUI right now? I didn’t run the code, yet.
>>>>>>>>>>>>
>>>>>>>>>>>> We should document the lists like we do it with the rulesets of the IPS. People might ignore this, but that is on them.
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Tim
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 05/12/2019 22:25, Michael Tremer wrote:
>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 4 Dec 2019, at 17:05, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hello Tim, hello Michael,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> please see my responses inline...
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We could periodically update the blacklists on our main mirror (and
>>>>>>>>>>>>>>>>>> wait for the network to sync it), make sure it is signed and write
>>>>>>>>>>>>>>>>>> a small downloader that fetches, validates and installs them.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> @All: Thoughts on this?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I think there are a number of points here.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Firstly, from the point of a third party using IPFire, is this really
>>>>>>>>>>>>>>>>> solving the privacy disclosure problem? There's no way round disclosing
>>>>>>>>>>>>>>>>> your public IP address to someone you're downloading from; all this does
>>>>>>>>>>>>>>>>> is change who that information is being disclosed to. For the user
>>>>>>>>>>>>>>>>> there's no way of knowing whether the source is more or less protective
>>>>>>>>>>>>>>>>> of the user's privacy than the blacklist provider. Indeed it won't be
>>>>>>>>>>>>>>>>> possible to know who the lists are being downloaded from until the
>>>>>>>>>>>>>>>>> download starts.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> There is a way: Tor. But that is a totally different story.
>>>>>>>>>>>>>>> Well, I see a third option on this: Use the mirror infrastructure we already
>>>>>>>>>>>>>>> have. Every IPFire installation discloses its public IP address to one of these
>>>>>>>>>>>>>>> servers sooner or later, so we do not disclose additional data if the blacklists
>>>>>>>>>>>>>>> were fetched from these.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Needless to say, Tor (hidden services) would be better, but that is a different
>>>>>>>>>>>>>>> story indeed. :-)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The point is rather that a forget list can be sent instead of the “real” one.
>>>>>>>>>>>>>>> I did not get this. Forget? Forged?? ???
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes, I meant to write forged, but auto-correct didn’t let me.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Secondly, latency; some of the lists are updated every 5 minutes. While
>>>>>>>>>>>>>>>>> I've limited the maximum check rate to hourly, will the updates
>>>>>>>>>>>>>>>>> propagate quickly enough. For reference on my main system the 24
>>>>>>>>>>>>>>>>> updates on the CIARMY list made 143 498 changes (additions or
>>>>>>>>>>>>>>>>> deletions). I've seem it do over 200 000.
>>>>>>>>>>>>>>> Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not
>>>>>>>>>>>>>>> document a recommended update interval anywhere, so we can only guess.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Personally, more static lists seem to be preferable for packet filtering. Highly
>>>>>>>>>>>>>>> dynamic ones such as CIArmy should be done via DNSBL queries or something similar
>>>>>>>>>>>>>>> - do we really want to have that list here?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It is not really an option to implement a DNSBL into a packet filter, but I get your point.
>>>>>>>>>>>>>
>>>>>>>>>>>>> One of the 'selling points' for an IP address blacklist is that it can
>>>>>>>>>>>>> respond quickly to new threats - or rather new attackers. While a new
>>>>>>>>>>>>> IDS/IPS rule needs time to analyse the threat, generate a rule and check
>>>>>>>>>>>>> it, it's easy to add an address to a list. So, I think the CIArmy list
>>>>>>>>>>>>> is potentially useful for protecting home systems etc. with budget
>>>>>>>>>>>>> hardware, but I would be very careful about using it for a protecting a
>>>>>>>>>>>>> general access website.
>>>>>>>>>>>>
>>>>>>>>>>>> If they are very volatile, we should honour that and update them often, too.
>>>>>>>>>>>>
>>>>>>>>>>>> It probably is more about false-positives being removed very quickly instead of adding threats very quickly. The average IPFire user is probably not under threats like these to need to react very quickly.
>>>>>>>>>>>>
>>>>>>>>>>>> So I do not see much value in adding those lists and then updating once a day. Does it have to be every 5 min? No. I would suggest 15 which should be good enough for everyone.
>>>>>>>>>>>>
>>>>>>>>>>>> Other lists should of course not be updated every 15 minutes when not needed.
>>>>>>>>>>>>
>>>>>>>>>>>> Running every 15 minutes would allow us to retry downloading lists that are on an hourly schedule if the download failed.
>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> How did you come up with the hour? Will it be retried more often if the download was not successful?
>>>>>>>>>>>>>>> One hour is the most common interval indeed, but adding some random time might
>>>>>>>>>>>>>>> be useful in order to reduce load on the servers providing a blacklist.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes, definitely. Otherwise we will shoot down our mirrors.
>>>>>>>>>>>>>
>>>>>>>>>>>>> When I implemented that section of code, specifying the minimum check
>>>>>>>>>>>>> period in hours seemed to provide a convenient way of allowing a check
>>>>>>>>>>>>> period covering a wide range, with an hour as the fastest and a week as
>>>>>>>>>>>>> the slowest. I didn't looked at the CIArmy list until much later. Most
>>>>>>>>>>>>> of the lists don't change nearly as much, but the CIArmy list is
>>>>>>>>>>>>> described as one that deliberately responds quickly.
>>>>>>>>>>>>>
>>>>>>>>>>>>> From my production system, for yesterday:
>>>>>>>>>>>>>
>>>>>>>>>>>>> The following block lists were updated:
>>>>>>>>>>>>> BLOCKLIST_DE: 24 Time(s) - 9341 change(s)
>>>>>>>>>>>>> BOGON_FULL: 1 Time(s) - 10 change(s)
>>>>>>>>>>>>> CIARMY: 24 Time(s) - 159134 change(s)
>>>>>>>>>>>>> DSHIELD: 7 Time(s) - 18 change(s)
>>>>>>>>>>>>> EMERGING_FWRULE: 1 Time(s) - 50 change(s)
>>>>>>>>>>>>> FEODO_AGGRESIVE: 24 Time(s) - 13 change(s)
>>>>>>>>>>>>> SHODAN: 1 Time(s) - 0 change(s)
>>>>>>>>>>>>> SPAMHAUS_DROP: 1 Time(s) - 0 change(s)
>>>>>>>>>>>>> TOR_EXIT: 24 Time(s) - 162 change(s)
>>>>>>>>>>>>
>>>>>>>>>>>> Very interesting statistics.
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> and my test system:
>>>>>>>>>>>>>
>>>>>>>>>>>>> The following block lists were updated:
>>>>>>>>>>>>> ALIENVAULT: 19 Time(s) - 5331 change(s)
>>>>>>>>>>>>> EMERGING_COMPROMISED: 1 Time(s) - 26 change(s)
>>>>>>>>>>>>> TALOS_MALICIOUS: 1 Time(s) - 36 change(s)
>>>>>>>>>>>>>
>>>>>>>>>>>>> That covers most of the lists. From the WUI, since 1 Dec:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Blacklist Entries pkts bytes Last updated
>>>>>>>>>>>>> in in
>>>>>>>>>>>>> AUTOBLACKLIST 0 731 51144 Sun Dec 8 18:40:02 2019
>>>>>>>>>>>>> BLOCKLIST_DE 28020 857 46735 Sun Dec 8 18:40:02 2019
>>>>>>>>>>>>> BOGON_FULL 214 5255 189K Sun Dec 8 16:50:04 2019
>>>>>>>>>>>>> CIARMY 15000 19774 976K Sun Dec 8 18:04:01 2019
>>>>>>>>>>>>> DSHIELD 20 7992 321K Sun Dec 8 16:45:13 2019
>>>>>>>>>>>>> EMERGING_FWRULE 1647 197 8383 Fri Dec 6 05:29:07 2019
>>>>>>>>>>>>> FEODO_AGGRESIVE 7169 0 0 Sun Dec 8 18:50:07 2019
>>>>>>>>>>>>> SHODAN 32 34 1530 Sun Dec 8 17:54:09 2019
>>>>>>>>>>>>> SPAMHAUS_DROP 823 0 0 Fri Dec 6 18:22:35 2019
>>>>>>>>>>>>> SPAMHAUS_EDROP 111 82 16433 Thu Dec 5 17:27:09 2019
>>>>>>>>>>>>> TOR_EXIT 1055 0 0 Sun Dec 8 18:31:02 2019
>>>>>>>>>>>>
>>>>>>>>>>>> This as well.
>>>>>>>>>>>>
>>>>>>>>>>>> Those are more packets than I would have expected.
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> (I've left out the pkts/bytes out fields which were all 0)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Note that where possible I do a HEAD request first and then only
>>>>>>>>>>>>> download the list if the modification time has changed since the last
>>>>>>>>>>>>> check. For dynamically generated lists this isn't possible.
>>>>>>>>>>>>
>>>>>>>>>>>> You won’t need a HEAD request for it. You can include it in the GET request.
>>>>>>>>>>>>
>>>>>>>>>>>> Have a look at location downloader where I use that. The server will respond with 304 and not send any payload.
>>>>>>>>>>>>
>>>>>>>>>>>> https://git.ipfire.org/?p=location/libloc.git;a=blob;f=src/python/location-downloader.in;h=1b5932d5822e03737d32b2a27816815b2f7e74dd;hb=HEAD#l151
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> If the download isn't successful it just gives up and waits for the next
>>>>>>>>>>>>> attempt (apart from the usual retries in the library). I probably
>>>>>>>>>>>>> should to change that so that it only applies the per list minimum
>>>>>>>>>>>>> update period in this case (specified in the sources file) rather than
>>>>>>>>>>>>> the user specified value as well.
>>>>>>>>>>>>
>>>>>>>>>>>> I think it is not the worst if an update fails. It might just happen every once in a while.
>>>>>>>>>>>>
>>>>>>>>>>>> So I would suggest to just re-run the script more often and when the mtime of the file is older than the threshold, a download is attempted. You can use that timestamp for the GET request.
>>>>>>>>>>>>
>>>>>>>>>>>>> I already use a time offset on the downloads - when it's started from
>>>>>>>>>>>>> boot, backup restore or WUI enable, it checks to see if it's installed
>>>>>>>>>>>>> in the fcrontab, and if not adds itself at a randomly generated offset
>>>>>>>>>>>>> in the hour.
>>>>>>>>>>>>
>>>>>>>>>>>> That should potentially go to red.up, or if we can settle on 15 minutes, I would consider that often enough to quickly update all lists after a reboot.
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is
>>>>>>>>>>>>>>>>> the largest at a few MB), there are going to be a lot of them. How will
>>>>>>>>>>>>>>>>> this affect the willingness of people to mirror IPFire?
>>>>>>>>>>>>>>> I do not consider this being a problem as we do not generate that much traffic
>>>>>>>>>>>>>>> to them. Of course, that depends on the update interval again.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> That depends on your point of view.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I do not have a problem with this at all in my data center, but there are plenty of people with a volume-based LTE plan or simply a 128 kBit/s connection. It will take a longer time to download the lists for them. We need to mind that.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Talking about the preference of packet filter and IPS, I prefer to
>>>>>>>>>>>>>>>>>> use the latter as well as it gains more insight in what kind of malicious
>>>>>>>>>>>>>>>>>> traffic tried to pass a firewall machine. On systems with low resources,
>>>>>>>>>>>>>>>>>> this might be problematic and removing load from the IPS can be preferred
>>>>>>>>>>>>>>>>>> (make this configurable?!), on others, people might want to have both
>>>>>>>>>>>>>>>>>> results.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> You're only going to get one result for a packet whichever way round the
>>>>>>>>>>>>>>>>> IP blacklist and IPS are since whichever comes first will drop the
>>>>>>>>>>>>>>>>> packet before it reaches the second (well it would be possible to put
>>>>>>>>>>>>>>>>> the IP blacklist first and get it to log and mark packets which are then
>>>>>>>>>>>>>>>>> dropped after the IPS, but I think that's getting a little complicated.
>>>>>>>>>>>>>>>>> In addition I've seen the messages about the trouble marking was
>>>>>>>>>>>>>>>>> causing in the QoS).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I think it's a 50/50 choice as to which is more valuable first; it's
>>>>>>>>>>>>>>>>> probably going to differ from packet to packet. For me the possibility
>>>>>>>>>>>>>>>>> of reducing the IPS load means I prefer putting the IP blacklist first
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It should be fairly easy to add the choice of where to put the IP
>>>>>>>>>>>>>>>>> blacklist. I think it'll have to be in the main firewall script, so
>>>>>>>>>>>>>>>>> it'll require a firewall restart, but it's not something that'll be
>>>>>>>>>>>>>>>>> changed often.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier.
>>>>>>>>>>>>>>> Agreed.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily.
>>>>>>>>>>>>>>> So dropping packets from blacklisted IP addresses/networks before IPS is it, then.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I do not even think it makes sense to swap the order in the outgoing direction.
>>>>>>>>>>>>>>> Me too.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this.
>>>>>>>>>>>>>>> Based on my personal experience, firewall hits observed on a single machine exposed
>>>>>>>>>>>>>>> to the internet are interesting, but the overall situation across multiple machines
>>>>>>>>>>>>>>> is even more interesting. Very quickly, you'll end on something like a centralised
>>>>>>>>>>>>>>> logging server and custom statistical analysis here...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Probably a project for IPFire 4.0 :)
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Or use one of the existing services, like the DSHIELD client
>>>>>>>>>>>>> https://dshield.org/howto.html (subject to privacy concerns again).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default.
>>>>>>>>>>>>>>>>>> I would love to see the bogon ruleset here, too (think about 8chan successor
>>>>>>>>>>>>>>>>>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause
>>>>>>>>>>>>>>>>>> interference if RED does not have a public IP address assigned.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I can add a field to the options file that controls whether a list is
>>>>>>>>>>>>>>>>> enabled by default.
>>>>>>>>>>>>>>> Thank you. :-)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat.
>>>>>>>>>>>>>>> This is my only objection against this patchset. Now, what can we do about it?
>>>>>>>>>>>>>>> One possibility is to apply the patchset now and implement a custom download
>>>>>>>>>>>>>>> source thing later on, or do that before releasing Core Update 139 (or which version
>>>>>>>>>>>>>>> the patchset will be to) after we agreed on something.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I do not see this being merged for 139. But that is not important. We need to get it right first and then release it.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> As far as I know, nobody has tested this, yet.
>>>>>>>>>>>>>
>>>>>>>>>>>>> There are a number of people who have been running an earlier version
>>>>>>>>>>>>> which I shared on GitHub. There were a few early issues, but it seems
>>>>>>>>>>>>> to be OK now.
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://forum.ipfire.org/viewtopic.php?f=27&t=21845
>>>>>>>>>>>>>
>>>>>>>>>>>>> This version wasn't integrated into IPFire, so (for example) it inserted
>>>>>>>>>>>>> itself into the INPUT IPTables chain rather than having it's chains
>>>>>>>>>>>>> created as part of the firewall start-up script.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have huge concerns about the automatic blacklist. @Peter: What is your opinion on this?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> While I implemented it, I'm aware of its potential to cause problems,
>>>>>>>>>>>>> which is why it has to be separately enabled. It's not caused me any
>>>>>>>>>>>>> issues at the default settings (blocks at over 10 packets per hour until
>>>>>>>>>>>>> 1 hour has passed without seeing packets from the address), but I've not
>>>>>>>>>>>>> used it on a site with publicly announced services. If I was going to
>>>>>>>>>>>>> use it on a web site I would want to, at the very minimum, drop the
>>>>>>>>>>>>> block period drastically.
>>>>>>>>>>>>
>>>>>>>>>>>> I suppose this is entirely unusable on an IPFire box in a data center that hosts things. Let’s say our rack in Hanover.
>>>>>>>>>>>>
>>>>>>>>>>>> You will have hits from some broken IP stacks and people might just end up on this without doing anything wrong.
>>>>>>>>>>>>
>>>>>>>>>>>> You can denial-of-service easily as well and I suppose without aggregation of data from many many systems, it does not make sense to instantly block addresses. And even then you probably would block an entire subnet.
>>>>>>>>>>>>
>>>>>>>>>>>> So I am not sure how I should feel about it. I do not think that it adds anything because the packets would have been blocked anyways.
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On the other hand, it's good at responding quickly. Usually I see only
>>>>>>>>>>>>> 1-2% of blocks from the automatic list:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Reason Count % First Last
>>>>>>>>>>>>> CIARMY 2416 45 Dec 6 00:00 Dec 6 23:59
>>>>>>>>>>>>> DSHIELD 1353 25 Dec 6 00:00 Dec 6 23:59
>>>>>>>>>>>>> INPUT 1294 24 Dec 6 00:00 Dec 6 23:59
>>>>>>>>>>>>> AUTOBLACKLIST 122 2 Dec 6 00:20 Dec 6 16:28
>>>>>>>>>>>>> BLOCKLIST_DE 89 2 Dec 6 00:20 Dec 6 23:46
>>>>>>>>>>>>>
>>>>>>>>>>>>> and sometimes none at all, but one one occasion it blocked over 8000
>>>>>>>>>>>>> packets. Again I'm aware this is for a home system, which is rather
>>>>>>>>>>>>> different than from a Web server.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not
>>>>>>>>>>>>>>> seem to be problematic, as they are hosted on 3rd party servers, too).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> One of them will be, sooner or later. And one is enough I suppose.
>>>>>>>>>>>>>
>>>>>>>>>>>>> DShield (https://dshield.org/api/#threatfeeds) and firehol
>>>>>>>>>>>>> (http://iplists.firehol.org/) seem to host copies of most of the lists
>>>>>>>>>>>>> as well.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I do not really want to overthink this - we didn’t do this with clamav for example either. But it probably is a decision to be made by the user what they want to enable and we should not enable anything by default. So no data will be leaked as long as the user does not consent.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks, and best regards,
>>>>>>>>>>>>>>> Peter Müller
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> <Screenshot_2019-12-16_IP_Address_Blacklists.png>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <Screenshot_2019-12-17_IP_Address_Blacklists.png>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> <IP_Address_Blacklists.png><IP_Address_Blacklist_Logs.png><Firewall_log_blacklist.png><Log_Summary.png>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 0/5] ipblacklist: IP Address Blacklists
2020-01-28 17:14 ` Michael Tremer
@ 2020-01-29 20:40 ` Tim FitzGeorge
2020-01-30 12:54 ` Michael Tremer
0 siblings, 1 reply; 35+ messages in thread
From: Tim FitzGeorge @ 2020-01-29 20:40 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 38890 bytes --]
Hi,
On 28/01/2020 17:14, Michael Tremer wrote:
> Hi,
>
> Apologies for my late reply again. I marked this email in my inbox and then it scrolled down because of loads of new stuff coming in. Please feel free to ping me if I do not respond in time.
That's OK. It took me a little longer than expected to check the patches.
>
>> On 24 Jan 2020, at 19:40, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>
>> I'm now nearly ready to submit the updated patches, but before then I'd
>> like to check that you're happy with some of my design decisions.
>
> Cool.
>
>> _IPTables_Chains_
>>
>> When setting up the firewall (in iptables_init()), four new chains are
>> created, IPBLACKLISTIN, IPBLACKLISTOUT, IPBLACKLISTREDIN and
>> IPBLACKLISTREDOUT. The first two are inserted into the INPUT, OUTPUT
>> and FORWARD chains, before the IPS chains. The main script is called to
>> restore the previously saved blacklists, creating the rules in the
>> latter two chains.
>
> For readability you can simply call them BLACKLIST*. The IP is a given.
Good point. I'll make the change.
>
>> In iptables_red_up(), rules are created to forward traffic to/from the
>> red interface from IPBLACKLISTIN/IPBLACKLISTOUT to
>> IPBLACKLISTREDIN/IPBLACKLISTREDOUT.
>> I did consider putting the blacklist rules into the
>> IPBLACKLISTIN/IPBLACKLISTOUT, but this couldn't be done until red comes
>> up and would require much more processing at this point - not a problem
>> if your red interface comes up and stays up, but if you've got one that
>> disconnects on no traffic you probably don't want the extra overhead.
>> The way I've done it also has the side effects of 1) the main script
>> doesn't have to worry about what the red interface is, and 2) if you're
>> using multiple blacklists, the check for the red interface is only done
>> once.
>
> You would only want to check the interface once and then jump into the chain. Can we not realise it that way?
I'm not sure what you're suggesting.
I insert BLACKLISTIN into INPUT & FORWARD and BLACKLISTOUT into OUTPUT &
FORWARD. These receive all non-ICMP traffic. On red up traffic from
the red interface is sent from BLACKLISTIN to BLACKLISTREDIN and traffic
to the red interface is sent from BLACKLISTOUT to BLACKLISTREDOUT.
The rules to check against the IPSets are put into the *RED* chain,
checking against the source or destination addresses as appropriate.
>
>> _Log_WUI_Pages_
>>
>> The two Log WUI pages are edited from existing pages. I've resisted the
>> temptation to reformat them, which means that they can be diff'ed with
>> the corresponding pages.
>
> Loads of the UI code is messy and deliberately not touched any more. I guess this is fine.
Yes, it's not pretty, but it is pragmatic.
>
>> _Update_
>>
>> The update is fairly simple - it calculates whether the check period is
>> passed, tries a download specifying the If-Modified-Since and parses the
>> returned file. It then compares the new list with the existing list,
>> adding new addresses/networks and deleting those that have been removed.
>> There's a check to handle a list changing from a list of addresses to a
>> list of networks, in which case it changes the type of the IPSet - this
>> should be a vary rare event, but could happen for a composite list.
>
> Why do you need the diff? Has that been in the first patchset?
It was, yes.
Originally I did it for performance. It took over a minute to process
the ALIENVAULT list (over 80 000 entries); just calling ipset with
changes reduced this to a few seconds. I've since modified the code to
pipe commands to a process running 'ipset restore', which has
drastically speed up the processing, but it's still a little bit quicker
to just do changes.
The second reason is that it doesn't feel right to flush the old list,
thus removing its protection while the new one is loaded, especially if
the new list is the same as the old list (not all sources implement
IP-Modified-Since). The actual period of reduced protection may be
short, but I prefer to avoid it.
>
> Just reloading the whole ipset should be easier.
>
>> _Enabling/Disabling_Blacklists_
>>
>> For changes to the existing firewall rules, the firewall is shut down
>> and restarted from scratch. The blacklist script does this on the fly,
>> since the possibilities are simpler than all the options available for
>> the firewall and also the necessary code for doing updates means that
>> adding this code is fairly simple. This includes turning logging on and
>> off.
>
> What do you mean by that? /etc/init.d/firewall restart?
>
> That is something we cannot do because IPsec and some other services create temporary rules that would be lost.
No. I'm actually trying to say the opposite (badly). The existing
IPFire code only implements some firewall changes by shutting down and
restarting the firewall, but all the IP Blacklist functions
(enable/disable/update lists, enable/disable logging) can be done
without this - the necessary rules can be added to/deleted from the
BLACKLISTRED* chains without affecting any of the rest of the firewall
system.
Tim
>
>> _Logging_
>>
>> The main scripts logs to syslog as ipblacklist, but if you run it from
>> the command line it'll also output to the terminal.
>> There's also some debug code which can be turned on by setting a value
>> in the settings file - it's turned on at the lowest level automatically
>> if you run from the command line.
>
> Yes that is fine.
>
> Best,
> -Michael
>
>> Tim
>>
>> On 06/01/2020 11:27, Michael Tremer wrote:
>>> Hello all,
>>>
>>>> On 4 Jan 2020, at 19:02, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>>
>>>> Hello,
>>>>
>>>> I've now updated the code - screenshots are attached.
>>>>
>>>> IP_Address_Blacklists.png
>>>>
>>>> This is the settings page. I don't think there's anything unexpected here.
>>>
>>> Yes, that looks a lot like the other IPFire pages :)
>>>
>>>> IP_Address_Blacklist_Logs.png
>>>>
>>>> The log page, replacing the status information that was previously at
>>>> the top of the settings page.
>>>
>>> Looks great!
>>>
>>>> I've sorted the lists alphabetically, rather than by the number of hits
>>>> since the worst threat is not necessary the largest number of blocked
>>>> packets (which could implied by sorting it by number).
>>>
>>> Yes, that is indeed a very good idea.
>>>
>>> You could add a % sign in the percentage column, so it cannot be confused so easily with the other column.
>>>
>>>> Firewall_log_blacklist.png
>>>>
>>>> Details page accessed from the log. Almost identical to the other
>>>> similar pages.
>>>
>>> *thumbs up*
>>>
>>>> Log_Summary.png
>>>>
>>>> This is an extract from the Log Summary page produced by logwatch.
>>>>
>>>> I still need to do some tidying, remove unused language strings etc.,
>>>> and then update and test the patches, so it'll be some time before I'm
>>>> ready to submit updated patches.
>>>
>>> Feel free to send some RFC to the list. It might be slightly noisy, but it is easier to review changes bit after bit.
>>>
>>> Great great work!
>>>
>>> -Michael
>>>
>>>>
>>>> Tim
>>>>
>>>> On 28/12/2019 20:59, Tim FitzGeorge wrote:
>>>>> Hi,
>>>>>
>>>>> I'll code and test what we've agreed on. In the meantime I'm going to
>>>>> reply on the sub-thread started by Tom Rymes as to what we call the list
>>>>> categories, so that we can keep that discussion separate from further
>>>>> functionality etc. discussion.
>>>>>
>>>>> Tim
>>>>>
>>>>> On 21/12/2019 18:34, Tim FitzGeorge wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I've got my systems checking for downloads at 15 minute intervals and
>>>>>> using If-Modified-Since. This all seems to be working OK.
>>>>>>
>>>>>> Responses to comments below:
>>>>>>
>>>>>> On 18/12/2019 12:07, Michael Tremer wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>>> On 17 Dec 2019, at 18:21, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I think we've agreed on the following:
>>>>>>>>
>>>>>>>> - Checks to be made every 15 minutes, subject to per-list minimum rate.
>>>>>>>> - Modify download to use If-Modified-Since rather than separate HEAD
>>>>>>>> request.
>>>>>>>> - Remove automatic blacklist.
>>>>>>>
>>>>>>> I thought someone wants to counter my argument. This is not a dictatorship :)
>>>>>>>
>>>>>>>>
>>>>>>>> other comments are below.
>>>>>>>>
>>>>>>>> Tim
>>>>>>>>
>>>>>>>> On 16/12/2019 22:20, Michael Tremer wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>>> On 16 Dec 2019, at 20:06, Tim FitzGeorge <lists(a)tfitzgeorge.me.uk> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I've attached the current GUI screenshot.
>>>>>>>>>
>>>>>>>>> Thanks for that.
>>>>>>>>>
>>>>>>>>> I have a couple of suggestions/concerns about it:
>>>>>>>>>
>>>>>>>>> a) I am not sure what the value is about the top box that is called “Status”. It is basically a summary of the iptables output, but will it help the user? A blacklist with more or fewer entries is not better or worse, blocking more packets isn’t better than blocking fewer. It is about the quality of the blocks.
>>>>>>>>
>>>>>>>> I agree that the size of the lists is not all that useful, but the
>>>>>>>> information about the number of blocked packets can be an indication of
>>>>>>>> compromise. It depends on the type of the list, but a list that targets
>>>>>>>> C&C channels (e.g. the FEODO lists) should not show any input packets
>>>>>>>> being blocked, unless one of the computers being protected is infected.
>>>>>>>> Apart from the BOGON lists, none of the lists should show any output
>>>>>>>> packets being blocked; again it's a potential indication of infection if
>>>>>>>> otherwise.
>>>>>>>
>>>>>>> The Shodan list will definitely show some activity unless you are behind CG NAT.
>>>>>>>
>>>>>>> Should we not have this rather in the logging section?
>>>>>>>
>>>>>>> And these counters here will be reset when the firewall is being reloaded. Should this data therefore not be collected from the logs so that you can go back in time? That would be very important to identify any start of compromise.
>>>>>>>
>>>>>>
>>>>>> That makes sense. I'll remove the status from this page and create a
>>>>>> log page for the information. If anyone actually wants the iptables
>>>>>> information it's available in from Firewall > iptables anyway.
>>>>>>
>>>>>> Do you think it makes more sense to show packets blocked in/out or
>>>>>> packets blocked per source interface? The latter would of (limited)
>>>>>> help in tracking down potential compromise.
>>>>>>
>>>>>>>>>
>>>>>>>>> b) I would prefer to move the settings box to the top. If you like you can show the size of the blacklists there and when they have been updated, too.
>>>>>>>>
>>>>>>>> I put the status at the top since it's usually what I want to see when I
>>>>>>>> go to the page. Under normal circumstances I don't change the settings,
>>>>>>>> so I'd rather not have to scroll past them to get to the information I'm
>>>>>>>> actually interested in. The update times are just useful to show that
>>>>>>>> the lists are actually being updated. The size is interesting to me,
>>>>>>>> but as you say it probably won't help the average user, especially if
>>>>>>>> you don't know which lists block individual addresses and which block
>>>>>>>> nets (BOGON, DSHIELD and EMERGING_COMPROMISED).
>>>>>>>
>>>>>>> See my comment on the order above. Splitting the page might make sense.
>>>>>>>
>>>>>>> When you show size, does that mean lines in the blacklists? So it could be networks or single IP addresses? Does it not make sense to count IP addresses (i.e. 256 for a /24 network and so on) and put it in the table?
>>>>>> It's lines in the blacklist.
>>>>>>
>>>>>>>
>>>>>>> I think I would only be interested in the size for two things:
>>>>>>>
>>>>>>> a) Is there any chance for overblocking? i.e. does the list have 2 billion entries which would be half the public IPv4 address space or so. I cannot really come up with a good example here.
>>>>>>>
>>>>>>> Or
>>>>>>>
>>>>>>> b) What is the performance impact/memory consumption of the list?
>>>>>>
>>>>>> It would be interesting to show the type (single address or net), number
>>>>>> of entries and number of addresses; the first two affect performance,
>>>>>> the second memory use and the latter the number of blocked addresses,
>>>>>> but that's really only going to be of interest to a very small number of
>>>>>> people, who can always do 'ipset list -t' to get the information. So
>>>>>> I'll remove the size column.
>>>>>>
>>>>>>>
>>>>>>>> I originally had the status and settings in one list, but I decided that
>>>>>>>> it was getting a bit messy and difficult to follow. I've attached a
>>>>>>>> screenshot of a quickly hacked page. (Note that the status area only
>>>>>>>> appears once the blacklists have been enabled.)
>>>>>>>
>>>>>>> You are right. This is getting quite tight there.
>>>>>>>
>>>>>>> You could write packets in/out into on column like 2k/1023, but see my concerns about this above.
>>>>>>
>>>>>> Actually I've come to the conclusion that the number of bytes blocked
>>>>>> isn't all that important; it's sufficient to show the number of packets.
>>>>>>
>>>>>>>
>>>>>>>>>
>>>>>>>>> c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
>>>>>>>>
>>>>>>>> I agree that it's it's an over simplistic summary of the lists. I note
>>>>>>>> that Tom Rymes has asked to keep it, but I think its only value is if it
>>>>>>>> suggests to users that things are more complicated than they appear and
>>>>>>>> therefore prompts them to read the Wiki.
>>>>>>>
>>>>>>> I will reply to that separately.
>>>>>>
>>>>>> Looking at that reply, I think it makes sense to include a category
>>>>>> column in both the settings and the log summary page - the former to
>>>>>> help selection of which lists to use, and the latter as a help when
>>>>>> deciding whether the information could indicate compromise or not.
>>>>>>
>>>>>>>
>>>>>>>>>
>>>>>>>>> d) I do not understand what the “update check rate” means. I suppose we should show the update interval of the lists in the table and just refresh them according to that.
>>>>>>>>
>>>>>>>> It's the minimum check period for updates. The system checks for
>>>>>>>> updates at either the rate specific to the list or this rate, whichever
>>>>>>>> is slower. This is a hangover from the early days, copied from the IPS
>>>>>>>> updates. I suppose it could have a value for some people who may wish
>>>>>>>> to lower the check rate, but I'm not sure that it has sufficient value
>>>>>>>> to be worth keeping.
>>>>>>>
>>>>>>> No, I would not let the user decide when and how often to update the lists.
>>>>>>
>>>>>> OK. I'll remove it.
>>>>>>
>>>>>>>
>>>>>>> Generally there is this argument internally how often some databases should be updated. People with slow internet connections or volume based data plans will find this rather expensive to update those lists too often.
>>>>>>>
>>>>>>> If we ever want to go ahead with that in IPFire 2, there should be a global switch for a “low data mode” which then delays updating these things.
>>>>>>
>>>>>> That makes sense. Maybe also the ability to set when downloads take
>>>>>> place - but as you say that's for the future.
>>>>>>
>>>>>>>
>>>>>>>>>
>>>>>>>>> e) Do we want to keep the automatic blacklists now? I do not think it will actually improve the security of IPFire.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I'll remove them. I think they do have a use, in that they can detect
>>>>>>>> someone doing a port scan and block them before they find an open port,
>>>>>>>> but as you say there is also the possibility of a DOS.
>>>>>>>
>>>>>>> How would this block a DOS?
>>>>>>
>>>>>> Sorry, I didn't make myself clear - there's the possibility that someone
>>>>>> could cause a DOS with the automatic list by faking the source IP
>>>>>> address and sending a few packets on a random port.
>>>>>>
>>>>>>>
>>>>>>> -Michael
>>>>>>
>>>>>> Tim
>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I'll update the code based on our discussions, and submit an updated set
>>>>>>>>>> of patches - I imagine there will have to be at least one more iteration.
>>>>>>>>>
>>>>>>>>> Let’s wait until we have come to decisions :)
>>>>>>>>>
>>>>>>>>> -Michael
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Tim
>>>>>>>>>>
>>>>>>>>>> (No additional comments below)
>>>>>>>>>>
>>>>>>>>>> On 13/12/2019 23:11, Michael Tremer wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> Again my apologies for my late reply. Busy busy weeks.
>>>>>>>>>>>
>>>>>>>>>>>> On 8 Dec 2019, at 20:50, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>
>>>>>>>>>>>> It's my turn to apologise for being slow to respond - I've had a busy
>>>>>>>>>>>> week, but I should have plenty of time over the next couple of weeks.
>>>>>>>>>>>
>>>>>>>>>>> No worries. Turns out we all do :)
>>>>>>>>>>>
>>>>>>>>>>>> I've made most of the comments inline, however I think Michael had a
>>>>>>>>>>>> question (which I can't find now) about what happens if someone enables
>>>>>>>>>>>> all the lists. One thing which would perhaps make this less likely is
>>>>>>>>>>>> that the WUI tags the available lists with whether they're safe or not,
>>>>>>>>>>>> with a footnote that safe means that the list only blocks malicious
>>>>>>>>>>>> traffic. This won't guarantee that a user won't still try to enable all
>>>>>>>>>>>> the lists, but it should make them realise that they should think first.
>>>>>>>>>>>
>>>>>>>>>>> We had a couple of features going slightly wrong or being “misunderstood” by some users. People still seem to panic when they see “local recursor”. To this day I do not know why.
>>>>>>>>>>>
>>>>>>>>>>> We cannot make everything idiot-proof. And when some user if of that category, they probably should shutdown their IPFire box, educate themselves and then come back again. So I do not want to limit people, but make things as easy as possible.
>>>>>>>>>>>
>>>>>>>>>>> If someone enables all the lists, good luck with passing packets :)
>>>>>>>>>>>
>>>>>>>>>>>> I have considered replacing this tag with a risk high/medium/low and
>>>>>>>>>>>> maybe adding a category (invalid/application/scanner/C&C or something
>>>>>>>>>>>> like that), but that may provide too much information and dissuade them
>>>>>>>>>>>> from actually following the links to checkout what the list actually does.
>>>>>>>>>>>
>>>>>>>>>>> Can we have a screenshot of the GUI right now? I didn’t run the code, yet.
>>>>>>>>>>>
>>>>>>>>>>> We should document the lists like we do it with the rulesets of the IPS. People might ignore this, but that is on them.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Tim
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 05/12/2019 22:25, Michael Tremer wrote:
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 4 Dec 2019, at 17:05, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hello Tim, hello Michael,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> please see my responses inline...
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We could periodically update the blacklists on our main mirror (and
>>>>>>>>>>>>>>>>> wait for the network to sync it), make sure it is signed and write
>>>>>>>>>>>>>>>>> a small downloader that fetches, validates and installs them.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> @All: Thoughts on this?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I think there are a number of points here.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Firstly, from the point of a third party using IPFire, is this really
>>>>>>>>>>>>>>>> solving the privacy disclosure problem? There's no way round disclosing
>>>>>>>>>>>>>>>> your public IP address to someone you're downloading from; all this does
>>>>>>>>>>>>>>>> is change who that information is being disclosed to. For the user
>>>>>>>>>>>>>>>> there's no way of knowing whether the source is more or less protective
>>>>>>>>>>>>>>>> of the user's privacy than the blacklist provider. Indeed it won't be
>>>>>>>>>>>>>>>> possible to know who the lists are being downloaded from until the
>>>>>>>>>>>>>>>> download starts.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> There is a way: Tor. But that is a totally different story.
>>>>>>>>>>>>>> Well, I see a third option on this: Use the mirror infrastructure we already
>>>>>>>>>>>>>> have. Every IPFire installation discloses its public IP address to one of these
>>>>>>>>>>>>>> servers sooner or later, so we do not disclose additional data if the blacklists
>>>>>>>>>>>>>> were fetched from these.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Needless to say, Tor (hidden services) would be better, but that is a different
>>>>>>>>>>>>>> story indeed. :-)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The point is rather that a forget list can be sent instead of the “real” one.
>>>>>>>>>>>>>> I did not get this. Forget? Forged?? ???
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes, I meant to write forged, but auto-correct didn’t let me.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Secondly, latency; some of the lists are updated every 5 minutes. While
>>>>>>>>>>>>>>>> I've limited the maximum check rate to hourly, will the updates
>>>>>>>>>>>>>>>> propagate quickly enough. For reference on my main system the 24
>>>>>>>>>>>>>>>> updates on the CIARMY list made 143 498 changes (additions or
>>>>>>>>>>>>>>>> deletions). I've seem it do over 200 000.
>>>>>>>>>>>>>> Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not
>>>>>>>>>>>>>> document a recommended update interval anywhere, so we can only guess.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Personally, more static lists seem to be preferable for packet filtering. Highly
>>>>>>>>>>>>>> dynamic ones such as CIArmy should be done via DNSBL queries or something similar
>>>>>>>>>>>>>> - do we really want to have that list here?
>>>>>>>>>>>>>
>>>>>>>>>>>>> It is not really an option to implement a DNSBL into a packet filter, but I get your point.
>>>>>>>>>>>>
>>>>>>>>>>>> One of the 'selling points' for an IP address blacklist is that it can
>>>>>>>>>>>> respond quickly to new threats - or rather new attackers. While a new
>>>>>>>>>>>> IDS/IPS rule needs time to analyse the threat, generate a rule and check
>>>>>>>>>>>> it, it's easy to add an address to a list. So, I think the CIArmy list
>>>>>>>>>>>> is potentially useful for protecting home systems etc. with budget
>>>>>>>>>>>> hardware, but I would be very careful about using it for a protecting a
>>>>>>>>>>>> general access website.
>>>>>>>>>>>
>>>>>>>>>>> If they are very volatile, we should honour that and update them often, too.
>>>>>>>>>>>
>>>>>>>>>>> It probably is more about false-positives being removed very quickly instead of adding threats very quickly. The average IPFire user is probably not under threats like these to need to react very quickly.
>>>>>>>>>>>
>>>>>>>>>>> So I do not see much value in adding those lists and then updating once a day. Does it have to be every 5 min? No. I would suggest 15 which should be good enough for everyone.
>>>>>>>>>>>
>>>>>>>>>>> Other lists should of course not be updated every 15 minutes when not needed.
>>>>>>>>>>>
>>>>>>>>>>> Running every 15 minutes would allow us to retry downloading lists that are on an hourly schedule if the download failed.
>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>> How did you come up with the hour? Will it be retried more often if the download was not successful?
>>>>>>>>>>>>>> One hour is the most common interval indeed, but adding some random time might
>>>>>>>>>>>>>> be useful in order to reduce load on the servers providing a blacklist.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes, definitely. Otherwise we will shoot down our mirrors.
>>>>>>>>>>>>
>>>>>>>>>>>> When I implemented that section of code, specifying the minimum check
>>>>>>>>>>>> period in hours seemed to provide a convenient way of allowing a check
>>>>>>>>>>>> period covering a wide range, with an hour as the fastest and a week as
>>>>>>>>>>>> the slowest. I didn't looked at the CIArmy list until much later. Most
>>>>>>>>>>>> of the lists don't change nearly as much, but the CIArmy list is
>>>>>>>>>>>> described as one that deliberately responds quickly.
>>>>>>>>>>>>
>>>>>>>>>>>> From my production system, for yesterday:
>>>>>>>>>>>>
>>>>>>>>>>>> The following block lists were updated:
>>>>>>>>>>>> BLOCKLIST_DE: 24 Time(s) - 9341 change(s)
>>>>>>>>>>>> BOGON_FULL: 1 Time(s) - 10 change(s)
>>>>>>>>>>>> CIARMY: 24 Time(s) - 159134 change(s)
>>>>>>>>>>>> DSHIELD: 7 Time(s) - 18 change(s)
>>>>>>>>>>>> EMERGING_FWRULE: 1 Time(s) - 50 change(s)
>>>>>>>>>>>> FEODO_AGGRESIVE: 24 Time(s) - 13 change(s)
>>>>>>>>>>>> SHODAN: 1 Time(s) - 0 change(s)
>>>>>>>>>>>> SPAMHAUS_DROP: 1 Time(s) - 0 change(s)
>>>>>>>>>>>> TOR_EXIT: 24 Time(s) - 162 change(s)
>>>>>>>>>>>
>>>>>>>>>>> Very interesting statistics.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> and my test system:
>>>>>>>>>>>>
>>>>>>>>>>>> The following block lists were updated:
>>>>>>>>>>>> ALIENVAULT: 19 Time(s) - 5331 change(s)
>>>>>>>>>>>> EMERGING_COMPROMISED: 1 Time(s) - 26 change(s)
>>>>>>>>>>>> TALOS_MALICIOUS: 1 Time(s) - 36 change(s)
>>>>>>>>>>>>
>>>>>>>>>>>> That covers most of the lists. From the WUI, since 1 Dec:
>>>>>>>>>>>>
>>>>>>>>>>>> Blacklist Entries pkts bytes Last updated
>>>>>>>>>>>> in in
>>>>>>>>>>>> AUTOBLACKLIST 0 731 51144 Sun Dec 8 18:40:02 2019
>>>>>>>>>>>> BLOCKLIST_DE 28020 857 46735 Sun Dec 8 18:40:02 2019
>>>>>>>>>>>> BOGON_FULL 214 5255 189K Sun Dec 8 16:50:04 2019
>>>>>>>>>>>> CIARMY 15000 19774 976K Sun Dec 8 18:04:01 2019
>>>>>>>>>>>> DSHIELD 20 7992 321K Sun Dec 8 16:45:13 2019
>>>>>>>>>>>> EMERGING_FWRULE 1647 197 8383 Fri Dec 6 05:29:07 2019
>>>>>>>>>>>> FEODO_AGGRESIVE 7169 0 0 Sun Dec 8 18:50:07 2019
>>>>>>>>>>>> SHODAN 32 34 1530 Sun Dec 8 17:54:09 2019
>>>>>>>>>>>> SPAMHAUS_DROP 823 0 0 Fri Dec 6 18:22:35 2019
>>>>>>>>>>>> SPAMHAUS_EDROP 111 82 16433 Thu Dec 5 17:27:09 2019
>>>>>>>>>>>> TOR_EXIT 1055 0 0 Sun Dec 8 18:31:02 2019
>>>>>>>>>>>
>>>>>>>>>>> This as well.
>>>>>>>>>>>
>>>>>>>>>>> Those are more packets than I would have expected.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> (I've left out the pkts/bytes out fields which were all 0)
>>>>>>>>>>>>
>>>>>>>>>>>> Note that where possible I do a HEAD request first and then only
>>>>>>>>>>>> download the list if the modification time has changed since the last
>>>>>>>>>>>> check. For dynamically generated lists this isn't possible.
>>>>>>>>>>>
>>>>>>>>>>> You won’t need a HEAD request for it. You can include it in the GET request.
>>>>>>>>>>>
>>>>>>>>>>> Have a look at location downloader where I use that. The server will respond with 304 and not send any payload.
>>>>>>>>>>>
>>>>>>>>>>> https://git.ipfire.org/?p=location/libloc.git;a=blob;f=src/python/location-downloader.in;h=1b5932d5822e03737d32b2a27816815b2f7e74dd;hb=HEAD#l151
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> If the download isn't successful it just gives up and waits for the next
>>>>>>>>>>>> attempt (apart from the usual retries in the library). I probably
>>>>>>>>>>>> should to change that so that it only applies the per list minimum
>>>>>>>>>>>> update period in this case (specified in the sources file) rather than
>>>>>>>>>>>> the user specified value as well.
>>>>>>>>>>>
>>>>>>>>>>> I think it is not the worst if an update fails. It might just happen every once in a while.
>>>>>>>>>>>
>>>>>>>>>>> So I would suggest to just re-run the script more often and when the mtime of the file is older than the threshold, a download is attempted. You can use that timestamp for the GET request.
>>>>>>>>>>>
>>>>>>>>>>>> I already use a time offset on the downloads - when it's started from
>>>>>>>>>>>> boot, backup restore or WUI enable, it checks to see if it's installed
>>>>>>>>>>>> in the fcrontab, and if not adds itself at a randomly generated offset
>>>>>>>>>>>> in the hour.
>>>>>>>>>>>
>>>>>>>>>>> That should potentially go to red.up, or if we can settle on 15 minutes, I would consider that often enough to quickly update all lists after a reboot.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is
>>>>>>>>>>>>>>>> the largest at a few MB), there are going to be a lot of them. How will
>>>>>>>>>>>>>>>> this affect the willingness of people to mirror IPFire?
>>>>>>>>>>>>>> I do not consider this being a problem as we do not generate that much traffic
>>>>>>>>>>>>>> to them. Of course, that depends on the update interval again.
>>>>>>>>>>>>>
>>>>>>>>>>>>> That depends on your point of view.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I do not have a problem with this at all in my data center, but there are plenty of people with a volume-based LTE plan or simply a 128 kBit/s connection. It will take a longer time to download the lists for them. We need to mind that.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Talking about the preference of packet filter and IPS, I prefer to
>>>>>>>>>>>>>>>>> use the latter as well as it gains more insight in what kind of malicious
>>>>>>>>>>>>>>>>> traffic tried to pass a firewall machine. On systems with low resources,
>>>>>>>>>>>>>>>>> this might be problematic and removing load from the IPS can be preferred
>>>>>>>>>>>>>>>>> (make this configurable?!), on others, people might want to have both
>>>>>>>>>>>>>>>>> results.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> You're only going to get one result for a packet whichever way round the
>>>>>>>>>>>>>>>> IP blacklist and IPS are since whichever comes first will drop the
>>>>>>>>>>>>>>>> packet before it reaches the second (well it would be possible to put
>>>>>>>>>>>>>>>> the IP blacklist first and get it to log and mark packets which are then
>>>>>>>>>>>>>>>> dropped after the IPS, but I think that's getting a little complicated.
>>>>>>>>>>>>>>>> In addition I've seen the messages about the trouble marking was
>>>>>>>>>>>>>>>> causing in the QoS).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I think it's a 50/50 choice as to which is more valuable first; it's
>>>>>>>>>>>>>>>> probably going to differ from packet to packet. For me the possibility
>>>>>>>>>>>>>>>> of reducing the IPS load means I prefer putting the IP blacklist first
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It should be fairly easy to add the choice of where to put the IP
>>>>>>>>>>>>>>>> blacklist. I think it'll have to be in the main firewall script, so
>>>>>>>>>>>>>>>> it'll require a firewall restart, but it's not something that'll be
>>>>>>>>>>>>>>>> changed often.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier.
>>>>>>>>>>>>>> Agreed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily.
>>>>>>>>>>>>>> So dropping packets from blacklisted IP addresses/networks before IPS is it, then.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I do not even think it makes sense to swap the order in the outgoing direction.
>>>>>>>>>>>>>> Me too.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this.
>>>>>>>>>>>>>> Based on my personal experience, firewall hits observed on a single machine exposed
>>>>>>>>>>>>>> to the internet are interesting, but the overall situation across multiple machines
>>>>>>>>>>>>>> is even more interesting. Very quickly, you'll end on something like a centralised
>>>>>>>>>>>>>> logging server and custom statistical analysis here...
>>>>>>>>>>>>>
>>>>>>>>>>>>> Probably a project for IPFire 4.0 :)
>>>>>>>>>>>>>
>>>>>>>>>>>> Or use one of the existing services, like the DSHIELD client
>>>>>>>>>>>> https://dshield.org/howto.html (subject to privacy concerns again).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default.
>>>>>>>>>>>>>>>>> I would love to see the bogon ruleset here, too (think about 8chan successor
>>>>>>>>>>>>>>>>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause
>>>>>>>>>>>>>>>>> interference if RED does not have a public IP address assigned.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I can add a field to the options file that controls whether a list is
>>>>>>>>>>>>>>>> enabled by default.
>>>>>>>>>>>>>> Thank you. :-)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat.
>>>>>>>>>>>>>> This is my only objection against this patchset. Now, what can we do about it?
>>>>>>>>>>>>>> One possibility is to apply the patchset now and implement a custom download
>>>>>>>>>>>>>> source thing later on, or do that before releasing Core Update 139 (or which version
>>>>>>>>>>>>>> the patchset will be to) after we agreed on something.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I do not see this being merged for 139. But that is not important. We need to get it right first and then release it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> As far as I know, nobody has tested this, yet.
>>>>>>>>>>>>
>>>>>>>>>>>> There are a number of people who have been running an earlier version
>>>>>>>>>>>> which I shared on GitHub. There were a few early issues, but it seems
>>>>>>>>>>>> to be OK now.
>>>>>>>>>>>>
>>>>>>>>>>>> https://forum.ipfire.org/viewtopic.php?f=27&t=21845
>>>>>>>>>>>>
>>>>>>>>>>>> This version wasn't integrated into IPFire, so (for example) it inserted
>>>>>>>>>>>> itself into the INPUT IPTables chain rather than having it's chains
>>>>>>>>>>>> created as part of the firewall start-up script.
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have huge concerns about the automatic blacklist. @Peter: What is your opinion on this?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> While I implemented it, I'm aware of its potential to cause problems,
>>>>>>>>>>>> which is why it has to be separately enabled. It's not caused me any
>>>>>>>>>>>> issues at the default settings (blocks at over 10 packets per hour until
>>>>>>>>>>>> 1 hour has passed without seeing packets from the address), but I've not
>>>>>>>>>>>> used it on a site with publicly announced services. If I was going to
>>>>>>>>>>>> use it on a web site I would want to, at the very minimum, drop the
>>>>>>>>>>>> block period drastically.
>>>>>>>>>>>
>>>>>>>>>>> I suppose this is entirely unusable on an IPFire box in a data center that hosts things. Let’s say our rack in Hanover.
>>>>>>>>>>>
>>>>>>>>>>> You will have hits from some broken IP stacks and people might just end up on this without doing anything wrong.
>>>>>>>>>>>
>>>>>>>>>>> You can denial-of-service easily as well and I suppose without aggregation of data from many many systems, it does not make sense to instantly block addresses. And even then you probably would block an entire subnet.
>>>>>>>>>>>
>>>>>>>>>>> So I am not sure how I should feel about it. I do not think that it adds anything because the packets would have been blocked anyways.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On the other hand, it's good at responding quickly. Usually I see only
>>>>>>>>>>>> 1-2% of blocks from the automatic list:
>>>>>>>>>>>>
>>>>>>>>>>>> Reason Count % First Last
>>>>>>>>>>>> CIARMY 2416 45 Dec 6 00:00 Dec 6 23:59
>>>>>>>>>>>> DSHIELD 1353 25 Dec 6 00:00 Dec 6 23:59
>>>>>>>>>>>> INPUT 1294 24 Dec 6 00:00 Dec 6 23:59
>>>>>>>>>>>> AUTOBLACKLIST 122 2 Dec 6 00:20 Dec 6 16:28
>>>>>>>>>>>> BLOCKLIST_DE 89 2 Dec 6 00:20 Dec 6 23:46
>>>>>>>>>>>>
>>>>>>>>>>>> and sometimes none at all, but one one occasion it blocked over 8000
>>>>>>>>>>>> packets. Again I'm aware this is for a home system, which is rather
>>>>>>>>>>>> different than from a Web server.
>>>>>>>>>>>>
>>>>>>>>>>>>>> If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not
>>>>>>>>>>>>>> seem to be problematic, as they are hosted on 3rd party servers, too).
>>>>>>>>>>>>>
>>>>>>>>>>>>> One of them will be, sooner or later. And one is enough I suppose.
>>>>>>>>>>>>
>>>>>>>>>>>> DShield (https://dshield.org/api/#threatfeeds) and firehol
>>>>>>>>>>>> (http://iplists.firehol.org/) seem to host copies of most of the lists
>>>>>>>>>>>> as well.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I do not really want to overthink this - we didn’t do this with clamav for example either. But it probably is a decision to be made by the user what they want to enable and we should not enable anything by default. So no data will be leaked as long as the user does not consent.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks, and best regards,
>>>>>>>>>>>>>> Peter Müller
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> <Screenshot_2019-12-16_IP_Address_Blacklists.png>
>>>>>>>>>
>>>>>>>>
>>>>>>>> <Screenshot_2019-12-17_IP_Address_Blacklists.png>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>> <IP_Address_Blacklists.png><IP_Address_Blacklist_Logs.png><Firewall_log_blacklist.png><Log_Summary.png>
>>>
>>
>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 0/5] ipblacklist: IP Address Blacklists
2020-01-24 19:40 ` Tim FitzGeorge
@ 2020-01-28 17:14 ` Michael Tremer
2020-01-29 20:40 ` Tim FitzGeorge
0 siblings, 1 reply; 35+ messages in thread
From: Michael Tremer @ 2020-01-28 17:14 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 36570 bytes --]
Hi,
Apologies for my late reply again. I marked this email in my inbox and then it scrolled down because of loads of new stuff coming in. Please feel free to ping me if I do not respond in time.
> On 24 Jan 2020, at 19:40, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>
> I'm now nearly ready to submit the updated patches, but before then I'd
> like to check that you're happy with some of my design decisions.
Cool.
> _IPTables_Chains_
>
> When setting up the firewall (in iptables_init()), four new chains are
> created, IPBLACKLISTIN, IPBLACKLISTOUT, IPBLACKLISTREDIN and
> IPBLACKLISTREDOUT. The first two are inserted into the INPUT, OUTPUT
> and FORWARD chains, before the IPS chains. The main script is called to
> restore the previously saved blacklists, creating the rules in the
> latter two chains.
For readability you can simply call them BLACKLIST*. The IP is a given.
> In iptables_red_up(), rules are created to forward traffic to/from the
> red interface from IPBLACKLISTIN/IPBLACKLISTOUT to
> IPBLACKLISTREDIN/IPBLACKLISTREDOUT.
> I did consider putting the blacklist rules into the
> IPBLACKLISTIN/IPBLACKLISTOUT, but this couldn't be done until red comes
> up and would require much more processing at this point - not a problem
> if your red interface comes up and stays up, but if you've got one that
> disconnects on no traffic you probably don't want the extra overhead.
> The way I've done it also has the side effects of 1) the main script
> doesn't have to worry about what the red interface is, and 2) if you're
> using multiple blacklists, the check for the red interface is only done
> once.
You would only want to check the interface once and then jump into the chain. Can we not realise it that way?
> _Log_WUI_Pages_
>
> The two Log WUI pages are edited from existing pages. I've resisted the
> temptation to reformat them, which means that they can be diff'ed with
> the corresponding pages.
Loads of the UI code is messy and deliberately not touched any more. I guess this is fine.
> _Update_
>
> The update is fairly simple - it calculates whether the check period is
> passed, tries a download specifying the If-Modified-Since and parses the
> returned file. It then compares the new list with the existing list,
> adding new addresses/networks and deleting those that have been removed.
> There's a check to handle a list changing from a list of addresses to a
> list of networks, in which case it changes the type of the IPSet - this
> should be a vary rare event, but could happen for a composite list.
Why do you need the diff? Has that been in the first patchset?
Just reloading the whole ipset should be easier.
> _Enabling/Disabling_Blacklists_
>
> For changes to the existing firewall rules, the firewall is shut down
> and restarted from scratch. The blacklist script does this on the fly,
> since the possibilities are simpler than all the options available for
> the firewall and also the necessary code for doing updates means that
> adding this code is fairly simple. This includes turning logging on and
> off.
What do you mean by that? /etc/init.d/firewall restart?
That is something we cannot do because IPsec and some other services create temporary rules that would be lost.
> _Logging_
>
> The main scripts logs to syslog as ipblacklist, but if you run it from
> the command line it'll also output to the terminal.
> There's also some debug code which can be turned on by setting a value
> in the settings file - it's turned on at the lowest level automatically
> if you run from the command line.
Yes that is fine.
Best,
-Michael
> Tim
>
> On 06/01/2020 11:27, Michael Tremer wrote:
>> Hello all,
>>
>>> On 4 Jan 2020, at 19:02, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>
>>> Hello,
>>>
>>> I've now updated the code - screenshots are attached.
>>>
>>> IP_Address_Blacklists.png
>>>
>>> This is the settings page. I don't think there's anything unexpected here.
>>
>> Yes, that looks a lot like the other IPFire pages :)
>>
>>> IP_Address_Blacklist_Logs.png
>>>
>>> The log page, replacing the status information that was previously at
>>> the top of the settings page.
>>
>> Looks great!
>>
>>> I've sorted the lists alphabetically, rather than by the number of hits
>>> since the worst threat is not necessary the largest number of blocked
>>> packets (which could implied by sorting it by number).
>>
>> Yes, that is indeed a very good idea.
>>
>> You could add a % sign in the percentage column, so it cannot be confused so easily with the other column.
>>
>>> Firewall_log_blacklist.png
>>>
>>> Details page accessed from the log. Almost identical to the other
>>> similar pages.
>>
>> *thumbs up*
>>
>>> Log_Summary.png
>>>
>>> This is an extract from the Log Summary page produced by logwatch.
>>>
>>> I still need to do some tidying, remove unused language strings etc.,
>>> and then update and test the patches, so it'll be some time before I'm
>>> ready to submit updated patches.
>>
>> Feel free to send some RFC to the list. It might be slightly noisy, but it is easier to review changes bit after bit.
>>
>> Great great work!
>>
>> -Michael
>>
>>>
>>> Tim
>>>
>>> On 28/12/2019 20:59, Tim FitzGeorge wrote:
>>>> Hi,
>>>>
>>>> I'll code and test what we've agreed on. In the meantime I'm going to
>>>> reply on the sub-thread started by Tom Rymes as to what we call the list
>>>> categories, so that we can keep that discussion separate from further
>>>> functionality etc. discussion.
>>>>
>>>> Tim
>>>>
>>>> On 21/12/2019 18:34, Tim FitzGeorge wrote:
>>>>> Hi,
>>>>>
>>>>> I've got my systems checking for downloads at 15 minute intervals and
>>>>> using If-Modified-Since. This all seems to be working OK.
>>>>>
>>>>> Responses to comments below:
>>>>>
>>>>> On 18/12/2019 12:07, Michael Tremer wrote:
>>>>>> Hi,
>>>>>>
>>>>>>> On 17 Dec 2019, at 18:21, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I think we've agreed on the following:
>>>>>>>
>>>>>>> - Checks to be made every 15 minutes, subject to per-list minimum rate.
>>>>>>> - Modify download to use If-Modified-Since rather than separate HEAD
>>>>>>> request.
>>>>>>> - Remove automatic blacklist.
>>>>>>
>>>>>> I thought someone wants to counter my argument. This is not a dictatorship :)
>>>>>>
>>>>>>>
>>>>>>> other comments are below.
>>>>>>>
>>>>>>> Tim
>>>>>>>
>>>>>>> On 16/12/2019 22:20, Michael Tremer wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>>> On 16 Dec 2019, at 20:06, Tim FitzGeorge <lists(a)tfitzgeorge.me.uk> wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I've attached the current GUI screenshot.
>>>>>>>>
>>>>>>>> Thanks for that.
>>>>>>>>
>>>>>>>> I have a couple of suggestions/concerns about it:
>>>>>>>>
>>>>>>>> a) I am not sure what the value is about the top box that is called “Status”. It is basically a summary of the iptables output, but will it help the user? A blacklist with more or fewer entries is not better or worse, blocking more packets isn’t better than blocking fewer. It is about the quality of the blocks.
>>>>>>>
>>>>>>> I agree that the size of the lists is not all that useful, but the
>>>>>>> information about the number of blocked packets can be an indication of
>>>>>>> compromise. It depends on the type of the list, but a list that targets
>>>>>>> C&C channels (e.g. the FEODO lists) should not show any input packets
>>>>>>> being blocked, unless one of the computers being protected is infected.
>>>>>>> Apart from the BOGON lists, none of the lists should show any output
>>>>>>> packets being blocked; again it's a potential indication of infection if
>>>>>>> otherwise.
>>>>>>
>>>>>> The Shodan list will definitely show some activity unless you are behind CG NAT.
>>>>>>
>>>>>> Should we not have this rather in the logging section?
>>>>>>
>>>>>> And these counters here will be reset when the firewall is being reloaded. Should this data therefore not be collected from the logs so that you can go back in time? That would be very important to identify any start of compromise.
>>>>>>
>>>>>
>>>>> That makes sense. I'll remove the status from this page and create a
>>>>> log page for the information. If anyone actually wants the iptables
>>>>> information it's available in from Firewall > iptables anyway.
>>>>>
>>>>> Do you think it makes more sense to show packets blocked in/out or
>>>>> packets blocked per source interface? The latter would of (limited)
>>>>> help in tracking down potential compromise.
>>>>>
>>>>>>>>
>>>>>>>> b) I would prefer to move the settings box to the top. If you like you can show the size of the blacklists there and when they have been updated, too.
>>>>>>>
>>>>>>> I put the status at the top since it's usually what I want to see when I
>>>>>>> go to the page. Under normal circumstances I don't change the settings,
>>>>>>> so I'd rather not have to scroll past them to get to the information I'm
>>>>>>> actually interested in. The update times are just useful to show that
>>>>>>> the lists are actually being updated. The size is interesting to me,
>>>>>>> but as you say it probably won't help the average user, especially if
>>>>>>> you don't know which lists block individual addresses and which block
>>>>>>> nets (BOGON, DSHIELD and EMERGING_COMPROMISED).
>>>>>>
>>>>>> See my comment on the order above. Splitting the page might make sense.
>>>>>>
>>>>>> When you show size, does that mean lines in the blacklists? So it could be networks or single IP addresses? Does it not make sense to count IP addresses (i.e. 256 for a /24 network and so on) and put it in the table?
>>>>> It's lines in the blacklist.
>>>>>
>>>>>>
>>>>>> I think I would only be interested in the size for two things:
>>>>>>
>>>>>> a) Is there any chance for overblocking? i.e. does the list have 2 billion entries which would be half the public IPv4 address space or so. I cannot really come up with a good example here.
>>>>>>
>>>>>> Or
>>>>>>
>>>>>> b) What is the performance impact/memory consumption of the list?
>>>>>
>>>>> It would be interesting to show the type (single address or net), number
>>>>> of entries and number of addresses; the first two affect performance,
>>>>> the second memory use and the latter the number of blocked addresses,
>>>>> but that's really only going to be of interest to a very small number of
>>>>> people, who can always do 'ipset list -t' to get the information. So
>>>>> I'll remove the size column.
>>>>>
>>>>>>
>>>>>>> I originally had the status and settings in one list, but I decided that
>>>>>>> it was getting a bit messy and difficult to follow. I've attached a
>>>>>>> screenshot of a quickly hacked page. (Note that the status area only
>>>>>>> appears once the blacklists have been enabled.)
>>>>>>
>>>>>> You are right. This is getting quite tight there.
>>>>>>
>>>>>> You could write packets in/out into on column like 2k/1023, but see my concerns about this above.
>>>>>
>>>>> Actually I've come to the conclusion that the number of bytes blocked
>>>>> isn't all that important; it's sufficient to show the number of packets.
>>>>>
>>>>>>
>>>>>>>>
>>>>>>>> c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
>>>>>>>
>>>>>>> I agree that it's it's an over simplistic summary of the lists. I note
>>>>>>> that Tom Rymes has asked to keep it, but I think its only value is if it
>>>>>>> suggests to users that things are more complicated than they appear and
>>>>>>> therefore prompts them to read the Wiki.
>>>>>>
>>>>>> I will reply to that separately.
>>>>>
>>>>> Looking at that reply, I think it makes sense to include a category
>>>>> column in both the settings and the log summary page - the former to
>>>>> help selection of which lists to use, and the latter as a help when
>>>>> deciding whether the information could indicate compromise or not.
>>>>>
>>>>>>
>>>>>>>>
>>>>>>>> d) I do not understand what the “update check rate” means. I suppose we should show the update interval of the lists in the table and just refresh them according to that.
>>>>>>>
>>>>>>> It's the minimum check period for updates. The system checks for
>>>>>>> updates at either the rate specific to the list or this rate, whichever
>>>>>>> is slower. This is a hangover from the early days, copied from the IPS
>>>>>>> updates. I suppose it could have a value for some people who may wish
>>>>>>> to lower the check rate, but I'm not sure that it has sufficient value
>>>>>>> to be worth keeping.
>>>>>>
>>>>>> No, I would not let the user decide when and how often to update the lists.
>>>>>
>>>>> OK. I'll remove it.
>>>>>
>>>>>>
>>>>>> Generally there is this argument internally how often some databases should be updated. People with slow internet connections or volume based data plans will find this rather expensive to update those lists too often.
>>>>>>
>>>>>> If we ever want to go ahead with that in IPFire 2, there should be a global switch for a “low data mode” which then delays updating these things.
>>>>>
>>>>> That makes sense. Maybe also the ability to set when downloads take
>>>>> place - but as you say that's for the future.
>>>>>
>>>>>>
>>>>>>>>
>>>>>>>> e) Do we want to keep the automatic blacklists now? I do not think it will actually improve the security of IPFire.
>>>>>>>>
>>>>>>>
>>>>>>> I'll remove them. I think they do have a use, in that they can detect
>>>>>>> someone doing a port scan and block them before they find an open port,
>>>>>>> but as you say there is also the possibility of a DOS.
>>>>>>
>>>>>> How would this block a DOS?
>>>>>
>>>>> Sorry, I didn't make myself clear - there's the possibility that someone
>>>>> could cause a DOS with the automatic list by faking the source IP
>>>>> address and sending a few packets on a random port.
>>>>>
>>>>>>
>>>>>> -Michael
>>>>>
>>>>> Tim
>>>>>>
>>>>>>>>>
>>>>>>>>> I'll update the code based on our discussions, and submit an updated set
>>>>>>>>> of patches - I imagine there will have to be at least one more iteration.
>>>>>>>>
>>>>>>>> Let’s wait until we have come to decisions :)
>>>>>>>>
>>>>>>>> -Michael
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Tim
>>>>>>>>>
>>>>>>>>> (No additional comments below)
>>>>>>>>>
>>>>>>>>> On 13/12/2019 23:11, Michael Tremer wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Again my apologies for my late reply. Busy busy weeks.
>>>>>>>>>>
>>>>>>>>>>> On 8 Dec 2019, at 20:50, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>> It's my turn to apologise for being slow to respond - I've had a busy
>>>>>>>>>>> week, but I should have plenty of time over the next couple of weeks.
>>>>>>>>>>
>>>>>>>>>> No worries. Turns out we all do :)
>>>>>>>>>>
>>>>>>>>>>> I've made most of the comments inline, however I think Michael had a
>>>>>>>>>>> question (which I can't find now) about what happens if someone enables
>>>>>>>>>>> all the lists. One thing which would perhaps make this less likely is
>>>>>>>>>>> that the WUI tags the available lists with whether they're safe or not,
>>>>>>>>>>> with a footnote that safe means that the list only blocks malicious
>>>>>>>>>>> traffic. This won't guarantee that a user won't still try to enable all
>>>>>>>>>>> the lists, but it should make them realise that they should think first.
>>>>>>>>>>
>>>>>>>>>> We had a couple of features going slightly wrong or being “misunderstood” by some users. People still seem to panic when they see “local recursor”. To this day I do not know why.
>>>>>>>>>>
>>>>>>>>>> We cannot make everything idiot-proof. And when some user if of that category, they probably should shutdown their IPFire box, educate themselves and then come back again. So I do not want to limit people, but make things as easy as possible.
>>>>>>>>>>
>>>>>>>>>> If someone enables all the lists, good luck with passing packets :)
>>>>>>>>>>
>>>>>>>>>>> I have considered replacing this tag with a risk high/medium/low and
>>>>>>>>>>> maybe adding a category (invalid/application/scanner/C&C or something
>>>>>>>>>>> like that), but that may provide too much information and dissuade them
>>>>>>>>>>> from actually following the links to checkout what the list actually does.
>>>>>>>>>>
>>>>>>>>>> Can we have a screenshot of the GUI right now? I didn’t run the code, yet.
>>>>>>>>>>
>>>>>>>>>> We should document the lists like we do it with the rulesets of the IPS. People might ignore this, but that is on them.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Tim
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 05/12/2019 22:25, Michael Tremer wrote:
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>
>>>>>>>>>>>>> On 4 Dec 2019, at 17:05, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hello Tim, hello Michael,
>>>>>>>>>>>>>
>>>>>>>>>>>>> please see my responses inline...
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We could periodically update the blacklists on our main mirror (and
>>>>>>>>>>>>>>>> wait for the network to sync it), make sure it is signed and write
>>>>>>>>>>>>>>>> a small downloader that fetches, validates and installs them.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> @All: Thoughts on this?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think there are a number of points here.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Firstly, from the point of a third party using IPFire, is this really
>>>>>>>>>>>>>>> solving the privacy disclosure problem? There's no way round disclosing
>>>>>>>>>>>>>>> your public IP address to someone you're downloading from; all this does
>>>>>>>>>>>>>>> is change who that information is being disclosed to. For the user
>>>>>>>>>>>>>>> there's no way of knowing whether the source is more or less protective
>>>>>>>>>>>>>>> of the user's privacy than the blacklist provider. Indeed it won't be
>>>>>>>>>>>>>>> possible to know who the lists are being downloaded from until the
>>>>>>>>>>>>>>> download starts.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> There is a way: Tor. But that is a totally different story.
>>>>>>>>>>>>> Well, I see a third option on this: Use the mirror infrastructure we already
>>>>>>>>>>>>> have. Every IPFire installation discloses its public IP address to one of these
>>>>>>>>>>>>> servers sooner or later, so we do not disclose additional data if the blacklists
>>>>>>>>>>>>> were fetched from these.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Needless to say, Tor (hidden services) would be better, but that is a different
>>>>>>>>>>>>> story indeed. :-)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The point is rather that a forget list can be sent instead of the “real” one.
>>>>>>>>>>>>> I did not get this. Forget? Forged?? ???
>>>>>>>>>>>>
>>>>>>>>>>>> Yes, I meant to write forged, but auto-correct didn’t let me.
>>>>>>>>>>>>
>>>>>>>>>>>>>>> Secondly, latency; some of the lists are updated every 5 minutes. While
>>>>>>>>>>>>>>> I've limited the maximum check rate to hourly, will the updates
>>>>>>>>>>>>>>> propagate quickly enough. For reference on my main system the 24
>>>>>>>>>>>>>>> updates on the CIARMY list made 143 498 changes (additions or
>>>>>>>>>>>>>>> deletions). I've seem it do over 200 000.
>>>>>>>>>>>>> Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not
>>>>>>>>>>>>> document a recommended update interval anywhere, so we can only guess.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Personally, more static lists seem to be preferable for packet filtering. Highly
>>>>>>>>>>>>> dynamic ones such as CIArmy should be done via DNSBL queries or something similar
>>>>>>>>>>>>> - do we really want to have that list here?
>>>>>>>>>>>>
>>>>>>>>>>>> It is not really an option to implement a DNSBL into a packet filter, but I get your point.
>>>>>>>>>>>
>>>>>>>>>>> One of the 'selling points' for an IP address blacklist is that it can
>>>>>>>>>>> respond quickly to new threats - or rather new attackers. While a new
>>>>>>>>>>> IDS/IPS rule needs time to analyse the threat, generate a rule and check
>>>>>>>>>>> it, it's easy to add an address to a list. So, I think the CIArmy list
>>>>>>>>>>> is potentially useful for protecting home systems etc. with budget
>>>>>>>>>>> hardware, but I would be very careful about using it for a protecting a
>>>>>>>>>>> general access website.
>>>>>>>>>>
>>>>>>>>>> If they are very volatile, we should honour that and update them often, too.
>>>>>>>>>>
>>>>>>>>>> It probably is more about false-positives being removed very quickly instead of adding threats very quickly. The average IPFire user is probably not under threats like these to need to react very quickly.
>>>>>>>>>>
>>>>>>>>>> So I do not see much value in adding those lists and then updating once a day. Does it have to be every 5 min? No. I would suggest 15 which should be good enough for everyone.
>>>>>>>>>>
>>>>>>>>>> Other lists should of course not be updated every 15 minutes when not needed.
>>>>>>>>>>
>>>>>>>>>> Running every 15 minutes would allow us to retry downloading lists that are on an hourly schedule if the download failed.
>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>> How did you come up with the hour? Will it be retried more often if the download was not successful?
>>>>>>>>>>>>> One hour is the most common interval indeed, but adding some random time might
>>>>>>>>>>>>> be useful in order to reduce load on the servers providing a blacklist.
>>>>>>>>>>>>
>>>>>>>>>>>> Yes, definitely. Otherwise we will shoot down our mirrors.
>>>>>>>>>>>
>>>>>>>>>>> When I implemented that section of code, specifying the minimum check
>>>>>>>>>>> period in hours seemed to provide a convenient way of allowing a check
>>>>>>>>>>> period covering a wide range, with an hour as the fastest and a week as
>>>>>>>>>>> the slowest. I didn't looked at the CIArmy list until much later. Most
>>>>>>>>>>> of the lists don't change nearly as much, but the CIArmy list is
>>>>>>>>>>> described as one that deliberately responds quickly.
>>>>>>>>>>>
>>>>>>>>>>> From my production system, for yesterday:
>>>>>>>>>>>
>>>>>>>>>>> The following block lists were updated:
>>>>>>>>>>> BLOCKLIST_DE: 24 Time(s) - 9341 change(s)
>>>>>>>>>>> BOGON_FULL: 1 Time(s) - 10 change(s)
>>>>>>>>>>> CIARMY: 24 Time(s) - 159134 change(s)
>>>>>>>>>>> DSHIELD: 7 Time(s) - 18 change(s)
>>>>>>>>>>> EMERGING_FWRULE: 1 Time(s) - 50 change(s)
>>>>>>>>>>> FEODO_AGGRESIVE: 24 Time(s) - 13 change(s)
>>>>>>>>>>> SHODAN: 1 Time(s) - 0 change(s)
>>>>>>>>>>> SPAMHAUS_DROP: 1 Time(s) - 0 change(s)
>>>>>>>>>>> TOR_EXIT: 24 Time(s) - 162 change(s)
>>>>>>>>>>
>>>>>>>>>> Very interesting statistics.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> and my test system:
>>>>>>>>>>>
>>>>>>>>>>> The following block lists were updated:
>>>>>>>>>>> ALIENVAULT: 19 Time(s) - 5331 change(s)
>>>>>>>>>>> EMERGING_COMPROMISED: 1 Time(s) - 26 change(s)
>>>>>>>>>>> TALOS_MALICIOUS: 1 Time(s) - 36 change(s)
>>>>>>>>>>>
>>>>>>>>>>> That covers most of the lists. From the WUI, since 1 Dec:
>>>>>>>>>>>
>>>>>>>>>>> Blacklist Entries pkts bytes Last updated
>>>>>>>>>>> in in
>>>>>>>>>>> AUTOBLACKLIST 0 731 51144 Sun Dec 8 18:40:02 2019
>>>>>>>>>>> BLOCKLIST_DE 28020 857 46735 Sun Dec 8 18:40:02 2019
>>>>>>>>>>> BOGON_FULL 214 5255 189K Sun Dec 8 16:50:04 2019
>>>>>>>>>>> CIARMY 15000 19774 976K Sun Dec 8 18:04:01 2019
>>>>>>>>>>> DSHIELD 20 7992 321K Sun Dec 8 16:45:13 2019
>>>>>>>>>>> EMERGING_FWRULE 1647 197 8383 Fri Dec 6 05:29:07 2019
>>>>>>>>>>> FEODO_AGGRESIVE 7169 0 0 Sun Dec 8 18:50:07 2019
>>>>>>>>>>> SHODAN 32 34 1530 Sun Dec 8 17:54:09 2019
>>>>>>>>>>> SPAMHAUS_DROP 823 0 0 Fri Dec 6 18:22:35 2019
>>>>>>>>>>> SPAMHAUS_EDROP 111 82 16433 Thu Dec 5 17:27:09 2019
>>>>>>>>>>> TOR_EXIT 1055 0 0 Sun Dec 8 18:31:02 2019
>>>>>>>>>>
>>>>>>>>>> This as well.
>>>>>>>>>>
>>>>>>>>>> Those are more packets than I would have expected.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> (I've left out the pkts/bytes out fields which were all 0)
>>>>>>>>>>>
>>>>>>>>>>> Note that where possible I do a HEAD request first and then only
>>>>>>>>>>> download the list if the modification time has changed since the last
>>>>>>>>>>> check. For dynamically generated lists this isn't possible.
>>>>>>>>>>
>>>>>>>>>> You won’t need a HEAD request for it. You can include it in the GET request.
>>>>>>>>>>
>>>>>>>>>> Have a look at location downloader where I use that. The server will respond with 304 and not send any payload.
>>>>>>>>>>
>>>>>>>>>> https://git.ipfire.org/?p=location/libloc.git;a=blob;f=src/python/location-downloader.in;h=1b5932d5822e03737d32b2a27816815b2f7e74dd;hb=HEAD#l151
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> If the download isn't successful it just gives up and waits for the next
>>>>>>>>>>> attempt (apart from the usual retries in the library). I probably
>>>>>>>>>>> should to change that so that it only applies the per list minimum
>>>>>>>>>>> update period in this case (specified in the sources file) rather than
>>>>>>>>>>> the user specified value as well.
>>>>>>>>>>
>>>>>>>>>> I think it is not the worst if an update fails. It might just happen every once in a while.
>>>>>>>>>>
>>>>>>>>>> So I would suggest to just re-run the script more often and when the mtime of the file is older than the threshold, a download is attempted. You can use that timestamp for the GET request.
>>>>>>>>>>
>>>>>>>>>>> I already use a time offset on the downloads - when it's started from
>>>>>>>>>>> boot, backup restore or WUI enable, it checks to see if it's installed
>>>>>>>>>>> in the fcrontab, and if not adds itself at a randomly generated offset
>>>>>>>>>>> in the hour.
>>>>>>>>>>
>>>>>>>>>> That should potentially go to red.up, or if we can settle on 15 minutes, I would consider that often enough to quickly update all lists after a reboot.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>>> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is
>>>>>>>>>>>>>>> the largest at a few MB), there are going to be a lot of them. How will
>>>>>>>>>>>>>>> this affect the willingness of people to mirror IPFire?
>>>>>>>>>>>>> I do not consider this being a problem as we do not generate that much traffic
>>>>>>>>>>>>> to them. Of course, that depends on the update interval again.
>>>>>>>>>>>>
>>>>>>>>>>>> That depends on your point of view.
>>>>>>>>>>>>
>>>>>>>>>>>> I do not have a problem with this at all in my data center, but there are plenty of people with a volume-based LTE plan or simply a 128 kBit/s connection. It will take a longer time to download the lists for them. We need to mind that.
>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Talking about the preference of packet filter and IPS, I prefer to
>>>>>>>>>>>>>>>> use the latter as well as it gains more insight in what kind of malicious
>>>>>>>>>>>>>>>> traffic tried to pass a firewall machine. On systems with low resources,
>>>>>>>>>>>>>>>> this might be problematic and removing load from the IPS can be preferred
>>>>>>>>>>>>>>>> (make this configurable?!), on others, people might want to have both
>>>>>>>>>>>>>>>> results.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> You're only going to get one result for a packet whichever way round the
>>>>>>>>>>>>>>> IP blacklist and IPS are since whichever comes first will drop the
>>>>>>>>>>>>>>> packet before it reaches the second (well it would be possible to put
>>>>>>>>>>>>>>> the IP blacklist first and get it to log and mark packets which are then
>>>>>>>>>>>>>>> dropped after the IPS, but I think that's getting a little complicated.
>>>>>>>>>>>>>>> In addition I've seen the messages about the trouble marking was
>>>>>>>>>>>>>>> causing in the QoS).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think it's a 50/50 choice as to which is more valuable first; it's
>>>>>>>>>>>>>>> probably going to differ from packet to packet. For me the possibility
>>>>>>>>>>>>>>> of reducing the IPS load means I prefer putting the IP blacklist first
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It should be fairly easy to add the choice of where to put the IP
>>>>>>>>>>>>>>> blacklist. I think it'll have to be in the main firewall script, so
>>>>>>>>>>>>>>> it'll require a firewall restart, but it's not something that'll be
>>>>>>>>>>>>>>> changed often.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier.
>>>>>>>>>>>>> Agreed.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily.
>>>>>>>>>>>>> So dropping packets from blacklisted IP addresses/networks before IPS is it, then.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I do not even think it makes sense to swap the order in the outgoing direction.
>>>>>>>>>>>>> Me too.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this.
>>>>>>>>>>>>> Based on my personal experience, firewall hits observed on a single machine exposed
>>>>>>>>>>>>> to the internet are interesting, but the overall situation across multiple machines
>>>>>>>>>>>>> is even more interesting. Very quickly, you'll end on something like a centralised
>>>>>>>>>>>>> logging server and custom statistical analysis here...
>>>>>>>>>>>>
>>>>>>>>>>>> Probably a project for IPFire 4.0 :)
>>>>>>>>>>>>
>>>>>>>>>>> Or use one of the existing services, like the DSHIELD client
>>>>>>>>>>> https://dshield.org/howto.html (subject to privacy concerns again).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default.
>>>>>>>>>>>>>>>> I would love to see the bogon ruleset here, too (think about 8chan successor
>>>>>>>>>>>>>>>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause
>>>>>>>>>>>>>>>> interference if RED does not have a public IP address assigned.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I can add a field to the options file that controls whether a list is
>>>>>>>>>>>>>>> enabled by default.
>>>>>>>>>>>>> Thank you. :-)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat.
>>>>>>>>>>>>> This is my only objection against this patchset. Now, what can we do about it?
>>>>>>>>>>>>> One possibility is to apply the patchset now and implement a custom download
>>>>>>>>>>>>> source thing later on, or do that before releasing Core Update 139 (or which version
>>>>>>>>>>>>> the patchset will be to) after we agreed on something.
>>>>>>>>>>>>
>>>>>>>>>>>> I do not see this being merged for 139. But that is not important. We need to get it right first and then release it.
>>>>>>>>>>>>
>>>>>>>>>>>> As far as I know, nobody has tested this, yet.
>>>>>>>>>>>
>>>>>>>>>>> There are a number of people who have been running an earlier version
>>>>>>>>>>> which I shared on GitHub. There were a few early issues, but it seems
>>>>>>>>>>> to be OK now.
>>>>>>>>>>>
>>>>>>>>>>> https://forum.ipfire.org/viewtopic.php?f=27&t=21845
>>>>>>>>>>>
>>>>>>>>>>> This version wasn't integrated into IPFire, so (for example) it inserted
>>>>>>>>>>> itself into the INPUT IPTables chain rather than having it's chains
>>>>>>>>>>> created as part of the firewall start-up script.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I have huge concerns about the automatic blacklist. @Peter: What is your opinion on this?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> While I implemented it, I'm aware of its potential to cause problems,
>>>>>>>>>>> which is why it has to be separately enabled. It's not caused me any
>>>>>>>>>>> issues at the default settings (blocks at over 10 packets per hour until
>>>>>>>>>>> 1 hour has passed without seeing packets from the address), but I've not
>>>>>>>>>>> used it on a site with publicly announced services. If I was going to
>>>>>>>>>>> use it on a web site I would want to, at the very minimum, drop the
>>>>>>>>>>> block period drastically.
>>>>>>>>>>
>>>>>>>>>> I suppose this is entirely unusable on an IPFire box in a data center that hosts things. Let’s say our rack in Hanover.
>>>>>>>>>>
>>>>>>>>>> You will have hits from some broken IP stacks and people might just end up on this without doing anything wrong.
>>>>>>>>>>
>>>>>>>>>> You can denial-of-service easily as well and I suppose without aggregation of data from many many systems, it does not make sense to instantly block addresses. And even then you probably would block an entire subnet.
>>>>>>>>>>
>>>>>>>>>> So I am not sure how I should feel about it. I do not think that it adds anything because the packets would have been blocked anyways.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On the other hand, it's good at responding quickly. Usually I see only
>>>>>>>>>>> 1-2% of blocks from the automatic list:
>>>>>>>>>>>
>>>>>>>>>>> Reason Count % First Last
>>>>>>>>>>> CIARMY 2416 45 Dec 6 00:00 Dec 6 23:59
>>>>>>>>>>> DSHIELD 1353 25 Dec 6 00:00 Dec 6 23:59
>>>>>>>>>>> INPUT 1294 24 Dec 6 00:00 Dec 6 23:59
>>>>>>>>>>> AUTOBLACKLIST 122 2 Dec 6 00:20 Dec 6 16:28
>>>>>>>>>>> BLOCKLIST_DE 89 2 Dec 6 00:20 Dec 6 23:46
>>>>>>>>>>>
>>>>>>>>>>> and sometimes none at all, but one one occasion it blocked over 8000
>>>>>>>>>>> packets. Again I'm aware this is for a home system, which is rather
>>>>>>>>>>> different than from a Web server.
>>>>>>>>>>>
>>>>>>>>>>>>> If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not
>>>>>>>>>>>>> seem to be problematic, as they are hosted on 3rd party servers, too).
>>>>>>>>>>>>
>>>>>>>>>>>> One of them will be, sooner or later. And one is enough I suppose.
>>>>>>>>>>>
>>>>>>>>>>> DShield (https://dshield.org/api/#threatfeeds) and firehol
>>>>>>>>>>> (http://iplists.firehol.org/) seem to host copies of most of the lists
>>>>>>>>>>> as well.
>>>>>>>>>>>>
>>>>>>>>>>>> I do not really want to overthink this - we didn’t do this with clamav for example either. But it probably is a decision to be made by the user what they want to enable and we should not enable anything by default. So no data will be leaked as long as the user does not consent.
>>>>>>>>>>>>
>>>>>>>>>>>> -Michael
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks, and best regards,
>>>>>>>>>>>>> Peter Müller
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <Screenshot_2019-12-16_IP_Address_Blacklists.png>
>>>>>>>>
>>>>>>>
>>>>>>> <Screenshot_2019-12-17_IP_Address_Blacklists.png>
>>>>>>
>>>>>
>>>>
>>>
>>> <IP_Address_Blacklists.png><IP_Address_Blacklist_Logs.png><Firewall_log_blacklist.png><Log_Summary.png>
>>
>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 0/5] ipblacklist: IP Address Blacklists
2020-01-06 11:27 ` Michael Tremer
@ 2020-01-24 19:40 ` Tim FitzGeorge
2020-01-28 17:14 ` Michael Tremer
0 siblings, 1 reply; 35+ messages in thread
From: Tim FitzGeorge @ 2020-01-24 19:40 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 34757 bytes --]
I'm now nearly ready to submit the updated patches, but before then I'd
like to check that you're happy with some of my design decisions.
_IPTables_Chains_
When setting up the firewall (in iptables_init()), four new chains are
created, IPBLACKLISTIN, IPBLACKLISTOUT, IPBLACKLISTREDIN and
IPBLACKLISTREDOUT. The first two are inserted into the INPUT, OUTPUT
and FORWARD chains, before the IPS chains. The main script is called to
restore the previously saved blacklists, creating the rules in the
latter two chains.
In iptables_red_up(), rules are created to forward traffic to/from the
red interface from IPBLACKLISTIN/IPBLACKLISTOUT to
IPBLACKLISTREDIN/IPBLACKLISTREDOUT.
I did consider putting the blacklist rules into the
IPBLACKLISTIN/IPBLACKLISTOUT, but this couldn't be done until red comes
up and would require much more processing at this point - not a problem
if your red interface comes up and stays up, but if you've got one that
disconnects on no traffic you probably don't want the extra overhead.
The way I've done it also has the side effects of 1) the main script
doesn't have to worry about what the red interface is, and 2) if you're
using multiple blacklists, the check for the red interface is only done
once.
_Log_WUI_Pages_
The two Log WUI pages are edited from existing pages. I've resisted the
temptation to reformat them, which means that they can be diff'ed with
the corresponding pages.
_Update_
The update is fairly simple - it calculates whether the check period is
passed, tries a download specifying the If-Modified-Since and parses the
returned file. It then compares the new list with the existing list,
adding new addresses/networks and deleting those that have been removed.
There's a check to handle a list changing from a list of addresses to a
list of networks, in which case it changes the type of the IPSet - this
should be a vary rare event, but could happen for a composite list.
_Enabling/Disabling_Blacklists_
For changes to the existing firewall rules, the firewall is shut down
and restarted from scratch. The blacklist script does this on the fly,
since the possibilities are simpler than all the options available for
the firewall and also the necessary code for doing updates means that
adding this code is fairly simple. This includes turning logging on and
off.
_Logging_
The main scripts logs to syslog as ipblacklist, but if you run it from
the command line it'll also output to the terminal.
There's also some debug code which can be turned on by setting a value
in the settings file - it's turned on at the lowest level automatically
if you run from the command line.
Tim
On 06/01/2020 11:27, Michael Tremer wrote:
> Hello all,
>
>> On 4 Jan 2020, at 19:02, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>
>> Hello,
>>
>> I've now updated the code - screenshots are attached.
>>
>> IP_Address_Blacklists.png
>>
>> This is the settings page. I don't think there's anything unexpected here.
>
> Yes, that looks a lot like the other IPFire pages :)
>
>> IP_Address_Blacklist_Logs.png
>>
>> The log page, replacing the status information that was previously at
>> the top of the settings page.
>
> Looks great!
>
>> I've sorted the lists alphabetically, rather than by the number of hits
>> since the worst threat is not necessary the largest number of blocked
>> packets (which could implied by sorting it by number).
>
> Yes, that is indeed a very good idea.
>
> You could add a % sign in the percentage column, so it cannot be confused so easily with the other column.
>
>> Firewall_log_blacklist.png
>>
>> Details page accessed from the log. Almost identical to the other
>> similar pages.
>
> *thumbs up*
>
>> Log_Summary.png
>>
>> This is an extract from the Log Summary page produced by logwatch.
>>
>> I still need to do some tidying, remove unused language strings etc.,
>> and then update and test the patches, so it'll be some time before I'm
>> ready to submit updated patches.
>
> Feel free to send some RFC to the list. It might be slightly noisy, but it is easier to review changes bit after bit.
>
> Great great work!
>
> -Michael
>
>>
>> Tim
>>
>> On 28/12/2019 20:59, Tim FitzGeorge wrote:
>>> Hi,
>>>
>>> I'll code and test what we've agreed on. In the meantime I'm going to
>>> reply on the sub-thread started by Tom Rymes as to what we call the list
>>> categories, so that we can keep that discussion separate from further
>>> functionality etc. discussion.
>>>
>>> Tim
>>>
>>> On 21/12/2019 18:34, Tim FitzGeorge wrote:
>>>> Hi,
>>>>
>>>> I've got my systems checking for downloads at 15 minute intervals and
>>>> using If-Modified-Since. This all seems to be working OK.
>>>>
>>>> Responses to comments below:
>>>>
>>>> On 18/12/2019 12:07, Michael Tremer wrote:
>>>>> Hi,
>>>>>
>>>>>> On 17 Dec 2019, at 18:21, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I think we've agreed on the following:
>>>>>>
>>>>>> - Checks to be made every 15 minutes, subject to per-list minimum rate.
>>>>>> - Modify download to use If-Modified-Since rather than separate HEAD
>>>>>> request.
>>>>>> - Remove automatic blacklist.
>>>>>
>>>>> I thought someone wants to counter my argument. This is not a dictatorship :)
>>>>>
>>>>>>
>>>>>> other comments are below.
>>>>>>
>>>>>> Tim
>>>>>>
>>>>>> On 16/12/2019 22:20, Michael Tremer wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>>> On 16 Dec 2019, at 20:06, Tim FitzGeorge <lists(a)tfitzgeorge.me.uk> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I've attached the current GUI screenshot.
>>>>>>>
>>>>>>> Thanks for that.
>>>>>>>
>>>>>>> I have a couple of suggestions/concerns about it:
>>>>>>>
>>>>>>> a) I am not sure what the value is about the top box that is called “Status”. It is basically a summary of the iptables output, but will it help the user? A blacklist with more or fewer entries is not better or worse, blocking more packets isn’t better than blocking fewer. It is about the quality of the blocks.
>>>>>>
>>>>>> I agree that the size of the lists is not all that useful, but the
>>>>>> information about the number of blocked packets can be an indication of
>>>>>> compromise. It depends on the type of the list, but a list that targets
>>>>>> C&C channels (e.g. the FEODO lists) should not show any input packets
>>>>>> being blocked, unless one of the computers being protected is infected.
>>>>>> Apart from the BOGON lists, none of the lists should show any output
>>>>>> packets being blocked; again it's a potential indication of infection if
>>>>>> otherwise.
>>>>>
>>>>> The Shodan list will definitely show some activity unless you are behind CG NAT.
>>>>>
>>>>> Should we not have this rather in the logging section?
>>>>>
>>>>> And these counters here will be reset when the firewall is being reloaded. Should this data therefore not be collected from the logs so that you can go back in time? That would be very important to identify any start of compromise.
>>>>>
>>>>
>>>> That makes sense. I'll remove the status from this page and create a
>>>> log page for the information. If anyone actually wants the iptables
>>>> information it's available in from Firewall > iptables anyway.
>>>>
>>>> Do you think it makes more sense to show packets blocked in/out or
>>>> packets blocked per source interface? The latter would of (limited)
>>>> help in tracking down potential compromise.
>>>>
>>>>>>>
>>>>>>> b) I would prefer to move the settings box to the top. If you like you can show the size of the blacklists there and when they have been updated, too.
>>>>>>
>>>>>> I put the status at the top since it's usually what I want to see when I
>>>>>> go to the page. Under normal circumstances I don't change the settings,
>>>>>> so I'd rather not have to scroll past them to get to the information I'm
>>>>>> actually interested in. The update times are just useful to show that
>>>>>> the lists are actually being updated. The size is interesting to me,
>>>>>> but as you say it probably won't help the average user, especially if
>>>>>> you don't know which lists block individual addresses and which block
>>>>>> nets (BOGON, DSHIELD and EMERGING_COMPROMISED).
>>>>>
>>>>> See my comment on the order above. Splitting the page might make sense.
>>>>>
>>>>> When you show size, does that mean lines in the blacklists? So it could be networks or single IP addresses? Does it not make sense to count IP addresses (i.e. 256 for a /24 network and so on) and put it in the table?
>>>> It's lines in the blacklist.
>>>>
>>>>>
>>>>> I think I would only be interested in the size for two things:
>>>>>
>>>>> a) Is there any chance for overblocking? i.e. does the list have 2 billion entries which would be half the public IPv4 address space or so. I cannot really come up with a good example here.
>>>>>
>>>>> Or
>>>>>
>>>>> b) What is the performance impact/memory consumption of the list?
>>>>
>>>> It would be interesting to show the type (single address or net), number
>>>> of entries and number of addresses; the first two affect performance,
>>>> the second memory use and the latter the number of blocked addresses,
>>>> but that's really only going to be of interest to a very small number of
>>>> people, who can always do 'ipset list -t' to get the information. So
>>>> I'll remove the size column.
>>>>
>>>>>
>>>>>> I originally had the status and settings in one list, but I decided that
>>>>>> it was getting a bit messy and difficult to follow. I've attached a
>>>>>> screenshot of a quickly hacked page. (Note that the status area only
>>>>>> appears once the blacklists have been enabled.)
>>>>>
>>>>> You are right. This is getting quite tight there.
>>>>>
>>>>> You could write packets in/out into on column like 2k/1023, but see my concerns about this above.
>>>>
>>>> Actually I've come to the conclusion that the number of bytes blocked
>>>> isn't all that important; it's sufficient to show the number of packets.
>>>>
>>>>>
>>>>>>>
>>>>>>> c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
>>>>>>
>>>>>> I agree that it's it's an over simplistic summary of the lists. I note
>>>>>> that Tom Rymes has asked to keep it, but I think its only value is if it
>>>>>> suggests to users that things are more complicated than they appear and
>>>>>> therefore prompts them to read the Wiki.
>>>>>
>>>>> I will reply to that separately.
>>>>
>>>> Looking at that reply, I think it makes sense to include a category
>>>> column in both the settings and the log summary page - the former to
>>>> help selection of which lists to use, and the latter as a help when
>>>> deciding whether the information could indicate compromise or not.
>>>>
>>>>>
>>>>>>>
>>>>>>> d) I do not understand what the “update check rate” means. I suppose we should show the update interval of the lists in the table and just refresh them according to that.
>>>>>>
>>>>>> It's the minimum check period for updates. The system checks for
>>>>>> updates at either the rate specific to the list or this rate, whichever
>>>>>> is slower. This is a hangover from the early days, copied from the IPS
>>>>>> updates. I suppose it could have a value for some people who may wish
>>>>>> to lower the check rate, but I'm not sure that it has sufficient value
>>>>>> to be worth keeping.
>>>>>
>>>>> No, I would not let the user decide when and how often to update the lists.
>>>>
>>>> OK. I'll remove it.
>>>>
>>>>>
>>>>> Generally there is this argument internally how often some databases should be updated. People with slow internet connections or volume based data plans will find this rather expensive to update those lists too often.
>>>>>
>>>>> If we ever want to go ahead with that in IPFire 2, there should be a global switch for a “low data mode” which then delays updating these things.
>>>>
>>>> That makes sense. Maybe also the ability to set when downloads take
>>>> place - but as you say that's for the future.
>>>>
>>>>>
>>>>>>>
>>>>>>> e) Do we want to keep the automatic blacklists now? I do not think it will actually improve the security of IPFire.
>>>>>>>
>>>>>>
>>>>>> I'll remove them. I think they do have a use, in that they can detect
>>>>>> someone doing a port scan and block them before they find an open port,
>>>>>> but as you say there is also the possibility of a DOS.
>>>>>
>>>>> How would this block a DOS?
>>>>
>>>> Sorry, I didn't make myself clear - there's the possibility that someone
>>>> could cause a DOS with the automatic list by faking the source IP
>>>> address and sending a few packets on a random port.
>>>>
>>>>>
>>>>> -Michael
>>>>
>>>> Tim
>>>>>
>>>>>>>>
>>>>>>>> I'll update the code based on our discussions, and submit an updated set
>>>>>>>> of patches - I imagine there will have to be at least one more iteration.
>>>>>>>
>>>>>>> Let’s wait until we have come to decisions :)
>>>>>>>
>>>>>>> -Michael
>>>>>>>
>>>>>>>>
>>>>>>>> Tim
>>>>>>>>
>>>>>>>> (No additional comments below)
>>>>>>>>
>>>>>>>> On 13/12/2019 23:11, Michael Tremer wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Again my apologies for my late reply. Busy busy weeks.
>>>>>>>>>
>>>>>>>>>> On 8 Dec 2019, at 20:50, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> It's my turn to apologise for being slow to respond - I've had a busy
>>>>>>>>>> week, but I should have plenty of time over the next couple of weeks.
>>>>>>>>>
>>>>>>>>> No worries. Turns out we all do :)
>>>>>>>>>
>>>>>>>>>> I've made most of the comments inline, however I think Michael had a
>>>>>>>>>> question (which I can't find now) about what happens if someone enables
>>>>>>>>>> all the lists. One thing which would perhaps make this less likely is
>>>>>>>>>> that the WUI tags the available lists with whether they're safe or not,
>>>>>>>>>> with a footnote that safe means that the list only blocks malicious
>>>>>>>>>> traffic. This won't guarantee that a user won't still try to enable all
>>>>>>>>>> the lists, but it should make them realise that they should think first.
>>>>>>>>>
>>>>>>>>> We had a couple of features going slightly wrong or being “misunderstood” by some users. People still seem to panic when they see “local recursor”. To this day I do not know why.
>>>>>>>>>
>>>>>>>>> We cannot make everything idiot-proof. And when some user if of that category, they probably should shutdown their IPFire box, educate themselves and then come back again. So I do not want to limit people, but make things as easy as possible.
>>>>>>>>>
>>>>>>>>> If someone enables all the lists, good luck with passing packets :)
>>>>>>>>>
>>>>>>>>>> I have considered replacing this tag with a risk high/medium/low and
>>>>>>>>>> maybe adding a category (invalid/application/scanner/C&C or something
>>>>>>>>>> like that), but that may provide too much information and dissuade them
>>>>>>>>>> from actually following the links to checkout what the list actually does.
>>>>>>>>>
>>>>>>>>> Can we have a screenshot of the GUI right now? I didn’t run the code, yet.
>>>>>>>>>
>>>>>>>>> We should document the lists like we do it with the rulesets of the IPS. People might ignore this, but that is on them.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Tim
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 05/12/2019 22:25, Michael Tremer wrote:
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>>> On 4 Dec 2019, at 17:05, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hello Tim, hello Michael,
>>>>>>>>>>>>
>>>>>>>>>>>> please see my responses inline...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We could periodically update the blacklists on our main mirror (and
>>>>>>>>>>>>>>> wait for the network to sync it), make sure it is signed and write
>>>>>>>>>>>>>>> a small downloader that fetches, validates and installs them.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> @All: Thoughts on this?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think there are a number of points here.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Firstly, from the point of a third party using IPFire, is this really
>>>>>>>>>>>>>> solving the privacy disclosure problem? There's no way round disclosing
>>>>>>>>>>>>>> your public IP address to someone you're downloading from; all this does
>>>>>>>>>>>>>> is change who that information is being disclosed to. For the user
>>>>>>>>>>>>>> there's no way of knowing whether the source is more or less protective
>>>>>>>>>>>>>> of the user's privacy than the blacklist provider. Indeed it won't be
>>>>>>>>>>>>>> possible to know who the lists are being downloaded from until the
>>>>>>>>>>>>>> download starts.
>>>>>>>>>>>>>
>>>>>>>>>>>>> There is a way: Tor. But that is a totally different story.
>>>>>>>>>>>> Well, I see a third option on this: Use the mirror infrastructure we already
>>>>>>>>>>>> have. Every IPFire installation discloses its public IP address to one of these
>>>>>>>>>>>> servers sooner or later, so we do not disclose additional data if the blacklists
>>>>>>>>>>>> were fetched from these.
>>>>>>>>>>>>
>>>>>>>>>>>> Needless to say, Tor (hidden services) would be better, but that is a different
>>>>>>>>>>>> story indeed. :-)
>>>>>>>>>>>>>
>>>>>>>>>>>>> The point is rather that a forget list can be sent instead of the “real” one.
>>>>>>>>>>>> I did not get this. Forget? Forged?? ???
>>>>>>>>>>>
>>>>>>>>>>> Yes, I meant to write forged, but auto-correct didn’t let me.
>>>>>>>>>>>
>>>>>>>>>>>>>> Secondly, latency; some of the lists are updated every 5 minutes. While
>>>>>>>>>>>>>> I've limited the maximum check rate to hourly, will the updates
>>>>>>>>>>>>>> propagate quickly enough. For reference on my main system the 24
>>>>>>>>>>>>>> updates on the CIARMY list made 143 498 changes (additions or
>>>>>>>>>>>>>> deletions). I've seem it do over 200 000.
>>>>>>>>>>>> Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not
>>>>>>>>>>>> document a recommended update interval anywhere, so we can only guess.
>>>>>>>>>>>>
>>>>>>>>>>>> Personally, more static lists seem to be preferable for packet filtering. Highly
>>>>>>>>>>>> dynamic ones such as CIArmy should be done via DNSBL queries or something similar
>>>>>>>>>>>> - do we really want to have that list here?
>>>>>>>>>>>
>>>>>>>>>>> It is not really an option to implement a DNSBL into a packet filter, but I get your point.
>>>>>>>>>>
>>>>>>>>>> One of the 'selling points' for an IP address blacklist is that it can
>>>>>>>>>> respond quickly to new threats - or rather new attackers. While a new
>>>>>>>>>> IDS/IPS rule needs time to analyse the threat, generate a rule and check
>>>>>>>>>> it, it's easy to add an address to a list. So, I think the CIArmy list
>>>>>>>>>> is potentially useful for protecting home systems etc. with budget
>>>>>>>>>> hardware, but I would be very careful about using it for a protecting a
>>>>>>>>>> general access website.
>>>>>>>>>
>>>>>>>>> If they are very volatile, we should honour that and update them often, too.
>>>>>>>>>
>>>>>>>>> It probably is more about false-positives being removed very quickly instead of adding threats very quickly. The average IPFire user is probably not under threats like these to need to react very quickly.
>>>>>>>>>
>>>>>>>>> So I do not see much value in adding those lists and then updating once a day. Does it have to be every 5 min? No. I would suggest 15 which should be good enough for everyone.
>>>>>>>>>
>>>>>>>>> Other lists should of course not be updated every 15 minutes when not needed.
>>>>>>>>>
>>>>>>>>> Running every 15 minutes would allow us to retry downloading lists that are on an hourly schedule if the download failed.
>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>> How did you come up with the hour? Will it be retried more often if the download was not successful?
>>>>>>>>>>>> One hour is the most common interval indeed, but adding some random time might
>>>>>>>>>>>> be useful in order to reduce load on the servers providing a blacklist.
>>>>>>>>>>>
>>>>>>>>>>> Yes, definitely. Otherwise we will shoot down our mirrors.
>>>>>>>>>>
>>>>>>>>>> When I implemented that section of code, specifying the minimum check
>>>>>>>>>> period in hours seemed to provide a convenient way of allowing a check
>>>>>>>>>> period covering a wide range, with an hour as the fastest and a week as
>>>>>>>>>> the slowest. I didn't looked at the CIArmy list until much later. Most
>>>>>>>>>> of the lists don't change nearly as much, but the CIArmy list is
>>>>>>>>>> described as one that deliberately responds quickly.
>>>>>>>>>>
>>>>>>>>>> From my production system, for yesterday:
>>>>>>>>>>
>>>>>>>>>> The following block lists were updated:
>>>>>>>>>> BLOCKLIST_DE: 24 Time(s) - 9341 change(s)
>>>>>>>>>> BOGON_FULL: 1 Time(s) - 10 change(s)
>>>>>>>>>> CIARMY: 24 Time(s) - 159134 change(s)
>>>>>>>>>> DSHIELD: 7 Time(s) - 18 change(s)
>>>>>>>>>> EMERGING_FWRULE: 1 Time(s) - 50 change(s)
>>>>>>>>>> FEODO_AGGRESIVE: 24 Time(s) - 13 change(s)
>>>>>>>>>> SHODAN: 1 Time(s) - 0 change(s)
>>>>>>>>>> SPAMHAUS_DROP: 1 Time(s) - 0 change(s)
>>>>>>>>>> TOR_EXIT: 24 Time(s) - 162 change(s)
>>>>>>>>>
>>>>>>>>> Very interesting statistics.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> and my test system:
>>>>>>>>>>
>>>>>>>>>> The following block lists were updated:
>>>>>>>>>> ALIENVAULT: 19 Time(s) - 5331 change(s)
>>>>>>>>>> EMERGING_COMPROMISED: 1 Time(s) - 26 change(s)
>>>>>>>>>> TALOS_MALICIOUS: 1 Time(s) - 36 change(s)
>>>>>>>>>>
>>>>>>>>>> That covers most of the lists. From the WUI, since 1 Dec:
>>>>>>>>>>
>>>>>>>>>> Blacklist Entries pkts bytes Last updated
>>>>>>>>>> in in
>>>>>>>>>> AUTOBLACKLIST 0 731 51144 Sun Dec 8 18:40:02 2019
>>>>>>>>>> BLOCKLIST_DE 28020 857 46735 Sun Dec 8 18:40:02 2019
>>>>>>>>>> BOGON_FULL 214 5255 189K Sun Dec 8 16:50:04 2019
>>>>>>>>>> CIARMY 15000 19774 976K Sun Dec 8 18:04:01 2019
>>>>>>>>>> DSHIELD 20 7992 321K Sun Dec 8 16:45:13 2019
>>>>>>>>>> EMERGING_FWRULE 1647 197 8383 Fri Dec 6 05:29:07 2019
>>>>>>>>>> FEODO_AGGRESIVE 7169 0 0 Sun Dec 8 18:50:07 2019
>>>>>>>>>> SHODAN 32 34 1530 Sun Dec 8 17:54:09 2019
>>>>>>>>>> SPAMHAUS_DROP 823 0 0 Fri Dec 6 18:22:35 2019
>>>>>>>>>> SPAMHAUS_EDROP 111 82 16433 Thu Dec 5 17:27:09 2019
>>>>>>>>>> TOR_EXIT 1055 0 0 Sun Dec 8 18:31:02 2019
>>>>>>>>>
>>>>>>>>> This as well.
>>>>>>>>>
>>>>>>>>> Those are more packets than I would have expected.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> (I've left out the pkts/bytes out fields which were all 0)
>>>>>>>>>>
>>>>>>>>>> Note that where possible I do a HEAD request first and then only
>>>>>>>>>> download the list if the modification time has changed since the last
>>>>>>>>>> check. For dynamically generated lists this isn't possible.
>>>>>>>>>
>>>>>>>>> You won’t need a HEAD request for it. You can include it in the GET request.
>>>>>>>>>
>>>>>>>>> Have a look at location downloader where I use that. The server will respond with 304 and not send any payload.
>>>>>>>>>
>>>>>>>>> https://git.ipfire.org/?p=location/libloc.git;a=blob;f=src/python/location-downloader.in;h=1b5932d5822e03737d32b2a27816815b2f7e74dd;hb=HEAD#l151
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> If the download isn't successful it just gives up and waits for the next
>>>>>>>>>> attempt (apart from the usual retries in the library). I probably
>>>>>>>>>> should to change that so that it only applies the per list minimum
>>>>>>>>>> update period in this case (specified in the sources file) rather than
>>>>>>>>>> the user specified value as well.
>>>>>>>>>
>>>>>>>>> I think it is not the worst if an update fails. It might just happen every once in a while.
>>>>>>>>>
>>>>>>>>> So I would suggest to just re-run the script more often and when the mtime of the file is older than the threshold, a download is attempted. You can use that timestamp for the GET request.
>>>>>>>>>
>>>>>>>>>> I already use a time offset on the downloads - when it's started from
>>>>>>>>>> boot, backup restore or WUI enable, it checks to see if it's installed
>>>>>>>>>> in the fcrontab, and if not adds itself at a randomly generated offset
>>>>>>>>>> in the hour.
>>>>>>>>>
>>>>>>>>> That should potentially go to red.up, or if we can settle on 15 minutes, I would consider that often enough to quickly update all lists after a reboot.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>>> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is
>>>>>>>>>>>>>> the largest at a few MB), there are going to be a lot of them. How will
>>>>>>>>>>>>>> this affect the willingness of people to mirror IPFire?
>>>>>>>>>>>> I do not consider this being a problem as we do not generate that much traffic
>>>>>>>>>>>> to them. Of course, that depends on the update interval again.
>>>>>>>>>>>
>>>>>>>>>>> That depends on your point of view.
>>>>>>>>>>>
>>>>>>>>>>> I do not have a problem with this at all in my data center, but there are plenty of people with a volume-based LTE plan or simply a 128 kBit/s connection. It will take a longer time to download the lists for them. We need to mind that.
>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Talking about the preference of packet filter and IPS, I prefer to
>>>>>>>>>>>>>>> use the latter as well as it gains more insight in what kind of malicious
>>>>>>>>>>>>>>> traffic tried to pass a firewall machine. On systems with low resources,
>>>>>>>>>>>>>>> this might be problematic and removing load from the IPS can be preferred
>>>>>>>>>>>>>>> (make this configurable?!), on others, people might want to have both
>>>>>>>>>>>>>>> results.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You're only going to get one result for a packet whichever way round the
>>>>>>>>>>>>>> IP blacklist and IPS are since whichever comes first will drop the
>>>>>>>>>>>>>> packet before it reaches the second (well it would be possible to put
>>>>>>>>>>>>>> the IP blacklist first and get it to log and mark packets which are then
>>>>>>>>>>>>>> dropped after the IPS, but I think that's getting a little complicated.
>>>>>>>>>>>>>> In addition I've seen the messages about the trouble marking was
>>>>>>>>>>>>>> causing in the QoS).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think it's a 50/50 choice as to which is more valuable first; it's
>>>>>>>>>>>>>> probably going to differ from packet to packet. For me the possibility
>>>>>>>>>>>>>> of reducing the IPS load means I prefer putting the IP blacklist first
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It should be fairly easy to add the choice of where to put the IP
>>>>>>>>>>>>>> blacklist. I think it'll have to be in the main firewall script, so
>>>>>>>>>>>>>> it'll require a firewall restart, but it's not something that'll be
>>>>>>>>>>>>>> changed often.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier.
>>>>>>>>>>>> Agreed.
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads.
>>>>>>>>>>>>>
>>>>>>>>>>>>> We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily.
>>>>>>>>>>>> So dropping packets from blacklisted IP addresses/networks before IPS is it, then.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I do not even think it makes sense to swap the order in the outgoing direction.
>>>>>>>>>>>> Me too.
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this.
>>>>>>>>>>>> Based on my personal experience, firewall hits observed on a single machine exposed
>>>>>>>>>>>> to the internet are interesting, but the overall situation across multiple machines
>>>>>>>>>>>> is even more interesting. Very quickly, you'll end on something like a centralised
>>>>>>>>>>>> logging server and custom statistical analysis here...
>>>>>>>>>>>
>>>>>>>>>>> Probably a project for IPFire 4.0 :)
>>>>>>>>>>>
>>>>>>>>>> Or use one of the existing services, like the DSHIELD client
>>>>>>>>>> https://dshield.org/howto.html (subject to privacy concerns again).
>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default.
>>>>>>>>>>>>>>> I would love to see the bogon ruleset here, too (think about 8chan successor
>>>>>>>>>>>>>>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause
>>>>>>>>>>>>>>> interference if RED does not have a public IP address assigned.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I can add a field to the options file that controls whether a list is
>>>>>>>>>>>>>> enabled by default.
>>>>>>>>>>>> Thank you. :-)
>>>>>>>>>>>>>
>>>>>>>>>>>>> To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat.
>>>>>>>>>>>> This is my only objection against this patchset. Now, what can we do about it?
>>>>>>>>>>>> One possibility is to apply the patchset now and implement a custom download
>>>>>>>>>>>> source thing later on, or do that before releasing Core Update 139 (or which version
>>>>>>>>>>>> the patchset will be to) after we agreed on something.
>>>>>>>>>>>
>>>>>>>>>>> I do not see this being merged for 139. But that is not important. We need to get it right first and then release it.
>>>>>>>>>>>
>>>>>>>>>>> As far as I know, nobody has tested this, yet.
>>>>>>>>>>
>>>>>>>>>> There are a number of people who have been running an earlier version
>>>>>>>>>> which I shared on GitHub. There were a few early issues, but it seems
>>>>>>>>>> to be OK now.
>>>>>>>>>>
>>>>>>>>>> https://forum.ipfire.org/viewtopic.php?f=27&t=21845
>>>>>>>>>>
>>>>>>>>>> This version wasn't integrated into IPFire, so (for example) it inserted
>>>>>>>>>> itself into the INPUT IPTables chain rather than having it's chains
>>>>>>>>>> created as part of the firewall start-up script.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I have huge concerns about the automatic blacklist. @Peter: What is your opinion on this?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> While I implemented it, I'm aware of its potential to cause problems,
>>>>>>>>>> which is why it has to be separately enabled. It's not caused me any
>>>>>>>>>> issues at the default settings (blocks at over 10 packets per hour until
>>>>>>>>>> 1 hour has passed without seeing packets from the address), but I've not
>>>>>>>>>> used it on a site with publicly announced services. If I was going to
>>>>>>>>>> use it on a web site I would want to, at the very minimum, drop the
>>>>>>>>>> block period drastically.
>>>>>>>>>
>>>>>>>>> I suppose this is entirely unusable on an IPFire box in a data center that hosts things. Let’s say our rack in Hanover.
>>>>>>>>>
>>>>>>>>> You will have hits from some broken IP stacks and people might just end up on this without doing anything wrong.
>>>>>>>>>
>>>>>>>>> You can denial-of-service easily as well and I suppose without aggregation of data from many many systems, it does not make sense to instantly block addresses. And even then you probably would block an entire subnet.
>>>>>>>>>
>>>>>>>>> So I am not sure how I should feel about it. I do not think that it adds anything because the packets would have been blocked anyways.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On the other hand, it's good at responding quickly. Usually I see only
>>>>>>>>>> 1-2% of blocks from the automatic list:
>>>>>>>>>>
>>>>>>>>>> Reason Count % First Last
>>>>>>>>>> CIARMY 2416 45 Dec 6 00:00 Dec 6 23:59
>>>>>>>>>> DSHIELD 1353 25 Dec 6 00:00 Dec 6 23:59
>>>>>>>>>> INPUT 1294 24 Dec 6 00:00 Dec 6 23:59
>>>>>>>>>> AUTOBLACKLIST 122 2 Dec 6 00:20 Dec 6 16:28
>>>>>>>>>> BLOCKLIST_DE 89 2 Dec 6 00:20 Dec 6 23:46
>>>>>>>>>>
>>>>>>>>>> and sometimes none at all, but one one occasion it blocked over 8000
>>>>>>>>>> packets. Again I'm aware this is for a home system, which is rather
>>>>>>>>>> different than from a Web server.
>>>>>>>>>>
>>>>>>>>>>>> If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not
>>>>>>>>>>>> seem to be problematic, as they are hosted on 3rd party servers, too).
>>>>>>>>>>>
>>>>>>>>>>> One of them will be, sooner or later. And one is enough I suppose.
>>>>>>>>>>
>>>>>>>>>> DShield (https://dshield.org/api/#threatfeeds) and firehol
>>>>>>>>>> (http://iplists.firehol.org/) seem to host copies of most of the lists
>>>>>>>>>> as well.
>>>>>>>>>>>
>>>>>>>>>>> I do not really want to overthink this - we didn’t do this with clamav for example either. But it probably is a decision to be made by the user what they want to enable and we should not enable anything by default. So no data will be leaked as long as the user does not consent.
>>>>>>>>>>>
>>>>>>>>>>> -Michael
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks, and best regards,
>>>>>>>>>>>> Peter Müller
>>>>>>>>>
>>>>>>>>
>>>>>>>> <Screenshot_2019-12-16_IP_Address_Blacklists.png>
>>>>>>>
>>>>>>
>>>>>> <Screenshot_2019-12-17_IP_Address_Blacklists.png>
>>>>>
>>>>
>>>
>>
>> <IP_Address_Blacklists.png><IP_Address_Blacklist_Logs.png><Firewall_log_blacklist.png><Log_Summary.png>
>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 0/5] ipblacklist: IP Address Blacklists
2020-01-22 20:35 ` Tim FitzGeorge
@ 2020-01-23 10:53 ` Michael Tremer
0 siblings, 0 replies; 35+ messages in thread
From: Michael Tremer @ 2020-01-23 10:53 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 5324 bytes --]
> On 22 Jan 2020, at 20:35, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>
> Hello Michael,
>
> On 06/01/2020 11:21, Michael Tremer wrote:
>> Hello Tim,
>>
>>> On 28 Dec 2019, at 21:17, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>
>>> Hi,
>>>
>>> Having decided that we'll categorise the lists, the question is what
>>> categories to use. They need to be:
>>>
>>> - Short (to fit on the screen)
>>> - Easily translatable
>>> - and above all, useful.
>>>
>>> Looking at the lists the obvious categories are:
>>>
>>> - Invalid Address (on the public internet)
>>> BOGON, BOGON_FULL
>>>
>>> - Scanner (not by itself malicious)
>>> SHODAN
>>>
>>> - Application (potentially unwanted)
>>> TOR_ALL, TOR_EXIT
>>>
>>> - Malware C & C
>>> FEODO_RECOMMENDED, FEODO_IP, FEODO_AGGRESIVE
>>>
>>> - Composite
>>> EMERGING_FWRULE
>>
>> I like all these a lot.
>>
>>> Less obvious are:
>>>
>>> - Reputation
>>> ALIENVAULT, CIARMY, SPAMHAUS_DROP, SPAMHAUS_EDROP
>>>
>>> - Attacks
>>> BLOCKLIST_DE, DSHIELD, EMERGING_COMPROMISED
>>
>> I even like those two, although I would potentially consider merging
> “Invalid Address” and Reputation. They are kind of the same to me. IP
> addresses I under no circumstances I want to talk to.
>
> I think I'd rather keep them separate, since conceptually they're rather
> different. Also, if your red interface has a private address, you
> definitely wouldn't want to enable these lists.
Yes, that makes sense.
>
>>
>> I also like the Attacks category, although the name is very generic.
> But I cannot come up with anything better. The only thing that might be
> worth considering is to merge it with Malware and just call it “Malicious”.
>
> I suspect that anything more specific that Attacks is going to only
> describe a single list. It's (unfortunately) a catch-all category for
> anything that doesn't fit better somewhere else.
>
> I'd rather not merge it with Malware C&C since the behaviour of the two
> is rather different - under most circumstances lists in this category
> shouldn't block any packets, whereas the Attacks category is expected to
> block a lot of inbound traffic.
I can follow your argument and I am all for it.
However I am not sure if this is obvious from the category names. Instead of wasting too much time thinking about them (because I am sure that we can slice this cake in many different ways) I am okay with what you proposed if everybody else is.
-Michael
>
>>
>>> I'm not sure that the distinction between these two is going to be
>>> helpful to most people (I'm not sure I understand it myself).
>>>
>>> We could use:
>>>
>>> - Top attackers
>>> DSHIELD, EMERGING_COMPROMISED, SPAMHAUS_DROP, SPAMHAUS_EDROP
>>>
>>> - Other attackers
>>> ALIENVAULT, BLOCKLIST_DE, CIARMY
>>>
>>> but that might be making a distinction that is better made by the user.
>>
>> Agreed. It is not obvious why some are top attackers and others are not.
>>
>> So I would 100% prefer the first option from above.
>>
>> Best,
>> -Michael
>>
>
> Tim
>
>>>
>>> Any opinions?
>>>
>>> Tim
>>>
>>>
>>> On 18/12/2019 12:10, Michael Tremer wrote:
>>>> Hi,
>>>>
>>>>> On 16 Dec 2019, at 23:05, Tom Rymes <trymes(a)rymes.com> wrote:
>>>>>
>>>>> On 12/16/2019 5:20 PM, Michael Tremer wrote:> Hi,
>>>>>>
>>>>>>> On 16 Dec 2019, at 20:06, Tim FitzGeorge
> <lists(a)tfitzgeorge.me.uk> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I've attached the current GUI screenshot.
>>>>>>
>>>>>> Thanks for that.
>>>>>>
>>>>>> I have a couple of suggestions/concerns about it:
>>>>>
>>>>> [snip]
>>>>>
>>>>>> c) I would suggest to remove the “safe” column because that is a
> very hard summary of what the lists do. We should explain that on the
> wiki. I guess this is too complicated to explain to our users in one
> sentence and it needs at least a page of text. People who do not read
> that have you just lost out.
>>>>>
>>>>> [snip]
>>>>>
>>>>> May I opine that the "Safe" information would be helpful to me in
> the WUI. Perhaps we can be more explicit, or better explain, such as is
> often done with RBLs in mail server settings, where lists are sometimes
> described in terms of their likelihood to cause false-positives.
>>>>>
>>>>> It's all well and good in the documentation, but a quick
> "Safe|Moderate|Risky" listing in the WUI will prove handy, IMHO.
>>>>>
>>>>> Just my $0.02 as more of a user than a developer,
>>>>
>>>> I appreciate your input, but I still disagree with is that we take
> the decision if something is “risky” or not. There are too many things
> that need to be taken into account to make that decision and it probably
> varies for each user.
>>>>
>>>> What I take from your comment though is that we should categorise
> the lists, and that is something we can do.
>>>>
>>>> We can add a headline to the table and group the lists by “Blocking
> ambiguous packets”, “Blocking Malware”, etc.
>>>>
>>>> That makes it easier for the user to decide which lists are
> interesting or even necessary depending on what they want to achieve.
>>>>
>>>> How is that?
>>>>
>>>> -Michael
>>>>
>>>>>
>>>>> Tom
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 0/5] ipblacklist: IP Address Blacklists
2020-01-06 11:21 ` Michael Tremer
@ 2020-01-22 20:35 ` Tim FitzGeorge
2020-01-23 10:53 ` Michael Tremer
0 siblings, 1 reply; 35+ messages in thread
From: Tim FitzGeorge @ 2020-01-22 20:35 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 4666 bytes --]
Hello Michael,
On 06/01/2020 11:21, Michael Tremer wrote:
> Hello Tim,
>
>> On 28 Dec 2019, at 21:17, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>
>> Hi,
>>
>> Having decided that we'll categorise the lists, the question is what
>> categories to use. They need to be:
>>
>> - Short (to fit on the screen)
>> - Easily translatable
>> - and above all, useful.
>>
>> Looking at the lists the obvious categories are:
>>
>> - Invalid Address (on the public internet)
>> BOGON, BOGON_FULL
>>
>> - Scanner (not by itself malicious)
>> SHODAN
>>
>> - Application (potentially unwanted)
>> TOR_ALL, TOR_EXIT
>>
>> - Malware C & C
>> FEODO_RECOMMENDED, FEODO_IP, FEODO_AGGRESIVE
>>
>> - Composite
>> EMERGING_FWRULE
>
> I like all these a lot.
>
>> Less obvious are:
>>
>> - Reputation
>> ALIENVAULT, CIARMY, SPAMHAUS_DROP, SPAMHAUS_EDROP
>>
>> - Attacks
>> BLOCKLIST_DE, DSHIELD, EMERGING_COMPROMISED
>
> I even like those two, although I would potentially consider merging
“Invalid Address” and Reputation. They are kind of the same to me. IP
addresses I under no circumstances I want to talk to.
I think I'd rather keep them separate, since conceptually they're rather
different. Also, if your red interface has a private address, you
definitely wouldn't want to enable these lists.
>
> I also like the Attacks category, although the name is very generic.
But I cannot come up with anything better. The only thing that might be
worth considering is to merge it with Malware and just call it “Malicious”.
I suspect that anything more specific that Attacks is going to only
describe a single list. It's (unfortunately) a catch-all category for
anything that doesn't fit better somewhere else.
I'd rather not merge it with Malware C&C since the behaviour of the two
is rather different - under most circumstances lists in this category
shouldn't block any packets, whereas the Attacks category is expected to
block a lot of inbound traffic.
>
>> I'm not sure that the distinction between these two is going to be
>> helpful to most people (I'm not sure I understand it myself).
>>
>> We could use:
>>
>> - Top attackers
>> DSHIELD, EMERGING_COMPROMISED, SPAMHAUS_DROP, SPAMHAUS_EDROP
>>
>> - Other attackers
>> ALIENVAULT, BLOCKLIST_DE, CIARMY
>>
>> but that might be making a distinction that is better made by the user.
>
> Agreed. It is not obvious why some are top attackers and others are not.
>
> So I would 100% prefer the first option from above.
>
> Best,
> -Michael
>
Tim
>>
>> Any opinions?
>>
>> Tim
>>
>>
>> On 18/12/2019 12:10, Michael Tremer wrote:
>>> Hi,
>>>
>>>> On 16 Dec 2019, at 23:05, Tom Rymes <trymes(a)rymes.com> wrote:
>>>>
>>>> On 12/16/2019 5:20 PM, Michael Tremer wrote:> Hi,
>>>>>
>>>>>> On 16 Dec 2019, at 20:06, Tim FitzGeorge
<lists(a)tfitzgeorge.me.uk> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I've attached the current GUI screenshot.
>>>>>
>>>>> Thanks for that.
>>>>>
>>>>> I have a couple of suggestions/concerns about it:
>>>>
>>>> [snip]
>>>>
>>>>> c) I would suggest to remove the “safe” column because that is a
very hard summary of what the lists do. We should explain that on the
wiki. I guess this is too complicated to explain to our users in one
sentence and it needs at least a page of text. People who do not read
that have you just lost out.
>>>>
>>>> [snip]
>>>>
>>>> May I opine that the "Safe" information would be helpful to me in
the WUI. Perhaps we can be more explicit, or better explain, such as is
often done with RBLs in mail server settings, where lists are sometimes
described in terms of their likelihood to cause false-positives.
>>>>
>>>> It's all well and good in the documentation, but a quick
"Safe|Moderate|Risky" listing in the WUI will prove handy, IMHO.
>>>>
>>>> Just my $0.02 as more of a user than a developer,
>>>
>>> I appreciate your input, but I still disagree with is that we take
the decision if something is “risky” or not. There are too many things
that need to be taken into account to make that decision and it probably
varies for each user.
>>>
>>> What I take from your comment though is that we should categorise
the lists, and that is something we can do.
>>>
>>> We can add a headline to the table and group the lists by “Blocking
ambiguous packets”, “Blocking Malware”, etc.
>>>
>>> That makes it easier for the user to decide which lists are
interesting or even necessary depending on what they want to achieve.
>>>
>>> How is that?
>>>
>>> -Michael
>>>
>>>>
>>>> Tom
>>>
>>
>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 0/5] ipblacklist: IP Address Blacklists
[not found] <aeb93668-a4b6-5735-1f68-fd53cafa2210@tfitzgeorge.me.uk>
@ 2020-01-06 11:27 ` Michael Tremer
2020-01-24 19:40 ` Tim FitzGeorge
0 siblings, 1 reply; 35+ messages in thread
From: Michael Tremer @ 2020-01-06 11:27 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 31579 bytes --]
Hello all,
> On 4 Jan 2020, at 19:02, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>
> Hello,
>
> I've now updated the code - screenshots are attached.
>
> IP_Address_Blacklists.png
>
> This is the settings page. I don't think there's anything unexpected here.
Yes, that looks a lot like the other IPFire pages :)
> IP_Address_Blacklist_Logs.png
>
> The log page, replacing the status information that was previously at
> the top of the settings page.
Looks great!
> I've sorted the lists alphabetically, rather than by the number of hits
> since the worst threat is not necessary the largest number of blocked
> packets (which could implied by sorting it by number).
Yes, that is indeed a very good idea.
You could add a % sign in the percentage column, so it cannot be confused so easily with the other column.
> Firewall_log_blacklist.png
>
> Details page accessed from the log. Almost identical to the other
> similar pages.
*thumbs up*
> Log_Summary.png
>
> This is an extract from the Log Summary page produced by logwatch.
>
> I still need to do some tidying, remove unused language strings etc.,
> and then update and test the patches, so it'll be some time before I'm
> ready to submit updated patches.
Feel free to send some RFC to the list. It might be slightly noisy, but it is easier to review changes bit after bit.
Great great work!
-Michael
>
> Tim
>
> On 28/12/2019 20:59, Tim FitzGeorge wrote:
>> Hi,
>>
>> I'll code and test what we've agreed on. In the meantime I'm going to
>> reply on the sub-thread started by Tom Rymes as to what we call the list
>> categories, so that we can keep that discussion separate from further
>> functionality etc. discussion.
>>
>> Tim
>>
>> On 21/12/2019 18:34, Tim FitzGeorge wrote:
>>> Hi,
>>>
>>> I've got my systems checking for downloads at 15 minute intervals and
>>> using If-Modified-Since. This all seems to be working OK.
>>>
>>> Responses to comments below:
>>>
>>> On 18/12/2019 12:07, Michael Tremer wrote:
>>>> Hi,
>>>>
>>>>> On 17 Dec 2019, at 18:21, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I think we've agreed on the following:
>>>>>
>>>>> - Checks to be made every 15 minutes, subject to per-list minimum rate.
>>>>> - Modify download to use If-Modified-Since rather than separate HEAD
>>>>> request.
>>>>> - Remove automatic blacklist.
>>>>
>>>> I thought someone wants to counter my argument. This is not a dictatorship :)
>>>>
>>>>>
>>>>> other comments are below.
>>>>>
>>>>> Tim
>>>>>
>>>>> On 16/12/2019 22:20, Michael Tremer wrote:
>>>>>> Hi,
>>>>>>
>>>>>>> On 16 Dec 2019, at 20:06, Tim FitzGeorge <lists(a)tfitzgeorge.me.uk> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I've attached the current GUI screenshot.
>>>>>>
>>>>>> Thanks for that.
>>>>>>
>>>>>> I have a couple of suggestions/concerns about it:
>>>>>>
>>>>>> a) I am not sure what the value is about the top box that is called “Status”. It is basically a summary of the iptables output, but will it help the user? A blacklist with more or fewer entries is not better or worse, blocking more packets isn’t better than blocking fewer. It is about the quality of the blocks.
>>>>>
>>>>> I agree that the size of the lists is not all that useful, but the
>>>>> information about the number of blocked packets can be an indication of
>>>>> compromise. It depends on the type of the list, but a list that targets
>>>>> C&C channels (e.g. the FEODO lists) should not show any input packets
>>>>> being blocked, unless one of the computers being protected is infected.
>>>>> Apart from the BOGON lists, none of the lists should show any output
>>>>> packets being blocked; again it's a potential indication of infection if
>>>>> otherwise.
>>>>
>>>> The Shodan list will definitely show some activity unless you are behind CG NAT.
>>>>
>>>> Should we not have this rather in the logging section?
>>>>
>>>> And these counters here will be reset when the firewall is being reloaded. Should this data therefore not be collected from the logs so that you can go back in time? That would be very important to identify any start of compromise.
>>>>
>>>
>>> That makes sense. I'll remove the status from this page and create a
>>> log page for the information. If anyone actually wants the iptables
>>> information it's available in from Firewall > iptables anyway.
>>>
>>> Do you think it makes more sense to show packets blocked in/out or
>>> packets blocked per source interface? The latter would of (limited)
>>> help in tracking down potential compromise.
>>>
>>>>>>
>>>>>> b) I would prefer to move the settings box to the top. If you like you can show the size of the blacklists there and when they have been updated, too.
>>>>>
>>>>> I put the status at the top since it's usually what I want to see when I
>>>>> go to the page. Under normal circumstances I don't change the settings,
>>>>> so I'd rather not have to scroll past them to get to the information I'm
>>>>> actually interested in. The update times are just useful to show that
>>>>> the lists are actually being updated. The size is interesting to me,
>>>>> but as you say it probably won't help the average user, especially if
>>>>> you don't know which lists block individual addresses and which block
>>>>> nets (BOGON, DSHIELD and EMERGING_COMPROMISED).
>>>>
>>>> See my comment on the order above. Splitting the page might make sense.
>>>>
>>>> When you show size, does that mean lines in the blacklists? So it could be networks or single IP addresses? Does it not make sense to count IP addresses (i.e. 256 for a /24 network and so on) and put it in the table?
>>> It's lines in the blacklist.
>>>
>>>>
>>>> I think I would only be interested in the size for two things:
>>>>
>>>> a) Is there any chance for overblocking? i.e. does the list have 2 billion entries which would be half the public IPv4 address space or so. I cannot really come up with a good example here.
>>>>
>>>> Or
>>>>
>>>> b) What is the performance impact/memory consumption of the list?
>>>
>>> It would be interesting to show the type (single address or net), number
>>> of entries and number of addresses; the first two affect performance,
>>> the second memory use and the latter the number of blocked addresses,
>>> but that's really only going to be of interest to a very small number of
>>> people, who can always do 'ipset list -t' to get the information. So
>>> I'll remove the size column.
>>>
>>>>
>>>>> I originally had the status and settings in one list, but I decided that
>>>>> it was getting a bit messy and difficult to follow. I've attached a
>>>>> screenshot of a quickly hacked page. (Note that the status area only
>>>>> appears once the blacklists have been enabled.)
>>>>
>>>> You are right. This is getting quite tight there.
>>>>
>>>> You could write packets in/out into on column like 2k/1023, but see my concerns about this above.
>>>
>>> Actually I've come to the conclusion that the number of bytes blocked
>>> isn't all that important; it's sufficient to show the number of packets.
>>>
>>>>
>>>>>>
>>>>>> c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
>>>>>
>>>>> I agree that it's it's an over simplistic summary of the lists. I note
>>>>> that Tom Rymes has asked to keep it, but I think its only value is if it
>>>>> suggests to users that things are more complicated than they appear and
>>>>> therefore prompts them to read the Wiki.
>>>>
>>>> I will reply to that separately.
>>>
>>> Looking at that reply, I think it makes sense to include a category
>>> column in both the settings and the log summary page - the former to
>>> help selection of which lists to use, and the latter as a help when
>>> deciding whether the information could indicate compromise or not.
>>>
>>>>
>>>>>>
>>>>>> d) I do not understand what the “update check rate” means. I suppose we should show the update interval of the lists in the table and just refresh them according to that.
>>>>>
>>>>> It's the minimum check period for updates. The system checks for
>>>>> updates at either the rate specific to the list or this rate, whichever
>>>>> is slower. This is a hangover from the early days, copied from the IPS
>>>>> updates. I suppose it could have a value for some people who may wish
>>>>> to lower the check rate, but I'm not sure that it has sufficient value
>>>>> to be worth keeping.
>>>>
>>>> No, I would not let the user decide when and how often to update the lists.
>>>
>>> OK. I'll remove it.
>>>
>>>>
>>>> Generally there is this argument internally how often some databases should be updated. People with slow internet connections or volume based data plans will find this rather expensive to update those lists too often.
>>>>
>>>> If we ever want to go ahead with that in IPFire 2, there should be a global switch for a “low data mode” which then delays updating these things.
>>>
>>> That makes sense. Maybe also the ability to set when downloads take
>>> place - but as you say that's for the future.
>>>
>>>>
>>>>>>
>>>>>> e) Do we want to keep the automatic blacklists now? I do not think it will actually improve the security of IPFire.
>>>>>>
>>>>>
>>>>> I'll remove them. I think they do have a use, in that they can detect
>>>>> someone doing a port scan and block them before they find an open port,
>>>>> but as you say there is also the possibility of a DOS.
>>>>
>>>> How would this block a DOS?
>>>
>>> Sorry, I didn't make myself clear - there's the possibility that someone
>>> could cause a DOS with the automatic list by faking the source IP
>>> address and sending a few packets on a random port.
>>>
>>>>
>>>> -Michael
>>>
>>> Tim
>>>>
>>>>>>>
>>>>>>> I'll update the code based on our discussions, and submit an updated set
>>>>>>> of patches - I imagine there will have to be at least one more iteration.
>>>>>>
>>>>>> Let’s wait until we have come to decisions :)
>>>>>>
>>>>>> -Michael
>>>>>>
>>>>>>>
>>>>>>> Tim
>>>>>>>
>>>>>>> (No additional comments below)
>>>>>>>
>>>>>>> On 13/12/2019 23:11, Michael Tremer wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Again my apologies for my late reply. Busy busy weeks.
>>>>>>>>
>>>>>>>>> On 8 Dec 2019, at 20:50, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> It's my turn to apologise for being slow to respond - I've had a busy
>>>>>>>>> week, but I should have plenty of time over the next couple of weeks.
>>>>>>>>
>>>>>>>> No worries. Turns out we all do :)
>>>>>>>>
>>>>>>>>> I've made most of the comments inline, however I think Michael had a
>>>>>>>>> question (which I can't find now) about what happens if someone enables
>>>>>>>>> all the lists. One thing which would perhaps make this less likely is
>>>>>>>>> that the WUI tags the available lists with whether they're safe or not,
>>>>>>>>> with a footnote that safe means that the list only blocks malicious
>>>>>>>>> traffic. This won't guarantee that a user won't still try to enable all
>>>>>>>>> the lists, but it should make them realise that they should think first.
>>>>>>>>
>>>>>>>> We had a couple of features going slightly wrong or being “misunderstood” by some users. People still seem to panic when they see “local recursor”. To this day I do not know why.
>>>>>>>>
>>>>>>>> We cannot make everything idiot-proof. And when some user if of that category, they probably should shutdown their IPFire box, educate themselves and then come back again. So I do not want to limit people, but make things as easy as possible.
>>>>>>>>
>>>>>>>> If someone enables all the lists, good luck with passing packets :)
>>>>>>>>
>>>>>>>>> I have considered replacing this tag with a risk high/medium/low and
>>>>>>>>> maybe adding a category (invalid/application/scanner/C&C or something
>>>>>>>>> like that), but that may provide too much information and dissuade them
>>>>>>>>> from actually following the links to checkout what the list actually does.
>>>>>>>>
>>>>>>>> Can we have a screenshot of the GUI right now? I didn’t run the code, yet.
>>>>>>>>
>>>>>>>> We should document the lists like we do it with the rulesets of the IPS. People might ignore this, but that is on them.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Tim
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 05/12/2019 22:25, Michael Tremer wrote:
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>>> On 4 Dec 2019, at 17:05, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hello Tim, hello Michael,
>>>>>>>>>>>
>>>>>>>>>>> please see my responses inline...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We could periodically update the blacklists on our main mirror (and
>>>>>>>>>>>>>> wait for the network to sync it), make sure it is signed and write
>>>>>>>>>>>>>> a small downloader that fetches, validates and installs them.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> @All: Thoughts on this?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think there are a number of points here.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Firstly, from the point of a third party using IPFire, is this really
>>>>>>>>>>>>> solving the privacy disclosure problem? There's no way round disclosing
>>>>>>>>>>>>> your public IP address to someone you're downloading from; all this does
>>>>>>>>>>>>> is change who that information is being disclosed to. For the user
>>>>>>>>>>>>> there's no way of knowing whether the source is more or less protective
>>>>>>>>>>>>> of the user's privacy than the blacklist provider. Indeed it won't be
>>>>>>>>>>>>> possible to know who the lists are being downloaded from until the
>>>>>>>>>>>>> download starts.
>>>>>>>>>>>>
>>>>>>>>>>>> There is a way: Tor. But that is a totally different story.
>>>>>>>>>>> Well, I see a third option on this: Use the mirror infrastructure we already
>>>>>>>>>>> have. Every IPFire installation discloses its public IP address to one of these
>>>>>>>>>>> servers sooner or later, so we do not disclose additional data if the blacklists
>>>>>>>>>>> were fetched from these.
>>>>>>>>>>>
>>>>>>>>>>> Needless to say, Tor (hidden services) would be better, but that is a different
>>>>>>>>>>> story indeed. :-)
>>>>>>>>>>>>
>>>>>>>>>>>> The point is rather that a forget list can be sent instead of the “real” one.
>>>>>>>>>>> I did not get this. Forget? Forged?? ???
>>>>>>>>>>
>>>>>>>>>> Yes, I meant to write forged, but auto-correct didn’t let me.
>>>>>>>>>>
>>>>>>>>>>>>> Secondly, latency; some of the lists are updated every 5 minutes. While
>>>>>>>>>>>>> I've limited the maximum check rate to hourly, will the updates
>>>>>>>>>>>>> propagate quickly enough. For reference on my main system the 24
>>>>>>>>>>>>> updates on the CIARMY list made 143 498 changes (additions or
>>>>>>>>>>>>> deletions). I've seem it do over 200 000.
>>>>>>>>>>> Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not
>>>>>>>>>>> document a recommended update interval anywhere, so we can only guess.
>>>>>>>>>>>
>>>>>>>>>>> Personally, more static lists seem to be preferable for packet filtering. Highly
>>>>>>>>>>> dynamic ones such as CIArmy should be done via DNSBL queries or something similar
>>>>>>>>>>> - do we really want to have that list here?
>>>>>>>>>>
>>>>>>>>>> It is not really an option to implement a DNSBL into a packet filter, but I get your point.
>>>>>>>>>
>>>>>>>>> One of the 'selling points' for an IP address blacklist is that it can
>>>>>>>>> respond quickly to new threats - or rather new attackers. While a new
>>>>>>>>> IDS/IPS rule needs time to analyse the threat, generate a rule and check
>>>>>>>>> it, it's easy to add an address to a list. So, I think the CIArmy list
>>>>>>>>> is potentially useful for protecting home systems etc. with budget
>>>>>>>>> hardware, but I would be very careful about using it for a protecting a
>>>>>>>>> general access website.
>>>>>>>>
>>>>>>>> If they are very volatile, we should honour that and update them often, too.
>>>>>>>>
>>>>>>>> It probably is more about false-positives being removed very quickly instead of adding threats very quickly. The average IPFire user is probably not under threats like these to need to react very quickly.
>>>>>>>>
>>>>>>>> So I do not see much value in adding those lists and then updating once a day. Does it have to be every 5 min? No. I would suggest 15 which should be good enough for everyone.
>>>>>>>>
>>>>>>>> Other lists should of course not be updated every 15 minutes when not needed.
>>>>>>>>
>>>>>>>> Running every 15 minutes would allow us to retry downloading lists that are on an hourly schedule if the download failed.
>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>> How did you come up with the hour? Will it be retried more often if the download was not successful?
>>>>>>>>>>> One hour is the most common interval indeed, but adding some random time might
>>>>>>>>>>> be useful in order to reduce load on the servers providing a blacklist.
>>>>>>>>>>
>>>>>>>>>> Yes, definitely. Otherwise we will shoot down our mirrors.
>>>>>>>>>
>>>>>>>>> When I implemented that section of code, specifying the minimum check
>>>>>>>>> period in hours seemed to provide a convenient way of allowing a check
>>>>>>>>> period covering a wide range, with an hour as the fastest and a week as
>>>>>>>>> the slowest. I didn't looked at the CIArmy list until much later. Most
>>>>>>>>> of the lists don't change nearly as much, but the CIArmy list is
>>>>>>>>> described as one that deliberately responds quickly.
>>>>>>>>>
>>>>>>>>> From my production system, for yesterday:
>>>>>>>>>
>>>>>>>>> The following block lists were updated:
>>>>>>>>> BLOCKLIST_DE: 24 Time(s) - 9341 change(s)
>>>>>>>>> BOGON_FULL: 1 Time(s) - 10 change(s)
>>>>>>>>> CIARMY: 24 Time(s) - 159134 change(s)
>>>>>>>>> DSHIELD: 7 Time(s) - 18 change(s)
>>>>>>>>> EMERGING_FWRULE: 1 Time(s) - 50 change(s)
>>>>>>>>> FEODO_AGGRESIVE: 24 Time(s) - 13 change(s)
>>>>>>>>> SHODAN: 1 Time(s) - 0 change(s)
>>>>>>>>> SPAMHAUS_DROP: 1 Time(s) - 0 change(s)
>>>>>>>>> TOR_EXIT: 24 Time(s) - 162 change(s)
>>>>>>>>
>>>>>>>> Very interesting statistics.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> and my test system:
>>>>>>>>>
>>>>>>>>> The following block lists were updated:
>>>>>>>>> ALIENVAULT: 19 Time(s) - 5331 change(s)
>>>>>>>>> EMERGING_COMPROMISED: 1 Time(s) - 26 change(s)
>>>>>>>>> TALOS_MALICIOUS: 1 Time(s) - 36 change(s)
>>>>>>>>>
>>>>>>>>> That covers most of the lists. From the WUI, since 1 Dec:
>>>>>>>>>
>>>>>>>>> Blacklist Entries pkts bytes Last updated
>>>>>>>>> in in
>>>>>>>>> AUTOBLACKLIST 0 731 51144 Sun Dec 8 18:40:02 2019
>>>>>>>>> BLOCKLIST_DE 28020 857 46735 Sun Dec 8 18:40:02 2019
>>>>>>>>> BOGON_FULL 214 5255 189K Sun Dec 8 16:50:04 2019
>>>>>>>>> CIARMY 15000 19774 976K Sun Dec 8 18:04:01 2019
>>>>>>>>> DSHIELD 20 7992 321K Sun Dec 8 16:45:13 2019
>>>>>>>>> EMERGING_FWRULE 1647 197 8383 Fri Dec 6 05:29:07 2019
>>>>>>>>> FEODO_AGGRESIVE 7169 0 0 Sun Dec 8 18:50:07 2019
>>>>>>>>> SHODAN 32 34 1530 Sun Dec 8 17:54:09 2019
>>>>>>>>> SPAMHAUS_DROP 823 0 0 Fri Dec 6 18:22:35 2019
>>>>>>>>> SPAMHAUS_EDROP 111 82 16433 Thu Dec 5 17:27:09 2019
>>>>>>>>> TOR_EXIT 1055 0 0 Sun Dec 8 18:31:02 2019
>>>>>>>>
>>>>>>>> This as well.
>>>>>>>>
>>>>>>>> Those are more packets than I would have expected.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> (I've left out the pkts/bytes out fields which were all 0)
>>>>>>>>>
>>>>>>>>> Note that where possible I do a HEAD request first and then only
>>>>>>>>> download the list if the modification time has changed since the last
>>>>>>>>> check. For dynamically generated lists this isn't possible.
>>>>>>>>
>>>>>>>> You won’t need a HEAD request for it. You can include it in the GET request.
>>>>>>>>
>>>>>>>> Have a look at location downloader where I use that. The server will respond with 304 and not send any payload.
>>>>>>>>
>>>>>>>> https://git.ipfire.org/?p=location/libloc.git;a=blob;f=src/python/location-downloader.in;h=1b5932d5822e03737d32b2a27816815b2f7e74dd;hb=HEAD#l151
>>>>>>>>
>>>>>>>>>
>>>>>>>>> If the download isn't successful it just gives up and waits for the next
>>>>>>>>> attempt (apart from the usual retries in the library). I probably
>>>>>>>>> should to change that so that it only applies the per list minimum
>>>>>>>>> update period in this case (specified in the sources file) rather than
>>>>>>>>> the user specified value as well.
>>>>>>>>
>>>>>>>> I think it is not the worst if an update fails. It might just happen every once in a while.
>>>>>>>>
>>>>>>>> So I would suggest to just re-run the script more often and when the mtime of the file is older than the threshold, a download is attempted. You can use that timestamp for the GET request.
>>>>>>>>
>>>>>>>>> I already use a time offset on the downloads - when it's started from
>>>>>>>>> boot, backup restore or WUI enable, it checks to see if it's installed
>>>>>>>>> in the fcrontab, and if not adds itself at a randomly generated offset
>>>>>>>>> in the hour.
>>>>>>>>
>>>>>>>> That should potentially go to red.up, or if we can settle on 15 minutes, I would consider that often enough to quickly update all lists after a reboot.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is
>>>>>>>>>>>>> the largest at a few MB), there are going to be a lot of them. How will
>>>>>>>>>>>>> this affect the willingness of people to mirror IPFire?
>>>>>>>>>>> I do not consider this being a problem as we do not generate that much traffic
>>>>>>>>>>> to them. Of course, that depends on the update interval again.
>>>>>>>>>>
>>>>>>>>>> That depends on your point of view.
>>>>>>>>>>
>>>>>>>>>> I do not have a problem with this at all in my data center, but there are plenty of people with a volume-based LTE plan or simply a 128 kBit/s connection. It will take a longer time to download the lists for them. We need to mind that.
>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Talking about the preference of packet filter and IPS, I prefer to
>>>>>>>>>>>>>> use the latter as well as it gains more insight in what kind of malicious
>>>>>>>>>>>>>> traffic tried to pass a firewall machine. On systems with low resources,
>>>>>>>>>>>>>> this might be problematic and removing load from the IPS can be preferred
>>>>>>>>>>>>>> (make this configurable?!), on others, people might want to have both
>>>>>>>>>>>>>> results.
>>>>>>>>>>>>>>
>>>>>>>>>>>>> You're only going to get one result for a packet whichever way round the
>>>>>>>>>>>>> IP blacklist and IPS are since whichever comes first will drop the
>>>>>>>>>>>>> packet before it reaches the second (well it would be possible to put
>>>>>>>>>>>>> the IP blacklist first and get it to log and mark packets which are then
>>>>>>>>>>>>> dropped after the IPS, but I think that's getting a little complicated.
>>>>>>>>>>>>> In addition I've seen the messages about the trouble marking was
>>>>>>>>>>>>> causing in the QoS).
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think it's a 50/50 choice as to which is more valuable first; it's
>>>>>>>>>>>>> probably going to differ from packet to packet. For me the possibility
>>>>>>>>>>>>> of reducing the IPS load means I prefer putting the IP blacklist first
>>>>>>>>>>>>>
>>>>>>>>>>>>> It should be fairly easy to add the choice of where to put the IP
>>>>>>>>>>>>> blacklist. I think it'll have to be in the main firewall script, so
>>>>>>>>>>>>> it'll require a firewall restart, but it's not something that'll be
>>>>>>>>>>>>> changed often.
>>>>>>>>>>>>
>>>>>>>>>>>> I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier.
>>>>>>>>>>> Agreed.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads.
>>>>>>>>>>>>
>>>>>>>>>>>> We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily.
>>>>>>>>>>> So dropping packets from blacklisted IP addresses/networks before IPS is it, then.
>>>>>>>>>>>>
>>>>>>>>>>>> I do not even think it makes sense to swap the order in the outgoing direction.
>>>>>>>>>>> Me too.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this.
>>>>>>>>>>> Based on my personal experience, firewall hits observed on a single machine exposed
>>>>>>>>>>> to the internet are interesting, but the overall situation across multiple machines
>>>>>>>>>>> is even more interesting. Very quickly, you'll end on something like a centralised
>>>>>>>>>>> logging server and custom statistical analysis here...
>>>>>>>>>>
>>>>>>>>>> Probably a project for IPFire 4.0 :)
>>>>>>>>>>
>>>>>>>>> Or use one of the existing services, like the DSHIELD client
>>>>>>>>> https://dshield.org/howto.html (subject to privacy concerns again).
>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default.
>>>>>>>>>>>>>> I would love to see the bogon ruleset here, too (think about 8chan successor
>>>>>>>>>>>>>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause
>>>>>>>>>>>>>> interference if RED does not have a public IP address assigned.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I can add a field to the options file that controls whether a list is
>>>>>>>>>>>>> enabled by default.
>>>>>>>>>>> Thank you. :-)
>>>>>>>>>>>>
>>>>>>>>>>>> To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat.
>>>>>>>>>>> This is my only objection against this patchset. Now, what can we do about it?
>>>>>>>>>>> One possibility is to apply the patchset now and implement a custom download
>>>>>>>>>>> source thing later on, or do that before releasing Core Update 139 (or which version
>>>>>>>>>>> the patchset will be to) after we agreed on something.
>>>>>>>>>>
>>>>>>>>>> I do not see this being merged for 139. But that is not important. We need to get it right first and then release it.
>>>>>>>>>>
>>>>>>>>>> As far as I know, nobody has tested this, yet.
>>>>>>>>>
>>>>>>>>> There are a number of people who have been running an earlier version
>>>>>>>>> which I shared on GitHub. There were a few early issues, but it seems
>>>>>>>>> to be OK now.
>>>>>>>>>
>>>>>>>>> https://forum.ipfire.org/viewtopic.php?f=27&t=21845
>>>>>>>>>
>>>>>>>>> This version wasn't integrated into IPFire, so (for example) it inserted
>>>>>>>>> itself into the INPUT IPTables chain rather than having it's chains
>>>>>>>>> created as part of the firewall start-up script.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I have huge concerns about the automatic blacklist. @Peter: What is your opinion on this?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> While I implemented it, I'm aware of its potential to cause problems,
>>>>>>>>> which is why it has to be separately enabled. It's not caused me any
>>>>>>>>> issues at the default settings (blocks at over 10 packets per hour until
>>>>>>>>> 1 hour has passed without seeing packets from the address), but I've not
>>>>>>>>> used it on a site with publicly announced services. If I was going to
>>>>>>>>> use it on a web site I would want to, at the very minimum, drop the
>>>>>>>>> block period drastically.
>>>>>>>>
>>>>>>>> I suppose this is entirely unusable on an IPFire box in a data center that hosts things. Let’s say our rack in Hanover.
>>>>>>>>
>>>>>>>> You will have hits from some broken IP stacks and people might just end up on this without doing anything wrong.
>>>>>>>>
>>>>>>>> You can denial-of-service easily as well and I suppose without aggregation of data from many many systems, it does not make sense to instantly block addresses. And even then you probably would block an entire subnet.
>>>>>>>>
>>>>>>>> So I am not sure how I should feel about it. I do not think that it adds anything because the packets would have been blocked anyways.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> On the other hand, it's good at responding quickly. Usually I see only
>>>>>>>>> 1-2% of blocks from the automatic list:
>>>>>>>>>
>>>>>>>>> Reason Count % First Last
>>>>>>>>> CIARMY 2416 45 Dec 6 00:00 Dec 6 23:59
>>>>>>>>> DSHIELD 1353 25 Dec 6 00:00 Dec 6 23:59
>>>>>>>>> INPUT 1294 24 Dec 6 00:00 Dec 6 23:59
>>>>>>>>> AUTOBLACKLIST 122 2 Dec 6 00:20 Dec 6 16:28
>>>>>>>>> BLOCKLIST_DE 89 2 Dec 6 00:20 Dec 6 23:46
>>>>>>>>>
>>>>>>>>> and sometimes none at all, but one one occasion it blocked over 8000
>>>>>>>>> packets. Again I'm aware this is for a home system, which is rather
>>>>>>>>> different than from a Web server.
>>>>>>>>>
>>>>>>>>>>> If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not
>>>>>>>>>>> seem to be problematic, as they are hosted on 3rd party servers, too).
>>>>>>>>>>
>>>>>>>>>> One of them will be, sooner or later. And one is enough I suppose.
>>>>>>>>>
>>>>>>>>> DShield (https://dshield.org/api/#threatfeeds) and firehol
>>>>>>>>> (http://iplists.firehol.org/) seem to host copies of most of the lists
>>>>>>>>> as well.
>>>>>>>>>>
>>>>>>>>>> I do not really want to overthink this - we didn’t do this with clamav for example either. But it probably is a decision to be made by the user what they want to enable and we should not enable anything by default. So no data will be leaked as long as the user does not consent.
>>>>>>>>>>
>>>>>>>>>> -Michael
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks, and best regards,
>>>>>>>>>>> Peter Müller
>>>>>>>>
>>>>>>>
>>>>>>> <Screenshot_2019-12-16_IP_Address_Blacklists.png>
>>>>>>
>>>>>
>>>>> <Screenshot_2019-12-17_IP_Address_Blacklists.png>
>>>>
>>>
>>
>
> <IP_Address_Blacklists.png><IP_Address_Blacklist_Logs.png><Firewall_log_blacklist.png><Log_Summary.png>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 0/5] ipblacklist: IP Address Blacklists
2019-12-28 21:17 ` Tim FitzGeorge
@ 2020-01-06 11:21 ` Michael Tremer
2020-01-22 20:35 ` Tim FitzGeorge
0 siblings, 1 reply; 35+ messages in thread
From: Michael Tremer @ 2020-01-06 11:21 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 3849 bytes --]
Hello Tim,
> On 28 Dec 2019, at 21:17, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>
> Hi,
>
> Having decided that we'll categorise the lists, the question is what
> categories to use. They need to be:
>
> - Short (to fit on the screen)
> - Easily translatable
> - and above all, useful.
>
> Looking at the lists the obvious categories are:
>
> - Invalid Address (on the public internet)
> BOGON, BOGON_FULL
>
> - Scanner (not by itself malicious)
> SHODAN
>
> - Application (potentially unwanted)
> TOR_ALL, TOR_EXIT
>
> - Malware C & C
> FEODO_RECOMMENDED, FEODO_IP, FEODO_AGGRESIVE
>
> - Composite
> EMERGING_FWRULE
I like all these a lot.
> Less obvious are:
>
> - Reputation
> ALIENVAULT, CIARMY, SPAMHAUS_DROP, SPAMHAUS_EDROP
>
> - Attacks
> BLOCKLIST_DE, DSHIELD, EMERGING_COMPROMISED
I even like those two, although I would potentially consider merging “Invalid Address” and Reputation. They are kind of the same to me. IP addresses I under no circumstances I want to talk to.
I also like the Attacks category, although the name is very generic. But I cannot come up with anything better. The only thing that might be worth considering is to merge it with Malware and just call it “Malicious”.
> I'm not sure that the distinction between these two is going to be
> helpful to most people (I'm not sure I understand it myself).
>
> We could use:
>
> - Top attackers
> DSHIELD, EMERGING_COMPROMISED, SPAMHAUS_DROP, SPAMHAUS_EDROP
>
> - Other attackers
> ALIENVAULT, BLOCKLIST_DE, CIARMY
>
> but that might be making a distinction that is better made by the user.
Agreed. It is not obvious why some are top attackers and others are not.
So I would 100% prefer the first option from above.
Best,
-Michael
>
> Any opinions?
>
> Tim
>
>
> On 18/12/2019 12:10, Michael Tremer wrote:
>> Hi,
>>
>>> On 16 Dec 2019, at 23:05, Tom Rymes <trymes(a)rymes.com> wrote:
>>>
>>> On 12/16/2019 5:20 PM, Michael Tremer wrote:> Hi,
>>>>
>>>>> On 16 Dec 2019, at 20:06, Tim FitzGeorge <lists(a)tfitzgeorge.me.uk> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I've attached the current GUI screenshot.
>>>>
>>>> Thanks for that.
>>>>
>>>> I have a couple of suggestions/concerns about it:
>>>
>>> [snip]
>>>
>>>> c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
>>>
>>> [snip]
>>>
>>> May I opine that the "Safe" information would be helpful to me in the WUI. Perhaps we can be more explicit, or better explain, such as is often done with RBLs in mail server settings, where lists are sometimes described in terms of their likelihood to cause false-positives.
>>>
>>> It's all well and good in the documentation, but a quick "Safe|Moderate|Risky" listing in the WUI will prove handy, IMHO.
>>>
>>> Just my $0.02 as more of a user than a developer,
>>
>> I appreciate your input, but I still disagree with is that we take the decision if something is “risky” or not. There are too many things that need to be taken into account to make that decision and it probably varies for each user.
>>
>> What I take from your comment though is that we should categorise the lists, and that is something we can do.
>>
>> We can add a headline to the table and group the lists by “Blocking ambiguous packets”, “Blocking Malware”, etc.
>>
>> That makes it easier for the user to decide which lists are interesting or even necessary depending on what they want to achieve.
>>
>> How is that?
>>
>> -Michael
>>
>>>
>>> Tom
>>
>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 0/5] ipblacklist: IP Address Blacklists
2019-12-18 12:10 ` Michael Tremer
@ 2019-12-28 21:17 ` Tim FitzGeorge
2020-01-06 11:21 ` Michael Tremer
0 siblings, 1 reply; 35+ messages in thread
From: Tim FitzGeorge @ 2019-12-28 21:17 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 2988 bytes --]
Hi,
Having decided that we'll categorise the lists, the question is what
categories to use. They need to be:
- Short (to fit on the screen)
- Easily translatable
- and above all, useful.
Looking at the lists the obvious categories are:
- Invalid Address (on the public internet)
BOGON, BOGON_FULL
- Scanner (not by itself malicious)
SHODAN
- Application (potentially unwanted)
TOR_ALL, TOR_EXIT
- Malware C & C
FEODO_RECOMMENDED, FEODO_IP, FEODO_AGGRESIVE
- Composite
EMERGING_FWRULE
Less obvious are:
- Reputation
ALIENVAULT, CIARMY, SPAMHAUS_DROP, SPAMHAUS_EDROP
- Attacks
BLOCKLIST_DE, DSHIELD, EMERGING_COMPROMISED
I'm not sure that the distinction between these two is going to be
helpful to most people (I'm not sure I understand it myself).
We could use:
- Top attackers
DSHIELD, EMERGING_COMPROMISED, SPAMHAUS_DROP, SPAMHAUS_EDROP
- Other attackers
ALIENVAULT, BLOCKLIST_DE, CIARMY
but that might be making a distinction that is better made by the user.
Any opinions?
Tim
On 18/12/2019 12:10, Michael Tremer wrote:
> Hi,
>
>> On 16 Dec 2019, at 23:05, Tom Rymes <trymes(a)rymes.com> wrote:
>>
>> On 12/16/2019 5:20 PM, Michael Tremer wrote:> Hi,
>>>
>>>> On 16 Dec 2019, at 20:06, Tim FitzGeorge <lists(a)tfitzgeorge.me.uk> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I've attached the current GUI screenshot.
>>>
>>> Thanks for that.
>>>
>>> I have a couple of suggestions/concerns about it:
>>
>> [snip]
>>
>>> c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
>>
>> [snip]
>>
>> May I opine that the "Safe" information would be helpful to me in the WUI. Perhaps we can be more explicit, or better explain, such as is often done with RBLs in mail server settings, where lists are sometimes described in terms of their likelihood to cause false-positives.
>>
>> It's all well and good in the documentation, but a quick "Safe|Moderate|Risky" listing in the WUI will prove handy, IMHO.
>>
>> Just my $0.02 as more of a user than a developer,
>
> I appreciate your input, but I still disagree with is that we take the decision if something is “risky” or not. There are too many things that need to be taken into account to make that decision and it probably varies for each user.
>
> What I take from your comment though is that we should categorise the lists, and that is something we can do.
>
> We can add a headline to the table and group the lists by “Blocking ambiguous packets”, “Blocking Malware”, etc.
>
> That makes it easier for the user to decide which lists are interesting or even necessary depending on what they want to achieve.
>
> How is that?
>
> -Michael
>
>>
>> Tom
>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 0/5] ipblacklist: IP Address Blacklists
2019-12-21 18:34 ` Tim FitzGeorge
2019-12-24 10:29 ` Michael Tremer
@ 2019-12-28 20:59 ` Tim FitzGeorge
1 sibling, 0 replies; 35+ messages in thread
From: Tim FitzGeorge @ 2019-12-28 20:59 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 28706 bytes --]
Hi,
I'll code and test what we've agreed on. In the meantime I'm going to
reply on the sub-thread started by Tom Rymes as to what we call the list
categories, so that we can keep that discussion separate from further
functionality etc. discussion.
Tim
On 21/12/2019 18:34, Tim FitzGeorge wrote:
> Hi,
>
> I've got my systems checking for downloads at 15 minute intervals and
> using If-Modified-Since. This all seems to be working OK.
>
> Responses to comments below:
>
> On 18/12/2019 12:07, Michael Tremer wrote:
>> Hi,
>>
>>> On 17 Dec 2019, at 18:21, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>
>>> Hi,
>>>
>>> I think we've agreed on the following:
>>>
>>> - Checks to be made every 15 minutes, subject to per-list minimum rate.
>>> - Modify download to use If-Modified-Since rather than separate HEAD
>>> request.
>>> - Remove automatic blacklist.
>>
>> I thought someone wants to counter my argument. This is not a dictatorship :)
>>
>>>
>>> other comments are below.
>>>
>>> Tim
>>>
>>> On 16/12/2019 22:20, Michael Tremer wrote:
>>>> Hi,
>>>>
>>>>> On 16 Dec 2019, at 20:06, Tim FitzGeorge <lists(a)tfitzgeorge.me.uk> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I've attached the current GUI screenshot.
>>>>
>>>> Thanks for that.
>>>>
>>>> I have a couple of suggestions/concerns about it:
>>>>
>>>> a) I am not sure what the value is about the top box that is called “Status”. It is basically a summary of the iptables output, but will it help the user? A blacklist with more or fewer entries is not better or worse, blocking more packets isn’t better than blocking fewer. It is about the quality of the blocks.
>>>
>>> I agree that the size of the lists is not all that useful, but the
>>> information about the number of blocked packets can be an indication of
>>> compromise. It depends on the type of the list, but a list that targets
>>> C&C channels (e.g. the FEODO lists) should not show any input packets
>>> being blocked, unless one of the computers being protected is infected.
>>> Apart from the BOGON lists, none of the lists should show any output
>>> packets being blocked; again it's a potential indication of infection if
>>> otherwise.
>>
>> The Shodan list will definitely show some activity unless you are behind CG NAT.
>>
>> Should we not have this rather in the logging section?
>>
>> And these counters here will be reset when the firewall is being reloaded. Should this data therefore not be collected from the logs so that you can go back in time? That would be very important to identify any start of compromise.
>>
>
> That makes sense. I'll remove the status from this page and create a
> log page for the information. If anyone actually wants the iptables
> information it's available in from Firewall > iptables anyway.
>
> Do you think it makes more sense to show packets blocked in/out or
> packets blocked per source interface? The latter would of (limited)
> help in tracking down potential compromise.
>
>>>>
>>>> b) I would prefer to move the settings box to the top. If you like you can show the size of the blacklists there and when they have been updated, too.
>>>
>>> I put the status at the top since it's usually what I want to see when I
>>> go to the page. Under normal circumstances I don't change the settings,
>>> so I'd rather not have to scroll past them to get to the information I'm
>>> actually interested in. The update times are just useful to show that
>>> the lists are actually being updated. The size is interesting to me,
>>> but as you say it probably won't help the average user, especially if
>>> you don't know which lists block individual addresses and which block
>>> nets (BOGON, DSHIELD and EMERGING_COMPROMISED).
>>
>> See my comment on the order above. Splitting the page might make sense.
>>
>> When you show size, does that mean lines in the blacklists? So it could be networks or single IP addresses? Does it not make sense to count IP addresses (i.e. 256 for a /24 network and so on) and put it in the table?
> It's lines in the blacklist.
>
>>
>> I think I would only be interested in the size for two things:
>>
>> a) Is there any chance for overblocking? i.e. does the list have 2 billion entries which would be half the public IPv4 address space or so. I cannot really come up with a good example here.
>>
>> Or
>>
>> b) What is the performance impact/memory consumption of the list?
>
> It would be interesting to show the type (single address or net), number
> of entries and number of addresses; the first two affect performance,
> the second memory use and the latter the number of blocked addresses,
> but that's really only going to be of interest to a very small number of
> people, who can always do 'ipset list -t' to get the information. So
> I'll remove the size column.
>
>>
>>> I originally had the status and settings in one list, but I decided that
>>> it was getting a bit messy and difficult to follow. I've attached a
>>> screenshot of a quickly hacked page. (Note that the status area only
>>> appears once the blacklists have been enabled.)
>>
>> You are right. This is getting quite tight there.
>>
>> You could write packets in/out into on column like 2k/1023, but see my concerns about this above.
>
> Actually I've come to the conclusion that the number of bytes blocked
> isn't all that important; it's sufficient to show the number of packets.
>
>>
>>>>
>>>> c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
>>>
>>> I agree that it's it's an over simplistic summary of the lists. I note
>>> that Tom Rymes has asked to keep it, but I think its only value is if it
>>> suggests to users that things are more complicated than they appear and
>>> therefore prompts them to read the Wiki.
>>
>> I will reply to that separately.
>
> Looking at that reply, I think it makes sense to include a category
> column in both the settings and the log summary page - the former to
> help selection of which lists to use, and the latter as a help when
> deciding whether the information could indicate compromise or not.
>
>>
>>>>
>>>> d) I do not understand what the “update check rate” means. I suppose we should show the update interval of the lists in the table and just refresh them according to that.
>>>
>>> It's the minimum check period for updates. The system checks for
>>> updates at either the rate specific to the list or this rate, whichever
>>> is slower. This is a hangover from the early days, copied from the IPS
>>> updates. I suppose it could have a value for some people who may wish
>>> to lower the check rate, but I'm not sure that it has sufficient value
>>> to be worth keeping.
>>
>> No, I would not let the user decide when and how often to update the lists.
>
> OK. I'll remove it.
>
>>
>> Generally there is this argument internally how often some databases should be updated. People with slow internet connections or volume based data plans will find this rather expensive to update those lists too often.
>>
>> If we ever want to go ahead with that in IPFire 2, there should be a global switch for a “low data mode” which then delays updating these things.
>
> That makes sense. Maybe also the ability to set when downloads take
> place - but as you say that's for the future.
>
>>
>>>>
>>>> e) Do we want to keep the automatic blacklists now? I do not think it will actually improve the security of IPFire.
>>>>
>>>
>>> I'll remove them. I think they do have a use, in that they can detect
>>> someone doing a port scan and block them before they find an open port,
>>> but as you say there is also the possibility of a DOS.
>>
>> How would this block a DOS?
>
> Sorry, I didn't make myself clear - there's the possibility that someone
> could cause a DOS with the automatic list by faking the source IP
> address and sending a few packets on a random port.
>
>>
>> -Michael
>
> Tim
>>
>>>>>
>>>>> I'll update the code based on our discussions, and submit an updated set
>>>>> of patches - I imagine there will have to be at least one more iteration.
>>>>
>>>> Let’s wait until we have come to decisions :)
>>>>
>>>> -Michael
>>>>
>>>>>
>>>>> Tim
>>>>>
>>>>> (No additional comments below)
>>>>>
>>>>> On 13/12/2019 23:11, Michael Tremer wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Again my apologies for my late reply. Busy busy weeks.
>>>>>>
>>>>>>> On 8 Dec 2019, at 20:50, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> It's my turn to apologise for being slow to respond - I've had a busy
>>>>>>> week, but I should have plenty of time over the next couple of weeks.
>>>>>>
>>>>>> No worries. Turns out we all do :)
>>>>>>
>>>>>>> I've made most of the comments inline, however I think Michael had a
>>>>>>> question (which I can't find now) about what happens if someone enables
>>>>>>> all the lists. One thing which would perhaps make this less likely is
>>>>>>> that the WUI tags the available lists with whether they're safe or not,
>>>>>>> with a footnote that safe means that the list only blocks malicious
>>>>>>> traffic. This won't guarantee that a user won't still try to enable all
>>>>>>> the lists, but it should make them realise that they should think first.
>>>>>>
>>>>>> We had a couple of features going slightly wrong or being “misunderstood” by some users. People still seem to panic when they see “local recursor”. To this day I do not know why.
>>>>>>
>>>>>> We cannot make everything idiot-proof. And when some user if of that category, they probably should shutdown their IPFire box, educate themselves and then come back again. So I do not want to limit people, but make things as easy as possible.
>>>>>>
>>>>>> If someone enables all the lists, good luck with passing packets :)
>>>>>>
>>>>>>> I have considered replacing this tag with a risk high/medium/low and
>>>>>>> maybe adding a category (invalid/application/scanner/C&C or something
>>>>>>> like that), but that may provide too much information and dissuade them
>>>>>>> from actually following the links to checkout what the list actually does.
>>>>>>
>>>>>> Can we have a screenshot of the GUI right now? I didn’t run the code, yet.
>>>>>>
>>>>>> We should document the lists like we do it with the rulesets of the IPS. People might ignore this, but that is on them.
>>>>>>
>>>>>>>
>>>>>>> Tim
>>>>>>>
>>>>>>>
>>>>>>> On 05/12/2019 22:25, Michael Tremer wrote:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>>> On 4 Dec 2019, at 17:05, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>>>>>>>>
>>>>>>>>> Hello Tim, hello Michael,
>>>>>>>>>
>>>>>>>>> please see my responses inline...
>>>>>>>>>>>>
>>>>>>>>>>>> We could periodically update the blacklists on our main mirror (and
>>>>>>>>>>>> wait for the network to sync it), make sure it is signed and write
>>>>>>>>>>>> a small downloader that fetches, validates and installs them.
>>>>>>>>>>>>
>>>>>>>>>>>> @All: Thoughts on this?
>>>>>>>>>>>
>>>>>>>>>>> I think there are a number of points here.
>>>>>>>>>>>
>>>>>>>>>>> Firstly, from the point of a third party using IPFire, is this really
>>>>>>>>>>> solving the privacy disclosure problem? There's no way round disclosing
>>>>>>>>>>> your public IP address to someone you're downloading from; all this does
>>>>>>>>>>> is change who that information is being disclosed to. For the user
>>>>>>>>>>> there's no way of knowing whether the source is more or less protective
>>>>>>>>>>> of the user's privacy than the blacklist provider. Indeed it won't be
>>>>>>>>>>> possible to know who the lists are being downloaded from until the
>>>>>>>>>>> download starts.
>>>>>>>>>>
>>>>>>>>>> There is a way: Tor. But that is a totally different story.
>>>>>>>>> Well, I see a third option on this: Use the mirror infrastructure we already
>>>>>>>>> have. Every IPFire installation discloses its public IP address to one of these
>>>>>>>>> servers sooner or later, so we do not disclose additional data if the blacklists
>>>>>>>>> were fetched from these.
>>>>>>>>>
>>>>>>>>> Needless to say, Tor (hidden services) would be better, but that is a different
>>>>>>>>> story indeed. :-)
>>>>>>>>>>
>>>>>>>>>> The point is rather that a forget list can be sent instead of the “real” one.
>>>>>>>>> I did not get this. Forget? Forged?? ???
>>>>>>>>
>>>>>>>> Yes, I meant to write forged, but auto-correct didn’t let me.
>>>>>>>>
>>>>>>>>>>> Secondly, latency; some of the lists are updated every 5 minutes. While
>>>>>>>>>>> I've limited the maximum check rate to hourly, will the updates
>>>>>>>>>>> propagate quickly enough. For reference on my main system the 24
>>>>>>>>>>> updates on the CIARMY list made 143 498 changes (additions or
>>>>>>>>>>> deletions). I've seem it do over 200 000.
>>>>>>>>> Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not
>>>>>>>>> document a recommended update interval anywhere, so we can only guess.
>>>>>>>>>
>>>>>>>>> Personally, more static lists seem to be preferable for packet filtering. Highly
>>>>>>>>> dynamic ones such as CIArmy should be done via DNSBL queries or something similar
>>>>>>>>> - do we really want to have that list here?
>>>>>>>>
>>>>>>>> It is not really an option to implement a DNSBL into a packet filter, but I get your point.
>>>>>>>
>>>>>>> One of the 'selling points' for an IP address blacklist is that it can
>>>>>>> respond quickly to new threats - or rather new attackers. While a new
>>>>>>> IDS/IPS rule needs time to analyse the threat, generate a rule and check
>>>>>>> it, it's easy to add an address to a list. So, I think the CIArmy list
>>>>>>> is potentially useful for protecting home systems etc. with budget
>>>>>>> hardware, but I would be very careful about using it for a protecting a
>>>>>>> general access website.
>>>>>>
>>>>>> If they are very volatile, we should honour that and update them often, too.
>>>>>>
>>>>>> It probably is more about false-positives being removed very quickly instead of adding threats very quickly. The average IPFire user is probably not under threats like these to need to react very quickly.
>>>>>>
>>>>>> So I do not see much value in adding those lists and then updating once a day. Does it have to be every 5 min? No. I would suggest 15 which should be good enough for everyone.
>>>>>>
>>>>>> Other lists should of course not be updated every 15 minutes when not needed.
>>>>>>
>>>>>> Running every 15 minutes would allow us to retry downloading lists that are on an hourly schedule if the download failed.
>>>>>>
>>>>>>>>
>>>>>>>>>> How did you come up with the hour? Will it be retried more often if the download was not successful?
>>>>>>>>> One hour is the most common interval indeed, but adding some random time might
>>>>>>>>> be useful in order to reduce load on the servers providing a blacklist.
>>>>>>>>
>>>>>>>> Yes, definitely. Otherwise we will shoot down our mirrors.
>>>>>>>
>>>>>>> When I implemented that section of code, specifying the minimum check
>>>>>>> period in hours seemed to provide a convenient way of allowing a check
>>>>>>> period covering a wide range, with an hour as the fastest and a week as
>>>>>>> the slowest. I didn't looked at the CIArmy list until much later. Most
>>>>>>> of the lists don't change nearly as much, but the CIArmy list is
>>>>>>> described as one that deliberately responds quickly.
>>>>>>>
>>>>>>> From my production system, for yesterday:
>>>>>>>
>>>>>>> The following block lists were updated:
>>>>>>> BLOCKLIST_DE: 24 Time(s) - 9341 change(s)
>>>>>>> BOGON_FULL: 1 Time(s) - 10 change(s)
>>>>>>> CIARMY: 24 Time(s) - 159134 change(s)
>>>>>>> DSHIELD: 7 Time(s) - 18 change(s)
>>>>>>> EMERGING_FWRULE: 1 Time(s) - 50 change(s)
>>>>>>> FEODO_AGGRESIVE: 24 Time(s) - 13 change(s)
>>>>>>> SHODAN: 1 Time(s) - 0 change(s)
>>>>>>> SPAMHAUS_DROP: 1 Time(s) - 0 change(s)
>>>>>>> TOR_EXIT: 24 Time(s) - 162 change(s)
>>>>>>
>>>>>> Very interesting statistics.
>>>>>>
>>>>>>>
>>>>>>> and my test system:
>>>>>>>
>>>>>>> The following block lists were updated:
>>>>>>> ALIENVAULT: 19 Time(s) - 5331 change(s)
>>>>>>> EMERGING_COMPROMISED: 1 Time(s) - 26 change(s)
>>>>>>> TALOS_MALICIOUS: 1 Time(s) - 36 change(s)
>>>>>>>
>>>>>>> That covers most of the lists. From the WUI, since 1 Dec:
>>>>>>>
>>>>>>> Blacklist Entries pkts bytes Last updated
>>>>>>> in in
>>>>>>> AUTOBLACKLIST 0 731 51144 Sun Dec 8 18:40:02 2019
>>>>>>> BLOCKLIST_DE 28020 857 46735 Sun Dec 8 18:40:02 2019
>>>>>>> BOGON_FULL 214 5255 189K Sun Dec 8 16:50:04 2019
>>>>>>> CIARMY 15000 19774 976K Sun Dec 8 18:04:01 2019
>>>>>>> DSHIELD 20 7992 321K Sun Dec 8 16:45:13 2019
>>>>>>> EMERGING_FWRULE 1647 197 8383 Fri Dec 6 05:29:07 2019
>>>>>>> FEODO_AGGRESIVE 7169 0 0 Sun Dec 8 18:50:07 2019
>>>>>>> SHODAN 32 34 1530 Sun Dec 8 17:54:09 2019
>>>>>>> SPAMHAUS_DROP 823 0 0 Fri Dec 6 18:22:35 2019
>>>>>>> SPAMHAUS_EDROP 111 82 16433 Thu Dec 5 17:27:09 2019
>>>>>>> TOR_EXIT 1055 0 0 Sun Dec 8 18:31:02 2019
>>>>>>
>>>>>> This as well.
>>>>>>
>>>>>> Those are more packets than I would have expected.
>>>>>>
>>>>>>>
>>>>>>> (I've left out the pkts/bytes out fields which were all 0)
>>>>>>>
>>>>>>> Note that where possible I do a HEAD request first and then only
>>>>>>> download the list if the modification time has changed since the last
>>>>>>> check. For dynamically generated lists this isn't possible.
>>>>>>
>>>>>> You won’t need a HEAD request for it. You can include it in the GET request.
>>>>>>
>>>>>> Have a look at location downloader where I use that. The server will respond with 304 and not send any payload.
>>>>>>
>>>>>> https://git.ipfire.org/?p=location/libloc.git;a=blob;f=src/python/location-downloader.in;h=1b5932d5822e03737d32b2a27816815b2f7e74dd;hb=HEAD#l151
>>>>>>
>>>>>>>
>>>>>>> If the download isn't successful it just gives up and waits for the next
>>>>>>> attempt (apart from the usual retries in the library). I probably
>>>>>>> should to change that so that it only applies the per list minimum
>>>>>>> update period in this case (specified in the sources file) rather than
>>>>>>> the user specified value as well.
>>>>>>
>>>>>> I think it is not the worst if an update fails. It might just happen every once in a while.
>>>>>>
>>>>>> So I would suggest to just re-run the script more often and when the mtime of the file is older than the threshold, a download is attempted. You can use that timestamp for the GET request.
>>>>>>
>>>>>>> I already use a time offset on the downloads - when it's started from
>>>>>>> boot, backup restore or WUI enable, it checks to see if it's installed
>>>>>>> in the fcrontab, and if not adds itself at a randomly generated offset
>>>>>>> in the hour.
>>>>>>
>>>>>> That should potentially go to red.up, or if we can settle on 15 minutes, I would consider that often enough to quickly update all lists after a reboot.
>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>>>> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is
>>>>>>>>>>> the largest at a few MB), there are going to be a lot of them. How will
>>>>>>>>>>> this affect the willingness of people to mirror IPFire?
>>>>>>>>> I do not consider this being a problem as we do not generate that much traffic
>>>>>>>>> to them. Of course, that depends on the update interval again.
>>>>>>>>
>>>>>>>> That depends on your point of view.
>>>>>>>>
>>>>>>>> I do not have a problem with this at all in my data center, but there are plenty of people with a volume-based LTE plan or simply a 128 kBit/s connection. It will take a longer time to download the lists for them. We need to mind that.
>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Talking about the preference of packet filter and IPS, I prefer to
>>>>>>>>>>>> use the latter as well as it gains more insight in what kind of malicious
>>>>>>>>>>>> traffic tried to pass a firewall machine. On systems with low resources,
>>>>>>>>>>>> this might be problematic and removing load from the IPS can be preferred
>>>>>>>>>>>> (make this configurable?!), on others, people might want to have both
>>>>>>>>>>>> results.
>>>>>>>>>>>>
>>>>>>>>>>> You're only going to get one result for a packet whichever way round the
>>>>>>>>>>> IP blacklist and IPS are since whichever comes first will drop the
>>>>>>>>>>> packet before it reaches the second (well it would be possible to put
>>>>>>>>>>> the IP blacklist first and get it to log and mark packets which are then
>>>>>>>>>>> dropped after the IPS, but I think that's getting a little complicated.
>>>>>>>>>>> In addition I've seen the messages about the trouble marking was
>>>>>>>>>>> causing in the QoS).
>>>>>>>>>>>
>>>>>>>>>>> I think it's a 50/50 choice as to which is more valuable first; it's
>>>>>>>>>>> probably going to differ from packet to packet. For me the possibility
>>>>>>>>>>> of reducing the IPS load means I prefer putting the IP blacklist first
>>>>>>>>>>>
>>>>>>>>>>> It should be fairly easy to add the choice of where to put the IP
>>>>>>>>>>> blacklist. I think it'll have to be in the main firewall script, so
>>>>>>>>>>> it'll require a firewall restart, but it's not something that'll be
>>>>>>>>>>> changed often.
>>>>>>>>>>
>>>>>>>>>> I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier.
>>>>>>>>> Agreed.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads.
>>>>>>>>>>
>>>>>>>>>> We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily.
>>>>>>>>> So dropping packets from blacklisted IP addresses/networks before IPS is it, then.
>>>>>>>>>>
>>>>>>>>>> I do not even think it makes sense to swap the order in the outgoing direction.
>>>>>>>>> Me too.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this.
>>>>>>>>> Based on my personal experience, firewall hits observed on a single machine exposed
>>>>>>>>> to the internet are interesting, but the overall situation across multiple machines
>>>>>>>>> is even more interesting. Very quickly, you'll end on something like a centralised
>>>>>>>>> logging server and custom statistical analysis here...
>>>>>>>>
>>>>>>>> Probably a project for IPFire 4.0 :)
>>>>>>>>
>>>>>>> Or use one of the existing services, like the DSHIELD client
>>>>>>> https://dshield.org/howto.html (subject to privacy concerns again).
>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default.
>>>>>>>>>>>> I would love to see the bogon ruleset here, too (think about 8chan successor
>>>>>>>>>>>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause
>>>>>>>>>>>> interference if RED does not have a public IP address assigned.
>>>>>>>>>>>
>>>>>>>>>>> I can add a field to the options file that controls whether a list is
>>>>>>>>>>> enabled by default.
>>>>>>>>> Thank you. :-)
>>>>>>>>>>
>>>>>>>>>> To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat.
>>>>>>>>> This is my only objection against this patchset. Now, what can we do about it?
>>>>>>>>> One possibility is to apply the patchset now and implement a custom download
>>>>>>>>> source thing later on, or do that before releasing Core Update 139 (or which version
>>>>>>>>> the patchset will be to) after we agreed on something.
>>>>>>>>
>>>>>>>> I do not see this being merged for 139. But that is not important. We need to get it right first and then release it.
>>>>>>>>
>>>>>>>> As far as I know, nobody has tested this, yet.
>>>>>>>
>>>>>>> There are a number of people who have been running an earlier version
>>>>>>> which I shared on GitHub. There were a few early issues, but it seems
>>>>>>> to be OK now.
>>>>>>>
>>>>>>> https://forum.ipfire.org/viewtopic.php?f=27&t=21845
>>>>>>>
>>>>>>> This version wasn't integrated into IPFire, so (for example) it inserted
>>>>>>> itself into the INPUT IPTables chain rather than having it's chains
>>>>>>> created as part of the firewall start-up script.
>>>>>>>
>>>>>>>>
>>>>>>>> I have huge concerns about the automatic blacklist. @Peter: What is your opinion on this?
>>>>>>>>
>>>>>>>
>>>>>>> While I implemented it, I'm aware of its potential to cause problems,
>>>>>>> which is why it has to be separately enabled. It's not caused me any
>>>>>>> issues at the default settings (blocks at over 10 packets per hour until
>>>>>>> 1 hour has passed without seeing packets from the address), but I've not
>>>>>>> used it on a site with publicly announced services. If I was going to
>>>>>>> use it on a web site I would want to, at the very minimum, drop the
>>>>>>> block period drastically.
>>>>>>
>>>>>> I suppose this is entirely unusable on an IPFire box in a data center that hosts things. Let’s say our rack in Hanover.
>>>>>>
>>>>>> You will have hits from some broken IP stacks and people might just end up on this without doing anything wrong.
>>>>>>
>>>>>> You can denial-of-service easily as well and I suppose without aggregation of data from many many systems, it does not make sense to instantly block addresses. And even then you probably would block an entire subnet.
>>>>>>
>>>>>> So I am not sure how I should feel about it. I do not think that it adds anything because the packets would have been blocked anyways.
>>>>>>
>>>>>>>
>>>>>>> On the other hand, it's good at responding quickly. Usually I see only
>>>>>>> 1-2% of blocks from the automatic list:
>>>>>>>
>>>>>>> Reason Count % First Last
>>>>>>> CIARMY 2416 45 Dec 6 00:00 Dec 6 23:59
>>>>>>> DSHIELD 1353 25 Dec 6 00:00 Dec 6 23:59
>>>>>>> INPUT 1294 24 Dec 6 00:00 Dec 6 23:59
>>>>>>> AUTOBLACKLIST 122 2 Dec 6 00:20 Dec 6 16:28
>>>>>>> BLOCKLIST_DE 89 2 Dec 6 00:20 Dec 6 23:46
>>>>>>>
>>>>>>> and sometimes none at all, but one one occasion it blocked over 8000
>>>>>>> packets. Again I'm aware this is for a home system, which is rather
>>>>>>> different than from a Web server.
>>>>>>>
>>>>>>>>> If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not
>>>>>>>>> seem to be problematic, as they are hosted on 3rd party servers, too).
>>>>>>>>
>>>>>>>> One of them will be, sooner or later. And one is enough I suppose.
>>>>>>>
>>>>>>> DShield (https://dshield.org/api/#threatfeeds) and firehol
>>>>>>> (http://iplists.firehol.org/) seem to host copies of most of the lists
>>>>>>> as well.
>>>>>>>>
>>>>>>>> I do not really want to overthink this - we didn’t do this with clamav for example either. But it probably is a decision to be made by the user what they want to enable and we should not enable anything by default. So no data will be leaked as long as the user does not consent.
>>>>>>>>
>>>>>>>> -Michael
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks, and best regards,
>>>>>>>>> Peter Müller
>>>>>>
>>>>>
>>>>> <Screenshot_2019-12-16_IP_Address_Blacklists.png>
>>>>
>>>
>>> <Screenshot_2019-12-17_IP_Address_Blacklists.png>
>>
>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 0/5] ipblacklist: IP Address Blacklists
2019-12-21 18:34 ` Tim FitzGeorge
@ 2019-12-24 10:29 ` Michael Tremer
2019-12-28 20:59 ` Tim FitzGeorge
1 sibling, 0 replies; 35+ messages in thread
From: Michael Tremer @ 2019-12-24 10:29 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 29706 bytes --]
Hi,
> On 21 Dec 2019, at 19:34, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>
> Hi,
>
> I've got my systems checking for downloads at 15 minute intervals and
> using If-Modified-Since. This all seems to be working OK.
Great!
> Responses to comments below:
>
> On 18/12/2019 12:07, Michael Tremer wrote:
>> Hi,
>>
>>> On 17 Dec 2019, at 18:21, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>
>>> Hi,
>>>
>>> I think we've agreed on the following:
>>>
>>> - Checks to be made every 15 minutes, subject to per-list minimum rate.
>>> - Modify download to use If-Modified-Since rather than separate HEAD
>>> request.
>>> - Remove automatic blacklist.
>>
>> I thought someone wants to counter my argument. This is not a dictatorship :)
>>
>>>
>>> other comments are below.
>>>
>>> Tim
>>>
>>> On 16/12/2019 22:20, Michael Tremer wrote:
>>>> Hi,
>>>>
>>>>> On 16 Dec 2019, at 20:06, Tim FitzGeorge <lists(a)tfitzgeorge.me.uk> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I've attached the current GUI screenshot.
>>>>
>>>> Thanks for that.
>>>>
>>>> I have a couple of suggestions/concerns about it:
>>>>
>>>> a) I am not sure what the value is about the top box that is called “Status”. It is basically a summary of the iptables output, but will it help the user? A blacklist with more or fewer entries is not better or worse, blocking more packets isn’t better than blocking fewer. It is about the quality of the blocks.
>>>
>>> I agree that the size of the lists is not all that useful, but the
>>> information about the number of blocked packets can be an indication of
>>> compromise. It depends on the type of the list, but a list that targets
>>> C&C channels (e.g. the FEODO lists) should not show any input packets
>>> being blocked, unless one of the computers being protected is infected.
>>> Apart from the BOGON lists, none of the lists should show any output
>>> packets being blocked; again it's a potential indication of infection if
>>> otherwise.
>>
>> The Shodan list will definitely show some activity unless you are behind CG NAT.
>>
>> Should we not have this rather in the logging section?
>>
>> And these counters here will be reset when the firewall is being reloaded. Should this data therefore not be collected from the logs so that you can go back in time? That would be very important to identify any start of compromise.
>>
>
> That makes sense. I'll remove the status from this page and create a
> log page for the information. If anyone actually wants the iptables
> information it's available in from Firewall > iptables anyway.
>
> Do you think it makes more sense to show packets blocked in/out or
> packets blocked per source interface? The latter would of (limited)
> help in tracking down potential compromise.
I would only care for packets.
The size of the packets would only be interesting in a DoS, but it is still worthless information unless you say in what time it happened. i.e. you would want to show MBit/s or something like it, so you can estimate how much bandwidth is left. I would consider that out of scope here.
>
>>>>
>>>> b) I would prefer to move the settings box to the top. If you like you can show the size of the blacklists there and when they have been updated, too.
>>>
>>> I put the status at the top since it's usually what I want to see when I
>>> go to the page. Under normal circumstances I don't change the settings,
>>> so I'd rather not have to scroll past them to get to the information I'm
>>> actually interested in. The update times are just useful to show that
>>> the lists are actually being updated. The size is interesting to me,
>>> but as you say it probably won't help the average user, especially if
>>> you don't know which lists block individual addresses and which block
>>> nets (BOGON, DSHIELD and EMERGING_COMPROMISED).
>>
>> See my comment on the order above. Splitting the page might make sense.
>>
>> When you show size, does that mean lines in the blacklists? So it could be networks or single IP addresses? Does it not make sense to count IP addresses (i.e. 256 for a /24 network and so on) and put it in the table?
> It's lines in the blacklist.
Okay, do we think this is what the user will expect?
>
>>
>> I think I would only be interested in the size for two things:
>>
>> a) Is there any chance for overblocking? i.e. does the list have 2 billion entries which would be half the public IPv4 address space or so. I cannot really come up with a good example here.
>>
>> Or
>>
>> b) What is the performance impact/memory consumption of the list?
>
> It would be interesting to show the type (single address or net), number
> of entries and number of addresses; the first two affect performance,
> the second memory use and the latter the number of blocked addresses,
> but that's really only going to be of interest to a very small number of
> people, who can always do 'ipset list -t' to get the information. So
> I'll remove the size column.
I would consider this all to be entirely irrelevant for choice of hardware.
It is really cheap to throw the blacklists into ipset (unless somebody else knows better than me).
>>
>>> I originally had the status and settings in one list, but I decided that
>>> it was getting a bit messy and difficult to follow. I've attached a
>>> screenshot of a quickly hacked page. (Note that the status area only
>>> appears once the blacklists have been enabled.)
>>
>> You are right. This is getting quite tight there.
>>
>> You could write packets in/out into on column like 2k/1023, but see my concerns about this above.
>
> Actually I've come to the conclusion that the number of bytes blocked
> isn't all that important; it's sufficient to show the number of packets.
Cool :)
>
>>
>>>>
>>>> c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
>>>
>>> I agree that it's it's an over simplistic summary of the lists. I note
>>> that Tom Rymes has asked to keep it, but I think its only value is if it
>>> suggests to users that things are more complicated than they appear and
>>> therefore prompts them to read the Wiki.
>>
>> I will reply to that separately.
>
> Looking at that reply, I think it makes sense to include a category
> column in both the settings and the log summary page - the former to
> help selection of which lists to use, and the latter as a help when
> deciding whether the information could indicate compromise or not.
Agreed.
>
>>
>>>>
>>>> d) I do not understand what the “update check rate” means. I suppose we should show the update interval of the lists in the table and just refresh them according to that.
>>>
>>> It's the minimum check period for updates. The system checks for
>>> updates at either the rate specific to the list or this rate, whichever
>>> is slower. This is a hangover from the early days, copied from the IPS
>>> updates. I suppose it could have a value for some people who may wish
>>> to lower the check rate, but I'm not sure that it has sufficient value
>>> to be worth keeping.
>>
>> No, I would not let the user decide when and how often to update the lists.
>
> OK. I'll remove it.
>
>>
>> Generally there is this argument internally how often some databases should be updated. People with slow internet connections or volume based data plans will find this rather expensive to update those lists too often.
>>
>> If we ever want to go ahead with that in IPFire 2, there should be a global switch for a “low data mode” which then delays updating these things.
>
> That makes sense. Maybe also the ability to set when downloads take
> place - but as you say that's for the future.
Why would that be relevant to the user?
>
>>
>>>>
>>>> e) Do we want to keep the automatic blacklists now? I do not think it will actually improve the security of IPFire.
>>>>
>>>
>>> I'll remove them. I think they do have a use, in that they can detect
>>> someone doing a port scan and block them before they find an open port,
>>> but as you say there is also the possibility of a DOS.
>>
>> How would this block a DOS?
>
> Sorry, I didn't make myself clear - there's the possibility that someone
> could cause a DOS with the automatic list by faking the source IP
> address and sending a few packets on a random port.
Yes, that is what I would consider the biggest risk here as well - especially when hosting public services behind the firewall. Those triggers are probably not malicious, but rather bugs in network stacks, browsers being too keen on opening too many connections at the same time, etc.
I think this is all taking great shape now and I am very happy about this feature and cannot wait to have it in IPFire!
Best,
-Michael
>
>>
>> -Michael
>
> Tim
>>
>>>>>
>>>>> I'll update the code based on our discussions, and submit an updated set
>>>>> of patches - I imagine there will have to be at least one more iteration.
>>>>
>>>> Let’s wait until we have come to decisions :)
>>>>
>>>> -Michael
>>>>
>>>>>
>>>>> Tim
>>>>>
>>>>> (No additional comments below)
>>>>>
>>>>> On 13/12/2019 23:11, Michael Tremer wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Again my apologies for my late reply. Busy busy weeks.
>>>>>>
>>>>>>> On 8 Dec 2019, at 20:50, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> It's my turn to apologise for being slow to respond - I've had a busy
>>>>>>> week, but I should have plenty of time over the next couple of weeks.
>>>>>>
>>>>>> No worries. Turns out we all do :)
>>>>>>
>>>>>>> I've made most of the comments inline, however I think Michael had a
>>>>>>> question (which I can't find now) about what happens if someone enables
>>>>>>> all the lists. One thing which would perhaps make this less likely is
>>>>>>> that the WUI tags the available lists with whether they're safe or not,
>>>>>>> with a footnote that safe means that the list only blocks malicious
>>>>>>> traffic. This won't guarantee that a user won't still try to enable all
>>>>>>> the lists, but it should make them realise that they should think first.
>>>>>>
>>>>>> We had a couple of features going slightly wrong or being “misunderstood” by some users. People still seem to panic when they see “local recursor”. To this day I do not know why.
>>>>>>
>>>>>> We cannot make everything idiot-proof. And when some user if of that category, they probably should shutdown their IPFire box, educate themselves and then come back again. So I do not want to limit people, but make things as easy as possible.
>>>>>>
>>>>>> If someone enables all the lists, good luck with passing packets :)
>>>>>>
>>>>>>> I have considered replacing this tag with a risk high/medium/low and
>>>>>>> maybe adding a category (invalid/application/scanner/C&C or something
>>>>>>> like that), but that may provide too much information and dissuade them
>>>>>>> from actually following the links to checkout what the list actually does.
>>>>>>
>>>>>> Can we have a screenshot of the GUI right now? I didn’t run the code, yet.
>>>>>>
>>>>>> We should document the lists like we do it with the rulesets of the IPS. People might ignore this, but that is on them.
>>>>>>
>>>>>>>
>>>>>>> Tim
>>>>>>>
>>>>>>>
>>>>>>> On 05/12/2019 22:25, Michael Tremer wrote:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>>> On 4 Dec 2019, at 17:05, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>>>>>>>>
>>>>>>>>> Hello Tim, hello Michael,
>>>>>>>>>
>>>>>>>>> please see my responses inline...
>>>>>>>>>>>>
>>>>>>>>>>>> We could periodically update the blacklists on our main mirror (and
>>>>>>>>>>>> wait for the network to sync it), make sure it is signed and write
>>>>>>>>>>>> a small downloader that fetches, validates and installs them.
>>>>>>>>>>>>
>>>>>>>>>>>> @All: Thoughts on this?
>>>>>>>>>>>
>>>>>>>>>>> I think there are a number of points here.
>>>>>>>>>>>
>>>>>>>>>>> Firstly, from the point of a third party using IPFire, is this really
>>>>>>>>>>> solving the privacy disclosure problem? There's no way round disclosing
>>>>>>>>>>> your public IP address to someone you're downloading from; all this does
>>>>>>>>>>> is change who that information is being disclosed to. For the user
>>>>>>>>>>> there's no way of knowing whether the source is more or less protective
>>>>>>>>>>> of the user's privacy than the blacklist provider. Indeed it won't be
>>>>>>>>>>> possible to know who the lists are being downloaded from until the
>>>>>>>>>>> download starts.
>>>>>>>>>>
>>>>>>>>>> There is a way: Tor. But that is a totally different story.
>>>>>>>>> Well, I see a third option on this: Use the mirror infrastructure we already
>>>>>>>>> have. Every IPFire installation discloses its public IP address to one of these
>>>>>>>>> servers sooner or later, so we do not disclose additional data if the blacklists
>>>>>>>>> were fetched from these.
>>>>>>>>>
>>>>>>>>> Needless to say, Tor (hidden services) would be better, but that is a different
>>>>>>>>> story indeed. :-)
>>>>>>>>>>
>>>>>>>>>> The point is rather that a forget list can be sent instead of the “real” one.
>>>>>>>>> I did not get this. Forget? Forged?? ???
>>>>>>>>
>>>>>>>> Yes, I meant to write forged, but auto-correct didn’t let me.
>>>>>>>>
>>>>>>>>>>> Secondly, latency; some of the lists are updated every 5 minutes. While
>>>>>>>>>>> I've limited the maximum check rate to hourly, will the updates
>>>>>>>>>>> propagate quickly enough. For reference on my main system the 24
>>>>>>>>>>> updates on the CIARMY list made 143 498 changes (additions or
>>>>>>>>>>> deletions). I've seem it do over 200 000.
>>>>>>>>> Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not
>>>>>>>>> document a recommended update interval anywhere, so we can only guess.
>>>>>>>>>
>>>>>>>>> Personally, more static lists seem to be preferable for packet filtering. Highly
>>>>>>>>> dynamic ones such as CIArmy should be done via DNSBL queries or something similar
>>>>>>>>> - do we really want to have that list here?
>>>>>>>>
>>>>>>>> It is not really an option to implement a DNSBL into a packet filter, but I get your point.
>>>>>>>
>>>>>>> One of the 'selling points' for an IP address blacklist is that it can
>>>>>>> respond quickly to new threats - or rather new attackers. While a new
>>>>>>> IDS/IPS rule needs time to analyse the threat, generate a rule and check
>>>>>>> it, it's easy to add an address to a list. So, I think the CIArmy list
>>>>>>> is potentially useful for protecting home systems etc. with budget
>>>>>>> hardware, but I would be very careful about using it for a protecting a
>>>>>>> general access website.
>>>>>>
>>>>>> If they are very volatile, we should honour that and update them often, too.
>>>>>>
>>>>>> It probably is more about false-positives being removed very quickly instead of adding threats very quickly. The average IPFire user is probably not under threats like these to need to react very quickly.
>>>>>>
>>>>>> So I do not see much value in adding those lists and then updating once a day. Does it have to be every 5 min? No. I would suggest 15 which should be good enough for everyone.
>>>>>>
>>>>>> Other lists should of course not be updated every 15 minutes when not needed.
>>>>>>
>>>>>> Running every 15 minutes would allow us to retry downloading lists that are on an hourly schedule if the download failed.
>>>>>>
>>>>>>>>
>>>>>>>>>> How did you come up with the hour? Will it be retried more often if the download was not successful?
>>>>>>>>> One hour is the most common interval indeed, but adding some random time might
>>>>>>>>> be useful in order to reduce load on the servers providing a blacklist.
>>>>>>>>
>>>>>>>> Yes, definitely. Otherwise we will shoot down our mirrors.
>>>>>>>
>>>>>>> When I implemented that section of code, specifying the minimum check
>>>>>>> period in hours seemed to provide a convenient way of allowing a check
>>>>>>> period covering a wide range, with an hour as the fastest and a week as
>>>>>>> the slowest. I didn't looked at the CIArmy list until much later. Most
>>>>>>> of the lists don't change nearly as much, but the CIArmy list is
>>>>>>> described as one that deliberately responds quickly.
>>>>>>>
>>>>>>> From my production system, for yesterday:
>>>>>>>
>>>>>>> The following block lists were updated:
>>>>>>> BLOCKLIST_DE: 24 Time(s) - 9341 change(s)
>>>>>>> BOGON_FULL: 1 Time(s) - 10 change(s)
>>>>>>> CIARMY: 24 Time(s) - 159134 change(s)
>>>>>>> DSHIELD: 7 Time(s) - 18 change(s)
>>>>>>> EMERGING_FWRULE: 1 Time(s) - 50 change(s)
>>>>>>> FEODO_AGGRESIVE: 24 Time(s) - 13 change(s)
>>>>>>> SHODAN: 1 Time(s) - 0 change(s)
>>>>>>> SPAMHAUS_DROP: 1 Time(s) - 0 change(s)
>>>>>>> TOR_EXIT: 24 Time(s) - 162 change(s)
>>>>>>
>>>>>> Very interesting statistics.
>>>>>>
>>>>>>>
>>>>>>> and my test system:
>>>>>>>
>>>>>>> The following block lists were updated:
>>>>>>> ALIENVAULT: 19 Time(s) - 5331 change(s)
>>>>>>> EMERGING_COMPROMISED: 1 Time(s) - 26 change(s)
>>>>>>> TALOS_MALICIOUS: 1 Time(s) - 36 change(s)
>>>>>>>
>>>>>>> That covers most of the lists. From the WUI, since 1 Dec:
>>>>>>>
>>>>>>> Blacklist Entries pkts bytes Last updated
>>>>>>> in in
>>>>>>> AUTOBLACKLIST 0 731 51144 Sun Dec 8 18:40:02 2019
>>>>>>> BLOCKLIST_DE 28020 857 46735 Sun Dec 8 18:40:02 2019
>>>>>>> BOGON_FULL 214 5255 189K Sun Dec 8 16:50:04 2019
>>>>>>> CIARMY 15000 19774 976K Sun Dec 8 18:04:01 2019
>>>>>>> DSHIELD 20 7992 321K Sun Dec 8 16:45:13 2019
>>>>>>> EMERGING_FWRULE 1647 197 8383 Fri Dec 6 05:29:07 2019
>>>>>>> FEODO_AGGRESIVE 7169 0 0 Sun Dec 8 18:50:07 2019
>>>>>>> SHODAN 32 34 1530 Sun Dec 8 17:54:09 2019
>>>>>>> SPAMHAUS_DROP 823 0 0 Fri Dec 6 18:22:35 2019
>>>>>>> SPAMHAUS_EDROP 111 82 16433 Thu Dec 5 17:27:09 2019
>>>>>>> TOR_EXIT 1055 0 0 Sun Dec 8 18:31:02 2019
>>>>>>
>>>>>> This as well.
>>>>>>
>>>>>> Those are more packets than I would have expected.
>>>>>>
>>>>>>>
>>>>>>> (I've left out the pkts/bytes out fields which were all 0)
>>>>>>>
>>>>>>> Note that where possible I do a HEAD request first and then only
>>>>>>> download the list if the modification time has changed since the last
>>>>>>> check. For dynamically generated lists this isn't possible.
>>>>>>
>>>>>> You won’t need a HEAD request for it. You can include it in the GET request.
>>>>>>
>>>>>> Have a look at location downloader where I use that. The server will respond with 304 and not send any payload.
>>>>>>
>>>>>> https://git.ipfire.org/?p=location/libloc.git;a=blob;f=src/python/location-downloader.in;h=1b5932d5822e03737d32b2a27816815b2f7e74dd;hb=HEAD#l151
>>>>>>
>>>>>>>
>>>>>>> If the download isn't successful it just gives up and waits for the next
>>>>>>> attempt (apart from the usual retries in the library). I probably
>>>>>>> should to change that so that it only applies the per list minimum
>>>>>>> update period in this case (specified in the sources file) rather than
>>>>>>> the user specified value as well.
>>>>>>
>>>>>> I think it is not the worst if an update fails. It might just happen every once in a while.
>>>>>>
>>>>>> So I would suggest to just re-run the script more often and when the mtime of the file is older than the threshold, a download is attempted. You can use that timestamp for the GET request.
>>>>>>
>>>>>>> I already use a time offset on the downloads - when it's started from
>>>>>>> boot, backup restore or WUI enable, it checks to see if it's installed
>>>>>>> in the fcrontab, and if not adds itself at a randomly generated offset
>>>>>>> in the hour.
>>>>>>
>>>>>> That should potentially go to red.up, or if we can settle on 15 minutes, I would consider that often enough to quickly update all lists after a reboot.
>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>>>> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is
>>>>>>>>>>> the largest at a few MB), there are going to be a lot of them. How will
>>>>>>>>>>> this affect the willingness of people to mirror IPFire?
>>>>>>>>> I do not consider this being a problem as we do not generate that much traffic
>>>>>>>>> to them. Of course, that depends on the update interval again.
>>>>>>>>
>>>>>>>> That depends on your point of view.
>>>>>>>>
>>>>>>>> I do not have a problem with this at all in my data center, but there are plenty of people with a volume-based LTE plan or simply a 128 kBit/s connection. It will take a longer time to download the lists for them. We need to mind that.
>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Talking about the preference of packet filter and IPS, I prefer to
>>>>>>>>>>>> use the latter as well as it gains more insight in what kind of malicious
>>>>>>>>>>>> traffic tried to pass a firewall machine. On systems with low resources,
>>>>>>>>>>>> this might be problematic and removing load from the IPS can be preferred
>>>>>>>>>>>> (make this configurable?!), on others, people might want to have both
>>>>>>>>>>>> results.
>>>>>>>>>>>>
>>>>>>>>>>> You're only going to get one result for a packet whichever way round the
>>>>>>>>>>> IP blacklist and IPS are since whichever comes first will drop the
>>>>>>>>>>> packet before it reaches the second (well it would be possible to put
>>>>>>>>>>> the IP blacklist first and get it to log and mark packets which are then
>>>>>>>>>>> dropped after the IPS, but I think that's getting a little complicated.
>>>>>>>>>>> In addition I've seen the messages about the trouble marking was
>>>>>>>>>>> causing in the QoS).
>>>>>>>>>>>
>>>>>>>>>>> I think it's a 50/50 choice as to which is more valuable first; it's
>>>>>>>>>>> probably going to differ from packet to packet. For me the possibility
>>>>>>>>>>> of reducing the IPS load means I prefer putting the IP blacklist first
>>>>>>>>>>>
>>>>>>>>>>> It should be fairly easy to add the choice of where to put the IP
>>>>>>>>>>> blacklist. I think it'll have to be in the main firewall script, so
>>>>>>>>>>> it'll require a firewall restart, but it's not something that'll be
>>>>>>>>>>> changed often.
>>>>>>>>>>
>>>>>>>>>> I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier.
>>>>>>>>> Agreed.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads.
>>>>>>>>>>
>>>>>>>>>> We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily.
>>>>>>>>> So dropping packets from blacklisted IP addresses/networks before IPS is it, then.
>>>>>>>>>>
>>>>>>>>>> I do not even think it makes sense to swap the order in the outgoing direction.
>>>>>>>>> Me too.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this.
>>>>>>>>> Based on my personal experience, firewall hits observed on a single machine exposed
>>>>>>>>> to the internet are interesting, but the overall situation across multiple machines
>>>>>>>>> is even more interesting. Very quickly, you'll end on something like a centralised
>>>>>>>>> logging server and custom statistical analysis here...
>>>>>>>>
>>>>>>>> Probably a project for IPFire 4.0 :)
>>>>>>>>
>>>>>>> Or use one of the existing services, like the DSHIELD client
>>>>>>> https://dshield.org/howto.html (subject to privacy concerns again).
>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default.
>>>>>>>>>>>> I would love to see the bogon ruleset here, too (think about 8chan successor
>>>>>>>>>>>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause
>>>>>>>>>>>> interference if RED does not have a public IP address assigned.
>>>>>>>>>>>
>>>>>>>>>>> I can add a field to the options file that controls whether a list is
>>>>>>>>>>> enabled by default.
>>>>>>>>> Thank you. :-)
>>>>>>>>>>
>>>>>>>>>> To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat.
>>>>>>>>> This is my only objection against this patchset. Now, what can we do about it?
>>>>>>>>> One possibility is to apply the patchset now and implement a custom download
>>>>>>>>> source thing later on, or do that before releasing Core Update 139 (or which version
>>>>>>>>> the patchset will be to) after we agreed on something.
>>>>>>>>
>>>>>>>> I do not see this being merged for 139. But that is not important. We need to get it right first and then release it.
>>>>>>>>
>>>>>>>> As far as I know, nobody has tested this, yet.
>>>>>>>
>>>>>>> There are a number of people who have been running an earlier version
>>>>>>> which I shared on GitHub. There were a few early issues, but it seems
>>>>>>> to be OK now.
>>>>>>>
>>>>>>> https://forum.ipfire.org/viewtopic.php?f=27&t=21845
>>>>>>>
>>>>>>> This version wasn't integrated into IPFire, so (for example) it inserted
>>>>>>> itself into the INPUT IPTables chain rather than having it's chains
>>>>>>> created as part of the firewall start-up script.
>>>>>>>
>>>>>>>>
>>>>>>>> I have huge concerns about the automatic blacklist. @Peter: What is your opinion on this?
>>>>>>>>
>>>>>>>
>>>>>>> While I implemented it, I'm aware of its potential to cause problems,
>>>>>>> which is why it has to be separately enabled. It's not caused me any
>>>>>>> issues at the default settings (blocks at over 10 packets per hour until
>>>>>>> 1 hour has passed without seeing packets from the address), but I've not
>>>>>>> used it on a site with publicly announced services. If I was going to
>>>>>>> use it on a web site I would want to, at the very minimum, drop the
>>>>>>> block period drastically.
>>>>>>
>>>>>> I suppose this is entirely unusable on an IPFire box in a data center that hosts things. Let’s say our rack in Hanover.
>>>>>>
>>>>>> You will have hits from some broken IP stacks and people might just end up on this without doing anything wrong.
>>>>>>
>>>>>> You can denial-of-service easily as well and I suppose without aggregation of data from many many systems, it does not make sense to instantly block addresses. And even then you probably would block an entire subnet.
>>>>>>
>>>>>> So I am not sure how I should feel about it. I do not think that it adds anything because the packets would have been blocked anyways.
>>>>>>
>>>>>>>
>>>>>>> On the other hand, it's good at responding quickly. Usually I see only
>>>>>>> 1-2% of blocks from the automatic list:
>>>>>>>
>>>>>>> Reason Count % First Last
>>>>>>> CIARMY 2416 45 Dec 6 00:00 Dec 6 23:59
>>>>>>> DSHIELD 1353 25 Dec 6 00:00 Dec 6 23:59
>>>>>>> INPUT 1294 24 Dec 6 00:00 Dec 6 23:59
>>>>>>> AUTOBLACKLIST 122 2 Dec 6 00:20 Dec 6 16:28
>>>>>>> BLOCKLIST_DE 89 2 Dec 6 00:20 Dec 6 23:46
>>>>>>>
>>>>>>> and sometimes none at all, but one one occasion it blocked over 8000
>>>>>>> packets. Again I'm aware this is for a home system, which is rather
>>>>>>> different than from a Web server.
>>>>>>>
>>>>>>>>> If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not
>>>>>>>>> seem to be problematic, as they are hosted on 3rd party servers, too).
>>>>>>>>
>>>>>>>> One of them will be, sooner or later. And one is enough I suppose.
>>>>>>>
>>>>>>> DShield (https://dshield.org/api/#threatfeeds) and firehol
>>>>>>> (http://iplists.firehol.org/) seem to host copies of most of the lists
>>>>>>> as well.
>>>>>>>>
>>>>>>>> I do not really want to overthink this - we didn’t do this with clamav for example either. But it probably is a decision to be made by the user what they want to enable and we should not enable anything by default. So no data will be leaked as long as the user does not consent.
>>>>>>>>
>>>>>>>> -Michael
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks, and best regards,
>>>>>>>>> Peter Müller
>>>>>>
>>>>>
>>>>> <Screenshot_2019-12-16_IP_Address_Blacklists.png>
>>>>
>>>
>>> <Screenshot_2019-12-17_IP_Address_Blacklists.png>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 0/5] ipblacklist: IP Address Blacklists
2019-12-18 12:07 ` Michael Tremer
@ 2019-12-21 18:34 ` Tim FitzGeorge
2019-12-24 10:29 ` Michael Tremer
2019-12-28 20:59 ` Tim FitzGeorge
0 siblings, 2 replies; 35+ messages in thread
From: Tim FitzGeorge @ 2019-12-21 18:34 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 27832 bytes --]
Hi,
I've got my systems checking for downloads at 15 minute intervals and
using If-Modified-Since. This all seems to be working OK.
Responses to comments below:
On 18/12/2019 12:07, Michael Tremer wrote:
> Hi,
>
>> On 17 Dec 2019, at 18:21, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>
>> Hi,
>>
>> I think we've agreed on the following:
>>
>> - Checks to be made every 15 minutes, subject to per-list minimum rate.
>> - Modify download to use If-Modified-Since rather than separate HEAD
>> request.
>> - Remove automatic blacklist.
>
> I thought someone wants to counter my argument. This is not a dictatorship :)
>
>>
>> other comments are below.
>>
>> Tim
>>
>> On 16/12/2019 22:20, Michael Tremer wrote:
>>> Hi,
>>>
>>>> On 16 Dec 2019, at 20:06, Tim FitzGeorge <lists(a)tfitzgeorge.me.uk> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I've attached the current GUI screenshot.
>>>
>>> Thanks for that.
>>>
>>> I have a couple of suggestions/concerns about it:
>>>
>>> a) I am not sure what the value is about the top box that is called “Status”. It is basically a summary of the iptables output, but will it help the user? A blacklist with more or fewer entries is not better or worse, blocking more packets isn’t better than blocking fewer. It is about the quality of the blocks.
>>
>> I agree that the size of the lists is not all that useful, but the
>> information about the number of blocked packets can be an indication of
>> compromise. It depends on the type of the list, but a list that targets
>> C&C channels (e.g. the FEODO lists) should not show any input packets
>> being blocked, unless one of the computers being protected is infected.
>> Apart from the BOGON lists, none of the lists should show any output
>> packets being blocked; again it's a potential indication of infection if
>> otherwise.
>
> The Shodan list will definitely show some activity unless you are behind CG NAT.
>
> Should we not have this rather in the logging section?
>
> And these counters here will be reset when the firewall is being reloaded. Should this data therefore not be collected from the logs so that you can go back in time? That would be very important to identify any start of compromise.
>
That makes sense. I'll remove the status from this page and create a
log page for the information. If anyone actually wants the iptables
information it's available in from Firewall > iptables anyway.
Do you think it makes more sense to show packets blocked in/out or
packets blocked per source interface? The latter would of (limited)
help in tracking down potential compromise.
>>>
>>> b) I would prefer to move the settings box to the top. If you like you can show the size of the blacklists there and when they have been updated, too.
>>
>> I put the status at the top since it's usually what I want to see when I
>> go to the page. Under normal circumstances I don't change the settings,
>> so I'd rather not have to scroll past them to get to the information I'm
>> actually interested in. The update times are just useful to show that
>> the lists are actually being updated. The size is interesting to me,
>> but as you say it probably won't help the average user, especially if
>> you don't know which lists block individual addresses and which block
>> nets (BOGON, DSHIELD and EMERGING_COMPROMISED).
>
> See my comment on the order above. Splitting the page might make sense.
>
> When you show size, does that mean lines in the blacklists? So it could be networks or single IP addresses? Does it not make sense to count IP addresses (i.e. 256 for a /24 network and so on) and put it in the table?
It's lines in the blacklist.
>
> I think I would only be interested in the size for two things:
>
> a) Is there any chance for overblocking? i.e. does the list have 2 billion entries which would be half the public IPv4 address space or so. I cannot really come up with a good example here.
>
> Or
>
> b) What is the performance impact/memory consumption of the list?
It would be interesting to show the type (single address or net), number
of entries and number of addresses; the first two affect performance,
the second memory use and the latter the number of blocked addresses,
but that's really only going to be of interest to a very small number of
people, who can always do 'ipset list -t' to get the information. So
I'll remove the size column.
>
>> I originally had the status and settings in one list, but I decided that
>> it was getting a bit messy and difficult to follow. I've attached a
>> screenshot of a quickly hacked page. (Note that the status area only
>> appears once the blacklists have been enabled.)
>
> You are right. This is getting quite tight there.
>
> You could write packets in/out into on column like 2k/1023, but see my concerns about this above.
Actually I've come to the conclusion that the number of bytes blocked
isn't all that important; it's sufficient to show the number of packets.
>
>>>
>>> c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
>>
>> I agree that it's it's an over simplistic summary of the lists. I note
>> that Tom Rymes has asked to keep it, but I think its only value is if it
>> suggests to users that things are more complicated than they appear and
>> therefore prompts them to read the Wiki.
>
> I will reply to that separately.
Looking at that reply, I think it makes sense to include a category
column in both the settings and the log summary page - the former to
help selection of which lists to use, and the latter as a help when
deciding whether the information could indicate compromise or not.
>
>>>
>>> d) I do not understand what the “update check rate” means. I suppose we should show the update interval of the lists in the table and just refresh them according to that.
>>
>> It's the minimum check period for updates. The system checks for
>> updates at either the rate specific to the list or this rate, whichever
>> is slower. This is a hangover from the early days, copied from the IPS
>> updates. I suppose it could have a value for some people who may wish
>> to lower the check rate, but I'm not sure that it has sufficient value
>> to be worth keeping.
>
> No, I would not let the user decide when and how often to update the lists.
OK. I'll remove it.
>
> Generally there is this argument internally how often some databases should be updated. People with slow internet connections or volume based data plans will find this rather expensive to update those lists too often.
>
> If we ever want to go ahead with that in IPFire 2, there should be a global switch for a “low data mode” which then delays updating these things.
That makes sense. Maybe also the ability to set when downloads take
place - but as you say that's for the future.
>
>>>
>>> e) Do we want to keep the automatic blacklists now? I do not think it will actually improve the security of IPFire.
>>>
>>
>> I'll remove them. I think they do have a use, in that they can detect
>> someone doing a port scan and block them before they find an open port,
>> but as you say there is also the possibility of a DOS.
>
> How would this block a DOS?
Sorry, I didn't make myself clear - there's the possibility that someone
could cause a DOS with the automatic list by faking the source IP
address and sending a few packets on a random port.
>
> -Michael
Tim
>
>>>>
>>>> I'll update the code based on our discussions, and submit an updated set
>>>> of patches - I imagine there will have to be at least one more iteration.
>>>
>>> Let’s wait until we have come to decisions :)
>>>
>>> -Michael
>>>
>>>>
>>>> Tim
>>>>
>>>> (No additional comments below)
>>>>
>>>> On 13/12/2019 23:11, Michael Tremer wrote:
>>>>> Hi,
>>>>>
>>>>> Again my apologies for my late reply. Busy busy weeks.
>>>>>
>>>>>> On 8 Dec 2019, at 20:50, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> It's my turn to apologise for being slow to respond - I've had a busy
>>>>>> week, but I should have plenty of time over the next couple of weeks.
>>>>>
>>>>> No worries. Turns out we all do :)
>>>>>
>>>>>> I've made most of the comments inline, however I think Michael had a
>>>>>> question (which I can't find now) about what happens if someone enables
>>>>>> all the lists. One thing which would perhaps make this less likely is
>>>>>> that the WUI tags the available lists with whether they're safe or not,
>>>>>> with a footnote that safe means that the list only blocks malicious
>>>>>> traffic. This won't guarantee that a user won't still try to enable all
>>>>>> the lists, but it should make them realise that they should think first.
>>>>>
>>>>> We had a couple of features going slightly wrong or being “misunderstood” by some users. People still seem to panic when they see “local recursor”. To this day I do not know why.
>>>>>
>>>>> We cannot make everything idiot-proof. And when some user if of that category, they probably should shutdown their IPFire box, educate themselves and then come back again. So I do not want to limit people, but make things as easy as possible.
>>>>>
>>>>> If someone enables all the lists, good luck with passing packets :)
>>>>>
>>>>>> I have considered replacing this tag with a risk high/medium/low and
>>>>>> maybe adding a category (invalid/application/scanner/C&C or something
>>>>>> like that), but that may provide too much information and dissuade them
>>>>>> from actually following the links to checkout what the list actually does.
>>>>>
>>>>> Can we have a screenshot of the GUI right now? I didn’t run the code, yet.
>>>>>
>>>>> We should document the lists like we do it with the rulesets of the IPS. People might ignore this, but that is on them.
>>>>>
>>>>>>
>>>>>> Tim
>>>>>>
>>>>>>
>>>>>> On 05/12/2019 22:25, Michael Tremer wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>>> On 4 Dec 2019, at 17:05, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>>>>>>>
>>>>>>>> Hello Tim, hello Michael,
>>>>>>>>
>>>>>>>> please see my responses inline...
>>>>>>>>>>>
>>>>>>>>>>> We could periodically update the blacklists on our main mirror (and
>>>>>>>>>>> wait for the network to sync it), make sure it is signed and write
>>>>>>>>>>> a small downloader that fetches, validates and installs them.
>>>>>>>>>>>
>>>>>>>>>>> @All: Thoughts on this?
>>>>>>>>>>
>>>>>>>>>> I think there are a number of points here.
>>>>>>>>>>
>>>>>>>>>> Firstly, from the point of a third party using IPFire, is this really
>>>>>>>>>> solving the privacy disclosure problem? There's no way round disclosing
>>>>>>>>>> your public IP address to someone you're downloading from; all this does
>>>>>>>>>> is change who that information is being disclosed to. For the user
>>>>>>>>>> there's no way of knowing whether the source is more or less protective
>>>>>>>>>> of the user's privacy than the blacklist provider. Indeed it won't be
>>>>>>>>>> possible to know who the lists are being downloaded from until the
>>>>>>>>>> download starts.
>>>>>>>>>
>>>>>>>>> There is a way: Tor. But that is a totally different story.
>>>>>>>> Well, I see a third option on this: Use the mirror infrastructure we already
>>>>>>>> have. Every IPFire installation discloses its public IP address to one of these
>>>>>>>> servers sooner or later, so we do not disclose additional data if the blacklists
>>>>>>>> were fetched from these.
>>>>>>>>
>>>>>>>> Needless to say, Tor (hidden services) would be better, but that is a different
>>>>>>>> story indeed. :-)
>>>>>>>>>
>>>>>>>>> The point is rather that a forget list can be sent instead of the “real” one.
>>>>>>>> I did not get this. Forget? Forged?? ???
>>>>>>>
>>>>>>> Yes, I meant to write forged, but auto-correct didn’t let me.
>>>>>>>
>>>>>>>>>> Secondly, latency; some of the lists are updated every 5 minutes. While
>>>>>>>>>> I've limited the maximum check rate to hourly, will the updates
>>>>>>>>>> propagate quickly enough. For reference on my main system the 24
>>>>>>>>>> updates on the CIARMY list made 143 498 changes (additions or
>>>>>>>>>> deletions). I've seem it do over 200 000.
>>>>>>>> Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not
>>>>>>>> document a recommended update interval anywhere, so we can only guess.
>>>>>>>>
>>>>>>>> Personally, more static lists seem to be preferable for packet filtering. Highly
>>>>>>>> dynamic ones such as CIArmy should be done via DNSBL queries or something similar
>>>>>>>> - do we really want to have that list here?
>>>>>>>
>>>>>>> It is not really an option to implement a DNSBL into a packet filter, but I get your point.
>>>>>>
>>>>>> One of the 'selling points' for an IP address blacklist is that it can
>>>>>> respond quickly to new threats - or rather new attackers. While a new
>>>>>> IDS/IPS rule needs time to analyse the threat, generate a rule and check
>>>>>> it, it's easy to add an address to a list. So, I think the CIArmy list
>>>>>> is potentially useful for protecting home systems etc. with budget
>>>>>> hardware, but I would be very careful about using it for a protecting a
>>>>>> general access website.
>>>>>
>>>>> If they are very volatile, we should honour that and update them often, too.
>>>>>
>>>>> It probably is more about false-positives being removed very quickly instead of adding threats very quickly. The average IPFire user is probably not under threats like these to need to react very quickly.
>>>>>
>>>>> So I do not see much value in adding those lists and then updating once a day. Does it have to be every 5 min? No. I would suggest 15 which should be good enough for everyone.
>>>>>
>>>>> Other lists should of course not be updated every 15 minutes when not needed.
>>>>>
>>>>> Running every 15 minutes would allow us to retry downloading lists that are on an hourly schedule if the download failed.
>>>>>
>>>>>>>
>>>>>>>>> How did you come up with the hour? Will it be retried more often if the download was not successful?
>>>>>>>> One hour is the most common interval indeed, but adding some random time might
>>>>>>>> be useful in order to reduce load on the servers providing a blacklist.
>>>>>>>
>>>>>>> Yes, definitely. Otherwise we will shoot down our mirrors.
>>>>>>
>>>>>> When I implemented that section of code, specifying the minimum check
>>>>>> period in hours seemed to provide a convenient way of allowing a check
>>>>>> period covering a wide range, with an hour as the fastest and a week as
>>>>>> the slowest. I didn't looked at the CIArmy list until much later. Most
>>>>>> of the lists don't change nearly as much, but the CIArmy list is
>>>>>> described as one that deliberately responds quickly.
>>>>>>
>>>>>> From my production system, for yesterday:
>>>>>>
>>>>>> The following block lists were updated:
>>>>>> BLOCKLIST_DE: 24 Time(s) - 9341 change(s)
>>>>>> BOGON_FULL: 1 Time(s) - 10 change(s)
>>>>>> CIARMY: 24 Time(s) - 159134 change(s)
>>>>>> DSHIELD: 7 Time(s) - 18 change(s)
>>>>>> EMERGING_FWRULE: 1 Time(s) - 50 change(s)
>>>>>> FEODO_AGGRESIVE: 24 Time(s) - 13 change(s)
>>>>>> SHODAN: 1 Time(s) - 0 change(s)
>>>>>> SPAMHAUS_DROP: 1 Time(s) - 0 change(s)
>>>>>> TOR_EXIT: 24 Time(s) - 162 change(s)
>>>>>
>>>>> Very interesting statistics.
>>>>>
>>>>>>
>>>>>> and my test system:
>>>>>>
>>>>>> The following block lists were updated:
>>>>>> ALIENVAULT: 19 Time(s) - 5331 change(s)
>>>>>> EMERGING_COMPROMISED: 1 Time(s) - 26 change(s)
>>>>>> TALOS_MALICIOUS: 1 Time(s) - 36 change(s)
>>>>>>
>>>>>> That covers most of the lists. From the WUI, since 1 Dec:
>>>>>>
>>>>>> Blacklist Entries pkts bytes Last updated
>>>>>> in in
>>>>>> AUTOBLACKLIST 0 731 51144 Sun Dec 8 18:40:02 2019
>>>>>> BLOCKLIST_DE 28020 857 46735 Sun Dec 8 18:40:02 2019
>>>>>> BOGON_FULL 214 5255 189K Sun Dec 8 16:50:04 2019
>>>>>> CIARMY 15000 19774 976K Sun Dec 8 18:04:01 2019
>>>>>> DSHIELD 20 7992 321K Sun Dec 8 16:45:13 2019
>>>>>> EMERGING_FWRULE 1647 197 8383 Fri Dec 6 05:29:07 2019
>>>>>> FEODO_AGGRESIVE 7169 0 0 Sun Dec 8 18:50:07 2019
>>>>>> SHODAN 32 34 1530 Sun Dec 8 17:54:09 2019
>>>>>> SPAMHAUS_DROP 823 0 0 Fri Dec 6 18:22:35 2019
>>>>>> SPAMHAUS_EDROP 111 82 16433 Thu Dec 5 17:27:09 2019
>>>>>> TOR_EXIT 1055 0 0 Sun Dec 8 18:31:02 2019
>>>>>
>>>>> This as well.
>>>>>
>>>>> Those are more packets than I would have expected.
>>>>>
>>>>>>
>>>>>> (I've left out the pkts/bytes out fields which were all 0)
>>>>>>
>>>>>> Note that where possible I do a HEAD request first and then only
>>>>>> download the list if the modification time has changed since the last
>>>>>> check. For dynamically generated lists this isn't possible.
>>>>>
>>>>> You won’t need a HEAD request for it. You can include it in the GET request.
>>>>>
>>>>> Have a look at location downloader where I use that. The server will respond with 304 and not send any payload.
>>>>>
>>>>> https://git.ipfire.org/?p=location/libloc.git;a=blob;f=src/python/location-downloader.in;h=1b5932d5822e03737d32b2a27816815b2f7e74dd;hb=HEAD#l151
>>>>>
>>>>>>
>>>>>> If the download isn't successful it just gives up and waits for the next
>>>>>> attempt (apart from the usual retries in the library). I probably
>>>>>> should to change that so that it only applies the per list minimum
>>>>>> update period in this case (specified in the sources file) rather than
>>>>>> the user specified value as well.
>>>>>
>>>>> I think it is not the worst if an update fails. It might just happen every once in a while.
>>>>>
>>>>> So I would suggest to just re-run the script more often and when the mtime of the file is older than the threshold, a download is attempted. You can use that timestamp for the GET request.
>>>>>
>>>>>> I already use a time offset on the downloads - when it's started from
>>>>>> boot, backup restore or WUI enable, it checks to see if it's installed
>>>>>> in the fcrontab, and if not adds itself at a randomly generated offset
>>>>>> in the hour.
>>>>>
>>>>> That should potentially go to red.up, or if we can settle on 15 minutes, I would consider that often enough to quickly update all lists after a reboot.
>>>>>
>>>>>>
>>>>>>>
>>>>>>>>>> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is
>>>>>>>>>> the largest at a few MB), there are going to be a lot of them. How will
>>>>>>>>>> this affect the willingness of people to mirror IPFire?
>>>>>>>> I do not consider this being a problem as we do not generate that much traffic
>>>>>>>> to them. Of course, that depends on the update interval again.
>>>>>>>
>>>>>>> That depends on your point of view.
>>>>>>>
>>>>>>> I do not have a problem with this at all in my data center, but there are plenty of people with a volume-based LTE plan or simply a 128 kBit/s connection. It will take a longer time to download the lists for them. We need to mind that.
>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Talking about the preference of packet filter and IPS, I prefer to
>>>>>>>>>>> use the latter as well as it gains more insight in what kind of malicious
>>>>>>>>>>> traffic tried to pass a firewall machine. On systems with low resources,
>>>>>>>>>>> this might be problematic and removing load from the IPS can be preferred
>>>>>>>>>>> (make this configurable?!), on others, people might want to have both
>>>>>>>>>>> results.
>>>>>>>>>>>
>>>>>>>>>> You're only going to get one result for a packet whichever way round the
>>>>>>>>>> IP blacklist and IPS are since whichever comes first will drop the
>>>>>>>>>> packet before it reaches the second (well it would be possible to put
>>>>>>>>>> the IP blacklist first and get it to log and mark packets which are then
>>>>>>>>>> dropped after the IPS, but I think that's getting a little complicated.
>>>>>>>>>> In addition I've seen the messages about the trouble marking was
>>>>>>>>>> causing in the QoS).
>>>>>>>>>>
>>>>>>>>>> I think it's a 50/50 choice as to which is more valuable first; it's
>>>>>>>>>> probably going to differ from packet to packet. For me the possibility
>>>>>>>>>> of reducing the IPS load means I prefer putting the IP blacklist first
>>>>>>>>>>
>>>>>>>>>> It should be fairly easy to add the choice of where to put the IP
>>>>>>>>>> blacklist. I think it'll have to be in the main firewall script, so
>>>>>>>>>> it'll require a firewall restart, but it's not something that'll be
>>>>>>>>>> changed often.
>>>>>>>>>
>>>>>>>>> I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier.
>>>>>>>> Agreed.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads.
>>>>>>>>>
>>>>>>>>> We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily.
>>>>>>>> So dropping packets from blacklisted IP addresses/networks before IPS is it, then.
>>>>>>>>>
>>>>>>>>> I do not even think it makes sense to swap the order in the outgoing direction.
>>>>>>>> Me too.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this.
>>>>>>>> Based on my personal experience, firewall hits observed on a single machine exposed
>>>>>>>> to the internet are interesting, but the overall situation across multiple machines
>>>>>>>> is even more interesting. Very quickly, you'll end on something like a centralised
>>>>>>>> logging server and custom statistical analysis here...
>>>>>>>
>>>>>>> Probably a project for IPFire 4.0 :)
>>>>>>>
>>>>>> Or use one of the existing services, like the DSHIELD client
>>>>>> https://dshield.org/howto.html (subject to privacy concerns again).
>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default.
>>>>>>>>>>> I would love to see the bogon ruleset here, too (think about 8chan successor
>>>>>>>>>>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause
>>>>>>>>>>> interference if RED does not have a public IP address assigned.
>>>>>>>>>>
>>>>>>>>>> I can add a field to the options file that controls whether a list is
>>>>>>>>>> enabled by default.
>>>>>>>> Thank you. :-)
>>>>>>>>>
>>>>>>>>> To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat.
>>>>>>>> This is my only objection against this patchset. Now, what can we do about it?
>>>>>>>> One possibility is to apply the patchset now and implement a custom download
>>>>>>>> source thing later on, or do that before releasing Core Update 139 (or which version
>>>>>>>> the patchset will be to) after we agreed on something.
>>>>>>>
>>>>>>> I do not see this being merged for 139. But that is not important. We need to get it right first and then release it.
>>>>>>>
>>>>>>> As far as I know, nobody has tested this, yet.
>>>>>>
>>>>>> There are a number of people who have been running an earlier version
>>>>>> which I shared on GitHub. There were a few early issues, but it seems
>>>>>> to be OK now.
>>>>>>
>>>>>> https://forum.ipfire.org/viewtopic.php?f=27&t=21845
>>>>>>
>>>>>> This version wasn't integrated into IPFire, so (for example) it inserted
>>>>>> itself into the INPUT IPTables chain rather than having it's chains
>>>>>> created as part of the firewall start-up script.
>>>>>>
>>>>>>>
>>>>>>> I have huge concerns about the automatic blacklist. @Peter: What is your opinion on this?
>>>>>>>
>>>>>>
>>>>>> While I implemented it, I'm aware of its potential to cause problems,
>>>>>> which is why it has to be separately enabled. It's not caused me any
>>>>>> issues at the default settings (blocks at over 10 packets per hour until
>>>>>> 1 hour has passed without seeing packets from the address), but I've not
>>>>>> used it on a site with publicly announced services. If I was going to
>>>>>> use it on a web site I would want to, at the very minimum, drop the
>>>>>> block period drastically.
>>>>>
>>>>> I suppose this is entirely unusable on an IPFire box in a data center that hosts things. Let’s say our rack in Hanover.
>>>>>
>>>>> You will have hits from some broken IP stacks and people might just end up on this without doing anything wrong.
>>>>>
>>>>> You can denial-of-service easily as well and I suppose without aggregation of data from many many systems, it does not make sense to instantly block addresses. And even then you probably would block an entire subnet.
>>>>>
>>>>> So I am not sure how I should feel about it. I do not think that it adds anything because the packets would have been blocked anyways.
>>>>>
>>>>>>
>>>>>> On the other hand, it's good at responding quickly. Usually I see only
>>>>>> 1-2% of blocks from the automatic list:
>>>>>>
>>>>>> Reason Count % First Last
>>>>>> CIARMY 2416 45 Dec 6 00:00 Dec 6 23:59
>>>>>> DSHIELD 1353 25 Dec 6 00:00 Dec 6 23:59
>>>>>> INPUT 1294 24 Dec 6 00:00 Dec 6 23:59
>>>>>> AUTOBLACKLIST 122 2 Dec 6 00:20 Dec 6 16:28
>>>>>> BLOCKLIST_DE 89 2 Dec 6 00:20 Dec 6 23:46
>>>>>>
>>>>>> and sometimes none at all, but one one occasion it blocked over 8000
>>>>>> packets. Again I'm aware this is for a home system, which is rather
>>>>>> different than from a Web server.
>>>>>>
>>>>>>>> If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not
>>>>>>>> seem to be problematic, as they are hosted on 3rd party servers, too).
>>>>>>>
>>>>>>> One of them will be, sooner or later. And one is enough I suppose.
>>>>>>
>>>>>> DShield (https://dshield.org/api/#threatfeeds) and firehol
>>>>>> (http://iplists.firehol.org/) seem to host copies of most of the lists
>>>>>> as well.
>>>>>>>
>>>>>>> I do not really want to overthink this - we didn’t do this with clamav for example either. But it probably is a decision to be made by the user what they want to enable and we should not enable anything by default. So no data will be leaked as long as the user does not consent.
>>>>>>>
>>>>>>> -Michael
>>>>>>>
>>>>>>>>
>>>>>>>> Thanks, and best regards,
>>>>>>>> Peter Müller
>>>>>
>>>>
>>>> <Screenshot_2019-12-16_IP_Address_Blacklists.png>
>>>
>>
>> <Screenshot_2019-12-17_IP_Address_Blacklists.png>
>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 0/5] ipblacklist: IP Address Blacklists
2019-12-16 23:05 ` Tom Rymes
@ 2019-12-18 12:10 ` Michael Tremer
2019-12-28 21:17 ` Tim FitzGeorge
0 siblings, 1 reply; 35+ messages in thread
From: Michael Tremer @ 2019-12-18 12:10 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 1840 bytes --]
Hi,
> On 16 Dec 2019, at 23:05, Tom Rymes <trymes(a)rymes.com> wrote:
>
> On 12/16/2019 5:20 PM, Michael Tremer wrote:> Hi,
> >
> >> On 16 Dec 2019, at 20:06, Tim FitzGeorge <lists(a)tfitzgeorge.me.uk> wrote:
> >>
> >> Hi,
> >>
> >> I've attached the current GUI screenshot.
> >
> > Thanks for that.
> >
> > I have a couple of suggestions/concerns about it:
>
> [snip]
>
> > c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
>
> [snip]
>
> May I opine that the "Safe" information would be helpful to me in the WUI. Perhaps we can be more explicit, or better explain, such as is often done with RBLs in mail server settings, where lists are sometimes described in terms of their likelihood to cause false-positives.
>
> It's all well and good in the documentation, but a quick "Safe|Moderate|Risky" listing in the WUI will prove handy, IMHO.
>
> Just my $0.02 as more of a user than a developer,
I appreciate your input, but I still disagree with is that we take the decision if something is “risky” or not. There are too many things that need to be taken into account to make that decision and it probably varies for each user.
What I take from your comment though is that we should categorise the lists, and that is something we can do.
We can add a headline to the table and group the lists by “Blocking ambiguous packets”, “Blocking Malware”, etc.
That makes it easier for the user to decide which lists are interesting or even necessary depending on what they want to achieve.
How is that?
-Michael
>
> Tom
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 0/5] ipblacklist: IP Address Blacklists
[not found] <6583305a-cd0d-94a5-aa8e-5456622de824@tfitzgeorge.me.uk>
@ 2019-12-18 12:07 ` Michael Tremer
2019-12-21 18:34 ` Tim FitzGeorge
0 siblings, 1 reply; 35+ messages in thread
From: Michael Tremer @ 2019-12-18 12:07 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 25625 bytes --]
Hi,
> On 17 Dec 2019, at 18:21, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>
> Hi,
>
> I think we've agreed on the following:
>
> - Checks to be made every 15 minutes, subject to per-list minimum rate.
> - Modify download to use If-Modified-Since rather than separate HEAD
> request.
> - Remove automatic blacklist.
I thought someone wants to counter my argument. This is not a dictatorship :)
>
> other comments are below.
>
> Tim
>
> On 16/12/2019 22:20, Michael Tremer wrote:
>> Hi,
>>
>>> On 16 Dec 2019, at 20:06, Tim FitzGeorge <lists(a)tfitzgeorge.me.uk> wrote:
>>>
>>> Hi,
>>>
>>> I've attached the current GUI screenshot.
>>
>> Thanks for that.
>>
>> I have a couple of suggestions/concerns about it:
>>
>> a) I am not sure what the value is about the top box that is called “Status”. It is basically a summary of the iptables output, but will it help the user? A blacklist with more or fewer entries is not better or worse, blocking more packets isn’t better than blocking fewer. It is about the quality of the blocks.
>
> I agree that the size of the lists is not all that useful, but the
> information about the number of blocked packets can be an indication of
> compromise. It depends on the type of the list, but a list that targets
> C&C channels (e.g. the FEODO lists) should not show any input packets
> being blocked, unless one of the computers being protected is infected.
> Apart from the BOGON lists, none of the lists should show any output
> packets being blocked; again it's a potential indication of infection if
> otherwise.
The Shodan list will definitely show some activity unless you are behind CG NAT.
Should we not have this rather in the logging section?
And these counters here will be reset when the firewall is being reloaded. Should this data therefore not be collected from the logs so that you can go back in time? That would be very important to identify any start of compromise.
>>
>> b) I would prefer to move the settings box to the top. If you like you can show the size of the blacklists there and when they have been updated, too.
>
> I put the status at the top since it's usually what I want to see when I
> go to the page. Under normal circumstances I don't change the settings,
> so I'd rather not have to scroll past them to get to the information I'm
> actually interested in. The update times are just useful to show that
> the lists are actually being updated. The size is interesting to me,
> but as you say it probably won't help the average user, especially if
> you don't know which lists block individual addresses and which block
> nets (BOGON, DSHIELD and EMERGING_COMPROMISED).
See my comment on the order above. Splitting the page might make sense.
When you show size, does that mean lines in the blacklists? So it could be networks or single IP addresses? Does it not make sense to count IP addresses (i.e. 256 for a /24 network and so on) and put it in the table?
I think I would only be interested in the size for two things:
a) Is there any chance for overblocking? i.e. does the list have 2 billion entries which would be half the public IPv4 address space or so. I cannot really come up with a good example here.
Or
b) What is the performance impact/memory consumption of the list?
> I originally had the status and settings in one list, but I decided that
> it was getting a bit messy and difficult to follow. I've attached a
> screenshot of a quickly hacked page. (Note that the status area only
> appears once the blacklists have been enabled.)
You are right. This is getting quite tight there.
You could write packets in/out into on column like 2k/1023, but see my concerns about this above.
>>
>> c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
>
> I agree that it's it's an over simplistic summary of the lists. I note
> that Tom Rymes has asked to keep it, but I think its only value is if it
> suggests to users that things are more complicated than they appear and
> therefore prompts them to read the Wiki.
I will reply to that separately.
>>
>> d) I do not understand what the “update check rate” means. I suppose we should show the update interval of the lists in the table and just refresh them according to that.
>
> It's the minimum check period for updates. The system checks for
> updates at either the rate specific to the list or this rate, whichever
> is slower. This is a hangover from the early days, copied from the IPS
> updates. I suppose it could have a value for some people who may wish
> to lower the check rate, but I'm not sure that it has sufficient value
> to be worth keeping.
No, I would not let the user decide when and how often to update the lists.
Generally there is this argument internally how often some databases should be updated. People with slow internet connections or volume based data plans will find this rather expensive to update those lists too often.
If we ever want to go ahead with that in IPFire 2, there should be a global switch for a “low data mode” which then delays updating these things.
>>
>> e) Do we want to keep the automatic blacklists now? I do not think it will actually improve the security of IPFire.
>>
>
> I'll remove them. I think they do have a use, in that they can detect
> someone doing a port scan and block them before they find an open port,
> but as you say there is also the possibility of a DOS.
How would this block a DOS?
-Michael
>>>
>>> I'll update the code based on our discussions, and submit an updated set
>>> of patches - I imagine there will have to be at least one more iteration.
>>
>> Let’s wait until we have come to decisions :)
>>
>> -Michael
>>
>>>
>>> Tim
>>>
>>> (No additional comments below)
>>>
>>> On 13/12/2019 23:11, Michael Tremer wrote:
>>>> Hi,
>>>>
>>>> Again my apologies for my late reply. Busy busy weeks.
>>>>
>>>>> On 8 Dec 2019, at 20:50, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> It's my turn to apologise for being slow to respond - I've had a busy
>>>>> week, but I should have plenty of time over the next couple of weeks.
>>>>
>>>> No worries. Turns out we all do :)
>>>>
>>>>> I've made most of the comments inline, however I think Michael had a
>>>>> question (which I can't find now) about what happens if someone enables
>>>>> all the lists. One thing which would perhaps make this less likely is
>>>>> that the WUI tags the available lists with whether they're safe or not,
>>>>> with a footnote that safe means that the list only blocks malicious
>>>>> traffic. This won't guarantee that a user won't still try to enable all
>>>>> the lists, but it should make them realise that they should think first.
>>>>
>>>> We had a couple of features going slightly wrong or being “misunderstood” by some users. People still seem to panic when they see “local recursor”. To this day I do not know why.
>>>>
>>>> We cannot make everything idiot-proof. And when some user if of that category, they probably should shutdown their IPFire box, educate themselves and then come back again. So I do not want to limit people, but make things as easy as possible.
>>>>
>>>> If someone enables all the lists, good luck with passing packets :)
>>>>
>>>>> I have considered replacing this tag with a risk high/medium/low and
>>>>> maybe adding a category (invalid/application/scanner/C&C or something
>>>>> like that), but that may provide too much information and dissuade them
>>>>> from actually following the links to checkout what the list actually does.
>>>>
>>>> Can we have a screenshot of the GUI right now? I didn’t run the code, yet.
>>>>
>>>> We should document the lists like we do it with the rulesets of the IPS. People might ignore this, but that is on them.
>>>>
>>>>>
>>>>> Tim
>>>>>
>>>>>
>>>>> On 05/12/2019 22:25, Michael Tremer wrote:
>>>>>> Hello,
>>>>>>
>>>>>>> On 4 Dec 2019, at 17:05, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>>>>>>
>>>>>>> Hello Tim, hello Michael,
>>>>>>>
>>>>>>> please see my responses inline...
>>>>>>>>>>
>>>>>>>>>> We could periodically update the blacklists on our main mirror (and
>>>>>>>>>> wait for the network to sync it), make sure it is signed and write
>>>>>>>>>> a small downloader that fetches, validates and installs them.
>>>>>>>>>>
>>>>>>>>>> @All: Thoughts on this?
>>>>>>>>>
>>>>>>>>> I think there are a number of points here.
>>>>>>>>>
>>>>>>>>> Firstly, from the point of a third party using IPFire, is this really
>>>>>>>>> solving the privacy disclosure problem? There's no way round disclosing
>>>>>>>>> your public IP address to someone you're downloading from; all this does
>>>>>>>>> is change who that information is being disclosed to. For the user
>>>>>>>>> there's no way of knowing whether the source is more or less protective
>>>>>>>>> of the user's privacy than the blacklist provider. Indeed it won't be
>>>>>>>>> possible to know who the lists are being downloaded from until the
>>>>>>>>> download starts.
>>>>>>>>
>>>>>>>> There is a way: Tor. But that is a totally different story.
>>>>>>> Well, I see a third option on this: Use the mirror infrastructure we already
>>>>>>> have. Every IPFire installation discloses its public IP address to one of these
>>>>>>> servers sooner or later, so we do not disclose additional data if the blacklists
>>>>>>> were fetched from these.
>>>>>>>
>>>>>>> Needless to say, Tor (hidden services) would be better, but that is a different
>>>>>>> story indeed. :-)
>>>>>>>>
>>>>>>>> The point is rather that a forget list can be sent instead of the “real” one.
>>>>>>> I did not get this. Forget? Forged?? ???
>>>>>>
>>>>>> Yes, I meant to write forged, but auto-correct didn’t let me.
>>>>>>
>>>>>>>>> Secondly, latency; some of the lists are updated every 5 minutes. While
>>>>>>>>> I've limited the maximum check rate to hourly, will the updates
>>>>>>>>> propagate quickly enough. For reference on my main system the 24
>>>>>>>>> updates on the CIARMY list made 143 498 changes (additions or
>>>>>>>>> deletions). I've seem it do over 200 000.
>>>>>>> Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not
>>>>>>> document a recommended update interval anywhere, so we can only guess.
>>>>>>>
>>>>>>> Personally, more static lists seem to be preferable for packet filtering. Highly
>>>>>>> dynamic ones such as CIArmy should be done via DNSBL queries or something similar
>>>>>>> - do we really want to have that list here?
>>>>>>
>>>>>> It is not really an option to implement a DNSBL into a packet filter, but I get your point.
>>>>>
>>>>> One of the 'selling points' for an IP address blacklist is that it can
>>>>> respond quickly to new threats - or rather new attackers. While a new
>>>>> IDS/IPS rule needs time to analyse the threat, generate a rule and check
>>>>> it, it's easy to add an address to a list. So, I think the CIArmy list
>>>>> is potentially useful for protecting home systems etc. with budget
>>>>> hardware, but I would be very careful about using it for a protecting a
>>>>> general access website.
>>>>
>>>> If they are very volatile, we should honour that and update them often, too.
>>>>
>>>> It probably is more about false-positives being removed very quickly instead of adding threats very quickly. The average IPFire user is probably not under threats like these to need to react very quickly.
>>>>
>>>> So I do not see much value in adding those lists and then updating once a day. Does it have to be every 5 min? No. I would suggest 15 which should be good enough for everyone.
>>>>
>>>> Other lists should of course not be updated every 15 minutes when not needed.
>>>>
>>>> Running every 15 minutes would allow us to retry downloading lists that are on an hourly schedule if the download failed.
>>>>
>>>>>>
>>>>>>>> How did you come up with the hour? Will it be retried more often if the download was not successful?
>>>>>>> One hour is the most common interval indeed, but adding some random time might
>>>>>>> be useful in order to reduce load on the servers providing a blacklist.
>>>>>>
>>>>>> Yes, definitely. Otherwise we will shoot down our mirrors.
>>>>>
>>>>> When I implemented that section of code, specifying the minimum check
>>>>> period in hours seemed to provide a convenient way of allowing a check
>>>>> period covering a wide range, with an hour as the fastest and a week as
>>>>> the slowest. I didn't looked at the CIArmy list until much later. Most
>>>>> of the lists don't change nearly as much, but the CIArmy list is
>>>>> described as one that deliberately responds quickly.
>>>>>
>>>>> From my production system, for yesterday:
>>>>>
>>>>> The following block lists were updated:
>>>>> BLOCKLIST_DE: 24 Time(s) - 9341 change(s)
>>>>> BOGON_FULL: 1 Time(s) - 10 change(s)
>>>>> CIARMY: 24 Time(s) - 159134 change(s)
>>>>> DSHIELD: 7 Time(s) - 18 change(s)
>>>>> EMERGING_FWRULE: 1 Time(s) - 50 change(s)
>>>>> FEODO_AGGRESIVE: 24 Time(s) - 13 change(s)
>>>>> SHODAN: 1 Time(s) - 0 change(s)
>>>>> SPAMHAUS_DROP: 1 Time(s) - 0 change(s)
>>>>> TOR_EXIT: 24 Time(s) - 162 change(s)
>>>>
>>>> Very interesting statistics.
>>>>
>>>>>
>>>>> and my test system:
>>>>>
>>>>> The following block lists were updated:
>>>>> ALIENVAULT: 19 Time(s) - 5331 change(s)
>>>>> EMERGING_COMPROMISED: 1 Time(s) - 26 change(s)
>>>>> TALOS_MALICIOUS: 1 Time(s) - 36 change(s)
>>>>>
>>>>> That covers most of the lists. From the WUI, since 1 Dec:
>>>>>
>>>>> Blacklist Entries pkts bytes Last updated
>>>>> in in
>>>>> AUTOBLACKLIST 0 731 51144 Sun Dec 8 18:40:02 2019
>>>>> BLOCKLIST_DE 28020 857 46735 Sun Dec 8 18:40:02 2019
>>>>> BOGON_FULL 214 5255 189K Sun Dec 8 16:50:04 2019
>>>>> CIARMY 15000 19774 976K Sun Dec 8 18:04:01 2019
>>>>> DSHIELD 20 7992 321K Sun Dec 8 16:45:13 2019
>>>>> EMERGING_FWRULE 1647 197 8383 Fri Dec 6 05:29:07 2019
>>>>> FEODO_AGGRESIVE 7169 0 0 Sun Dec 8 18:50:07 2019
>>>>> SHODAN 32 34 1530 Sun Dec 8 17:54:09 2019
>>>>> SPAMHAUS_DROP 823 0 0 Fri Dec 6 18:22:35 2019
>>>>> SPAMHAUS_EDROP 111 82 16433 Thu Dec 5 17:27:09 2019
>>>>> TOR_EXIT 1055 0 0 Sun Dec 8 18:31:02 2019
>>>>
>>>> This as well.
>>>>
>>>> Those are more packets than I would have expected.
>>>>
>>>>>
>>>>> (I've left out the pkts/bytes out fields which were all 0)
>>>>>
>>>>> Note that where possible I do a HEAD request first and then only
>>>>> download the list if the modification time has changed since the last
>>>>> check. For dynamically generated lists this isn't possible.
>>>>
>>>> You won’t need a HEAD request for it. You can include it in the GET request.
>>>>
>>>> Have a look at location downloader where I use that. The server will respond with 304 and not send any payload.
>>>>
>>>> https://git.ipfire.org/?p=location/libloc.git;a=blob;f=src/python/location-downloader.in;h=1b5932d5822e03737d32b2a27816815b2f7e74dd;hb=HEAD#l151
>>>>
>>>>>
>>>>> If the download isn't successful it just gives up and waits for the next
>>>>> attempt (apart from the usual retries in the library). I probably
>>>>> should to change that so that it only applies the per list minimum
>>>>> update period in this case (specified in the sources file) rather than
>>>>> the user specified value as well.
>>>>
>>>> I think it is not the worst if an update fails. It might just happen every once in a while.
>>>>
>>>> So I would suggest to just re-run the script more often and when the mtime of the file is older than the threshold, a download is attempted. You can use that timestamp for the GET request.
>>>>
>>>>> I already use a time offset on the downloads - when it's started from
>>>>> boot, backup restore or WUI enable, it checks to see if it's installed
>>>>> in the fcrontab, and if not adds itself at a randomly generated offset
>>>>> in the hour.
>>>>
>>>> That should potentially go to red.up, or if we can settle on 15 minutes, I would consider that often enough to quickly update all lists after a reboot.
>>>>
>>>>>
>>>>>>
>>>>>>>>> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is
>>>>>>>>> the largest at a few MB), there are going to be a lot of them. How will
>>>>>>>>> this affect the willingness of people to mirror IPFire?
>>>>>>> I do not consider this being a problem as we do not generate that much traffic
>>>>>>> to them. Of course, that depends on the update interval again.
>>>>>>
>>>>>> That depends on your point of view.
>>>>>>
>>>>>> I do not have a problem with this at all in my data center, but there are plenty of people with a volume-based LTE plan or simply a 128 kBit/s connection. It will take a longer time to download the lists for them. We need to mind that.
>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Talking about the preference of packet filter and IPS, I prefer to
>>>>>>>>>> use the latter as well as it gains more insight in what kind of malicious
>>>>>>>>>> traffic tried to pass a firewall machine. On systems with low resources,
>>>>>>>>>> this might be problematic and removing load from the IPS can be preferred
>>>>>>>>>> (make this configurable?!), on others, people might want to have both
>>>>>>>>>> results.
>>>>>>>>>>
>>>>>>>>> You're only going to get one result for a packet whichever way round the
>>>>>>>>> IP blacklist and IPS are since whichever comes first will drop the
>>>>>>>>> packet before it reaches the second (well it would be possible to put
>>>>>>>>> the IP blacklist first and get it to log and mark packets which are then
>>>>>>>>> dropped after the IPS, but I think that's getting a little complicated.
>>>>>>>>> In addition I've seen the messages about the trouble marking was
>>>>>>>>> causing in the QoS).
>>>>>>>>>
>>>>>>>>> I think it's a 50/50 choice as to which is more valuable first; it's
>>>>>>>>> probably going to differ from packet to packet. For me the possibility
>>>>>>>>> of reducing the IPS load means I prefer putting the IP blacklist first
>>>>>>>>>
>>>>>>>>> It should be fairly easy to add the choice of where to put the IP
>>>>>>>>> blacklist. I think it'll have to be in the main firewall script, so
>>>>>>>>> it'll require a firewall restart, but it's not something that'll be
>>>>>>>>> changed often.
>>>>>>>>
>>>>>>>> I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier.
>>>>>>> Agreed.
>>>>>>>
>>>>>>>>
>>>>>>>> I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads.
>>>>>>>>
>>>>>>>> We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily.
>>>>>>> So dropping packets from blacklisted IP addresses/networks before IPS is it, then.
>>>>>>>>
>>>>>>>> I do not even think it makes sense to swap the order in the outgoing direction.
>>>>>>> Me too.
>>>>>>>
>>>>>>>>
>>>>>>>> What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this.
>>>>>>> Based on my personal experience, firewall hits observed on a single machine exposed
>>>>>>> to the internet are interesting, but the overall situation across multiple machines
>>>>>>> is even more interesting. Very quickly, you'll end on something like a centralised
>>>>>>> logging server and custom statistical analysis here...
>>>>>>
>>>>>> Probably a project for IPFire 4.0 :)
>>>>>>
>>>>> Or use one of the existing services, like the DSHIELD client
>>>>> https://dshield.org/howto.html (subject to privacy concerns again).
>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default.
>>>>>>>>>> I would love to see the bogon ruleset here, too (think about 8chan successor
>>>>>>>>>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause
>>>>>>>>>> interference if RED does not have a public IP address assigned.
>>>>>>>>>
>>>>>>>>> I can add a field to the options file that controls whether a list is
>>>>>>>>> enabled by default.
>>>>>>> Thank you. :-)
>>>>>>>>
>>>>>>>> To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat.
>>>>>>> This is my only objection against this patchset. Now, what can we do about it?
>>>>>>> One possibility is to apply the patchset now and implement a custom download
>>>>>>> source thing later on, or do that before releasing Core Update 139 (or which version
>>>>>>> the patchset will be to) after we agreed on something.
>>>>>>
>>>>>> I do not see this being merged for 139. But that is not important. We need to get it right first and then release it.
>>>>>>
>>>>>> As far as I know, nobody has tested this, yet.
>>>>>
>>>>> There are a number of people who have been running an earlier version
>>>>> which I shared on GitHub. There were a few early issues, but it seems
>>>>> to be OK now.
>>>>>
>>>>> https://forum.ipfire.org/viewtopic.php?f=27&t=21845
>>>>>
>>>>> This version wasn't integrated into IPFire, so (for example) it inserted
>>>>> itself into the INPUT IPTables chain rather than having it's chains
>>>>> created as part of the firewall start-up script.
>>>>>
>>>>>>
>>>>>> I have huge concerns about the automatic blacklist. @Peter: What is your opinion on this?
>>>>>>
>>>>>
>>>>> While I implemented it, I'm aware of its potential to cause problems,
>>>>> which is why it has to be separately enabled. It's not caused me any
>>>>> issues at the default settings (blocks at over 10 packets per hour until
>>>>> 1 hour has passed without seeing packets from the address), but I've not
>>>>> used it on a site with publicly announced services. If I was going to
>>>>> use it on a web site I would want to, at the very minimum, drop the
>>>>> block period drastically.
>>>>
>>>> I suppose this is entirely unusable on an IPFire box in a data center that hosts things. Let’s say our rack in Hanover.
>>>>
>>>> You will have hits from some broken IP stacks and people might just end up on this without doing anything wrong.
>>>>
>>>> You can denial-of-service easily as well and I suppose without aggregation of data from many many systems, it does not make sense to instantly block addresses. And even then you probably would block an entire subnet.
>>>>
>>>> So I am not sure how I should feel about it. I do not think that it adds anything because the packets would have been blocked anyways.
>>>>
>>>>>
>>>>> On the other hand, it's good at responding quickly. Usually I see only
>>>>> 1-2% of blocks from the automatic list:
>>>>>
>>>>> Reason Count % First Last
>>>>> CIARMY 2416 45 Dec 6 00:00 Dec 6 23:59
>>>>> DSHIELD 1353 25 Dec 6 00:00 Dec 6 23:59
>>>>> INPUT 1294 24 Dec 6 00:00 Dec 6 23:59
>>>>> AUTOBLACKLIST 122 2 Dec 6 00:20 Dec 6 16:28
>>>>> BLOCKLIST_DE 89 2 Dec 6 00:20 Dec 6 23:46
>>>>>
>>>>> and sometimes none at all, but one one occasion it blocked over 8000
>>>>> packets. Again I'm aware this is for a home system, which is rather
>>>>> different than from a Web server.
>>>>>
>>>>>>> If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not
>>>>>>> seem to be problematic, as they are hosted on 3rd party servers, too).
>>>>>>
>>>>>> One of them will be, sooner or later. And one is enough I suppose.
>>>>>
>>>>> DShield (https://dshield.org/api/#threatfeeds) and firehol
>>>>> (http://iplists.firehol.org/) seem to host copies of most of the lists
>>>>> as well.
>>>>>>
>>>>>> I do not really want to overthink this - we didn’t do this with clamav for example either. But it probably is a decision to be made by the user what they want to enable and we should not enable anything by default. So no data will be leaked as long as the user does not consent.
>>>>>>
>>>>>> -Michael
>>>>>>
>>>>>>>
>>>>>>> Thanks, and best regards,
>>>>>>> Peter Müller
>>>>
>>>
>>> <Screenshot_2019-12-16_IP_Address_Blacklists.png>
>>
>
> <Screenshot_2019-12-17_IP_Address_Blacklists.png>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 0/5] ipblacklist: IP Address Blacklists
2019-12-16 22:20 ` Michael Tremer
@ 2019-12-16 23:05 ` Tom Rymes
2019-12-18 12:10 ` Michael Tremer
0 siblings, 1 reply; 35+ messages in thread
From: Tom Rymes @ 2019-12-16 23:05 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 1065 bytes --]
On 12/16/2019 5:20 PM, Michael Tremer wrote:> Hi,
>
>> On 16 Dec 2019, at 20:06, Tim FitzGeorge <lists(a)tfitzgeorge.me.uk>
wrote:
>>
>> Hi,
>>
>> I've attached the current GUI screenshot.
>
> Thanks for that.
>
> I have a couple of suggestions/concerns about it:
[snip]
> c) I would suggest to remove the “safe” column because that is a very
hard summary of what the lists do. We should explain that on the wiki. I
guess this is too complicated to explain to our users in one sentence
and it needs at least a page of text. People who do not read that have
you just lost out.
[snip]
May I opine that the "Safe" information would be helpful to me in the
WUI. Perhaps we can be more explicit, or better explain, such as is
often done with RBLs in mail server settings, where lists are sometimes
described in terms of their likelihood to cause false-positives.
It's all well and good in the documentation, but a quick
"Safe|Moderate|Risky" listing in the WUI will prove handy, IMHO.
Just my $0.02 as more of a user than a developer,
Tom
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 0/5] ipblacklist: IP Address Blacklists
[not found] <c0c3b48a-f773-8002-a004-82ff150ea1bb@tfitzgeorge.me.uk>
@ 2019-12-16 22:20 ` Michael Tremer
2019-12-16 23:05 ` Tom Rymes
0 siblings, 1 reply; 35+ messages in thread
From: Michael Tremer @ 2019-12-16 22:20 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 20310 bytes --]
Hi,
> On 16 Dec 2019, at 20:06, Tim FitzGeorge <lists(a)tfitzgeorge.me.uk> wrote:
>
> Hi,
>
> I've attached the current GUI screenshot.
Thanks for that.
I have a couple of suggestions/concerns about it:
a) I am not sure what the value is about the top box that is called “Status”. It is basically a summary of the iptables output, but will it help the user? A blacklist with more or fewer entries is not better or worse, blocking more packets isn’t better than blocking fewer. It is about the quality of the blocks.
b) I would prefer to move the settings box to the top. If you like you can show the size of the blacklists there and when they have been updated, too.
c) I would suggest to remove the “safe” column because that is a very hard summary of what the lists do. We should explain that on the wiki. I guess this is too complicated to explain to our users in one sentence and it needs at least a page of text. People who do not read that have you just lost out.
d) I do not understand what the “update check rate” means. I suppose we should show the update interval of the lists in the table and just refresh them according to that.
e) Do we want to keep the automatic blacklists now? I do not think it will actually improve the security of IPFire.
>
> I'll update the code based on our discussions, and submit an updated set
> of patches - I imagine there will have to be at least one more iteration.
Let’s wait until we have come to decisions :)
-Michael
>
> Tim
>
> (No additional comments below)
>
> On 13/12/2019 23:11, Michael Tremer wrote:
>> Hi,
>>
>> Again my apologies for my late reply. Busy busy weeks.
>>
>>> On 8 Dec 2019, at 20:50, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>>
>>> Hello,
>>>
>>> It's my turn to apologise for being slow to respond - I've had a busy
>>> week, but I should have plenty of time over the next couple of weeks.
>>
>> No worries. Turns out we all do :)
>>
>>> I've made most of the comments inline, however I think Michael had a
>>> question (which I can't find now) about what happens if someone enables
>>> all the lists. One thing which would perhaps make this less likely is
>>> that the WUI tags the available lists with whether they're safe or not,
>>> with a footnote that safe means that the list only blocks malicious
>>> traffic. This won't guarantee that a user won't still try to enable all
>>> the lists, but it should make them realise that they should think first.
>>
>> We had a couple of features going slightly wrong or being “misunderstood” by some users. People still seem to panic when they see “local recursor”. To this day I do not know why.
>>
>> We cannot make everything idiot-proof. And when some user if of that category, they probably should shutdown their IPFire box, educate themselves and then come back again. So I do not want to limit people, but make things as easy as possible.
>>
>> If someone enables all the lists, good luck with passing packets :)
>>
>>> I have considered replacing this tag with a risk high/medium/low and
>>> maybe adding a category (invalid/application/scanner/C&C or something
>>> like that), but that may provide too much information and dissuade them
>>> from actually following the links to checkout what the list actually does.
>>
>> Can we have a screenshot of the GUI right now? I didn’t run the code, yet.
>>
>> We should document the lists like we do it with the rulesets of the IPS. People might ignore this, but that is on them.
>>
>>>
>>> Tim
>>>
>>>
>>> On 05/12/2019 22:25, Michael Tremer wrote:
>>>> Hello,
>>>>
>>>>> On 4 Dec 2019, at 17:05, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>>>>
>>>>> Hello Tim, hello Michael,
>>>>>
>>>>> please see my responses inline...
>>>>>>>>
>>>>>>>> We could periodically update the blacklists on our main mirror (and
>>>>>>>> wait for the network to sync it), make sure it is signed and write
>>>>>>>> a small downloader that fetches, validates and installs them.
>>>>>>>>
>>>>>>>> @All: Thoughts on this?
>>>>>>>
>>>>>>> I think there are a number of points here.
>>>>>>>
>>>>>>> Firstly, from the point of a third party using IPFire, is this really
>>>>>>> solving the privacy disclosure problem? There's no way round disclosing
>>>>>>> your public IP address to someone you're downloading from; all this does
>>>>>>> is change who that information is being disclosed to. For the user
>>>>>>> there's no way of knowing whether the source is more or less protective
>>>>>>> of the user's privacy than the blacklist provider. Indeed it won't be
>>>>>>> possible to know who the lists are being downloaded from until the
>>>>>>> download starts.
>>>>>>
>>>>>> There is a way: Tor. But that is a totally different story.
>>>>> Well, I see a third option on this: Use the mirror infrastructure we already
>>>>> have. Every IPFire installation discloses its public IP address to one of these
>>>>> servers sooner or later, so we do not disclose additional data if the blacklists
>>>>> were fetched from these.
>>>>>
>>>>> Needless to say, Tor (hidden services) would be better, but that is a different
>>>>> story indeed. :-)
>>>>>>
>>>>>> The point is rather that a forget list can be sent instead of the “real” one.
>>>>> I did not get this. Forget? Forged?? ???
>>>>
>>>> Yes, I meant to write forged, but auto-correct didn’t let me.
>>>>
>>>>>>> Secondly, latency; some of the lists are updated every 5 minutes. While
>>>>>>> I've limited the maximum check rate to hourly, will the updates
>>>>>>> propagate quickly enough. For reference on my main system the 24
>>>>>>> updates on the CIARMY list made 143 498 changes (additions or
>>>>>>> deletions). I've seem it do over 200 000.
>>>>> Yes, I observe that behaviour for CINS/CIArmy too. Unfortunately, they do not
>>>>> document a recommended update interval anywhere, so we can only guess.
>>>>>
>>>>> Personally, more static lists seem to be preferable for packet filtering. Highly
>>>>> dynamic ones such as CIArmy should be done via DNSBL queries or something similar
>>>>> - do we really want to have that list here?
>>>>
>>>> It is not really an option to implement a DNSBL into a packet filter, but I get your point.
>>>
>>> One of the 'selling points' for an IP address blacklist is that it can
>>> respond quickly to new threats - or rather new attackers. While a new
>>> IDS/IPS rule needs time to analyse the threat, generate a rule and check
>>> it, it's easy to add an address to a list. So, I think the CIArmy list
>>> is potentially useful for protecting home systems etc. with budget
>>> hardware, but I would be very careful about using it for a protecting a
>>> general access website.
>>
>> If they are very volatile, we should honour that and update them often, too.
>>
>> It probably is more about false-positives being removed very quickly instead of adding threats very quickly. The average IPFire user is probably not under threats like these to need to react very quickly.
>>
>> So I do not see much value in adding those lists and then updating once a day. Does it have to be every 5 min? No. I would suggest 15 which should be good enough for everyone.
>>
>> Other lists should of course not be updated every 15 minutes when not needed.
>>
>> Running every 15 minutes would allow us to retry downloading lists that are on an hourly schedule if the download failed.
>>
>>>>
>>>>>> How did you come up with the hour? Will it be retried more often if the download was not successful?
>>>>> One hour is the most common interval indeed, but adding some random time might
>>>>> be useful in order to reduce load on the servers providing a blacklist.
>>>>
>>>> Yes, definitely. Otherwise we will shoot down our mirrors.
>>>
>>> When I implemented that section of code, specifying the minimum check
>>> period in hours seemed to provide a convenient way of allowing a check
>>> period covering a wide range, with an hour as the fastest and a week as
>>> the slowest. I didn't looked at the CIArmy list until much later. Most
>>> of the lists don't change nearly as much, but the CIArmy list is
>>> described as one that deliberately responds quickly.
>>>
>>> From my production system, for yesterday:
>>>
>>> The following block lists were updated:
>>> BLOCKLIST_DE: 24 Time(s) - 9341 change(s)
>>> BOGON_FULL: 1 Time(s) - 10 change(s)
>>> CIARMY: 24 Time(s) - 159134 change(s)
>>> DSHIELD: 7 Time(s) - 18 change(s)
>>> EMERGING_FWRULE: 1 Time(s) - 50 change(s)
>>> FEODO_AGGRESIVE: 24 Time(s) - 13 change(s)
>>> SHODAN: 1 Time(s) - 0 change(s)
>>> SPAMHAUS_DROP: 1 Time(s) - 0 change(s)
>>> TOR_EXIT: 24 Time(s) - 162 change(s)
>>
>> Very interesting statistics.
>>
>>>
>>> and my test system:
>>>
>>> The following block lists were updated:
>>> ALIENVAULT: 19 Time(s) - 5331 change(s)
>>> EMERGING_COMPROMISED: 1 Time(s) - 26 change(s)
>>> TALOS_MALICIOUS: 1 Time(s) - 36 change(s)
>>>
>>> That covers most of the lists. From the WUI, since 1 Dec:
>>>
>>> Blacklist Entries pkts bytes Last updated
>>> in in
>>> AUTOBLACKLIST 0 731 51144 Sun Dec 8 18:40:02 2019
>>> BLOCKLIST_DE 28020 857 46735 Sun Dec 8 18:40:02 2019
>>> BOGON_FULL 214 5255 189K Sun Dec 8 16:50:04 2019
>>> CIARMY 15000 19774 976K Sun Dec 8 18:04:01 2019
>>> DSHIELD 20 7992 321K Sun Dec 8 16:45:13 2019
>>> EMERGING_FWRULE 1647 197 8383 Fri Dec 6 05:29:07 2019
>>> FEODO_AGGRESIVE 7169 0 0 Sun Dec 8 18:50:07 2019
>>> SHODAN 32 34 1530 Sun Dec 8 17:54:09 2019
>>> SPAMHAUS_DROP 823 0 0 Fri Dec 6 18:22:35 2019
>>> SPAMHAUS_EDROP 111 82 16433 Thu Dec 5 17:27:09 2019
>>> TOR_EXIT 1055 0 0 Sun Dec 8 18:31:02 2019
>>
>> This as well.
>>
>> Those are more packets than I would have expected.
>>
>>>
>>> (I've left out the pkts/bytes out fields which were all 0)
>>>
>>> Note that where possible I do a HEAD request first and then only
>>> download the list if the modification time has changed since the last
>>> check. For dynamically generated lists this isn't possible.
>>
>> You won’t need a HEAD request for it. You can include it in the GET request.
>>
>> Have a look at location downloader where I use that. The server will respond with 304 and not send any payload.
>>
>> https://git.ipfire.org/?p=location/libloc.git;a=blob;f=src/python/location-downloader.in;h=1b5932d5822e03737d32b2a27816815b2f7e74dd;hb=HEAD#l151
>>
>>>
>>> If the download isn't successful it just gives up and waits for the next
>>> attempt (apart from the usual retries in the library). I probably
>>> should to change that so that it only applies the per list minimum
>>> update period in this case (specified in the sources file) rather than
>>> the user specified value as well.
>>
>> I think it is not the worst if an update fails. It might just happen every once in a while.
>>
>> So I would suggest to just re-run the script more often and when the mtime of the file is older than the threshold, a download is attempted. You can use that timestamp for the GET request.
>>
>>> I already use a time offset on the downloads - when it's started from
>>> boot, backup restore or WUI enable, it checks to see if it's installed
>>> in the fcrontab, and if not adds itself at a randomly generated offset
>>> in the hour.
>>
>> That should potentially go to red.up, or if we can settle on 15 minutes, I would consider that often enough to quickly update all lists after a reboot.
>>
>>>
>>>>
>>>>>>> Third, bandwidth; while the downloads are fairly small (ALIENVAULT is
>>>>>>> the largest at a few MB), there are going to be a lot of them. How will
>>>>>>> this affect the willingness of people to mirror IPFire?
>>>>> I do not consider this being a problem as we do not generate that much traffic
>>>>> to them. Of course, that depends on the update interval again.
>>>>
>>>> That depends on your point of view.
>>>>
>>>> I do not have a problem with this at all in my data center, but there are plenty of people with a volume-based LTE plan or simply a 128 kBit/s connection. It will take a longer time to download the lists for them. We need to mind that.
>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Talking about the preference of packet filter and IPS, I prefer to
>>>>>>>> use the latter as well as it gains more insight in what kind of malicious
>>>>>>>> traffic tried to pass a firewall machine. On systems with low resources,
>>>>>>>> this might be problematic and removing load from the IPS can be preferred
>>>>>>>> (make this configurable?!), on others, people might want to have both
>>>>>>>> results.
>>>>>>>>
>>>>>>> You're only going to get one result for a packet whichever way round the
>>>>>>> IP blacklist and IPS are since whichever comes first will drop the
>>>>>>> packet before it reaches the second (well it would be possible to put
>>>>>>> the IP blacklist first and get it to log and mark packets which are then
>>>>>>> dropped after the IPS, but I think that's getting a little complicated.
>>>>>>> In addition I've seen the messages about the trouble marking was
>>>>>>> causing in the QoS).
>>>>>>>
>>>>>>> I think it's a 50/50 choice as to which is more valuable first; it's
>>>>>>> probably going to differ from packet to packet. For me the possibility
>>>>>>> of reducing the IPS load means I prefer putting the IP blacklist first
>>>>>>>
>>>>>>> It should be fairly easy to add the choice of where to put the IP
>>>>>>> blacklist. I think it'll have to be in the main firewall script, so
>>>>>>> it'll require a firewall restart, but it's not something that'll be
>>>>>>> changed often.
>>>>>>
>>>>>> I do not think that the user should choose this. If we cannot easily make a decision, how can our users do this? Not saying they are stupid here, we are just giving them something so that they do not have to put the thought and research into things themselves and make their jobs easier.
>>>>> Agreed.
>>>>>
>>>>>>
>>>>>> I think performance matters. And if the IPS comes first, the most likely case would be that we are seeing a SYN packet that is being scanned and after that being dropped by the blacklist. It was pointless to even scan the empty packet (TCP fast open aside). This is only different for other packets with payloads.
>>>>>>
>>>>>> We would protect the IPS from a SYN flooding attack here at least and from scanning more packets unnecessarily.
>>>>> So dropping packets from blacklisted IP addresses/networks before IPS is it, then.
>>>>>>
>>>>>> I do not even think it makes sense to swap the order in the outgoing direction.
>>>>> Me too.
>>>>>
>>>>>>
>>>>>> What IPFire is lacking is a statistical analysis for those logs. Collecting more and more data isn’t helpful from my point of view. Only if you are looking at a very specific thing.This is true, but I am not sure if it makes sense to spend too much work on this.
>>>>> Based on my personal experience, firewall hits observed on a single machine exposed
>>>>> to the internet are interesting, but the overall situation across multiple machines
>>>>> is even more interesting. Very quickly, you'll end on something like a centralised
>>>>> logging server and custom statistical analysis here...
>>>>
>>>> Probably a project for IPFire 4.0 :)
>>>>
>>> Or use one of the existing services, like the DSHIELD client
>>> https://dshield.org/howto.html (subject to privacy concerns again).
>>>>>>
>>>>>>>>
>>>>>>>> Personally, I consider Spamhaus DROP/EDROP can be safely enabled by default.
>>>>>>>> I would love to see the bogon ruleset here, too (think about 8chan successor
>>>>>>>> hosted at unallocated RIPE space in Saint Petersburg), but that will likely cause
>>>>>>>> interference if RED does not have a public IP address assigned.
>>>>>>>
>>>>>>> I can add a field to the options file that controls whether a list is
>>>>>>> enabled by default.
>>>>> Thank you. :-)
>>>>>>
>>>>>> To stress the point from above again: We would then share all public IP addresses of all IPFire systems in the world with Spamhaus and who is hosting their infrastructure. That can be considered a threat.
>>>>> This is my only objection against this patchset. Now, what can we do about it?
>>>>> One possibility is to apply the patchset now and implement a custom download
>>>>> source thing later on, or do that before releasing Core Update 139 (or which version
>>>>> the patchset will be to) after we agreed on something.
>>>>
>>>> I do not see this being merged for 139. But that is not important. We need to get it right first and then release it.
>>>>
>>>> As far as I know, nobody has tested this, yet.
>>>
>>> There are a number of people who have been running an earlier version
>>> which I shared on GitHub. There were a few early issues, but it seems
>>> to be OK now.
>>>
>>> https://forum.ipfire.org/viewtopic.php?f=27&t=21845
>>>
>>> This version wasn't integrated into IPFire, so (for example) it inserted
>>> itself into the INPUT IPTables chain rather than having it's chains
>>> created as part of the firewall start-up script.
>>>
>>>>
>>>> I have huge concerns about the automatic blacklist. @Peter: What is your opinion on this?
>>>>
>>>
>>> While I implemented it, I'm aware of its potential to cause problems,
>>> which is why it has to be separately enabled. It's not caused me any
>>> issues at the default settings (blocks at over 10 packets per hour until
>>> 1 hour has passed without seeing packets from the address), but I've not
>>> used it on a site with publicly announced services. If I was going to
>>> use it on a web site I would want to, at the very minimum, drop the
>>> block period drastically.
>>
>> I suppose this is entirely unusable on an IPFire box in a data center that hosts things. Let’s say our rack in Hanover.
>>
>> You will have hits from some broken IP stacks and people might just end up on this without doing anything wrong.
>>
>> You can denial-of-service easily as well and I suppose without aggregation of data from many many systems, it does not make sense to instantly block addresses. And even then you probably would block an entire subnet.
>>
>> So I am not sure how I should feel about it. I do not think that it adds anything because the packets would have been blocked anyways.
>>
>>>
>>> On the other hand, it's good at responding quickly. Usually I see only
>>> 1-2% of blocks from the automatic list:
>>>
>>> Reason Count % First Last
>>> CIARMY 2416 45 Dec 6 00:00 Dec 6 23:59
>>> DSHIELD 1353 25 Dec 6 00:00 Dec 6 23:59
>>> INPUT 1294 24 Dec 6 00:00 Dec 6 23:59
>>> AUTOBLACKLIST 122 2 Dec 6 00:20 Dec 6 16:28
>>> BLOCKLIST_DE 89 2 Dec 6 00:20 Dec 6 23:46
>>>
>>> and sometimes none at all, but one one occasion it blocked over 8000
>>> packets. Again I'm aware this is for a home system, which is rather
>>> different than from a Web server.
>>>
>>>>> If we do, I will have a look at the licensing stuff (DShield and Spamhaus do not
>>>>> seem to be problematic, as they are hosted on 3rd party servers, too).
>>>>
>>>> One of them will be, sooner or later. And one is enough I suppose.
>>>
>>> DShield (https://dshield.org/api/#threatfeeds) and firehol
>>> (http://iplists.firehol.org/) seem to host copies of most of the lists
>>> as well.
>>>>
>>>> I do not really want to overthink this - we didn’t do this with clamav for example either. But it probably is a decision to be made by the user what they want to enable and we should not enable anything by default. So no data will be leaked as long as the user does not consent.
>>>>
>>>> -Michael
>>>>
>>>>>
>>>>> Thanks, and best regards,
>>>>> Peter Müller
>>
>
> <Screenshot_2019-12-16_IP_Address_Blacklists.png>
^ permalink raw reply [flat|nested] 35+ messages in thread
end of thread, other threads:[~2020-01-30 21:26 UTC | newest]
Thread overview: 35+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-11-25 20:13 [PATCH 0/5] ipblacklist: IP Address Blacklists Tim FitzGeorge
2019-11-25 20:13 ` [PATCH 1/5] ipblacklist: Main script Tim FitzGeorge
2019-11-25 20:13 ` [PATCH 2/5] ipblacklist: WUI and language file Tim FitzGeorge
2019-11-25 20:13 ` [PATCH 3/5] ipblacklist: Ancillary files Tim FitzGeorge
2019-11-25 20:13 ` [PATCH 4/5] ipblacklist: Modifications to system Tim FitzGeorge
2019-11-25 20:13 ` [PATCH 5/5] ipblacklist: Build infrastructure Tim FitzGeorge
2019-11-25 21:09 ` [PATCH 0/5] ipblacklist: IP Address Blacklists Peter Müller
2019-11-27 21:34 ` Tim FitzGeorge
2019-11-28 12:03 ` Michael Tremer
2019-11-28 21:39 ` Peter Müller
2019-11-29 23:25 ` Tim FitzGeorge
2019-12-02 11:17 ` Michael Tremer
2019-12-04 17:05 ` Peter Müller
2019-12-05 22:25 ` Michael Tremer
2019-12-08 20:50 ` Tim FitzGeorge
2019-12-13 23:11 ` Michael Tremer
2019-12-02 11:06 ` Michael Tremer
[not found] <c0c3b48a-f773-8002-a004-82ff150ea1bb@tfitzgeorge.me.uk>
2019-12-16 22:20 ` Michael Tremer
2019-12-16 23:05 ` Tom Rymes
2019-12-18 12:10 ` Michael Tremer
2019-12-28 21:17 ` Tim FitzGeorge
2020-01-06 11:21 ` Michael Tremer
2020-01-22 20:35 ` Tim FitzGeorge
2020-01-23 10:53 ` Michael Tremer
[not found] <6583305a-cd0d-94a5-aa8e-5456622de824@tfitzgeorge.me.uk>
2019-12-18 12:07 ` Michael Tremer
2019-12-21 18:34 ` Tim FitzGeorge
2019-12-24 10:29 ` Michael Tremer
2019-12-28 20:59 ` Tim FitzGeorge
[not found] <aeb93668-a4b6-5735-1f68-fd53cafa2210@tfitzgeorge.me.uk>
2020-01-06 11:27 ` Michael Tremer
2020-01-24 19:40 ` Tim FitzGeorge
2020-01-28 17:14 ` Michael Tremer
2020-01-29 20:40 ` Tim FitzGeorge
2020-01-30 12:54 ` Michael Tremer
2020-01-30 20:24 ` Tim FitzGeorge
2020-01-30 21:26 ` Michael Tremer
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox