Implements downloading of IP address blacklists and implementing them as IPSets. A separate IPSet is used for each blacklist; this simplifies handling of overlaps between different lists. Traffic to or from the red0/ppp0 interface is checked against the IPSets. The check is placed before the IPS check as the IPSet check is much lighter on CPU use which means that overall CPU use is reduced.
The available lists are defined in a separate file. A WUI page allows the desired lists to be enabled. A minimum update check interval is defined for each blacklist in the definition file.
Changes since Version 1:
- Changed Dshield download URL to preferred address. - Removed Abuse.ch blacklist (discontinued). - Removed Talos Malicious blacklist (not for production use). - Added Feodo recommended blacklist. - Added blocklist.de all blacklist. - Updated ignored messages in logwatch. - Modified sources file 'rate' to allow unit to be specified. - Updated sources file 'disable' to allow list to be specified. - Removed autoblacklist. - Added WUI log pages. - Removed status from settings WUI page.
Tim FitzGeorge (8): ipblacklist: Main script ipblacklist: WUI Settings page ipblacklist: WUI Log page ipblacklist: WUI Log details page ipblacklist: WUI menus, language file etc ipblacklist: Ancillary files ipblacklist: Modifications to system ipblacklist: Build infrastructure
config/backup/backup.pl | 1 + config/backup/include | 2 + config/ipblacklist/sources | 138 ++ config/logwatch/ipblacklist | 105 ++ config/logwatch/ipblacklist.conf | 34 + config/menu/50-firewall.menu | 5 + config/menu/70-log.menu | 5 + config/rootfiles/common/aarch64/stage2 | 1 + config/rootfiles/common/configroot | 2 + config/rootfiles/common/ipblacklist-sources | 1 + config/rootfiles/common/logwatch | 2 + config/rootfiles/common/misc-progs | 2 + config/rootfiles/common/stage2 | 1 + config/rootfiles/common/web-user-interface | 3 + config/rootfiles/common/x86_64/stage2 | 1 + html/cgi-bin/ipblacklist.cgi | 463 +++++++ html/cgi-bin/logs.cgi/ipblacklists.dat | 363 +++++ html/cgi-bin/logs.cgi/log.dat | 2 + html/cgi-bin/logs.cgi/showrequestfromblacklist.dat | 415 ++++++ langs/en/cgi-bin/en.pl | 27 +- lfs/configroot | 4 +- lfs/ipblacklist-sources | 53 + lfs/logwatch | 2 + make.sh | 1 + src/initscripts/system/firewall | 12 + src/misc-progs/Makefile | 2 +- src/misc-progs/getipsetstat.c | 25 + src/misc-progs/ipblacklistctrl.c | 48 + src/scripts/ipblacklist | 1382 ++++++++++++++++++++ 29 files changed, 3098 insertions(+), 4 deletions(-) create mode 100644 config/ipblacklist/sources create mode 100644 config/logwatch/ipblacklist create mode 100644 config/logwatch/ipblacklist.conf create mode 100644 config/rootfiles/common/ipblacklist-sources create mode 100644 html/cgi-bin/ipblacklist.cgi create mode 100755 html/cgi-bin/logs.cgi/ipblacklists.dat create mode 100755 html/cgi-bin/logs.cgi/showrequestfromblacklist.dat create mode 100644 lfs/ipblacklist-sources create mode 100644 src/misc-progs/getipsetstat.c create mode 100644 src/misc-progs/ipblacklistctrl.c create mode 100755 src/scripts/ipblacklist
Responsible for downloading blacklists and creating/modifying IPSets Does all work involving creating, deleting and changing IPTables and IPSets.
Signed-off-by: Tim FitzGeorge ipfr@tfitzgeorge.me.uk --- src/scripts/ipblacklist | 1382 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 1382 insertions(+) create mode 100755 src/scripts/ipblacklist
diff --git a/src/scripts/ipblacklist b/src/scripts/ipblacklist new file mode 100755 index 000000000..6f950214c --- /dev/null +++ b/src/scripts/ipblacklist @@ -0,0 +1,1382 @@ +#! /usr/bin/perl + +############################################################################ +# # +# IP Address blocklists for IPFire # +# # +# This is free software; you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation; either version 3 of the License, or # +# (at your option) any later version. # +# # +# This is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with IPFire; if not, write to the Free Software # +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # +# # +# Copyright (C) 2018 - 2020 The IPFire team # +# # +############################################################################ +# # +# This script uses a file containing blacklist details in # +# /var/ipfire/ipblacklist/sources as well as # +# /var/ipfire/ipblacklistsettings containing an enable/disable flag for # +# each source. # +# # +# Two IPTables chains are used: BLACKLISTIN and BLACKLISTOUT are inserted # +# inserted into the main INPUT, OUTPUT and FORWARD chains; they capture # +# packets other than for the ICMP protocol. # +# # +# For each blacklist that is loaded, a chain is created to optionally log # +# and then to drop matching packets. An IPSet is created containing the # +# addresses or networks blocked by the blacklist, and then rules are added # +# to the BLACKLISTIN and BLACKLISTOUT chains to jump to this chain if a # +# packet list matches the set. # +# # +# When checking for updates, the modification time is used for each source # +# and if necessary the list is downloaded. The downloaded list is # +# compared to the existing IPSet contents and entries created or deleted # +# as necessary. # +# # +############################################################################ + +use strict; +#use warnings; + +use Carp; +use Sys::Syslog qw(:standard :macros); +use HTTP::Request; +use LWP::UserAgent; + +require "/var/ipfire/general-functions.pl"; + +############################################################################ +# Configuration variables +# +# These variables give the locations of various files used by this script +############################################################################ + +my $settingsdir = "/var/ipfire/ipblacklist"; +my $savedir = "/var/lib/ipblacklist"; +my $tmpdir = "/var/tmp"; + +my $settings = "$settingsdir/settings"; +my $sources = "$settingsdir/sources"; +my $checked = "$settingsdir/checked"; +my $modified = "$settingsdir/modified"; +my $iptables_list = "/var/tmp/iptables.txt"; +my $getipstat = "/usr/local/bin/getipstat"; +my $iptables = "/sbin/iptables"; +my $ipset = "/usr/sbin/ipset"; +my $fcrontab = "/usr/bin/fcrontab"; +my $lockfile = "/var/run/ipblacklist.pid"; +my $proxy_settings = "${General::swroot}/proxy/settings"; +my $red_setting = "/var/ipfire/red/iface"; +my $detailed_log = "$tmpdir/ipblacklist_log.txt"; +my $active = "/var/ipfire/red/active"; + +# Other configuration items + +my $margin = 30; # Scheduling allowance for run time etc in seconds +my $count = 30; # Maximum time to wait for another instance (300s) +my $max_dl_fails = 3; # Ignore check rate limit for this number of failures +my $max_size_fraction = 0.7; # Maximum fill fraction of IPSet before enlarging. +my $min_ipset_entries = 1024; # The minimum size of an IPSet. +my $max_dl_bytes = 10_485_760; # Maximum number of bytes to download. +my %parsers = ( 'ip-or-net-list' => &parse_ip_or_net_list, + 'dshield' => &parse_dshield ); + +############################################################################ +# Default settings +# Should be overwritten by reading settings files +############################################################################ + +my %sources = ( ); + +my %settings = ( 'DEBUG' => 0, + 'LOGGING' => 'on', + 'ENABLE' => 'off' ); + +my %proxy_settings = ( 'UPSTREAM_PROXY' => '' ); # No Proxy in use + +############################################################################ +# Function prototypes +############################################################################ + +sub abort( $ ); +sub create_list( $ ); +sub create_ipset( $$$ ); +sub debug( $$ ); +sub delete_list( $ ); +sub disable_logging(); +sub disable_updates(); +sub do_delete(); +sub do_start(); +sub do_stop(); +sub do_update(); +sub download_list( $$$ ); +sub download_check_header_time( $$$ ); +sub download_wget( $$$ ); +sub enable_logging(); +sub enable_updates(); +sub get_ipsets(); +sub get_rate_seconds( $ ); +sub iptables( $ ); +sub ipset( $ ); +sub stop_ipset(); +sub is_connected(); +sub log_message( $$ ); +sub parse_dshield( $ ); +sub parse_ip_or_net_list( $ ); +sub read_ipset( $$$$ ); +sub update_list( $$$ ); + +############################################################################ +# Variables +############################################################################ + +my %chains; # The Blacklist IPSets already loaded +my %old_blacklist; # Already blocked IP Addresses and/or networks + # downloaded for current blacklist +my $update_status = 0; # Set to 1 to update status file +my $ipset_running = 0; # Set to 1 if IPSet process is running +my %status; # Status information +my %checked; # Time blacklists last changed +my %modified; # Time blacklists last modified +my $red_iface; # The name of the red interface + +############################################################################ +# Synchronise runs +############################################################################ + +# This script can be triggered either by cron or the WUI. If another +# instance is running, wait for it to finish or timeout. + +while (-r $lockfile and $count > 0) +{ + open LOCKFILE, '<', $lockfile or (abort "Can't open lockfile", last); + my $pid = <LOCKFILE>; + close LOCKFILE; + + chomp $pid; + + last unless (-e "/proc/$pid"); + + sleep 10; + $count--; +} + +# Create pid file before starting main processing + +open LOCKFILE, '>', '/var/run/ipblacklist.pid' or abort "Can't open PID file: $!"; +print LOCKFILE "$$\n"; +close LOCKFILE; + +############################################################################ +# Set up for update +############################################################################ + +mkdir $settingsdir unless (-d $settingsdir); + +# Connect to the system log + +openlog( "ipblacklist", "nofatal", LOG_USER); +log_message LOG_INFO, "Starting IP Blacklist processing"; + +# Read settings + +General::readhash( $settings, %settings ) if (-e $settings); +General::readhash( $checked, %checked ) if (-e $checked); +General::readhash( $modified, %modified ) if (-e $modified); +General::readhash( $proxy_settings, %proxy_settings ) if (-e $proxy_settings); + +if (-r $sources) +{ + debug 1, "Reading sources file"; + + eval qx|/bin/cat $sources|; +} + +if (-r $red_setting) +{ + open REDIF, '<', $red_setting or (abort "Can't open red interface file", exit); + $red_iface = <REDIF>; + chomp $red_iface; + close REDIF; +} + +if (@ARGV) +{ + foreach my $cmd (@ARGV) + { + if ('update' =~ m/^$cmd/i) + { + # Called hourly when enabled and on setting changes. + # Update the blacklists. + + if ($settings{'ENABLE'} eq 'on') + { + do_update(); + } + } + elsif ('start' =~ m/^$cmd/i) + { + # Called during system startup. + # Restore saved blacklists. + # Don't do an update since can take too long. + + do_start() if ($settings{'ENABLE'} eq 'on'); + } + elsif ('stop' =~ m/^$cmd/i) + { + # Called when shutting down. + # Delete IPSets and IPTables chains + + do_stop(); + } + elsif ('restore' =~ m/^$cmd/i) + { + # Called after restoring backup. + # Delete IPSets and IPTables chains, then re-create them with the new + # (restored) settings and make sure updates are enabled. + + do_stop(); + + if ($settings{'ENABLE'} eq 'on') + { + do_start(); + enable_updates(); + } + else + { + disable_updates(); + } + } + elsif ('log-on' =~ m/^$cmd/i) + { + # Called from WUI. + # Create entries in IPTables chains to log dropped packets. + + if ($settings{'ENABLE'} eq 'on') + { + enable_logging(); + } + } + elsif ('log-off' =~ m/^$cmd/i) + { + # Called from WUI + # Delete entries in IPTables chains to log dropped packets. + + disable_logging(); + } + elsif ('enable' =~ m/^$cmd/i) + { + # Called from WUI to enable blacklists + # Do an update and then enable automatic updates + + if ($settings{'ENABLE'} eq 'on') + { + do_update(); + enable_updates(); + } + } + elsif ('disable' =~ m/^$cmd/i) + { + # Called from WUI to disable blacklists. + # Disable updates, delete IPSets, IPTables chains and save files. + + disable_updates(); + do_delete(); + } + else + { + print "Usage: $0 [update|start|stop|restart|log-on|log-off|enable|disable]\n"; + } + } +} +elsif ($settings{'ENABLE'} eq 'on') # Default action if none specified +{ + do_update(); +} + +stop_ipset(); + +if ($update_status) +{ + debug 1, "Writing updated status file"; + + General::writehash( $checked, %checked ); + General::writehash( $modified, %modified ); +} + +# Remove the pid file + +unlink $lockfile; + +log_message LOG_INFO, "Finished IP Blacklist processing"; +closelog(); + + +#------------------------------------------------------------------------------ +# sub do_stop +# +# Deletes all the IPTables chains and the IPSets +#------------------------------------------------------------------------------ + +sub do_stop() +{ + get_ipsets(); + + log_message LOG_NOTICE, "Stopping IP Blacklists"; + + foreach my $list ( sort keys %sources ) + { + delete_list( $list ) if (exists $chains{$list}); + } +} + + +#------------------------------------------------------------------------------ +# sub do_start +# +# Recreates the IPTables chains and the IPSets from the saved values +#------------------------------------------------------------------------------ + +sub do_start() +{ + log_message LOG_NOTICE, "Starting IP Blacklists"; + + foreach my $list ( sort keys %sources ) + { + delete_list( $list ) if (exists $chains{$list}); # Make sure OK to start + + if ((-e "$savedir/$list.conf") and ($red_iface)) + { + log_message LOG_INFO, "Restoring blacklist $list"; + system( "$ipset restore -f $savedir/$list.conf" ); # Can't use the ipset + # function to do this + + create_list( $list ); + } + } +} + + +#------------------------------------------------------------------------------ +# sub do_delete +# +# Deletes the IPTables chains, the IPSets and the saved values. +#------------------------------------------------------------------------------ + +sub do_delete() +{ + # Get the list of current ipsets + + get_ipsets(); + + log_message LOG_NOTICE, "Deleting IP Blacklists"; + + foreach my $source ( sort keys %sources ) + { + if (exists $chains{$source}) + { + delete_list( $source ); + } + + if (-e "$savedir/$source.conf") + { + unlink "$savedir/$source.conf"; + } + } + + %modified = (); + $update_status = 1; +} + + +#------------------------------------------------------------------------------ +# sub do_update +# +# Updates all the blacklists. +# Creates or deletes the blacklist firewall rules as necessary and checks for +# updates to the blacklists. Each blacklist has its own minimum elapsed time +# between updates, which is specified in the sources file, so the time of each +# check is stored. +#------------------------------------------------------------------------------ + +sub do_update() +{ + return unless ($red_iface); + + # Get the list of current ipsets + + get_ipsets(); + + # Check sources + + debug 1, "Checking blacklist sources"; + + LIST: + foreach my $list ( sort keys %sources ) + { + my @new_blacklist = (); + my $name = $sources{$list}{'name'}; + my $last_checked = $checked{$list} || 0; + my $failures = $checked{"${list}_failures"} || 0; + my $enabled = 0; + + if (exists $settings{$list}) + { + $enabled = $settings{$list} eq 'on'; + } + + if ($enabled and is_connected()) + { + debug 1, "Checking blacklist source: $name"; + + # Calculate the per list rate + + my $rate = get_rate_seconds( $sources{$list}{'rate'} ); + + # Has enough time passed since the last time we checked the list? + # Ignore the limit if the last download failed + + if (($last_checked + $rate) < (time() + $margin) or + ($failures > 0 and $failures < $max_dl_fails)) + { + my $type = 'hash:ip'; + + download_list( $list, @new_blacklist, $type ); + + next LIST unless (@new_blacklist); + + if (not exists $chains{$list}) + { + # Doesn't currently exist: Create it. + + create_ipset( $list, $type, scalar @new_blacklist ); + create_list( $list ); + } + + update_list( $list, @new_blacklist, $type ); + } + } + elsif (exists $chains{$list}) + { + # Exists, but not enabled: Delete it. + + delete_list( $list ); + + # Delete the save file + # Don't delete the checked time from the status, in case the list is + # re-enabled quickly - don't want to exceed maximum allowed download + # rate. + + unlink "$savedir/$list.conf" if (-e "$savedir/$list.conf"); + + delete $modified{$list} if (exists $modified{$list}); + delete $checked{"${list}_failures"} if (exists $checked{"${list}_failures"}); + $update_status = 1; + } + } + + # Check for any lists that don't exist any more + + foreach my $list (keys %modified) + { + next if (exists $sources{$list}); + + delete_list( $list ); + + # Delete the save file + + unlink "$savedir/$list.conf" if (-e "$savedir/$list.conf"); + + # Delete from the status + + delete $modified{$list} if (exists $modified{$list}); + $update_status = 1; + } + + foreach my $list (keys %checked) + { + next if ($list =~ m/_failures/); + next if (exists $sources{$list}); + + delete $checked{$list}; + delete $checked{"${list}_failures"}; + delete $settings{$list} if (exists $settings{$list}); + $update_status = 1; + } +} + + +#------------------------------------------------------------------------------ +# sub get_rate_seconds( text ) +# +# Converts a check rate into seconds. A sanity check is made on the coverted +# value. +# +# Parameters: +# text The value to convert in the form nnnu, where nnn is a number and u +# is either m (minutes), h (hours) or d (days). Hours is assumed if +# not specified and everything after the first letter is ignored. +#------------------------------------------------------------------------------ + +sub get_rate_seconds( $ ) +{ + my ($text) = @_; + + my ($value, $unit) = (uc $text) =~ m/(\d+)([DHM]?)/; + + if ($unit eq 'D') # Days + { + $value *= 60 * 60 * 24; + } + elsif ($unit eq 'M') # Minutes + { + $value *= 60; + } + else # Everything else - assume hours + { + $value *= 60 * 60; + } + + # Sanity check - limit to range 5 min .. 1 week + + # d h m s + $value = 5 * 60 if ($value < 5 * 60); + $value = 7 * 24 * 60 * 60 if ($value > 7 * 24 * 60 * 60); + + return $value; +} + + +#------------------------------------------------------------------------------ +# sub is_connected() +# +# Checks that the system is connected to the internet. +# +# This looks for a file created by IPFire when connected to the internet +#------------------------------------------------------------------------------ + +sub is_connected() +{ + return (-e $active); +} + + +#------------------------------------------------------------------------------ +# sub create_list( list ) +# +# Creates a new IPTables chain for a blacklist source. +# The set must be created before calling this function. Two rules are added to +# the chain: +# (optional) 1 Log the packet +# 2 Drop the packet +# +# The log rule is only added when logging is enabled by the WUI. +# +# Rules are then added to the BLACKLISTIN and BLACKLISTOUT chains that check +# the packet's IP address against the IPSet and then jump to the newly created +# chain. +# +# Parameters: +# list The name of the blacklist +#------------------------------------------------------------------------------ + +sub create_list( $ ) +{ + my ($list) = @_; + + log_message LOG_INFO, "Create IPTables chains for blacklist $list"; + + # Create new chain in filter table + + iptables( "-N ${list}_DROP" ) == 0 or + ( abort "Could not create IPTables chain ${list}_DROP", return ); + + # Add the logging and drop rules + + if ($settings{'LOGGING'} eq 'on') + { + iptables( "-A ${list}_DROP -j LOG -m limit --limit 10/second --log-prefix 'BLKLST_$list'" ) == 0 or + ( abort "Could not create IPTables chain $list LOG rule", return ); + } + + iptables( "-A ${list}_DROP -j DROP" ) == 0 or + ( abort "Could not create IPTables chain $list drop rule", return ); + + # Add the rules to check against the set + + iptables( "-A BLACKLISTIN -p ALL -i $red_iface -m set --match-set $list src -j ${list}_DROP" ); + iptables( "-A BLACKLISTOUT -p ALL -o $red_iface -m set --match-set $list dst -j ${list}_DROP" ); +} + + +#------------------------------------------------------------------------------ +# sub delete_list( $list ) +# +# Deletes an IPTables chain when a blacklist source is disabled. Also flushes +# and destroys the IPSet. +# +# Parameters: +# list The name of the blacklist +#------------------------------------------------------------------------------ + +sub delete_list( $ ) +{ + my ($list) = @_; + + log_message LOG_INFO, "Delete IPTables chains for blacklist $list"; + + # Remove the blacklist chains from the main INPUT and OUTPUT chains + + iptables( "-D BLACKLISTIN -p ALL -i $red_iface -m set --match-set $list src -j ${list}_DROP" ); + iptables( "-D BLACKLISTOUT -p ALL -o $red_iface -m set --match-set $list dst -j ${list}_DROP" ); + + # Flush and delete the chain + + iptables( "-F ${list}_DROP" ); + iptables( "-X ${list}_DROP" ); + + # Flush and delete the set + + ipset( "flush $list" ); + ipset( "destroy $list" ); +} + + +#------------------------------------------------------------------------------ +# sub download_list( list, ref_list, ref_type ) +# +# Downloads the IP Addresses for a blacklist. +# +# Once downloaded the list is parsed to get the IP addresses and/or networks. +# +# Parameters: +# list The name of the blacklist +# ref_list A reference to an array to store the downloaded blacklist +# ref_type A reference to store the type of the blacklist +#------------------------------------------------------------------------------ + +sub download_list( $$$ ) +{ + my ($list, $new_blacklist, $type) = @_; + + $checked{$list} = time(); # Record that the list has been checked + $update_status = 1; + + # Check the parser for the blacklist + + if (not exists $parsers{ $sources{$list}{'parser'} }) + { + log_message LOG_ERR, "Can't find parser $sources{$list}{'parser'} for $list blacklist"; + return; + } + + # Add alternative download mechanisms here + + download_check_header_time( $list, $new_blacklist, $type ); +} + + +#------------------------------------------------------------------------------ +# sub download_check_header_time( list, ref_list, ref_type ) +# +# Updates the IP Addresses for a blacklist. The If-Modified-Since header is +# specified in the request so that only updated lists are downloaded (providing +# that the server supports this functionality). +# +# Once downloaded the list is parsed to get the IP addresses and/or networks. +# +# Parameters: +# list The name of the blacklist +# ref_list A reference to an array to store the downloaded blacklist +# ref_type A reference to store the type of the blacklist +# +# Returns: +# The list type: 'hash:ip' or 'hash:net' +#------------------------------------------------------------------------------ + +sub download_check_header_time( $$$ ) +{ + my ($list, $new_blacklist, $type) = @_; + my $found_ip = 0; + my $found_net = 0; + + # Get the parser for the blacklist + + my $parser = $parsers{ $sources{$list}{'parser'} }; + + debug 1, "Checking for blacklist $list updates with LWP"; + + # Create a user agent for downloading the blacklist + # Limit the download size for safety + + my $ua = LWP::UserAgent->new( max_size => $max_dl_bytes ); + + # Get the Proxy settings + + if ($proxy_settings{'UPSTREAM_PROXY'}) + { + if ($proxy_settings{'UPSTREAM_USER'}) + { + $ua->proxy( [["http", "https"] => "http://$proxy_settings%7B%27UPSTREAM_USER%27%7D:$proxy_settings%7B%27UPSTREA..."] ); + } + else + { + $ua->proxy( [["http", "https"] => "http://$proxy_settings%7B%27UPSTREAM_PROXY%27%7D/"] ); + } + } + + # Get the last modified time + + my $modified = gmtime( $modified{$list} || 0 ); + + # Download the blacklist + + my $response = $ua->get( $sources{$list}{'url'}, 'If-Modified-Since' => $modified ); + + if (not $response->is_success) + { + if ($response->code == 304) + { + # Not an error - list has not been modified + debug 1, "Blacklist $list not modified"; + + return; + } + + log_message LOG_WARNING, "Failed to download $list blacklist $sources{$list}{'url'}: ". $response->status_line; + $checked{"${list}_failures"}++; + + return; + } + + $modified{$list} = $response->last_modified; + $checked{"${list}_failures"} = 0; + + # Parse the downloaded list, checking if it's a list of addresses or nets + + foreach my $line (split /[\r\n]+/, $response->content) + { + chomp $line; + + my $address = &$parser( $line ); + + next unless ($address and $address =~ m/\d+.\d+.\d+.\d+/); + + if ($address =~ m|/32|) + { + $address =~ s|/32||; + $found_ip = 1; + } + elsif ($address =~ m|/\d+|) + { + $found_net = 1; + } + else + { + $found_ip = 1; + } + + push @{ $new_blacklist }, $address; + } + + if ($found_net and $found_ip) + { + # Convert mixed addresses and networks to all networks + + foreach my $address (@{ $new_blacklist }) + { + $address .= '/32' unless ($address =~ m|/\d+|); + } + + $found_ip = 0; + } + + $$type = $found_net ? 'hash:net' : 'hash:ip'; +} + + +#------------------------------------------------------------------------------ +# sub read_ipset( list, old, type, maxelem ) +# +# Reads the existing contents and type of the set. +# +# Parameters: +# list The name of the blacklist +# old Reference to array to contain blacklist +# type Reference to type +# maxelem Reference to maximum number of elements +#------------------------------------------------------------------------------ + +sub read_ipset( $$$$ ) +{ + my ($list, $old, $type, $maxelem) = @_; + my $found_net = 0; + my $found_ip = 0; + + debug 2, "Reading existing ipset for blacklist $list"; + + foreach my $line (qx/$ipset list $list/) + { + if ($line =~ m|Header:.*maxelem (\d+)|) + { + $$maxelem = $1; + next; + } + + next unless ($line =~ m|(\d+.\d+.\d+.\d+(?:/\d+)?)|); + + my $address = $1; + + if ($address =~ m|/32|) + { + $found_ip = 1; + $address =~ s|/32$||; + } + elsif ($address =~ m|/\d+$|) + { + $found_net = 1; + } + else + { + $found_ip = 1; + } + + $$old{$address} = 1; + } + + if ($found_ip and $found_net) + { + # Convert mixed addresses and networks to all networks + + my @ads_list = keys %{ $old }; + + foreach my $address (@ads_list) + { + unless ($address =~ m|/\d+|) + { + delete $$old{$address}; + $$old{"$address/32"} = 1; + } + } + + $found_ip = 0; + } + + $$type = $found_net ? 'hash:net' : 'hash:ip'; +} + + +#------------------------------------------------------------------------------ +# sub update_list( list, new, new_type ) +# +# Updates the IP Addresses for a blacklist +# +# The new list is compared to the existing list and new entries added or old +# entries deleted as necessary. If the list type ('hash:ip' or 'hash:net') has +# changed then the IPSet is deleted and re-created with the new type. +# +# Parameters: +# list The name of the blacklist +# new Reference to array of new blacklist entries +# new_type The type of the updated list (hash:ip or hash:net) +#------------------------------------------------------------------------------ + +sub update_list( $$$ ) +{ + my ($list, $new, $new_type) = @_; + my %old; + my $old_type; + my $changes = 0; + my $maxelem = 0; + + debug 1, "Checking for $list blacklist update from $sources{$list}{'url'}"; + + if (exists $chains{$list} ) + { + my $recreate_ipset = 0; + + read_ipset( $list, %old, $old_type, $maxelem ); + + # Check the IPSet type hasn't changed + + if ($new_type ne $old_type) + { + log_message LOG_NOTICE, "Blacklist $list changed type from $old_type to $new_type"; + $recreate_ipset = 1; + } + + if ($max_size_fraction * $maxelem < scalar @{ $new } ) + { + log_message LOG_NOTICE, "Blacklist $list changed size from $maxelem"; + $recreate_ipset = 1; + } + + if ($recreate_ipset) + { + # Change the IPSet type and/or size. This requires removing references + # to it first. We could delete and then create the chain, but doing it + # like this keeps the statistics. + + # Remove the IPSet from the IPTables chains + + iptables( "-D 'BLACKLISTIN' -p ALL -i $red_iface -m set --match-set $list src -j ${list}_DROP" ) == 0 or + log_message LOG_ERR, "Could not remove ${list} from BLACKLISTIN chain"; + + iptables( "-D 'BLACKLISTOUT' -p ALL -o $red_iface -m set --match-set $list dst -j ${list}_DROP" ) == 0 or + log_message LOG_ERR, "Could not remove ${list} from BLACKLISTOUT chain"; + + # Flush and delete the old set + + ipset( "flush $list" ); + ipset( "destroy $list" ); + + %old = (); # Since we've deleted the old set it can't have any entries. + + # Create the new ipset + + create_ipset( $list, $new_type, scalar @{ $new } ); + + # Add the rules to check against the set + + iptables( "-A 'BLACKLISTIN' -p ALL -i $red_iface -m set --match-set $list src -j ${list}_DROP" ) == 0 or + log_message LOG_ERR, "Could not add IPSet $list to BLACKLISTIN chain"; + + iptables( "-A 'BLACKLISTOUT' -p ALL -o $red_iface -m set --match-set $list dst -j ${list}_DROP" ) == 0 or + log_message LOG_ERR, "Could not add IPSet $list to BLACKLISTOUT chain"; + } + } + + # Process the blacklist + + foreach my $address ( @{ $new } ) + { + # We've got an address. Add to IPSet if it's new + + if (exists $old{$address}) + { + delete $old{$address}; # Not new - don't delete from chain later + } + else + { + ipset( "add $list $address -exist" ); # New - add it + + $changes++; + } + + debug 3, "Add net $address to blacklist $list"; + } + + # Delete old entries that aren't needed any more + + debug 2, "Removing deleted rules from IPTables chain for blacklist $list"; + + foreach my $address ( keys %old ) + { + ipset( "del $list $address" ); + + $changes++; + + debug 3, "Delete old net $address from blacklist $list"; + } + + log_message LOG_INFO, "Updated $list blacklist with $changes changes"; + + # Save the blacklist for the next reboot + + mkdir "$savedir" unless (-d "$savedir" ); + + ipset( "save $list -file $savedir/$list.conf" ) if ($changes > 0); + + stop_ipset(); +} + + +#------------------------------------------------------------------------------ +# sub get_ipsets( ) +# +# Gets a list of the current IPSets +#------------------------------------------------------------------------------ + +sub get_ipsets( ) +{ + debug 1, "Reading list of existing ipsets"; + + my @sets = qx($ipset -n list); + + # Parse the tables + + foreach my $line (@sets) + { + chomp $line; + + next unless ($line); + + $chains{$line} = 1; + } +} + + +#------------------------------------------------------------------------------ +# sub create_ipset( list, type, size ) +# +# Creates a new IPSet. The current size of the set is determined by taking the +# next power of two greater than the number of entries; the maximum size is set +# to double this, subject to a minimum size. This allows for future expansion. +# +# Parameters: +# list The name of the blacklist +# type The type of the blacklist (hash:ip or hash:net) +# size The number of entries in the lsit +#------------------------------------------------------------------------------ + +sub create_ipset( $$$ ) +{ + my ($list, $type, $size) = @_; + + my $hashsize = 1; + $hashsize <<= 1 while ($hashsize < $size); + my $maxsize = $hashsize * 2; + $maxsize = $min_ipset_entries if ($maxsize < $min_ipset_entries); + + # Create the new ipset + ipset( "create $list $type hashsize $hashsize maxelem $maxsize" ); + stop_ipset(); # Need to do this to action the IPSet commands +} + + +#------------------------------------------------------------------------------ +# sub enable_logging() +# +# Enable logging of packets dropped by IP Blacklist rules. +# This adds a rule to log the packet to each lists' IPTables chain. +#------------------------------------------------------------------------------ + +sub enable_logging() +{ + get_ipsets(); + + log_message LOG_NOTICE, "Enabling IP Blacklist logging"; + + foreach my $list ( sort keys %sources ) + { + if (exists $chains{$list}) + { + iptables( "-I ${list}_DROP 1 -j LOG -m limit --limit 10/second --log-prefix 'BLKLST_$list'" ); + } + } +} + + +#------------------------------------------------------------------------------ +# sub disable_logging() +# +# Disable logging of packets dropped by IP Blacklist rules. +# This deletes a rule to log the packet from each lists' IPTables chain. +#------------------------------------------------------------------------------ + +sub disable_logging() +{ + get_ipsets(); + + log_message LOG_NOTICE, "Disabling IP Blacklist logging"; + + foreach my $list ( sort keys %sources ) + { + if (exists $chains{$list}) + { + iptables( "-D ${list}_DROP -j LOG -m limit --limit 10/second --log-prefix 'BLKLST_$list'" ); + } + } +} + + +#------------------------------------------------------------------------------ +# sub enable_updates() +# +# Adds a command to the fcrontab to run the update hourly. +# If there is a command already in the fcrontab to do this it will be +# uncommented, otherwise a new line is added. +# +# The update is executed at an offset from the hour so that all the users don't +# try to download the updates at exactly the same time - the blacklists are +# provided free, so it's good manners to spread the load on the servers. The +# offset is initialised to a random number that avoids running on the hour +# (when a lot of other things happen), and every fifteen minutes thereafter. +#------------------------------------------------------------------------------ + +sub enable_updates() +{ + my @lines = qx/$fcrontab -l/; + my $found = 0; + + # Check for an existing fcrontab entry. + + foreach my $line (@lines) + { + if ($line =~ m|/usr/local/bin/ipblacklist|) + { + return if ($line !~ m/^#/); # Already enabled - do nothing + + # Already in fcrontab - uncomment the line + + $line =~ s/^#+//; + $found = 1; + log_message LOG_INFO, "Enable IP Address Blacklist update in crontab"; + last; + } + } + + if (not $found) + { + # Add a new entry to fcrontab + + my $start = int( rand(13) ) + 1; + + my $times = $start; + + for (my $offset = $times+15 ; $offset < 60 ; $offset += 15) + { + $times .= ",$offset"; + } + + push @lines, "\n"; + push @lines, "# IP Blacklist update\n"; + push @lines, "%nice(1) $times * * * * /usr/local/bin/ipblacklist\n"; + log_message LOG_INFO, "Add IP Address Blacklist update to crontab"; + } + + open FCRONTAB, "| $fcrontab -" or (abort "Can't open pipe to write fcrontab: $!", return); + print FCRONTAB @lines; + close FCRONTAB; +} + + +#------------------------------------------------------------------------------ +# sub disable_updates() +# +# Comments out the entry in the fcrontab that runs the updates. +#------------------------------------------------------------------------------ + +sub disable_updates() +{ + my @lines = qx/$fcrontab -l/; + my $found = 0; + + foreach my $line (@lines) + { + if ($line =~ m|/usr/local/bin/ipblacklist|) + { + return if ($line =~ m/^#/); # Already disabled - do nothing + + # In fcrontab - comment the line + + $line =~ s/^#*/#/; + $found = 1; + log_message LOG_INFO, "Disable IP Address Blacklist updates"; + last; + } + } + + return unless ($found); # Don't update crontab unnecessarily + + open FCRONTAB, "| $fcrontab -" or (abort "Can't open pipe to write fcrontab: $!", return); + print FCRONTAB @lines; + close FCRONTAB; +} + + +#------------------------------------------------------------------------------ +# sub parse_ip_or_net_list( line ) +# +# Parses an input line, looking for lines starting with an IP Address or +# Network specification. +# +# Parameters: +# line The line to parse +# +# Returns: +# Either an IP Address or a null string +#------------------------------------------------------------------------------ + +sub parse_ip_or_net_list( $ ) +{ + my ($line) = @_; + + $line =~ m|^(\d+.\d+.\d+.\d+(?:/\d+)?)|; + + return $1; +} + + +#------------------------------------------------------------------------------ +# sub parse_dshield( line ) +# +# Parses an input line removing comments. +# +# The format is: +# Start Addrs End Addrs Netmask Nb Attacks Network Name Country email +# We're only interested in the start address and netmask. +# +# Parameters: +# line The line to parse +# +# Returns: +# Either and IP Address or a null string +#------------------------------------------------------------------------------ + +sub parse_dshield( $ ) +{ + my ($line) = @_; + + return "" if ($line =~ m/^\s*#/); + + $line =~ s/#.*$//; + + # |Start addrs | |End Addrs | |Mask + $line =~ m|(\d+.\d+.\d+.\d+(?:/\d+)?)\s+\d+.\d+.\d+.\d+(?:/\d+)?\s+(\d+)|; + + return unless ($1); + return "$1/32" unless ($2); + + return "$1/$2"; +} + + +#------------------------------------------------------------------------------ +# sub iptables( cmd ) +# +# Executes an IPTables command, waiting for the internal lock to ensure only +# one change is made at a time. +# +# Parameters: +# cmd The command to execute +# +# Returns: +# Status of command +#------------------------------------------------------------------------------ + +sub iptables( $ ) +{ + my ($cmd) = @_; + + return system( "$iptables $cmd" ); +} + + +#------------------------------------------------------------------------------ +# sub ipset( cmd ) +# +# Executes an IPSet command. The command is piped to a sub-process running +# ipset, rather than exected separately. This saves the overhead of starting a +# new process for each command. The sub-process is started if it's not already +# running. +# +# Note that the pipe is buffered so commands are not necessarily executed +# immediately. Use ipset_stop() to force commands to be executed. This should +# be done before relying on anything that the ipset commands do, for example +# before referencing the IPSet in an IPTables command. +# +# Parameters: +# cmd The command to execute +#------------------------------------------------------------------------------ + +sub ipset( $ ) +{ + my ($cmd) = @_; + + if (not $ipset_running) + { + local $SIG{PIPE} = 'IGNORE'; + open IPSET, "|-", $ipset, "restore" or die "Can't start ipset: $!"; + $ipset_running = 1; + } + + print IPSET "$cmd\n"; +} + + +#------------------------------------------------------------------------------ +# sub stop_ipset( ) +# +# Stops the ipset sub-process. +# This causes any pending ipset commands to be executed. +#------------------------------------------------------------------------------ + +sub stop_ipset( ) +{ + if ($ipset_running) + { + close IPSET or abort "ipset process died: $! $?"; + $ipset_running = 0; + } +} + + +#------------------------------------------------------------------------------ +# sub abort( message, parameters... ) +# +# Used when aborting the current activity, printing out an error message. +# +# Parameters: +# message Message to be printed +#------------------------------------------------------------------------------ + +sub abort( $ ) +{ + my ($message) = @_; + + log_message( LOG_ERR, $message ); + carp $message; +} + + +#------------------------------------------------------------------------------ +# sub log_message( level, message ) +# +# Logs a message. If the script is run from a terminal messages are also +# output on STDOUT. +# +# Parameters: +# level Severity of message +# message Message to be logged +#------------------------------------------------------------------------------ + +sub log_message( $$ ) +{ + my ($level, $message) = @_; + + print "($level) $message\n" if (-t STDIN); + syslog( $level, $message ); +} + + +#------------------------------------------------------------------------------ +# sub debug( level, message ) +# +# Optionally logs a debug message. If the script is run from a terminal, level +# 1 debug messages are output regardless of the debug setting. +# +# Parameters: +# level Debug level +# message Message to be logged +#------------------------------------------------------------------------------ + +sub debug( $$ ) +{ + my ($level, $message) = @_; + + if (($level <= $settings{'DEBUG'}) or + ($level == 1 and -t STDIN)) + { + log_message LOG_DEBUG, $message; + } +}
Signed-off-by: Tim FitzGeorge ipfr@tfitzgeorge.me.uk --- html/cgi-bin/ipblacklist.cgi | 463 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 463 insertions(+) create mode 100644 html/cgi-bin/ipblacklist.cgi
diff --git a/html/cgi-bin/ipblacklist.cgi b/html/cgi-bin/ipblacklist.cgi new file mode 100644 index 000000000..28b42edf2 --- /dev/null +++ b/html/cgi-bin/ipblacklist.cgi @@ -0,0 +1,463 @@ +#!/usr/bin/perl + +############################################################################### +# # +# IPFire.org - A linux based firewall # +# # +# This program is free software: you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation, either version 3 of the License, or # +# (at your option) any later version. # +# # +# This program is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with this program. If not, see http://www.gnu.org/licenses/. # +# # +# Copyright (C) 2018 - 2020 The IPFire Team # +# # +############################################################################### + +use strict; +use CGI qw/:standard/; +# enable the following only for debugging purposes +#use warnings; +#use CGI::Carp 'fatalsToBrowser'; +use Sort::Naturally; +use Socket; + +require '/var/ipfire/general-functions.pl'; +require "${General::swroot}/lang.pl"; +require "${General::swroot}/header.pl"; + +############################################################################### +# Configuration variables +############################################################################### + +my $settings = "${General::swroot}/ipblacklist/settings"; +my $sources = "${General::swroot}/ipblacklist/sources"; +my $getipstat = '/usr/local/bin/getipstat'; +my $getipsetstat = '/usr/local/bin/getipsetstat'; +my $control = '/usr/local/bin/ipblacklistctrl'; +my $lockfile = '/var/run/ipblacklist.pid'; +my %cgiparams = ('ACTION' => ''); + +############################################################################### +# Variables +############################################################################### + +my $errormessage = ''; +my $updating = 0; +my %mainsettings; +my %color; +my %sources; +my %stats; + +# Default settings - normally overwritten by settings file + +my %settings = ( 'DEBUG' => 0, + 'LOGGING' => 'on', + 'ENABLE' => 'off' ); + +# Read all parameters + +Header::getcgihash( %cgiparams); +General::readhash( "${General::swroot}/main/settings", %mainsettings ); +General::readhash( "/srv/web/ipfire/html/themes/".$mainsettings{'THEME'}."/include/colors.txt", %color ); +General::readhash( $settings, %settings ) if (-r $settings); +eval qx|/bin/cat $sources| if (-r $sources); + +# Show Headers + +Header::showhttpheaders(); + +# Process actions + +if ($cgiparams{'ACTION'} eq "$Lang::tr{'save'}") +{ + # Save Button + + my %new_settings = ( 'ENABLE' => 'off', + 'LOGGING' => 'off', + 'DEBUG' => 0 ); + + foreach my $item ('LOGGING', 'ENABLE', keys %sources) + { + $new_settings{$item} = (exists $cgiparams{$item}) ? 'on' : 'off'; + + $updating = 1 if (not exists $settings{$item} or $new_settings{$item} ne $settings{$item}); + } + + # Check for redundant blacklists being enabled + + foreach my $list (keys %sources) + { + if (exists $new_settings{$list} and + $new_settings{$list} eq 'on' and + exists $sources{$list}{'disable'}) + { + my @disable; + + if ('ARRAY' eq ref $sources{$list}{'disable'}) + { + @disable = @{ $sources{$list}{'disable'} }; + } + else + { + @disable = ( $sources{$list}{'disable'} ); + } + + foreach my $disable (@disable) + { + if ($new_settings{$disable} eq 'on') + { + $new_settings{$disable} = 'off'; + + $updating = 1; + $errormessage .= "$Lang::tr{'ipblacklist disable pre'} $disable " . + "$Lang::tr{'ipblacklist disable mid'} $list $Lang::tr{'ipblacklist disable post'}<br>\n"; + } + } + } + } + + if ($settings{'LOGGING'} ne $new_settings{'LOGGING'}) + { + if ($new_settings{'LOGGING'} eq 'on') + { + system( "$control log-on" ); + } + else + { + system( "$control log-off" ); + } + } + + if ($settings{'ENABLE'} ne $new_settings{'ENABLE'}) + { + if ($new_settings{'ENABLE'} eq 'on') + { + system( "$control enable" ); + } + else + { + system( "$control disable" ); + } + + $updating = 1; + } + + %settings = %new_settings; + + if ($errormessage) + { + $updating = 0; + } + else + { + General::writehash($settings, %new_settings); + + if ($updating) + { + system( "$control update &" ); + show_running(); + exit 0; + } + } +} + +if (is_running()) +{ + show_running(); + exit 0; +} + +# Show site + +Header::openpage($Lang::tr{'ipblacklist'}, 1, ''); +Header::openbigbox('100%', 'left'); + +error() if ($errormessage); + +configsite(); + +# End of page + +Header::closebigbox(); +Header::closepage(); + +exit 0; + + +#------------------------------------------------------------------------------ +# sub configsite() +# +# Displays configuration +#------------------------------------------------------------------------------ + +sub configsite +{ + # Find preselections + + my $enable = 'checked'; + Header::openbox('100%', 'left', $Lang::tr{'settings'}); + + #### JAVA SCRIPT #### + + print<<END; +<script> + $(document).ready(function() + { + // Show/Hide elements when ENABLE checkbox is checked. + if ($("#ENABLE").attr("checked")) + { + $(".sources").show(); + } + else + { + $(".sources").hide(); + } + + // Toggle Source list elements when "ENABLE" checkbox is clicked + $("#ENABLE").change(function() + { + $(".sources").toggle(); + }); + }); +</script> +END + + ##### JAVA SCRIPT END #### + + # Enable checkbox + + $enable = ($settings{'ENABLE'} eq 'on') ? ' checked' : ''; + + print<<END; + <form method='post' action='$ENV{'SCRIPT_NAME'}'> + <table style='width:100%' border='0'> + <tr> + <td style='width:24em'>$Lang::tr{'ipblacklist use ipblacklists'}</td> + <td><input type='checkbox' name='ENABLE' id='ENABLE'$enable></td> + </tr> + </table><br> + +END + + # The following are only displayed if the blacklists are enabled + + $enable = ($settings{'LOGGING'} eq 'on') ? ' checked' : ''; + + print <<END; +<div class='sources'> + <table style='width:100%' border='0'> + <tr> + <td style='width:24em'>$Lang::tr{'ipblacklist log'}</td> + <td><input type='checkbox' name="LOGGING" id="LOGGING"$enable></td> + </tr> + </table> + <br><br> + <h2>$Lang::tr{'ipblacklist blacklist settings'}</h2> + <table width='100%' cellspacing='1' class='tbl'> + <tr> + <th align='left'>$Lang::tr{'ipblacklist id'}</th> + <th align='left'>$Lang::tr{'ipblacklist name'}</th> + <th align='left'>$Lang::tr{'ipblacklist category'}</th> + <th align='center'>$Lang::tr{'ipblacklist enable'}</th> + </tr> +END + + # Iterate through the list of sources + + my $lines = 0; + + foreach my $list (sort keys %sources) + { + my $name = escapeHTML( $sources{$list}{'name'} ); + my $category = $Lang::tr{"ipblacklist category $sources{$list}{'category'}"}; + $enable = ''; + my $col = ($lines++ % 2) ? "bgcolor='$color{'color20'}'" : "bgcolor='$color{'color22'}'"; + + $enable = ' checked' if (exists $settings{$list} and $settings{$list} eq 'on'); + + print <<END; + <tr $col> + <td> +END + + if ($sources{$list}{info}) + { + print "<a href='$sources{$list}{info}' target='_blank'>$list</a>\n"; + } + else + { + print "$list\n"; + } + + print <<END; + </td> + <td>$name</td> + <td>$category</td> + <td align='center'><input type='checkbox' name="$list" id="$list"$enable></td> + </tr>\n +END + } + + # The save button at the bottom of the table + + print <<END; + </table> + </div> + <table style='width:100%;'> + <tr> + <td colspan='3' display:inline align='right'><input type='submit' name='ACTION' value='$Lang::tr{'save'}'></td> + </tr> + </table> +END + + Header::closebox(); +} + + +#------------------------------------------------------------------------------ +# sub get_ipset_stats() +# +# Gets the number of entries in each IPSet. +#------------------------------------------------------------------------------ + +sub get_ipset_stats +{ + my $name; + + system( $getipsetstat ); + + if (-r '/var/tmp/ipsets.txt') + { + open STATS, '<', '/var/tmp/ipsets.txt' or die "Can't open IP Sets stats file: $!"; + + foreach my $line (<STATS>) + { + if ($line =~ m/Name: (\w+)/) + { + $name = $1; + next; + } + + if ($line =~ m/Number of entries: (\d+)/) + { + $stats{$name}{'size'} = $1; + } + } + + close STATS; + + unlink( '/var/tmp/ipsets.txt' ); + } +} + + +#------------------------------------------------------------------------------ +# sub is_running() +# +# Checks to see if the main script is running +#------------------------------------------------------------------------------ + +sub is_running +{ + return 0 unless (-r $lockfile); + + open LOCKFILE, '<', $lockfile or die "Can't open lockfile"; + my $pid = <LOCKFILE>; + close LOCKFILE; + + chomp $pid; + + return (-e "/proc/$pid"); +} + + +#------------------------------------------------------------------------------ +# sub show_running +# +# Displayed when update is running. +# Shows a 'working' message plus some information about the IPSets. +#------------------------------------------------------------------------------ + +sub show_running +{ + # Open site + + Header::openpage( $Lang::tr{'ipblacklist'}, 1, '<meta http-equiv="refresh" content="1;url=/cgi-bin/ipblacklist.cgi">' ); + Header::openbigbox( '100%', 'center' ); + error(); + Header::openbox( 'Working', 'center', "$Lang::tr{'ipblacklist working'}" ); + + print <<END; + <table width='100%'> + <tr> + <td align='center'> + <img src='/images/indicator.gif' alt='$Lang::tr{'aktiv'}'> + <td> + </tr> + </table> + <br> + <table cellspacing='1' align='center'> + <tr><th>$Lang::tr{'ipblacklist id'}</th><th>$Lang::tr{'ipblacklist entries'}</th></tr> +END + + get_ipset_stats(); + + foreach my $name (sort keys %stats) + { + print "<tr><td>$name</td><td align='right'>$stats{$name}{'size'}</td></tr>\n" if (exists $stats{$name}{'size'}); + } + + print <<END; + </table> +END + + Header::closebox(); + + Header::closebigbox(); + Header::closepage(); +} + + +#------------------------------------------------------------------------------ +# sub error() +# +# Shows error messages +#------------------------------------------------------------------------------ + +sub error +{ + Header::openbox('100%', 'left', $Lang::tr{'error messages'}); + print "<class name='base'>$errormessage\n"; + print " </class>\n"; + Header::closebox(); +} + + +#------------------------------------------------------------------------------ +# sub format_time( seconds ) +# +# Converts time in seconds to HH:MM:SS +#------------------------------------------------------------------------------ + +sub format_time($) { + my $time = shift; + + my $seconds = $time % 60; + my $minutes = $time / 60; + + my $hours = 0; + if ($minutes >= 60) { + $hours = $minutes / 60; + $minutes %= 60; + } + + return sprintf("%3d:%02d:%02d", $hours, $minutes, $seconds); +}
Signed-off-by: Tim FitzGeorge ipfr@tfitzgeorge.me.uk --- html/cgi-bin/logs.cgi/ipblacklists.dat | 363 +++++++++++++++++++++++++++++++++ 1 file changed, 363 insertions(+) create mode 100755 html/cgi-bin/logs.cgi/ipblacklists.dat
diff --git a/html/cgi-bin/logs.cgi/ipblacklists.dat b/html/cgi-bin/logs.cgi/ipblacklists.dat new file mode 100755 index 000000000..9b8db7062 --- /dev/null +++ b/html/cgi-bin/logs.cgi/ipblacklists.dat @@ -0,0 +1,363 @@ +#!/usr/bin/perl +# +# SmoothWall CGIs +# +# This code is distributed under the terms of the GPL +# +# JC HERITIER +# page inspired from the initial firewalllog.dat +# +# Modified for IPFire by Christian Schmidt +# and Michael Tremer (www.ipfire.org) + +use strict; +use Getopt::Std; + +# enable only the following on debugging purpose +#use warnings; +#use CGI::Carp 'fatalsToBrowser'; + +require '/var/ipfire/general-functions.pl'; +require "${General::swroot}/geoip-functions.pl"; +require "${General::swroot}/lang.pl"; +require "${General::swroot}/header.pl"; + +use POSIX(); + +my %cgiparams=(); +my $errormessage = ''; + +my @shortmonths = ( 'Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', + 'Sep', 'Oct', 'Nov', 'Dec' ); +my @longmonths = ( $Lang::tr{'january'}, $Lang::tr{'february'}, $Lang::tr{'march'}, + $Lang::tr{'april'}, $Lang::tr{'may'}, $Lang::tr{'june'}, $Lang::tr{'july'}, $Lang::tr{'august'}, + $Lang::tr{'september'}, $Lang::tr{'october'}, $Lang::tr{'november'}, + $Lang::tr{'december'} ); + +my @now = localtime(); +my $dow = $now[6]; +my $doy = $now[7]; +my $tdoy = $now[7]; +my $year = $now[5]+1900; + +$cgiparams{'DAY'} = $now[3]; +$cgiparams{'MONTH'} = $now[4]; +$cgiparams{'ACTION'} = ''; + +&Header::getcgihash(%cgiparams); + +my $start = -1; +if ($ENV{'QUERY_STRING'} && $cgiparams{'ACTION'} ne $Lang::tr{'update'}) +{ + my @temp = split(',',$ENV{'QUERY_STRING'}); + $start = $temp[0]; + $cgiparams{'MONTH'} = $temp[1]; + $cgiparams{'DAY'} = $temp[2]; +} + +if (!($cgiparams{'MONTH'} =~ /^(0|1|2|3|4|5|6|7|8|9|10|11)$/) || + !($cgiparams{'DAY'} =~ /^(1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31)$/)) +{ + $cgiparams{'DAY'} = $now[3]; + $cgiparams{'MONTH'} = $now[4]; +} +elsif($cgiparams{'ACTION'} eq '>>') +{ + my @temp_then=(); + my @temp_now = localtime(time); + $temp_now[4] = $cgiparams{'MONTH'}; + $temp_now[3] = $cgiparams{'DAY'}; + @temp_then = localtime(POSIX::mktime(@temp_now) + 86400); + ## Retrieve the same time on the next day - + ## 86400 seconds in a day + $cgiparams{'MONTH'} = $temp_then[4]; + $cgiparams{'DAY'} = $temp_then[3]; +} +elsif($cgiparams{'ACTION'} eq '<<') +{ + my @temp_then=(); + my @temp_now = localtime(time); + $temp_now[4] = $cgiparams{'MONTH'}; + $temp_now[3] = $cgiparams{'DAY'}; + @temp_then = localtime(POSIX::mktime(@temp_now) - 86400); + ## Retrieve the same time on the previous day - + ## 86400 seconds in a day + $cgiparams{'MONTH'} = $temp_then[4]; + $cgiparams{'DAY'} = $temp_then[3]; +} + +if (($cgiparams{'DAY'} ne $now[3]) || ($cgiparams{'MONTH'} ne $now[4])) +{ + my @then = (); + if ( ( $cgiparams{'MONTH'} eq $now[4]) && ($cgiparams{'DAY'} > $now[3]) || + ( $cgiparams{'MONTH'} > $now[4] ) ) { + @then = localtime(POSIX::mktime( 0, 0, 0, $cgiparams{'DAY'}, $cgiparams{'MONTH'}, $year - 1901 )); + } else { + @then = localtime(POSIX::mktime( 0, 0, 0, $cgiparams{'DAY'}, $cgiparams{'MONTH'}, $year - 1900 )); + } + $tdoy = $then[7]; + my $lastleap=($year-1)%4; + if ($tdoy>$doy) { + if ($lastleap == 0 && $tdoy < 60) { + $doy=$tdoy+366; + } else { + $doy=$doy+365; + } + } +} + +my $datediff=0; +my $dowd=0; +my $multifile=0; +if ($tdoy ne $doy) { + $datediff=int(($doy-$tdoy)/7); + $dowd=($doy-$tdoy)%7; + if (($dow-$dowd)<1) { + $datediff=$datediff+1; + } + if (($dow-$dowd)==0) { + $multifile=1; + } +} + +my $monthstr = $shortmonths[$cgiparams{'MONTH'}]; +my $longmonthstr = $longmonths[$cgiparams{'MONTH'}]; +my $day = $cgiparams{'DAY'}; +my $daystr=''; +if ($day <= 9) { + $daystr = " $day"; } +else { + $daystr = $day; +} + +my %lists; +my %directions; +my %sources = (); +my %settings = (); +&General::readhash("${General::swroot}/ipblacklist/settings", %settings); +eval qx|/bin/cat /var/ipfire/ipblacklist/sources|; + +foreach my $blacklist (keys %sources) +{ + $lists{$blacklist} = {} if ($settings{$blacklist} eq 'on'); +} + +my $skip=0; +my $filestr=''; +if ($datediff==0) { + $filestr="/var/log/messages"; +} else { + $filestr="/var/log/messages.$datediff"; + $filestr = "$filestr.gz" if -f "$filestr.gz"; +} + +if (!(open (FILE,($filestr =~ /.gz$/ ? "gzip -dc $filestr |" : $filestr)))) { + $errormessage = "$Lang::tr{'date not in logs'}: $filestr $Lang::tr{'could not be opened'}"; + $skip=1; + # Note: This is in case the log does not exist for that date +} + +my $lines = 0; +my $directions = 0; + +if (!$skip) +{ + while (<FILE>) + { + if (/^${monthstr} ${daystr} ..:..:.. [\w-]+ kernel:.*BLKLST_(\w+)\s*IN=(\w*)/) + { + my $list = $1; + + if ($2 =~ m/ppp|red/) + { + $lists{$list}{in}++; + $directions{in}++; + } + else + { + $lists{$list}{out}++; + $directions{out}++; + } + + $lines++; + } + + } + close (FILE); +} + +if ($multifile) { + $datediff=$datediff-1; + if ($datediff==0) { + $filestr="/var/log/messages"; + } else { + $filestr="/var/log/messages.$datediff"; + $filestr = "$filestr.gz" if -f "$filestr.gz"; + } + if (!(open (FILE,($filestr =~ /.gz$/ ? "gzip -dc $filestr |" : $filestr)))) { + $errormessage="$Lang::tr{'date not in logs'}: $filestr $Lang::tr{'could not be opened'}"; + $skip=1; + } + if (!$skip) { + while (<FILE>) { + if (/^${monthstr} ${daystr} ..:..:.. [\w-]+ kernel:.*BLKLST_(\w+)\s*IN=(\w+)/) + { + my $list = $1; + + if ($2 =~ m/ppp|red/) + { + $lists{$list}{in}++; + $directions{in}++; + } + else + { + $lists{$list}{out}++; + $directions{out}++; + } + + $lines++; + } + } + close (FILE); + } +} + +my $MODNAME="fwlogs"; + +&Header::showhttpheaders(); +&Header::openpage($Lang::tr{'ipblacklist logs'}, 1, ''); +&Header::openbigbox('100%', 'left', '', $errormessage); + + +if ($errormessage) { + &Header::openbox('100%', 'left', $Lang::tr{'error messages'}); + print "<font class='base'>$errormessage </font>\n"; + &Header::closebox(); +} + +&Header::openbox('100%', 'left', "$Lang::tr{'settings'}"); + +print <<END +<form method='post' action='$ENV{'SCRIPT_NAME'}'> +<table width='100%'> +<tr> + <td width='10%' class='base'>$Lang::tr{'month'}: </td> + <td width='10%'> + <select name='MONTH'> +END +; +my $month; +for ($month = 0; $month < 12; $month++) +{ + print "\t<option "; + if ($month == $cgiparams{'MONTH'}) { + print "selected='selected' "; + } + print "value='$month'>$longmonths[$month]</option>\n"; +} +print <<END + </select> + </td> + <td width='10%' class='base' align='right'> $Lang::tr{'day'}: </td> + <td width='40%'> + <select name='DAY'> +END +; +for ($day = 1; $day <= 31; $day++) +{ + print "\t<option "; + if ($day == $cgiparams{'DAY'}) { + print "selected='selected' "; + } + print "value='$day'>$day</option>\n"; +} + +print <<END +</select> +</td> +<td width='5%' align='center'><input type='submit' name='ACTION' title='$Lang::tr{'day before'}' value='<<' /></td> +<td width='5%' align='center'><input type='submit' name='ACTION' title='$Lang::tr{'day after'}' value='>>' /></td> +<td width='20%' align='right'><input type='submit' name='ACTION' value='$Lang::tr{'update'}' /></td> +</tr> +</table> +</form> +END +; + +&Header::closebox(); + +&Header::openbox('100%', 'left', $Lang::tr{'firewall log'}); +print "<p><b>$Lang::tr{'ipblacklist hits'} $longmonthstr $daystr: $lines</b></p>"; + +my %color = (); +my %mainsettings = (); +&General::readhash("${General::swroot}/main/settings", %mainsettings); +&General::readhash("/srv/web/ipfire/html/themes/".$mainsettings{'THEME'}."/include/colors.txt", %color); + +my @lists = sort keys (%lists); + +print <<END +<table width='100%' class='tbl'> +<tr> +<th align='center' class='boldbase' rowspan='2'></th> +<th align='left' class='boldbase' rowspan='2'><b>$Lang::tr{'ipblacklist id'}</b></th> +<th align='left' class='boldbase' rowspan='2'><b>$Lang::tr{'ipblacklist category'}</b></th> +<th align='center' class='boldbase' colspan='2'><b>$Lang::tr{'ipblacklist input'}</b></th> +<th align='center' class='boldbase' colspan='2'><b>$Lang::tr{'ipblacklist output'}</b></th> +</tr> +<tr> +<th align='center' class='boldbase'>$Lang::tr{'count'}</th> +<th align='center' class='boldbase'>$Lang::tr{'percentage'}</th> +<th align='center' class='boldbase'>$Lang::tr{'count'}</th> +<th align='center' class='boldbase'>$Lang::tr{'percentage'}</th> +</tr> +END +; + +$lines = 0; +my $lists = join ',', @lists; + +foreach my $list (@lists) +{ + my $col = ($lines++ % 2) ? "bgcolor='$color{'color20'}'" : "bgcolor='$color{'color22'}'"; + my $category = exists( $sources{$list}) ? $Lang::tr{"ipblacklist category $sources{$list}{'category'}"} : ' '; + + print "<tr>"; + + print "<td align='center' $col><form method='post' action='showrequestfromblacklist.dat'><input type='hidden' name='MONTH' value='$cgiparams{'MONTH'}'> <input type='hidden' name='DAY' value='$cgiparams{'DAY'}'> <input type='hidden' name='blacklist' value='$list'><input type='hidden' name='blacklists' value='$lists'> <input type='submit' value='$Lang::tr{'details'}'></form></td>"; + + if (exists($sources{$list}) and $sources{$list}{'info'}) + { + print "<td $col><a href='$sources{$list}{info}' target='_blank'>$list</a></td>"; + } + else + { + print "<td $col>$list</td>"; + } + + print "<td $col>$category</td>"; + + foreach my $direction ('in', 'out') + { + my $count = $lists{$list}{$direction} || 0; + my $percent = $directions{$direction} > 0 ? $count * 100 / $directions{$direction} : 0; + $percent = sprintf("%.f", $percent); + print "<td align='center' class='boldbase' $col>$count</th>"; + print "<td align='center' class='boldbase' $col>$percent%</th>"; + } + + print "</tr>"; +} +print <<END +</table> +END +; + +&Header::closebox(); +&Header::closebigbox(); +&Header::closepage(); + +sub checkversion { + #Automatic Updates is disabled + return "0","0"; +}
Signed-off-by: Tim FitzGeorge ipfr@tfitzgeorge.me.uk --- html/cgi-bin/logs.cgi/showrequestfromblacklist.dat | 415 +++++++++++++++++++++ 1 file changed, 415 insertions(+) create mode 100755 html/cgi-bin/logs.cgi/showrequestfromblacklist.dat
diff --git a/html/cgi-bin/logs.cgi/showrequestfromblacklist.dat b/html/cgi-bin/logs.cgi/showrequestfromblacklist.dat new file mode 100755 index 000000000..0de381a2d --- /dev/null +++ b/html/cgi-bin/logs.cgi/showrequestfromblacklist.dat @@ -0,0 +1,415 @@ +#!/usr/bin/perl +# SmoothWall CGIs +# +# This code is distributed under the terms of the GPL +# +# JC HERITIER +# page inspired from the initial firewalllog.dat +# +# Modified for IPFire by Christian Schmidt (www.ipfire.org) + +# enable only the following on debugging purpose +#use warnings; +#use CGI::Carp 'fatalsToBrowser'; + +require '/var/ipfire/general-functions.pl'; +require "${General::swroot}/lang.pl"; +require "${General::swroot}/header.pl"; + +use POSIX(); + +#workaround to suppress a warning when a variable is used only once +my @dummy = ( ${Header::table2colour} ); +undef (@dummy); + +my %cgiparams=(); +my %logsettings=(); +my $errormessage = ''; + +my @shortmonths = ( 'Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', + 'Sep', 'Oct', 'Nov', 'Dec' ); +my @longmonths = ( $Lang::tr{'january'}, $Lang::tr{'february'}, $Lang::tr{'march'}, + $Lang::tr{'april'}, $Lang::tr{'may'}, $Lang::tr{'june'}, $Lang::tr{'july'}, $Lang::tr{'august'}, + $Lang::tr{'september'}, $Lang::tr{'october'}, $Lang::tr{'november'}, + $Lang::tr{'december'} ); + +my @now = localtime(); +my $dow = $now[6]; +my $doy = $now[7]; +my $tdoy = $now[7]; +my $year = $now[5]+1900; + +$cgiparams{'DAY'} = $now[3]; +$cgiparams{'MONTH'} = $now[4]; +$cgiparams{'ACTION'} = ''; + +&Header::getcgihash(%cgiparams); + +$logsettings{'LOGVIEW_REVERSE'} = 'off'; +&General::readhash("${General::swroot}/logging/settings", %logsettings); + +my $start = -1; +my @blacklists; +if ($ENV{'QUERY_STRING'} && $cgiparams{'ACTION'} ne $Lang::tr{'update'}) +{ + my @temp = split(',',$ENV{'QUERY_STRING'}, 5); + $start = shift @temp; + $cgiparams{'MONTH'} = shift @temp; + $cgiparams{'DAY'} = shift @temp; + $cgiparams{'blacklist'} = shift @temp; + $cgiparams{'blacklists'} = shift @temp; +} + +if (!($cgiparams{'MONTH'} =~ /^(0|1|2|3|4|5|6|7|8|9|10|11)$/) || + !($cgiparams{'DAY'} =~ /^(1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31)$/)) +{ + $cgiparams{'DAY'} = $now[3]; + $cgiparams{'MONTH'} = $now[4]; +} +elsif($cgiparams{'ACTION'} eq '>>') +{ + my @temp_then=(); + my @temp_now = localtime(time); + $temp_now[4] = $cgiparams{'MONTH'}; + $temp_now[3] = $cgiparams{'DAY'}; + @temp_then = localtime(POSIX::mktime(@temp_now) + 86400); + ## Retrieve the same time on the next day - + ## 86400 seconds in a day + $cgiparams{'MONTH'} = $temp_then[4]; + $cgiparams{'DAY'} = $temp_then[3]; +} +elsif($cgiparams{'ACTION'} eq '<<') +{ + my @temp_then=(); + my @temp_now = localtime(time); + $temp_now[4] = $cgiparams{'MONTH'}; + $temp_now[3] = $cgiparams{'DAY'}; + @temp_then = localtime(POSIX::mktime(@temp_now) - 86400); + ## Retrieve the same time on the previous day - + ## 86400 seconds in a day + $cgiparams{'MONTH'} = $temp_then[4]; + $cgiparams{'DAY'} = $temp_then[3]; +} + +if (($cgiparams{'DAY'} ne $now[3]) || ($cgiparams{'MONTH'} ne $now[4])) +{ + my @then = (); + if ( ( $cgiparams{'MONTH'} eq $now[4]) && ($cgiparams{'DAY'} > $now[3]) || + ( $cgiparams{'MONTH'} > $now[4] ) ) { + @then = localtime(POSIX::mktime( 0, 0, 0, $cgiparams{'DAY'}, $cgiparams{'MONTH'}, $year - 1901 )); + } else { + @then = localtime(POSIX::mktime( 0, 0, 0, $cgiparams{'DAY'}, $cgiparams{'MONTH'}, $year - 1900 )); + } + $tdoy = $then[7]; + my $lastleap=($year-1)%4; + if ($tdoy>$doy) { + if ($lastleap == 0 && $tdoy < 60) { + $doy=$tdoy+366; + } else { + $doy=$doy+365; + } + } +} + +if ($cgiparams{'blacklists'}) +{ + @blacklists = split ',', $cgiparams{'blacklists'}; +} + +my $datediff=0; +my $dowd=0; +my $multifile=0; +if ($tdoy ne $doy) { + $datediff=int(($doy-$tdoy)/7); + $dowd=($doy-$tdoy)%7; + if (($dow-$dowd)<1) { + $datediff=$datediff+1; + } + if (($dow-$dowd)==0) { + $multifile=1; + } +} + +my $monthstr = $shortmonths[$cgiparams{'MONTH'}]; +my $longmonthstr = $longmonths[$cgiparams{'MONTH'}]; +my $day = $cgiparams{'DAY'}; +my $daystr=''; +if ($day <= 9) { + $daystr = " $day"; } +else { + $daystr = $day; +} + +my $skip=0; +my $filestr=''; +if ($datediff==0) { + $filestr="/var/log/messages"; +} else { + $filestr="/var/log/messages.$datediff"; + $filestr = "$filestr.gz" if -f "$filestr.gz"; +} + +if (!(open (FILE,($filestr =~ /.gz$/ ? "gzip -dc $filestr |" : $filestr)))) { + $errormessage = "$Lang::tr{'date not in logs'}: $filestr $Lang::tr{'could not be opened'}"; + $skip=1; + # Note: This is in case the log does not exist for that date +} +my $lines = 0; +my @log=(); +my $blacklist = $cgiparams{blacklist}; + +if (!$skip) +{ + while (<FILE>) { + if (/^${monthstr} ${daystr} ..:..:.. [\w-]+ kernel:.*BLKLST_(\w+)IN=.*/) { + if($1 eq $blacklist){ + $log[$lines] = $_; + $lines++; + } + } + } + close (FILE); +} + +$skip=0; +if ($multifile) { + $datediff=$datediff-1; + if ($datediff==0) { + $filestr="/var/log/messages"; + } else { + $filestr="/var/log/messages.$datediff"; + $filestr = "$filestr.gz" if -f "$filestr.gz"; + } + if (!(open (FILE,($filestr =~ /.gz$/ ? "gzip -dc $filestr |" : $filestr)))) { + $errormessage="$Lang::tr{'date not in logs'}: $filestr $Lang::tr{'could not be opened'}"; + $skip=1; + } + if (!$skip) { + while (<FILE>) { + if (/^${monthstr} ${daystr} ..:..:.. [\w-]+ kernel:.*BLKLST_(\w+)IN=.*/) { + if($1 eq $blacklist){ + $log[$lines] = $_; + $lines++; + } + } + } + close (FILE); + } +} + +&Header::showhttpheaders(); +&Header::openpage($Lang::tr{'ipblacklist log list'}, 1, ''); +&Header::openbigbox('100%', 'left', '', $errormessage); + +if ($errormessage) { + &Header::openbox('100%', 'left', $Lang::tr{'error messages'}); + print "<font class='base'>$errormessage </font>\n"; + &Header::closebox(); +} + +&Header::openbox('100%', 'left', "$Lang::tr{'settings'}:"); + +print <<END +<form method='post' action='$ENV{'SCRIPT_NAME'}'> +<input type='hidden' name='blacklists' value='$cgiparams{blacklists}'> +<table width='100%'> +<tr> + <td width='10%' class='base'>$Lang::tr{'month'}: </td> + <td width='10%'> + <select name='MONTH'> +END +; +my $month; +for ($month = 0; $month < 12; $month++) +{ + print "\t<option "; + if ($month == $cgiparams{'MONTH'}) { + print "selected='selected' "; } + print "value='$month'>$longmonths[$month]</option>\n"; +} +print <<END + </select> + </td> + <td width='10%' class='base' align='right'> $Lang::tr{'day'}: </td> + <td width='40%'> + <select name='DAY'> +END +; +for ($day = 1; $day <= 31; $day++) +{ + print "\t<option "; + if ($day == $cgiparams{'DAY'}) { + print "selected='selected' "; } + print "value='$day'>$day</option>\n"; +} +print <<END +</select> +</td> +<td width='5%' align='center'><input type='submit' name='ACTION' title='$Lang::tr{'day before'}' value='<<' /></td> +<td width='5%' align='center'><input type='submit' name='ACTION' title='$Lang::tr{'day after'}' value='>>' /></td> +<td width='10%' align='center'><input type='submit' name='ACTION' value='$Lang::tr{'update'}' /></td> +<tr><td width='15%'>$Lang::tr{'ipblacklist id'}</td><td><select name='blacklist'> +END +; + +foreach my $option (@blacklists) +{ + my $selected = $option eq $cgiparams{blacklist} ? ' selected' : ''; + print "<option value='$option'$selected>$option</option>"; +} + +print <<END +</select></td></tr> +</tr> +</table> +</form> +END +; + +&Header::closebox(); + +&Header::openbox('100%', 'left', $Lang::tr{'ipblacklist log list'}); +print "<p><b>$Lang::tr{'firewall hits'} $longmonthstr $daystr: $lines</b></p>"; + +if ($start == -1) { + $start = $lines - ${Header::viewsize}; +} +if ($start >= $lines - ${Header::viewsize}) { $start = $lines - ${Header::viewsize}; }; +if ($start < 0) { $start = 0; } + +my $prev = $start - ${Header::viewsize}; +my $next = $start + ${Header::viewsize}; + +if ($prev < 0) { $prev = 0; } +if ($next >= $lines) { $next = -1 } +if ($start == 0) { $prev = -1; } + +if ($lines != 0) { &oldernewer(); } + +print <<END +<table width='100%'> +<tr> +<td width='12%' align='center' class='boldbase'><b>$Lang::tr{'time'}</b></td> +<td width='6%' align='center' class='boldbase'><b>$Lang::tr{'iface'}</b></td> +<td width='6%' align='center' class='boldbase'><b>$Lang::tr{'proto'}</b></td> +<td width='18%' align='center' class='boldbase'><b>$Lang::tr{'source'}</b></td> +<td width='15%' align='center' class='boldbase'><b>$Lang::tr{'src port'}</b></td> +<td width='18%' align='center' class='boldbase'><b>$Lang::tr{'destination'}</b></td> +<td width='15%' align='center' class='boldbase'><b>$Lang::tr{'dst port'}</b></td> +</tr> +END +; + +my @slice = splice(@log, $start, ${Header::viewsize}); + +if ($logsettings{'LOGVIEW_REVERSE'} eq 'on') { @slice = reverse @slice; } + +$lines = 0; +foreach $_ (@slice) { + $a = $_; + # Check whether valid ipv4 or ipv6 address + if (($_ =~ /BLKLST_(\w+)IN=/)) { + if($1 eq $blacklist) { + + my $in = '-'; my $out = '-'; + my $srcaddr = ''; my $dstaddr = ''; + my $protostr = ''; + my $srcport = ''; my $dstport = ''; + + # If ipv6 uses bridge, the use PHYSIN, otherwise use IN + if ($_ =~ /(^.* ..:..:..) [\w-]+ kernel:.*(IN=.*)(PHYSIN=.*)$/) {} + elsif ($_ =~ /(^.* ..:..:..) [\w-]+ kernel:.*(IN=.*)$/) {} + my $timestamp = $1; my $packet = $2; + $timestamp =~ /(...) (..) (..:..:..)/; + my $month = $1; my $day = $2; my $time = $3; + + # If ipv6 uses bridge, the use PHYSIN and PHYSOUT, otherwise use IN and OUT + if ($a =~ /PHYSIN=(\w+)/) { $iface = $1; } elsif ($a =~ /IN=(\w+)/) { $iface = $1; } + if ($a =~ /PHYSOUT=(\w+)/) { $out = $1; } elsif ($a =~ /OUT=(\w+)/) { $out = $1; } + # Detect ipv4 and ipv6 addresses + if (($a =~ /SRC=(([\d]{1,3})(.([\d]{1,3})){3})/) or ($a =~ /SRC=(([0-9a-fA-F]{0,4})(:([0-9a-fA-F]{0,4})){2,7})/)) { $srcaddr = $1; } + if (($a =~ /DST=(([\d]{1,3})(.([\d]{1,3})){3})/) or ($a =~ /DST=(([0-9a-fA-F]{0,4})(:([0-9a-fA-F]{0,4})){2,7})/)) { $dstaddr = $1; } + if ($a =~ /PROTO=(\w+)/) { $protostr = $1; } + my $protostrlc = lc($protostr); + if ($a =~ /SPT=([\d.]+)/){ $srcport = $1; } + if ($a =~ /DPT=([\d.]+)/){ $dstport = $1; } + + if ($lines % 2) { + print "<tr bgcolor='${Header::table1colour}'>\n"; + } + else { + print "<tr bgcolor='${Header::table2colour}'>\n"; + } + print <<END + <td align='center'>$time</td> + <td align='center'>$iface</td> + <td align='center'>$protostr</td> + <td align='center'> + <table width='100%' cellpadding='0' cellspacing='0'><tr> + <td align='center'><a href='/cgi-bin/ipinfo.cgi?ip=$srcaddr'>$srcaddr</a></td> + </tr></table> + </td> + <td align='center'>$srcport</td> + <td align='center'> + <table width='100%' cellpadding='0' cellspacing='0'><tr> + <td align='center'><a href='/cgi-bin/ipinfo.cgi?ip=$dstaddr'>$dstaddr</a></td> + </tr></table> + </td> + <td align='center'>$dstport</td> + </tr> +END + ; + $lines++; + } + } +} + +print <<END +</table> +END +; + +&oldernewer(); + + print"<table width='100%'><tr><td align='center'><a href='/cgi-bin/logs.cgi/ipblacklists.dat'><img src='/images/back.png' alt='$Lang::tr{'back'}' title='$Lang::tr{'back'}' /></a></td></tr></table>"; + +&Header::closebox(); + +&Header::closebigbox(); + +&Header::closepage(); + +sub oldernewer +{ + print <<END + <table width='100%'> + <tr> +END + ; + + my $blacklists = join ',', @blacklists; + + print "<td align='center' width='50%'>"; + if ($prev != -1) { + print "<a href='/cgi-bin/logs.cgi/showrequestfromblacklist.dat?$prev,$cgiparams{'MONTH'},$cgiparams{'DAY'},$cgiparams{blacklist},$blacklists'>$Lang::tr{'older'}</a>"; + } + else { + print "$Lang::tr{'older'}"; + } + print "</td>\n"; + + print "<td align='center' width='50%'>"; + if ($next != -1) { + print "<a href='/cgi-bin/logs.cgi/showrequestfromblacklist.dat?$next,$cgiparams{'MONTH'},$cgiparams{'DAY'},$cgiparams{blacklist},$blacklists'>$Lang::tr{'newer'}</a>"; + } + else { + print "$Lang::tr{'newer'}"; + } + print "</td>\n"; + + print <<END + </tr> + </table> +END + ; +}
Signed-off-by: Tim FitzGeorge ipfr@tfitzgeorge.me.uk --- config/menu/50-firewall.menu | 5 +++++ config/menu/70-log.menu | 5 +++++ html/cgi-bin/logs.cgi/log.dat | 2 ++ langs/en/cgi-bin/en.pl | 27 ++++++++++++++++++++++++++- 4 files changed, 38 insertions(+), 1 deletion(-)
diff --git a/config/menu/50-firewall.menu b/config/menu/50-firewall.menu index 5ec1f67fc..3cfcde835 100644 --- a/config/menu/50-firewall.menu +++ b/config/menu/50-firewall.menu @@ -21,6 +21,11 @@ 'title' => "$Lang::tr{'intrusion detection system'}", 'enabled' => 1, }; + $subfirewall->{'45.ipblacklist'} = {'caption' => $Lang::tr{'ipblacklist'}, + 'uri' => '/cgi-bin/ipblacklist.cgi', + 'title' => "$Lang::tr{'ipblacklist'}", + 'enabled' => 1, + }; $subfirewall->{'50.p2p'} = { 'caption' => $Lang::tr{'p2p block'}, 'uri' => '/cgi-bin/p2p-block.cgi', diff --git a/config/menu/70-log.menu b/config/menu/70-log.menu index 08973de5a..67bbecc1a 100644 --- a/config/menu/70-log.menu +++ b/config/menu/70-log.menu @@ -43,6 +43,11 @@ 'title' => "$Lang::tr{'ids logs'}", 'enabled' => 1 }; + $sublogs->{'55.ipblacklist'} = {'caption' => $Lang::tr{'ipblacklist logs'}, + 'uri' => '/cgi-bin/logs.cgi/ipblacklists.dat', + 'title' => "$Lang::tr{'ipblacklist logs'}", + 'enabled' => 1 + }; $sublogs->{'60.urlfilter'} = { 'caption' => $Lang::tr{'urlfilter logs'}, 'uri' => '/cgi-bin/logs.cgi/urlfilter.dat', diff --git a/html/cgi-bin/logs.cgi/log.dat b/html/cgi-bin/logs.cgi/log.dat index 8ca32d675..ba1a482c8 100644 --- a/html/cgi-bin/logs.cgi/log.dat +++ b/html/cgi-bin/logs.cgi/log.dat @@ -59,6 +59,7 @@ my %sections = ( 'dhcp' => '(dhcpd: )', 'dma' => '(dma: |dma[.*]: |postfix/\w*[\d*]: )', 'guardian' => '(guardian[.*]: )', + 'ipblacklist' => '(ipblacklist: )', 'ipfire' => '(ipfire: )', 'ipsec' => '(ipsec_[\w_]+: |pluto[.*]: |charon: |vpnwatch: )', 'kernel' => '(kernel: (?!DROP_))', @@ -87,6 +88,7 @@ my %trsections = ( 'dhcp' => "$Lang::tr{'dhcp server'}", 'dma' => 'Mail', 'guardian' => "$Lang::tr{'guardian'}", + 'ipblacklist' => "$Lang::tr{'ipblacklist'}", 'ipfire' => 'IPFire', 'ipsec' => 'IPSec', 'kernel' => "$Lang::tr{'kernel'}", diff --git a/langs/en/cgi-bin/en.pl b/langs/en/cgi-bin/en.pl index 3f3e46641..57f52fa48 100644 --- a/langs/en/cgi-bin/en.pl +++ b/langs/en/cgi-bin/en.pl @@ -1,4 +1,4 @@ -%tr = ( +%tr = ( %tr,
'24 hours' => '24 Hours', @@ -1534,6 +1534,31 @@ 'ip alias changed' => 'External IP alias changed', 'ip alias removed' => 'External IP alias removed', 'ip info' => 'IP Information', +'ipblacklist' => 'IP Address Blacklists', +'ipblacklist blacklist settings' => 'Blacklist settings', +'ipblacklist category' => 'Category', +'ipblacklist category application' => 'Application', +'ipblacklist category attacker' => 'Attacker', +'ipblacklist category c and c' => 'Malware C&C', +'ipblacklist category composite' => 'Composite', +'ipblacklist category invalid' => 'Invalid Address', +'ipblacklist category reputation' => 'Reputation', +'ipblacklist category scanner' => 'Scanner', +'ipblacklist disable mid' => 'because it is included in', +'ipblacklist disable post' => '', +'ipblacklist disable pre' => 'Disabling', +'ipblacklist enable' => 'Enable', +'ipblacklist entries' => 'Entries', +'ipblacklist hits' => 'Total number of blacklist hits for', +'ipblacklist id' => 'Blacklist', +'ipblacklist input' => 'Packets Dropped In', +'ipblacklist log list' => 'Firewall log (blacklist)', +'ipblacklist log' => 'Log dropped packets', +'ipblacklist logs' => 'IP Address Blacklist Logs', +'ipblacklist name' => 'Name', +'ipblacklist output' => 'Packets Dropped Out', +'ipblacklist use ipblacklists' => 'Enable IP Blacklists', +'ipblacklist working' => 'Updating IP address blacklists...', 'ipfire has now rebooted' => 'IPFire is rebooting now.', 'ipfire has now shutdown' => 'IPFire is shutting down now.', 'ipfire side' => 'IPFire side:',
getipsetstat Gets information on IPSETs for WUI ipblacklistctrl Allows WUI to call main script as root sources List of blacklists used by main script and WUI
Signed-off-by: Tim FitzGeorge ipfr@tfitzgeorge.me.uk --- config/ipblacklist/sources | 138 +++++++++++++++++++++++++++++++++++++++ src/misc-progs/getipsetstat.c | 25 +++++++ src/misc-progs/ipblacklistctrl.c | 48 ++++++++++++++ 3 files changed, 211 insertions(+) create mode 100644 config/ipblacklist/sources create mode 100644 src/misc-progs/getipsetstat.c create mode 100644 src/misc-progs/ipblacklistctrl.c
diff --git a/config/ipblacklist/sources b/config/ipblacklist/sources new file mode 100644 index 000000000..3cfa7f7d4 --- /dev/null +++ b/config/ipblacklist/sources @@ -0,0 +1,138 @@ +############################################################################ +# # +# IP Address blacklists for IPFire # +# # +# This file contains a list of blacklist sources that will replace the one # +# internal to the updated if it is found at /var/ipfire/blacklist/sources. # +# The intention is to provide a common source of information for both the # +# updater and WUI. # +# # +# The chains created in the packet filter will be named by the top level # +# key and this will also be used in the log message to identify the reason # +# for the dropped packet. # +# # +# The fields are: # +# # +# name The blacklist's full name # +# url URL of the file containing the list # +# info URL giving information about the source # +# parser The parser function used to extract IP addresses from the # +# downloaded list # +# rate Minimum period between checks for updates. Can be specified in # +# days (d), hours (h) or minutes (m) # +# category Used for documentation on the WUI. Can be one of the following # +# 'application' Potentially unwanted applications # +# 'attacker' Generic source of malicious packets # +# 'c and c' Malware Command and Control source # +# 'composite' Composite of other lists # +# 'invalid' Invalid addresses on the public internet # +# 'scanner' Port scanner that is not initself malicious # +# disable Name of another list to disable if this one is enabled. Used # +# when the other list is a subset of this one. # +# # +# The info and category fields are purely for documentation. # +# # +############################################################################ + +%sources = ( 'EMERGING_FWRULE' => { 'name' => 'Emerging Threats Blocklist', + 'url' => 'https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt', + 'info' => 'https://doc.emergingthreats.net/bin/view/Main/EmergingFirewallRules', + 'parser' => 'ip-or-net-list', + 'rate' => '1h', + 'category' => 'composite', + 'disable' => ['FEODO_RECOMMENDED', 'FEODO_IP', 'FEODO_AGGRESIVE', 'SPAMHAUS_DROP', 'DSHIELD'] }, + 'EMERGING_COMPROMISED' => { 'name' => 'Emerging Threats Compromised IPs', + 'url' => 'https://rules.emergingthreats.net/blockrules/compromised-ips.txt', + 'info' => 'https://doc.emergingthreats.net/bin/view/Main/CompromisedHost', + 'parser' => 'ip-or-net-list', + 'rate' => '1h', + 'category' => 'attacker' }, + 'SPAMHAUS_DROP' => { 'name' => "Spamhaus Don't Route or Peer List", + 'url' => 'https://www.spamhaus.org/drop/drop.txt', + 'info' => 'https://www.spamhaus.org/drop/', + 'parser' => 'ip-or-net-list', + 'rate' => '12h', + 'category' => 'reputation' }, + 'SPAMHAUS_EDROP' => { 'name' => "Spamhaus Extended Don't Route or Peer List", + 'url' => 'https://www.spamhaus.org/drop/edrop.txt', + 'info' => 'https://www.spamhaus.org/drop/', + 'parser' => 'ip-or-net-list', + 'rate' => '1h', + 'category' => 'reputation' }, + 'DSHIELD' => { 'name' => 'Dshield.org Recommended Block List', + 'url' => 'https://www.dshield.org/block.txt', + 'info' => 'https://dshield.org/', + 'parser' => 'dshield', + 'rate' => '1h', + 'category' => 'attacker' }, + 'FEODO_RECOMMENDED'=> {'name' => 'Feodo Trojan IP Blocklist (Recommended)', + 'url' => 'https://feodotracker.abuse.ch/downloads/ipblocklist_recommended.txt', + 'info' => 'https://feodotracker.abuse.ch/blocklist', + 'parser' => 'ip-or-net-list', + 'rate' => '5m', + 'category' => 'c and c' }, + 'FEODO_IP' => { 'name' => 'Feodo Trojan IP Blocklist', + 'url' => 'https://feodotracker.abuse.ch/downloads/ipblocklist.txt', + 'info' => 'https://feodotracker.abuse.ch/blocklist', + 'parser' => 'ip-or-net-list', + 'rate' => '5m', + 'category' => 'c and c', + 'disable' => 'FEODO_RECOMMENDED' }, + 'FEODO_AGGRESIVE' => { 'name' => 'Feodo Trojan IP Blocklist (Aggresive)', + 'url' => 'https://feodotracker.abuse.ch/downloads/ipblocklist_aggressive.txt', + 'info' => 'https://feodotracker.abuse.ch/blocklist', + 'parser' => 'ip-or-net-list', + 'rate' => '5m', + 'category' => 'c and c', + 'disable' => ['FEODO_IP', 'FEODO_RECOMMENDED'] }, + 'CIARMY' => { 'name' => 'The CINS Army List', + 'url' => 'https://cinsscore.com/list/ci-badguys.txt', + 'info' => 'https://cinsscore.com/#list', + 'parser' => 'ip-or-net-list', + 'rate' => '15m', + 'category' => 'reputation' }, + 'TOR_ALL' => { 'name' => 'Known TOR Nodes', + 'url' => 'https://www.dan.me.uk/torlist', + 'info' => 'https://www.dan.me.uk/tornodes', + 'parser' => 'ip-or-net-list', + 'rate' => '1h', + 'category' => 'application', + 'disable' => 'TOR_EXIT' }, + 'TOR_EXIT' => { 'name' => 'Known TOR Exit Nodes', + 'url' => 'https://www.dan.me.uk/torlist/?exit', + 'info' => 'https://www.dan.me.uk/tornodes', + 'parser' => 'ip-or-net-list',, + 'rate' => '1h', + 'category' => 'application' }, + 'ALIENVAULT' => { 'name' => 'AlienVault IP Reputation database', + 'url' => 'https://reputation.alienvault.com/reputation.generic', + 'info' => 'https://www.alienvault.com/resource-center/videos/what-is-ip-domain-reputati...', + 'parser' => 'ip-or-net-list', + 'rate' => '1h', + 'category' => 'reputation' }, + 'BOGON' => { 'name' => 'Bogus address list (Martian)', + 'url' => 'https://www.team-cymru.org/Services/Bogons/bogon-bn-agg.txt', + 'info' => 'https://www.team-cymru.com/bogon-reference.html', + 'parser' => 'ip-or-net-list', + 'rate' => '1d', + 'category' => 'invalid' }, + 'BOGON_FULL' => { 'name' => 'Full Bogus Address List', + 'url' => 'https://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt', + 'info' => 'https://www.team-cymru.com/bogon-reference.html', + 'parser' => 'ip-or-net-list', + 'rate' => '4h', + 'category' => 'invalid', + 'disable' => 'BOGON' }, + 'SHODAN' => { 'name' => 'ISC Shodan scanner blacklist', + 'url' => 'https://isc.sans.edu/api/threatlist/shodan?tab', + 'info' => 'https://isc.sans.edu', + 'parser' => 'ip-or-net-list', + 'rate' => '1d', + 'category' => 'scanner' }, + 'BLOCKLIST_DE' => { 'name' => 'Blocklist.de all attacks list', + 'url' => 'https://lists.blocklist.de/lists/all.txt', + 'info' => 'https://www.blocklist.de', + 'parser' => 'ip-or-net-list', + 'rate' => '30m', + 'category' => 'attacker' } + ); diff --git a/src/misc-progs/getipsetstat.c b/src/misc-progs/getipsetstat.c new file mode 100644 index 000000000..781bfc55b --- /dev/null +++ b/src/misc-progs/getipsetstat.c @@ -0,0 +1,25 @@ +/* IPFire helper program - GetIPSetStat + * + * Get the list from IPSET LIST + * + */ + +#include <stdio.h> +#include <string.h> +#include <unistd.h> +#include <stdlib.h> +#include <sys/types.h> +#include <fcntl.h> +#include "setuid.h" + + +int main(void) +{ + if (!(initsetuid())) + exit(1); + + safe_system("/usr/sbin/ipset list -t -f /var/tmp/ipsets.txt"); + safe_system("chown nobody:nobody /var/tmp/ipsets.txt"); + + return 0; +} diff --git a/src/misc-progs/ipblacklistctrl.c b/src/misc-progs/ipblacklistctrl.c new file mode 100644 index 000000000..7536b1e97 --- /dev/null +++ b/src/misc-progs/ipblacklistctrl.c @@ -0,0 +1,48 @@ +/* This file is part of the IPFire Firewall. + * + * This program is distributed under the terms of the GNU General Public + * Licence. See the file COPYING for details. + * + */ + +#include <stdlib.h> +#include <stdio.h> +#include <string.h> +#include <unistd.h> +#include <sys/types.h> +#include <fcntl.h> +#include "setuid.h" + +int main(int argc, char *argv[]) { + + if (!(initsetuid())) + exit(1); + + if (argc < 2) { + fprintf(stderr, "\nNo argument given.\n" + "ipblacklistctrl (update|restore|log-on|log-off|" + "enable|disable)\n\n"); + exit(1); + } + + if (strcmp(argv[1], "update") == 0) { + safe_system("/usr/local/bin/ipblacklist update >/dev/null 2>&1 &"); + } else if (strcmp(argv[1], "restore") == 0) { + safe_system("/usr/local/bin/ipblacklist restore >/dev/null 2>&1 &"); + } else if (strcmp(argv[1], "log-on") == 0) { + safe_system("/usr/local/bin/ipblacklist log-on >/dev/null 2>&1 &"); + } else if (strcmp(argv[1], "log-off") == 0) { + safe_system("/usr/local/bin/ipblacklist log-off >/dev/null 2>&1 &"); + } else if (strcmp(argv[1], "enable") == 0) { + safe_system("/usr/local/bin/ipblacklist enable >/dev/null 2>&1 &"); + } else if (strcmp(argv[1], "disable") == 0) { + safe_system("/usr/local/bin/ipblacklist disable >/dev/null 2>&1 &"); + } else { + fprintf(stderr, "\nBad argument given.\n" + "ipblacklistctrl (update|restore|log-on|log-off|" + "enable|disable)\n\n"); + exit(1); + } + + return 0; +}
backup.pl Restart when restoring backup ipblacklist ) Adds ipblacklist stats, events and errors ipblacklist.conf ) to the daily log summary include Add blacklists and settings to backups firewall Add main IPTables used to invoke IPSet
Signed-off-by: Tim FitzGeorge ipfr@tfitzgeorge.me.uk --- config/backup/backup.pl | 1 + config/backup/include | 2 + config/logwatch/ipblacklist | 105 +++++++++++++++++++++++++++++++++++++++ config/logwatch/ipblacklist.conf | 34 +++++++++++++ src/initscripts/system/firewall | 12 +++++ 5 files changed, 154 insertions(+) create mode 100644 config/logwatch/ipblacklist create mode 100644 config/logwatch/ipblacklist.conf
diff --git a/config/backup/backup.pl b/config/backup/backup.pl index 5b5734044..d253bd90f 100644 --- a/config/backup/backup.pl +++ b/config/backup/backup.pl @@ -141,6 +141,7 @@ restore_backup() {
# Reload firewall firewallctrl + /usr/local/bin/ipblacklistctrl restore
# Convert old OpenVPN CCD files (CN change, Core Update 75) convert-ovpn diff --git a/config/backup/include b/config/backup/include index 5db452cda..1a63ef1b1 100644 --- a/config/backup/include +++ b/config/backup/include @@ -39,6 +39,7 @@ /var/ipfire/ethernet/wireless /var/ipfire/firewall /var/ipfire/fwhosts +/var/ipfire/ipblacklist/modified /var/ipfire/main/* /var/ipfire/ovpn /var/ipfire/ovpn/collectd.vpn @@ -54,6 +55,7 @@ /var/ipfire/time/ /var/ipfire/urlfilter /var/ipfire/vpn +/var/lib/ipblacklist /var/lib/suricata /var/log/ip-acct/* /var/log/rrd/* diff --git a/config/logwatch/ipblacklist b/config/logwatch/ipblacklist new file mode 100644 index 000000000..6d6c46188 --- /dev/null +++ b/config/logwatch/ipblacklist @@ -0,0 +1,105 @@ +########################################################################### +# ipblacklist script for Logwatch +# Analyzes the IPFire IP Blacklist log +# +######################################################################### + +######################################################## +## Copyright (c) 2008 Lars Skj�rlund +## Covered under the included MIT/X-Consortium License: +## http://www.opensource.org/licenses/mit-license.php +## All modifications and contributions by other persons to +## this script are assumed to have been donated to the +## Logwatch project and thus assume the above copyright +## and licensing terms. If you want to make contributions +## under your own copyright or a different license this +## must be explicitly stated in the contribution and the +## Logwatch project reserves the right to not accept such +## contributions. If you have made significant +## contributions to this script and want to claim +## copyright please contact logwatch-devel@lists.sourceforge.net. +######################################################### + +######################################################################### +# Files - all shown with default paths: +# +# /usr/share/logwatch/default.conf/logfiles/messages.conf +# /usr/share/logwatch/dist.conf/services/blacklist.conf +# /usr/share/logwatch/scripts/services/ipblacklist (this file) +# +# ... and of course +# +# /var/log/messages +######################################################################### + +use Logwatch ':dates'; + +my $Detail = $ENV{'LOGWATCH_DETAIL_LEVEL'}; + +my $SearchDate; + +my %Updates; +my %Errors; + +$SearchDate = TimeFilter("%b %e"); + +while (defined(my $ThisLine = <STDIN>)) +{ + next unless ($ThisLine =~ m/^\s*\w+\s+\w+\s+(..:..:..) .* ipblacklist: (.*)/); + + my $text = $2; + + if ($text =~ m/Updated (\w+) blacklist with (\d+) changes/) + { + $Updates{$1}{updates}++; + $Updates{$1}{changes} += $2; + } + elsif ($text !~ m/Starting IP Blacklists/ and + $text !~ m/Starting IP Blacklist processing/ and + $text !~ m/Stopping IP Blacklists/ and + $text !~ m/Deleting IP Blacklists/ and + $text !~ m/Finished IP Blacklist processing/ and + $text !~ m/Create IPTables chains for blacklist/ and + $text !~ m/Delete IPTables chains for blacklist/ and + $text !~ m/Add IP Address Blacklist update to crontab/ and + $text !~ m/Enable IP Address Blacklist update in crontab/ and + $text !~ m/Disable IP Address Blacklist updates/ and + $text !~ m/Restoring blacklist / and + $text !~ m/Blacklist \w+ changed type/ and + $text !~ m/Blacklist \w+ changed size/ and + $text !~ m/Enabling IP Blacklist logging/ and + $text !~ m/Disabling IP Blacklist logging/ ) + { + $Errors{$text}++; + } +} + +##################################################################### + +if (keys %Updates) +{ + print "\nThe following block lists were updated:\n"; + foreach my $Lists (sort keys %Updates) + { + print " $Lists: $Updates{$Lists}{updates} Time(s) - $Updates{$Lists}{changes} change(s)\n"; + } +} + +if (keys %Errors) +{ + print "\nThe following errors were detected:\n"; + + foreach my $Text (keys %Errors) + { + print " $Text: $Errors{$Text} Time(s)\n"; + } +} + +exit(0); + +# vi: shiftwidth=3 tabstop=3 syntax=perl et +# Local Variables: +# mode: perl +# perl-indent-level: 3 +# indent-tabs-mode: nil +# End: diff --git a/config/logwatch/ipblacklist.conf b/config/logwatch/ipblacklist.conf new file mode 100644 index 000000000..ed0ecc5f1 --- /dev/null +++ b/config/logwatch/ipblacklist.conf @@ -0,0 +1,34 @@ +######################################################################### +# ids-update script for Logwatch +# Analyzes the IPFire IP Blacklist update log +# +# Version: 1.0.0 +# Initial release +# +######################################################################### + +######################################################################### +# This script is subject to the same copyright as Logwatch itself +######################################################################### + +######################################################################### +# Files - all shown with default paths: +# +# /usr/share/logwatch/default.conf/logfiles/messages.conf +# /usr/share/logwatch/dist.conf/services/blacklist.conf (this file) +# /usr/share/logwatch/scripts/services/blacklist +# +# ... and of course +# +# /var/log/messages +######################################################################### + + +Title = "IP Blacklist" + +# Which logfile group... +LogFile = messages + +*applystddate + +# vi: shiftwidth=3 tabstop=3 et diff --git a/src/initscripts/system/firewall b/src/initscripts/system/firewall index ab144ea18..8e7c1c7ef 100644 --- a/src/initscripts/system/firewall +++ b/src/initscripts/system/firewall @@ -196,6 +196,14 @@ iptables_init() { iptables -A FORWARD -i tun+ -j OVPNBLOCK iptables -A FORWARD -o tun+ -j OVPNBLOCK
+ # IP Address Blacklist chains + iptables -N BLACKLISTIN + iptables -N BLACKLISTOUT + iptables -A INPUT ! -p icmp -j BLACKLISTIN + iptables -A FORWARD ! -p icmp -j BLACKLISTIN + iptables -A FORWARD ! -p icmp -j BLACKLISTOUT + iptables -A OUTPUT ! -p icmp -j BLACKLISTOUT + # IPS (suricata) chains iptables -N IPS_INPUT iptables -N IPS_FORWARD @@ -382,6 +390,9 @@ iptables_init() { # run captivectrl /usr/local/bin/captivectrl
+ # run IP Blacklist start + /usr/local/bin/ipblacklist start + # POLICY CHAIN iptables -N POLICYIN iptables -A INPUT -j POLICYIN @@ -504,6 +515,7 @@ case "$1" in evaluate_retval ;; restart) + /usr/local/bin/ipblacklist stop $0 start ;; *)
Signed-off-by: Tim FitzGeorge ipfr@tfitzgeorge.me.uk --- config/rootfiles/common/aarch64/stage2 | 1 + config/rootfiles/common/configroot | 2 ++ config/rootfiles/common/ipblacklist-sources | 1 + config/rootfiles/common/logwatch | 2 ++ config/rootfiles/common/misc-progs | 2 ++ config/rootfiles/common/stage2 | 1 + config/rootfiles/common/web-user-interface | 3 ++ config/rootfiles/common/x86_64/stage2 | 1 + lfs/configroot | 4 +-- lfs/ipblacklist-sources | 53 +++++++++++++++++++++++++++++ lfs/logwatch | 2 ++ make.sh | 1 + src/misc-progs/Makefile | 2 +- 13 files changed, 72 insertions(+), 3 deletions(-) create mode 100644 config/rootfiles/common/ipblacklist-sources create mode 100644 lfs/ipblacklist-sources
diff --git a/config/rootfiles/common/aarch64/stage2 b/config/rootfiles/common/aarch64/stage2 index 82e2c20d0..e78137d08 100644 --- a/config/rootfiles/common/aarch64/stage2 +++ b/config/rootfiles/common/aarch64/stage2 @@ -96,6 +96,7 @@ usr/local/bin/convert-dns-settings usr/local/bin/convert-ovpn usr/local/bin/filesystem-cleanup usr/local/bin/hddshutdown +usr/local/bin/ipblacklist usr/local/bin/ipsec-interfaces usr/local/bin/makegraphs usr/local/bin/qosd diff --git a/config/rootfiles/common/configroot b/config/rootfiles/common/configroot index 67c4abc75..e2ebf2c84 100644 --- a/config/rootfiles/common/configroot +++ b/config/rootfiles/common/configroot @@ -82,6 +82,8 @@ var/ipfire/geoip-functions.pl var/ipfire/graphs.pl var/ipfire/header.pl var/ipfire/ids-functions.pl +var/ipfire/ipblacklist +#var/ipfire/ipblacklist/settings var/ipfire/isdn #var/ipfire/isdn/settings var/ipfire/key diff --git a/config/rootfiles/common/ipblacklist-sources b/config/rootfiles/common/ipblacklist-sources new file mode 100644 index 000000000..7f54b1bbf --- /dev/null +++ b/config/rootfiles/common/ipblacklist-sources @@ -0,0 +1 @@ +var/ipfire/ipblacklist/sources diff --git a/config/rootfiles/common/logwatch b/config/rootfiles/common/logwatch index c47fb4199..8b4810d97 100644 --- a/config/rootfiles/common/logwatch +++ b/config/rootfiles/common/logwatch @@ -192,6 +192,7 @@ usr/share/logwatch/default.conf/services/zz-sys.conf usr/share/logwatch/dist.conf/logfiles usr/share/logwatch/dist.conf/services usr/share/logwatch/dist.conf/services/dialup.conf +usr/share/logwatch/dist.conf/services/ipblacklist.conf #usr/share/logwatch/lib usr/share/logwatch/lib/Logwatch.pm #usr/share/logwatch/scripts @@ -256,6 +257,7 @@ usr/share/logwatch/scripts/services/http usr/share/logwatch/scripts/services/imapd #usr/share/logwatch/scripts/services/in.qpopper usr/share/logwatch/scripts/services/init +usr/share/logwatch/scripts/services/ipblacklist usr/share/logwatch/scripts/services/ipop3d usr/share/logwatch/scripts/services/iptables usr/share/logwatch/scripts/services/kernel diff --git a/config/rootfiles/common/misc-progs b/config/rootfiles/common/misc-progs index c48a474b2..d17f3dd80 100644 --- a/config/rootfiles/common/misc-progs +++ b/config/rootfiles/common/misc-progs @@ -10,8 +10,10 @@ usr/local/bin/extrahdctrl usr/local/bin/fireinfoctrl usr/local/bin/firewallctrl usr/local/bin/getconntracktable +usr/local/bin/getipsetstat usr/local/bin/getipstat #usr/local/bin/iowrap +usr/local/bin/ipblacklistctrl usr/local/bin/ipfirereboot usr/local/bin/ipsecctrl usr/local/bin/launch-ether-wake diff --git a/config/rootfiles/common/stage2 b/config/rootfiles/common/stage2 index 8067df39b..78c55338d 100644 --- a/config/rootfiles/common/stage2 +++ b/config/rootfiles/common/stage2 @@ -95,6 +95,7 @@ usr/local/bin/convert-dns-settings usr/local/bin/convert-ovpn usr/local/bin/filesystem-cleanup usr/local/bin/hddshutdown +usr/local/bin/ipblacklist usr/local/bin/ipsec-interfaces usr/local/bin/makegraphs usr/local/bin/qosd diff --git a/config/rootfiles/common/web-user-interface b/config/rootfiles/common/web-user-interface index a3636002e..a3434128d 100644 --- a/config/rootfiles/common/web-user-interface +++ b/config/rootfiles/common/web-user-interface @@ -34,6 +34,7 @@ srv/web/ipfire/cgi-bin/hardwaregraphs.cgi srv/web/ipfire/cgi-bin/hosts.cgi srv/web/ipfire/cgi-bin/ids.cgi srv/web/ipfire/cgi-bin/index.cgi +srv/web/ipfire/cgi-bin/ipblacklist.cgi srv/web/ipfire/cgi-bin/ipinfo.cgi srv/web/ipfire/cgi-bin/iptables.cgi srv/web/ipfire/cgi-bin/logs.cgi @@ -44,8 +45,10 @@ srv/web/ipfire/cgi-bin/logs.cgi/firewalllogcountry.dat srv/web/ipfire/cgi-bin/logs.cgi/firewalllogip.dat srv/web/ipfire/cgi-bin/logs.cgi/firewalllogport.dat srv/web/ipfire/cgi-bin/logs.cgi/ids.dat +srv/web/ipfire/cgi-bin/logs.cgi/ipblacklists.dat srv/web/ipfire/cgi-bin/logs.cgi/log.dat srv/web/ipfire/cgi-bin/logs.cgi/proxylog.dat +srv/web/ipfire/cgi-bin/logs.cgi/showrequestfromblacklist.dat srv/web/ipfire/cgi-bin/logs.cgi/showrequestfromcountry.dat srv/web/ipfire/cgi-bin/logs.cgi/showrequestfromip.dat srv/web/ipfire/cgi-bin/logs.cgi/showrequestfromport.dat diff --git a/config/rootfiles/common/x86_64/stage2 b/config/rootfiles/common/x86_64/stage2 index 026532b8f..5aa177008 100644 --- a/config/rootfiles/common/x86_64/stage2 +++ b/config/rootfiles/common/x86_64/stage2 @@ -97,6 +97,7 @@ usr/local/bin/convert-dns-settings usr/local/bin/convert-ovpn usr/local/bin/filesystem-cleanup usr/local/bin/hddshutdown +usr/local/bin/ipblacklist usr/local/bin/ipsec-interfaces usr/local/bin/makegraphs usr/local/bin/qosd diff --git a/lfs/configroot b/lfs/configroot index 2c9dbe0e3..90b90eb3c 100644 --- a/lfs/configroot +++ b/lfs/configroot @@ -51,7 +51,7 @@ $(TARGET) :
# Create all directories for i in addon-lang auth backup ca captive certs connscheduler crls ddns dhcp dhcpc dns dnsforward \ - ethernet extrahd/bin fwlogs fwhosts firewall isdn key langs logging mac main \ + ethernet extrahd/bin fwlogs fwhosts firewall ipblacklist isdn key langs logging mac main \ menu.d modem optionsfw \ ovpn patches pakfire portfw ppp private proxy/advanced/cre \ proxy/calamaris/bin qos/bin red remote sensors suricata time \ @@ -65,7 +65,7 @@ $(TARGET) : captive/settings captive/agb.txt captive/clients captive/voucher_out certs/index.txt certs/index.txt.attr ddns/config ddns/settings ddns/ipcache dhcp/settings \ dhcp/fixleases dhcp/advoptions dhcp/dhcpd.conf.local dns/settings dns/servers dnsforward/config ethernet/aliases ethernet/settings ethernet/known_nics ethernet/scanned_nics \ ethernet/wireless extrahd/scan extrahd/devices extrahd/partitions extrahd/settings firewall/settings firewall/config firewall/geoipblock firewall/input firewall/outgoing \ - fwhosts/customnetworks fwhosts/customhosts fwhosts/customgroups fwhosts/customservicegrp fwhosts/customgeoipgrp fwlogs/ipsettings fwlogs/portsettings \ + fwhosts/customnetworks fwhosts/customhosts fwhosts/customgroups fwhosts/customservicegrp fwhosts/customgeoipgrp fwlogs/ipsettings fwlogs/portsettings ipblacklist/settings \ isdn/settings mac/settings main/hosts main/routing main/security main/settings optionsfw/settings \ ovpn/ccd.conf ovpn/ccdroute ovpn/ccdroute2 pakfire/settings portfw/config ppp/settings-1 ppp/settings-2 ppp/settings-3 ppp/settings-4 \ ppp/settings-5 ppp/settings proxy/settings proxy/squid.conf proxy/advanced/settings proxy/advanced/cre/enable remote/settings qos/settings qos/classes qos/subclasses qos/level7config qos/portconfig \ diff --git a/lfs/ipblacklist-sources b/lfs/ipblacklist-sources new file mode 100644 index 000000000..c9431285d --- /dev/null +++ b/lfs/ipblacklist-sources @@ -0,0 +1,53 @@ +############################################################################### +# # +# IPFire.org - A linux based firewall # +# Copyright (C) 2007 Michael Tremer & Christian Schmidt # +# # +# This program is free software: you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation, either version 3 of the License, or # +# (at your option) any later version. # +# # +# This program is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with this program. If not, see http://www.gnu.org/licenses/. # +# # +############################################################################### + +############################################################################### +# Definitions +############################################################################### + +include Config + +VER = ipfire + +THISAPP = ipblacklist-sources +TARGET = $(DIR_INFO)/$(THISAPP) + +############################################################################### +# Top-level Rules +############################################################################### + +install : $(TARGET) + +check : + +download : + +md5 : + +############################################################################### +# Installation Details +############################################################################### + +$(TARGET) : + @$(PREBUILD) + mkdir -p /var/ipfire/ipblacklist + install -v -m 0644 $(DIR_SRC)/config/ipblacklist/sources /var/ipfire/ipblacklist + + @$(POSTBUILD) diff --git a/lfs/logwatch b/lfs/logwatch index a980b1b40..a1b02f9d7 100644 --- a/lfs/logwatch +++ b/lfs/logwatch @@ -93,6 +93,8 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) # done cp -f $(DIR_SRC)/config/logwatch/dialup /usr/share/logwatch/scripts/services/dialup cp -f $(DIR_SRC)/config/logwatch/dialup.conf /usr/share/logwatch/dist.conf/services/dialup.conf + cp -f $(DIR_SRC)/config/logwatch/ipblacklist /usr/share/logwatch/scripts/services/ipblacklist + cp -f $(DIR_SRC)/config/logwatch/ipblacklist.conf /usr/share/logwatch/dist.conf/services/ipblacklist.conf -mkdir -p /var/cache/logwatch chmod -v 777 /var/cache/logwatch diff --git a/make.sh b/make.sh index f507c5584..8e062140b 100755 --- a/make.sh +++ b/make.sh @@ -1645,6 +1645,7 @@ buildipfire() { lfsmake2 speedtest-cli lfsmake2 rfkill lfsmake2 amazon-ssm-agent + lfsmake2 ipblacklist-sources }
buildinstaller() { diff --git a/src/misc-progs/Makefile b/src/misc-progs/Makefile index bea54e773..60b3965e0 100644 --- a/src/misc-progs/Makefile +++ b/src/misc-progs/Makefile @@ -32,7 +32,7 @@ SUID_PROGS = squidctrl sshctrl ipfirereboot \ smartctrl clamavctrl addonctrl pakfire mpfirectrl wlanapctrl \ setaliases urlfilterctrl updxlratorctrl fireinfoctrl rebuildroutes \ getconntracktable wirelessclient torctrl ddnsctrl unboundctrl \ - captivectrl + captivectrl ipblacklistctrl getipsetstat SUID_UPDX = updxsetperms
OBJS = $(patsubst %,%.o,$(PROGS) $(SUID_PROGS))
Hi Tim,
This has now been sitting in my inbox for almost a month. Nobody else has commented on it.
I tried a couple of times to review this, but I do not even know where to start. This patch is just too large.
First of all, although it is split across multiple patches, it is one large patch. It is not split by changes that belong together, but just one patch per file.
I have no idea what has changed since I last time looked at the code. It has been months since the previous patchset and I do not remember each individual line.
You Git repository also does not have any changes any more. The branch has been reset.
Are you able to send a diff with the changes since the first patchset?
How can we finally bring this into the distribution?
Best, -Michael
On 27 Apr 2020, at 15:31, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Implements downloading of IP address blacklists and implementing them as IPSets. A separate IPSet is used for each blacklist; this simplifies handling of overlaps between different lists. Traffic to or from the red0/ppp0 interface is checked against the IPSets. The check is placed before the IPS check as the IPSet check is much lighter on CPU use which means that overall CPU use is reduced.
The available lists are defined in a separate file. A WUI page allows the desired lists to be enabled. A minimum update check interval is defined for each blacklist in the definition file.
Changes since Version 1:
- Changed Dshield download URL to preferred address.
- Removed Abuse.ch blacklist (discontinued).
- Removed Talos Malicious blacklist (not for production use).
- Added Feodo recommended blacklist.
- Added blocklist.de all blacklist.
- Updated ignored messages in logwatch.
- Modified sources file 'rate' to allow unit to be specified.
- Updated sources file 'disable' to allow list to be specified.
- Removed autoblacklist.
- Added WUI log pages.
- Removed status from settings WUI page.
Tim FitzGeorge (8): ipblacklist: Main script ipblacklist: WUI Settings page ipblacklist: WUI Log page ipblacklist: WUI Log details page ipblacklist: WUI menus, language file etc ipblacklist: Ancillary files ipblacklist: Modifications to system ipblacklist: Build infrastructure
config/backup/backup.pl | 1 + config/backup/include | 2 + config/ipblacklist/sources | 138 ++ config/logwatch/ipblacklist | 105 ++ config/logwatch/ipblacklist.conf | 34 + config/menu/50-firewall.menu | 5 + config/menu/70-log.menu | 5 + config/rootfiles/common/aarch64/stage2 | 1 + config/rootfiles/common/configroot | 2 + config/rootfiles/common/ipblacklist-sources | 1 + config/rootfiles/common/logwatch | 2 + config/rootfiles/common/misc-progs | 2 + config/rootfiles/common/stage2 | 1 + config/rootfiles/common/web-user-interface | 3 + config/rootfiles/common/x86_64/stage2 | 1 + html/cgi-bin/ipblacklist.cgi | 463 +++++++ html/cgi-bin/logs.cgi/ipblacklists.dat | 363 +++++ html/cgi-bin/logs.cgi/log.dat | 2 + html/cgi-bin/logs.cgi/showrequestfromblacklist.dat | 415 ++++++ langs/en/cgi-bin/en.pl | 27 +- lfs/configroot | 4 +- lfs/ipblacklist-sources | 53 + lfs/logwatch | 2 + make.sh | 1 + src/initscripts/system/firewall | 12 + src/misc-progs/Makefile | 2 +- src/misc-progs/getipsetstat.c | 25 + src/misc-progs/ipblacklistctrl.c | 48 + src/scripts/ipblacklist | 1382 ++++++++++++++++++++ 29 files changed, 3098 insertions(+), 4 deletions(-) create mode 100644 config/ipblacklist/sources create mode 100644 config/logwatch/ipblacklist create mode 100644 config/logwatch/ipblacklist.conf create mode 100644 config/rootfiles/common/ipblacklist-sources create mode 100644 html/cgi-bin/ipblacklist.cgi create mode 100755 html/cgi-bin/logs.cgi/ipblacklists.dat create mode 100755 html/cgi-bin/logs.cgi/showrequestfromblacklist.dat create mode 100644 lfs/ipblacklist-sources create mode 100644 src/misc-progs/getipsetstat.c create mode 100644 src/misc-progs/ipblacklistctrl.c create mode 100755 src/scripts/ipblacklist
-- 2.16.4
Hi Michael,
The diff between the V1 and V2 patches is at the end of this email. I've not included html/cgi-bin/logs.cgi/ipblacklists.dat html/cgi-bin/logs.cgi/showrequestfromblacklist.dat (V2 patches 0003 and 0004) as they're completely new at V2 - they're modifications of the other similar log files.
There are a lot of lines here, but a lot of them are deletions due to moving the status etc. into the logging.
I hope the is OK.
Tim
On 16/05/2020 10:40, Michael Tremer wrote:
Hi Tim,
This has now been sitting in my inbox for almost a month. Nobody else has commented on it.
I tried a couple of times to review this, but I do not even know where to start. This patch is just too large.
First of all, although it is split across multiple patches, it is one large patch. It is not split by changes that belong together, but just one patch per file.
I have no idea what has changed since I last time looked at the code. It has been months since the previous patchset and I do not remember each individual line.
You Git repository also does not have any changes any more. The branch has been reset.
Are you able to send a diff with the changes since the first patchset?
How can we finally bring this into the distribution?
Best, -Michael
On 27 Apr 2020, at 15:31, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Implements downloading of IP address blacklists and implementing them as IPSets. A separate IPSet is used for each blacklist; this simplifies handling of overlaps between different lists. Traffic to or from the red0/ppp0 interface is checked against the IPSets. The check is placed before the IPS check as the IPSet check is much lighter on CPU use which means that overall CPU use is reduced.
The available lists are defined in a separate file. A WUI page allows the desired lists to be enabled. A minimum update check interval is defined for each blacklist in the definition file.
Changes since Version 1:
- Changed Dshield download URL to preferred address.
- Removed Abuse.ch blacklist (discontinued).
- Removed Talos Malicious blacklist (not for production use).
- Added Feodo recommended blacklist.
- Added blocklist.de all blacklist.
- Updated ignored messages in logwatch.
- Modified sources file 'rate' to allow unit to be specified.
- Updated sources file 'disable' to allow list to be specified.
- Removed autoblacklist.
- Added WUI log pages.
- Removed status from settings WUI page.
Tim FitzGeorge (8): ipblacklist: Main script ipblacklist: WUI Settings page ipblacklist: WUI Log page ipblacklist: WUI Log details page ipblacklist: WUI menus, language file etc ipblacklist: Ancillary files ipblacklist: Modifications to system ipblacklist: Build infrastructure
config/backup/backup.pl | 1 + config/backup/include | 2 + config/ipblacklist/sources | 138 ++ config/logwatch/ipblacklist | 105 ++ config/logwatch/ipblacklist.conf | 34 + config/menu/50-firewall.menu | 5 + config/menu/70-log.menu | 5 + config/rootfiles/common/aarch64/stage2 | 1 + config/rootfiles/common/configroot | 2 + config/rootfiles/common/ipblacklist-sources | 1 + config/rootfiles/common/logwatch | 2 + config/rootfiles/common/misc-progs | 2 + config/rootfiles/common/stage2 | 1 + config/rootfiles/common/web-user-interface | 3 + config/rootfiles/common/x86_64/stage2 | 1 + html/cgi-bin/ipblacklist.cgi | 463 +++++++ html/cgi-bin/logs.cgi/ipblacklists.dat | 363 +++++ html/cgi-bin/logs.cgi/log.dat | 2 + html/cgi-bin/logs.cgi/showrequestfromblacklist.dat | 415 ++++++ langs/en/cgi-bin/en.pl | 27 +- lfs/configroot | 4 +- lfs/ipblacklist-sources | 53 + lfs/logwatch | 2 + make.sh | 1 + src/initscripts/system/firewall | 12 + src/misc-progs/Makefile | 2 +- src/misc-progs/getipsetstat.c | 25 + src/misc-progs/ipblacklistctrl.c | 48 + src/scripts/ipblacklist | 1382 ++++++++++++++++++++ 29 files changed, 3098 insertions(+), 4 deletions(-) create mode 100644 config/ipblacklist/sources create mode 100644 config/logwatch/ipblacklist create mode 100644 config/logwatch/ipblacklist.conf create mode 100644 config/rootfiles/common/ipblacklist-sources create mode 100644 html/cgi-bin/ipblacklist.cgi create mode 100755 html/cgi-bin/logs.cgi/ipblacklists.dat create mode 100755 html/cgi-bin/logs.cgi/showrequestfromblacklist.dat create mode 100644 lfs/ipblacklist-sources create mode 100644 src/misc-progs/getipsetstat.c create mode 100644 src/misc-progs/ipblacklistctrl.c create mode 100755 src/scripts/ipblacklist
-- 2.16.4
diff --git a/config/firewall/firewall-policy b/config/firewall/firewall-policy index 1198d120f..21165e933 100755 --- a/config/firewall/firewall-policy +++ b/config/firewall/firewall-policy @@ -22,7 +22,6 @@ eval $(/usr/local/bin/readhash /var/ipfire/ethernet/settings) eval $(/usr/local/bin/readhash /var/ipfire/firewall/settings) eval $(/usr/local/bin/readhash /var/ipfire/optionsfw/settings) -eval $(/usr/local/bin/readhash /var/ipfire/ipblacklist/settings)
function iptables() { /sbin/iptables --wait "$@" @@ -98,10 +97,6 @@ case "${HAVE_OPENVPN},${POLICY}" in ;; esac
-if [ "${AUTOBLACKLIST}" = "on" ]; then - iptables -A POLICYIN -i ${IFACE} -m hashlimit --hashlimit-mode srcip --hashlimit-above ${BLOCK_THRESHOLD}/hour --hashlimit-name AUTOBLACKLIST -j SET --add-set AUTOBLACKLIST src -fi - case "${FWPOLICY2}" in REJECT) if [ "${DROPINPUT}" = "on" ]; then diff --git a/config/ipblacklist/sources b/config/ipblacklist/sources index ab991e12a..3cfa7f7d4 100644 --- a/config/ipblacklist/sources +++ b/config/ipblacklist/sources @@ -13,139 +13,126 @@ # # # The fields are: # # # -# name The blacklist's full name # -# url URL of the file containing the list # -# info URL giving information about the source # -# parser The parser function used to extract IP addresses from the # -# downloaded list # -# method Method used to download updates. # -# rate Minimum number of hours between checks for updates # -# safe 'yes' if the list is unlikely to contain addresses that can be # -# used for legitimate traffic, or 'no' otherwise # -# disable Name of another list to disable if this one is enabled. Used # -# when the other list is a subset of this one. # +# name The blacklist's full name # +# url URL of the file containing the list # +# info URL giving information about the source # +# parser The parser function used to extract IP addresses from the # +# downloaded list # +# rate Minimum period between checks for updates. Can be specified in # +# days (d), hours (h) or minutes (m) # +# category Used for documentation on the WUI. Can be one of the following # +# 'application' Potentially unwanted applications # +# 'attacker' Generic source of malicious packets # +# 'c and c' Malware Command and Control source # +# 'composite' Composite of other lists # +# 'invalid' Invalid addresses on the public internet # +# 'scanner' Port scanner that is not initself malicious # +# disable Name of another list to disable if this one is enabled. Used # +# when the other list is a subset of this one. # # # -# The info and safe fields are purely for documentation. # -# # -# Note that the Emerging Threats blacklist is a composite list containing # -# addresses from some of the other lists. It is unnecessary to enable # -# this list if the other lists are enabled. # +# The info and category fields are purely for documentation. # # # ############################################################################
%sources = ( 'EMERGING_FWRULE' => { 'name' => 'Emerging Threats Blocklist', 'url' => 'https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt', 'info' => 'https://doc.emergingthreats.net/bin/view/Main/EmergingFirewallRules', - 'parser' => 'text-with-hash-comments', - 'method' => 'check-header-time', - 'rate' => 1, - 'safe' => 'no' }, + 'parser' => 'ip-or-net-list', + 'rate' => '1h', + 'category' => 'composite', + 'disable' => ['FEODO_RECOMMENDED', 'FEODO_IP', 'FEODO_AGGRESIVE', 'SPAMHAUS_DROP', 'DSHIELD'] }, 'EMERGING_COMPROMISED' => { 'name' => 'Emerging Threats Compromised IPs', 'url' => 'https://rules.emergingthreats.net/blockrules/compromised-ips.txt', 'info' => 'https://doc.emergingthreats.net/bin/view/Main/CompromisedHost', - 'parser' => 'text-with-hash-comments', - 'method' => 'check-header-time', - 'rate' => 1, - 'safe' => 'no' }, + 'parser' => 'ip-or-net-list', + 'rate' => '1h', + 'category' => 'attacker' }, 'SPAMHAUS_DROP' => { 'name' => "Spamhaus Don't Route or Peer List", 'url' => 'https://www.spamhaus.org/drop/drop.txt', 'info' => 'https://www.spamhaus.org/drop/', - 'parser' => 'text-with-semicolon-comments', - 'method' => 'check-header-time', - 'rate' => 12, - 'safe' => 'yes' }, + 'parser' => 'ip-or-net-list', + 'rate' => '12h', + 'category' => 'reputation' }, 'SPAMHAUS_EDROP' => { 'name' => "Spamhaus Extended Don't Route or Peer List", 'url' => 'https://www.spamhaus.org/drop/edrop.txt', 'info' => 'https://www.spamhaus.org/drop/', - 'parser' => 'text-with-semicolon-comments', - 'method' => 'check-header-time', - 'rate' => 1, - 'safe' => 'no' }, + 'parser' => 'ip-or-net-list', + 'rate' => '1h', + 'category' => 'reputation' }, 'DSHIELD' => { 'name' => 'Dshield.org Recommended Block List', 'url' => 'https://www.dshield.org/block.txt', 'info' => 'https://dshield.org/', 'parser' => 'dshield', - 'method' => 'check-header-time', - 'rate' => 2, - 'safe' => 'no' }, + 'rate' => '1h', + 'category' => 'attacker' }, + 'FEODO_RECOMMENDED'=> {'name' => 'Feodo Trojan IP Blocklist (Recommended)', + 'url' => 'https://feodotracker.abuse.ch/downloads/ipblocklist_recommended.txt', + 'info' => 'https://feodotracker.abuse.ch/blocklist', + 'parser' => 'ip-or-net-list', + 'rate' => '5m', + 'category' => 'c and c' }, 'FEODO_IP' => { 'name' => 'Feodo Trojan IP Blocklist', 'url' => 'https://feodotracker.abuse.ch/downloads/ipblocklist.txt', 'info' => 'https://feodotracker.abuse.ch/blocklist', - 'parser' => 'text-with-hash-comments', - 'method' => 'check-header-time', - 'rate' => 1, - 'safe' => 'no' }, + 'parser' => 'ip-or-net-list', + 'rate' => '5m', + 'category' => 'c and c', + 'disable' => 'FEODO_RECOMMENDED' }, 'FEODO_AGGRESIVE' => { 'name' => 'Feodo Trojan IP Blocklist (Aggresive)', 'url' => 'https://feodotracker.abuse.ch/downloads/ipblocklist_aggressive.txt', 'info' => 'https://feodotracker.abuse.ch/blocklist', - 'parser' => 'text-with-hash-comments', - 'method' => 'check-header-time', - 'rate' => 1, - 'safe' => 'no', - 'disable' => 'FEODO_IP' }, - 'ABUSE_CH' => { 'name' => 'Abuse.ch Ransomware C&C Blocklist', - 'url' => 'https://ransomwaretracker.abuse.ch/downloads/RW_IPBL.txt', - 'info' => 'https://ransomwaretracker.abuse.ch/blocklist/', - 'parser' => 'text-with-hash-comments', - 'method' => 'check-header-time', - 'rate' => 1, - 'safe' => 'no' }, + 'parser' => 'ip-or-net-list', + 'rate' => '5m', + 'category' => 'c and c', + 'disable' => ['FEODO_IP', 'FEODO_RECOMMENDED'] }, 'CIARMY' => { 'name' => 'The CINS Army List', 'url' => 'https://cinsscore.com/list/ci-badguys.txt', 'info' => 'https://cinsscore.com/#list', - 'parser' => 'text-with-hash-comments', - 'method' => 'check-header-time', - 'rate' => 1, - 'safe' => 'no' }, + 'parser' => 'ip-or-net-list', + 'rate' => '15m', + 'category' => 'reputation' }, 'TOR_ALL' => { 'name' => 'Known TOR Nodes', 'url' => 'https://www.dan.me.uk/torlist', 'info' => 'https://www.dan.me.uk/tornodes', - 'parser' => 'text-with-hash-comments', - 'method' => 'wget', - 'rate' => 1, - 'safe' => 'no', + 'parser' => 'ip-or-net-list', + 'rate' => '1h', + 'category' => 'application', 'disable' => 'TOR_EXIT' }, 'TOR_EXIT' => { 'name' => 'Known TOR Exit Nodes', 'url' => 'https://www.dan.me.uk/torlist/?exit', 'info' => 'https://www.dan.me.uk/tornodes', - 'parser' => 'text-with-hash-comments', - 'method' => 'wget', - 'rate' => 1, - 'safe' => 'no' }, - 'TALOS_MALICIOUS' => { 'name' => 'Talos Malicious hosts list', - 'url' => 'https://www.talosintelligence.com/documents/ip-blacklist', - 'info' => 'https://www.talosintelligence.com/reputation', - 'parser' => 'text-with-hash-comments', - 'method' => 'wget', - 'rate' => 24, - 'safe' => 'no' }, + 'parser' => 'ip-or-net-list',, + 'rate' => '1h', + 'category' => 'application' }, 'ALIENVAULT' => { 'name' => 'AlienVault IP Reputation database', 'url' => 'https://reputation.alienvault.com/reputation.generic', 'info' => 'https://www.alienvault.com/resource-center/videos/what-is-ip-domain-reputati...', - 'parser' => 'text-with-hash-comments', - 'method' => 'check-header-time', - 'rate' => 1, - 'safe' => 'no' }, - 'BOGON' => { 'name' => 'Bogus address list (Martian)', + 'parser' => 'ip-or-net-list', + 'rate' => '1h', + 'category' => 'reputation' }, + 'BOGON' => { 'name' => 'Bogus address list (Martian)', 'url' => 'https://www.team-cymru.org/Services/Bogons/bogon-bn-agg.txt', 'info' => 'https://www.team-cymru.com/bogon-reference.html', - 'parser' => 'text-with-hash-comments', - 'method' => 'check-header-time', - 'rate' => 24, - 'safe' => 'yes' }, - 'BOGON_FULL' => { 'name' => 'Full Bogus Address List', + 'parser' => 'ip-or-net-list', + 'rate' => '1d', + 'category' => 'invalid' }, + 'BOGON_FULL' => { 'name' => 'Full Bogus Address List', 'url' => 'https://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt', 'info' => 'https://www.team-cymru.com/bogon-reference.html', - 'parser' => 'text-with-hash-comments', - 'method' => 'check-header-time', - 'rate' => 24, - 'safe' => 'yes', + 'parser' => 'ip-or-net-list', + 'rate' => '4h', + 'category' => 'invalid', 'disable' => 'BOGON' }, - 'SHODAN' => { 'name' => 'ISC Shodan scanner blacklist', + 'SHODAN' => { 'name' => 'ISC Shodan scanner blacklist', 'url' => 'https://isc.sans.edu/api/threatlist/shodan?tab', 'info' => 'https://isc.sans.edu', - 'parser' => 'text-with-hash-comments', - 'method' => 'wget', - 'rate' => 24, - 'safe' => 'no' } + 'parser' => 'ip-or-net-list', + 'rate' => '1d', + 'category' => 'scanner' }, + 'BLOCKLIST_DE' => { 'name' => 'Blocklist.de all attacks list', + 'url' => 'https://lists.blocklist.de/lists/all.txt', + 'info' => 'https://www.blocklist.de', + 'parser' => 'ip-or-net-list', + 'rate' => '30m', + 'category' => 'attacker' } ); diff --git a/config/logwatch/ipblacklist b/config/logwatch/ipblacklist index 0fadc6250..6d6c46188 100644 --- a/config/logwatch/ipblacklist +++ b/config/logwatch/ipblacklist @@ -49,24 +49,26 @@ while (defined(my $ThisLine = <STDIN>))
my $text = $2;
- if ($text =~ m/Finished updating (\w+) blacklist with (\d+) changes/) + if ($text =~ m/Updated (\w+) blacklist with (\d+) changes/) { $Updates{$1}{updates}++; $Updates{$1}{changes} += $2; } - elsif ($text !~ m/Starting IP Blacklists/ and - $text !~ m/Starting IP Blacklist processing/ and - $text !~ m/Updating \w+ blacklist/ and - $text !~ m/Stopping IP Blacklists/ and - $text !~ m/Deleting IP Blacklists/ and - $text !~ m/Completed IP Blacklist update/ and - $text !~ m/Finished IP Blacklist processing/ and - $text !~ m/Blacklist \w+ Modification times/ and - $text !~ m/Create IPTables chains for blacklist/ and - $text !~ m/Delete IPTables chains for blacklist/ and - $text !~ m/Checking modification time for blacklist/ and - $text !~ m/Restoring blacklist / and - $text !~ m/Downloading blacklist/ ) + elsif ($text !~ m/Starting IP Blacklists/ and + $text !~ m/Starting IP Blacklist processing/ and + $text !~ m/Stopping IP Blacklists/ and + $text !~ m/Deleting IP Blacklists/ and + $text !~ m/Finished IP Blacklist processing/ and + $text !~ m/Create IPTables chains for blacklist/ and + $text !~ m/Delete IPTables chains for blacklist/ and + $text !~ m/Add IP Address Blacklist update to crontab/ and + $text !~ m/Enable IP Address Blacklist update in crontab/ and + $text !~ m/Disable IP Address Blacklist updates/ and + $text !~ m/Restoring blacklist / and + $text !~ m/Blacklist \w+ changed type/ and + $text !~ m/Blacklist \w+ changed size/ and + $text !~ m/Enabling IP Blacklist logging/ and + $text !~ m/Disabling IP Blacklist logging/ ) { $Errors{$text}++; } diff --git a/config/menu/50-firewall.menu b/config/menu/50-firewall.menu index cd82bfaa3..3cfcde835 100644 --- a/config/menu/50-firewall.menu +++ b/config/menu/50-firewall.menu @@ -21,7 +21,7 @@ 'title' => "$Lang::tr{'intrusion detection system'}", 'enabled' => 1, }; - $subfirewall->{'45.ipblacklist'} = {'caption' => $Lang::tr{'ipblacklist'}, + $subfirewall->{'45.ipblacklist'} = {'caption' => $Lang::tr{'ipblacklist'}, 'uri' => '/cgi-bin/ipblacklist.cgi', 'title' => "$Lang::tr{'ipblacklist'}", 'enabled' => 1, diff --git a/config/menu/70-log.menu b/config/menu/70-log.menu index 2fa0e426e..c597de60a 100644 --- a/config/menu/70-log.menu +++ b/config/menu/70-log.menu @@ -43,12 +43,16 @@ 'title' => "$Lang::tr{'ids logs'}", 'enabled' => 1 }; + $sublogs->{'55.ipblacklist'} = {'caption' => $Lang::tr{'ipblacklist logs'}, + 'uri' => '/cgi-bin/logs.cgi/ipblacklists.dat', + 'title' => "$Lang::tr{'ipblacklist logs'}", + 'enabled' => 1 + }; $sublogs->{'55.ovpnclients'} = { 'caption' => $Lang::tr{'ovpn rw connection log'}, 'uri' => '/cgi-bin/logs.cgi/ovpnclients.dat', 'title' => "$Lang::tr{'ovpn rw connection log'}", 'enabled' => 1, - }; $sublogs->{'60.urlfilter'} = { 'caption' => $Lang::tr{'urlfilter logs'}, 'uri' => '/cgi-bin/logs.cgi/urlfilter.dat', diff --git a/config/rootfiles/common/web-user-interface b/config/rootfiles/common/web-user-interface index ea31a943a..37a62d357 100644 --- a/config/rootfiles/common/web-user-interface +++ b/config/rootfiles/common/web-user-interface @@ -45,9 +45,11 @@ srv/web/ipfire/cgi-bin/logs.cgi/firewalllogcountry.dat srv/web/ipfire/cgi-bin/logs.cgi/firewalllogip.dat srv/web/ipfire/cgi-bin/logs.cgi/firewalllogport.dat srv/web/ipfire/cgi-bin/logs.cgi/ids.dat +srv/web/ipfire/cgi-bin/logs.cgi/ipblacklists.dat srv/web/ipfire/cgi-bin/logs.cgi/log.dat srv/web/ipfire/cgi-bin/logs.cgi/ovpnclients.dat srv/web/ipfire/cgi-bin/logs.cgi/proxylog.dat +srv/web/ipfire/cgi-bin/logs.cgi/showrequestfromblacklist.dat srv/web/ipfire/cgi-bin/logs.cgi/showrequestfromcountry.dat srv/web/ipfire/cgi-bin/logs.cgi/showrequestfromip.dat srv/web/ipfire/cgi-bin/logs.cgi/showrequestfromport.dat diff --git a/html/cgi-bin/ipblacklist.cgi b/html/cgi-bin/ipblacklist.cgi index b2ccf7b3f..28b42edf2 100644 --- a/html/cgi-bin/ipblacklist.cgi +++ b/html/cgi-bin/ipblacklist.cgi @@ -17,13 +17,13 @@ # You should have received a copy of the GNU General Public License # # along with this program. If not, see http://www.gnu.org/licenses/. # # # -# Copyright (C) 2018 - 2019 The IPFire Team # +# Copyright (C) 2018 - 2020 The IPFire Team # # # ###############################################################################
use strict; use CGI qw/:standard/; -#enable only the following on debugging purpose +# enable the following only for debugging purposes #use warnings; #use CGI::Carp 'fatalsToBrowser'; use Sort::Naturally; @@ -34,41 +34,40 @@ require "${General::swroot}/lang.pl"; require "${General::swroot}/header.pl";
############################################################################### -# Initialize variables and hashes +# Configuration variables ###############################################################################
my $settings = "${General::swroot}/ipblacklist/settings"; -my $modified = "${General::swroot}/ipblacklist/modified"; my $sources = "${General::swroot}/ipblacklist/sources"; my $getipstat = '/usr/local/bin/getipstat'; my $getipsetstat = '/usr/local/bin/getipsetstat'; my $control = '/usr/local/bin/ipblacklistctrl'; my $lockfile = '/var/run/ipblacklist.pid'; -my $autoblacklist = 'AUTOBLACKLIST'; -my %cgiparams = ('ACTION' => '', 'AUTOACTION' => ''); +my %cgiparams = ('ACTION' => ''); + +############################################################################### +# Variables +############################################################################### + my $errormessage = ''; my $updating = 0; my %mainsettings; my %color; -my %modified; my %sources; my %stats; -my %autoblock_addresses; + +# Default settings - normally overwritten by settings file
my %settings = ( 'DEBUG' => 0, 'LOGGING' => 'on', - 'RATE' => 24, - 'ENABLE' => 'off', - 'BLOCK_THRESHOLD' => 10, - 'BLOCK_PERIOD' => 3600, - $autoblacklist => 'off' ); + 'ENABLE' => 'off' ); + +# Read all parameters
-# Read all parameters for site Header::getcgihash( %cgiparams); General::readhash( "${General::swroot}/main/settings", %mainsettings ); General::readhash( "/srv/web/ipfire/html/themes/".$mainsettings{'THEME'}."/include/colors.txt", %color ); General::readhash( $settings, %settings ) if (-r $settings); -General::readhash( $modified, %modified) if (-r $modified); eval qx|/bin/cat $sources| if (-r $sources);
# Show Headers @@ -79,22 +78,11 @@ Header::showhttpheaders();
if ($cgiparams{'ACTION'} eq "$Lang::tr{'save'}") { - #Save Button on configsite + # Save Button
my %new_settings = ( 'ENABLE' => 'off', - 'RATE' => 24, 'LOGGING' => 'off', - 'DEBUG' => 0, - 'BLOCK_THRESHOLD' => $settings{'BLOCK_THRESHOLD'} || 10, - 'BLOCK_PERIOD' => $settings{'BLOCK_PERIOD'} || 3600, - $autoblacklist => $settings{$autoblacklist} ); - - $errormessage .= "$Lang::tr{'ipblacklist invalid check rate'}<br>" if (($cgiparams{'RATE'} !~ m/^\d+$/) or - ($cgiparams{'RATE'} < 1) or - ($cgiparams{'RATE'} > 1000)); - - $new_settings{'RATE'} = $cgiparams{'RATE'}; - delete $cgiparams{'RATE'}; + 'DEBUG' => 0 );
foreach my $item ('LOGGING', 'ENABLE', keys %sources) { @@ -107,16 +95,32 @@ if ($cgiparams{'ACTION'} eq "$Lang::tr{'save'}")
foreach my $list (keys %sources) { - if (exists $settings{$list} and - $settings{$list} eq 'on' and - exists $sources{$list}{'override'} and - $settings{$sources{$list}{'override'}} eq 'on') + if (exists $new_settings{$list} and + $new_settings{$list} eq 'on' and + exists $sources{$list}{'disable'}) { - $settings{$sources{$list}{'override'}} = 'off'; + my @disable; + + if ('ARRAY' eq ref $sources{$list}{'disable'}) + { + @disable = @{ $sources{$list}{'disable'} }; + } + else + { + @disable = ( $sources{$list}{'disable'} ); + }
- $updating = 1; - $errormessage .= "$Lang::tr{'ipblacklist disable pre'} $sources{$list}{'override'} " . - "$Lang::tr{'ipblacklist disable mid'} $list $Lang::tr{'ipblacklist disable post'}<br>\n"; + foreach my $disable (@disable) + { + if ($new_settings{$disable} eq 'on') + { + $new_settings{$disable} = 'off'; + + $updating = 1; + $errormessage .= "$Lang::tr{'ipblacklist disable pre'} $disable " . + "$Lang::tr{'ipblacklist disable mid'} $list $Lang::tr{'ipblacklist disable post'}<br>\n"; + } + } } }
@@ -140,7 +144,6 @@ if ($cgiparams{'ACTION'} eq "$Lang::tr{'save'}") } else { - $settings{$autoblacklist} = 'off'; system( "$control disable" ); }
@@ -160,51 +163,11 @@ if ($cgiparams{'ACTION'} eq "$Lang::tr{'save'}") if ($updating) { system( "$control update &" ); - get_ipset_stats(); show_running(); exit 0; } } } -elsif ($cgiparams{'AUTOACTION'} eq "$Lang::tr{'save'}") -{ - $updating = 1 if ($settings{$autoblacklist} eq 'on' and not exists $cgiparams{$autoblacklist}); - $updating = 1 if ($settings{$autoblacklist} eq 'off' and exists $cgiparams{$autoblacklist}); - - $settings{$autoblacklist} = (exists $cgiparams{$autoblacklist}) ? 'on' : 'off'; - $settings{'BLOCK_THRESHOLD'} = $cgiparams{'BLOCK_THRESHOLD'}; - $settings{'BLOCK_PERIOD'} = $cgiparams{'BLOCK_PERIOD'} ; - - if (($cgiparams{'BLOCK_THRESHOLD'} !~ m/^\d+$/) or - ($cgiparams{'BLOCK_THRESHOLD'} < 1) or - ($cgiparams{'BLOCK_THRESHOLD'} > 1000000)) - { - $errormessage .= "$Lang::tr{'ipblacklist invalid threshold'}: $cgiparams{'BLOCK_THRESHOLD'}<br>"; - } - - if (($cgiparams{'BLOCK_PERIOD'} !~ m/^\d+$/) or - ($cgiparams{'BLOCK_PERIOD'} < 1) or - ($cgiparams{'BLOCK_PERIOD'} > 86400)) - { - $errormessage .= "$Lang::tr{'ipblacklist invalid block time'}: $cgiparams{'BLOCK_PERIOD'}<br>"; - } - - if ($errormessage) - { - $updating = 0; - } - else - { - General::writehash($settings, %settings); - system( "$control autoblacklist-update" ) if ($updating); - } -} -elsif ($cgiparams{'AUTOACTION'} eq "$Lang::tr{'unblock all'}") -{ - system( "$control autoblacklist-clear" ); -} - -get_ipset_stats();
if (is_running()) { @@ -212,17 +175,12 @@ if (is_running()) exit 0; }
-# Get blacklist statistics - -get_iptables_stats(); - # Show site
Header::openpage($Lang::tr{'ipblacklist'}, 1, ''); Header::openbigbox('100%', 'left'); -error();
-showstatus() if ($settings{ENABLE} eq 'on'); +error() if ($errormessage);
configsite();
@@ -299,10 +257,6 @@ END <tr> <td style='width:24em'>$Lang::tr{'ipblacklist log'}</td> <td><input type='checkbox' name="LOGGING" id="LOGGING"$enable></td> - <td style='width:24em'>$Lang::tr{'ipblacklist check rate'}</td> - <td> - <input type='number' name='RATE' min='1' max='1000' maxlength='7' pattern='\d+' value='$settings{'RATE'}'> - </td> </tr> </table> <br><br> @@ -311,7 +265,7 @@ END <tr> <th align='left'>$Lang::tr{'ipblacklist id'}</th> <th align='left'>$Lang::tr{'ipblacklist name'}</th> - <th align='center'>$Lang::tr{'ipblacklist safe'}</th> + <th align='left'>$Lang::tr{'ipblacklist category'}</th> <th align='center'>$Lang::tr{'ipblacklist enable'}</th> </tr> END @@ -322,10 +276,10 @@ END
foreach my $list (sort keys %sources) { - my $name = escapeHTML( $sources{$list}{'name'} ); - my $safe = $Lang::tr{$sources{$list}{safe}}; - $enable = ''; - my $col = ($lines++ % 2) ? "bgcolor='$color{'color20'}'" : "bgcolor='$color{'color22'}'"; + my $name = escapeHTML( $sources{$list}{'name'} ); + my $category = $Lang::tr{"ipblacklist category $sources{$list}{'category'}"}; + $enable = ''; + my $col = ($lines++ % 2) ? "bgcolor='$color{'color20'}'" : "bgcolor='$color{'color22'}'";
$enable = ' checked' if (exists $settings{$list} and $settings{$list} eq 'on');
@@ -346,7 +300,7 @@ END print <<END; </td> <td>$name</td> - <td align='center'>$safe</td> + <td>$category</td> <td align='center'><input type='checkbox' name="$list" id="$list"$enable></td> </tr>\n END @@ -356,165 +310,18 @@ END
print <<END; </table> - <p>$Lang::tr{'ipblacklist safe note'}</p> </div> <table style='width:100%;'> <tr> <td colspan='3' display:inline align='right'><input type='submit' name='ACTION' value='$Lang::tr{'save'}'></td> </tr> </table> - <div class='sources'> - <br> -END - - $enable = $settings{$autoblacklist} eq 'on' ? ' checked' : ''; - - print <<END; - <br><br> - <h2>$Lang::tr{'ipblacklist auto list'}</h2> - <table style='width:100%' border='0'> - <tr> - <td> - $Lang::tr{'ipblacklist autoblacklist enable'} - </td> - <td> - <input type='checkbox' name="$autoblacklist" id="$autoblacklist"$enable> - </td> - <td> - - </td> - </tr> - <tr> - <td> - $Lang::tr{'ipblacklist autoblacklist threshold'} - </td> - <td> - <input type='number' name='BLOCK_THRESHOLD' min='1' max='1000000' maxlength='7' pattern='\\d+' value='$settings{BLOCK_THRESHOLD}'> - </input> - </td> - </tr> - <tr> - <td> - $Lang::tr{'ipblacklist autoblacklist block time'} - </td> - <td> - <input type='number' name='BLOCK_PERIOD' min='1' max='86400' maxlength='7' pattern='\\d+' value='$settings{BLOCK_PERIOD}'> - </input> - </td> - </tr> - </table> - <table style='width:100%;'> - <tr> - <td colspan='3' display:inline align='right'><input type='submit' name='AUTOACTION' value='$Lang::tr{'save'}'></td> - </tr> - </table> - </form> - </div> END
Header::closebox(); }
-#------------------------------------------------------------------------------ -# sub showstatus() -# -# Displays current blacklist status -#------------------------------------------------------------------------------ - -sub showstatus -{ - Header::openbox('100%', 'center', $Lang::tr{'status'}); - - print <<END; - <table width='100%' cellspacing='1'> - <tr> - <th align='left'>$Lang::tr{'ipblacklist id'}</th> - <th align='right'>$Lang::tr{'ipblacklist entries'}</th> - <th align='right'>$Lang::tr{'ipblacklist pkts in'}</th> - <th align='right'>$Lang::tr{'ipblacklist bytes in'}</th> - <th align='right'>$Lang::tr{'ipblacklist pkts out'}</th> - <th align='right'>$Lang::tr{'ipblacklist bytes out'}</th> - <th align='center'>$Lang::tr{'ipblacklist updated'}</th> - </tr> -END - - # Iterate through the list of sources - - foreach my $list ($autoblacklist, sort keys %sources) - { - next unless ($settings{$list} eq 'on'); - - my $size = ' '; - my $pkts_in = ' '; - my $bytes_in = ' '; - my $pkts_out = ' '; - my $bytes_out = ' '; - my $updated = ' '; - - if (exists $stats{$list}) - { - ($pkts_in, $bytes_in) = @{ $stats{$list}{IPBLACKLISTREDIN} } if (exists $stats{$list}{IPBLACKLISTREDIN}); - ($pkts_out, $bytes_out) = @{ $stats{$list}{IPBLACKLISTREDOUT} } if (exists $stats{$list}{IPBLACKLISTREDOUT}); - $size = $stats{$list}{size} if (exists $stats{$list}{size}); - } - - if (exists $modified{$list} and $modified{$list} > 0) - { - $updated = localtime( $modified{$list} ); - } - - print <<END; - <tr> - <td>$list</td> - <td align='right'>$size</td> - <td align='right'>$pkts_in</td> - <td align='right'>$bytes_in</td> - <td align='right'>$pkts_out</td> - <td align='right'>$bytes_out</td> - <td align='center'>$updated</td> - </tr>\n -END - - } - - print <<END; - </table> -END - - if ($settings{$autoblacklist} eq 'on') - { - print <<END; - <br><br> - <h2>$Lang::tr{'ipblacklist auto list'}</h2> - <table width='60%' cellspacing='1'> - <tr> - <th align='left'>$Lang::tr{'ip address'}</th> - <th align='center'>$Lang::tr{'ipblacklist block time remaining'}</th> - </tr> -END - - foreach my $address (nsort keys %autoblock_addresses ) - { - print "<tr><td>$address</td><td align='center'>$autoblock_addresses{$address}</td></tr>\n"; - } - - print <<END; - </table> - <table width='60%' border='0'> - <form method='post' action='$ENV{'SCRIPT_NAME'}'> - <tr> - <td align='right' width='15%'><input type='submit' name='ACTION' value='$Lang::tr{'unblock all'}'></td> - </tr> - </form> - </table> -END - } - - Header::closebox(); -} - - #------------------------------------------------------------------------------ # sub get_ipset_stats() # @@ -527,8 +334,6 @@ sub get_ipset_stats
system( $getipsetstat );
- # Get the number of entries in each IP set - if (-r '/var/tmp/ipsets.txt') { open STATS, '<', '/var/tmp/ipsets.txt' or die "Can't open IP Sets stats file: $!"; @@ -551,66 +356,6 @@ sub get_ipset_stats
unlink( '/var/tmp/ipsets.txt' ); } - - # Get the IP addresses in the autoblacklist - - if (-r '/var/tmp/autoblacklist.txt') - { - open HASHTABLE, '<', '/var/tmp/autoblacklist.txt' or die "Can't open autoblacklist address file: $!"; - - # Iterate through the blocked addresses - - foreach my $line (<HASHTABLE>) - { - next unless ($line =~ m/(\d+.\d+.\d+.\d+) timeout (\d+)/); - - $autoblock_addresses{$1} = format_time( $2 ); - } - - close HASHTABLE; - - unlink( '/var/tmp/autoblacklist.txt' ); - } - -} - - -#------------------------------------------------------------------------------ -# sub get_iptables_stats() -# -# Gets information on the number of packets and bytes rejected by each -# blacklist -#------------------------------------------------------------------------------ - -sub get_iptables_stats -{ - system( $getipstat ); - unlink( '/var/tmp/iptablesmangle.txt' ); - unlink( '/var/tmp/iptablesnat.txt' ); - - return unless (-r '/var/tmp/iptables.txt'); - - open STATS, '<', '/var/tmp/iptables.txt' or die "Can't open IP Tables stats file: $!"; - - my $table = 'Unknown'; - - foreach my $line (<STATS>) - { - if ($line =~ m/^Chain (\w+)/) - { - $table = $1; - next; - } - - next unless ($line =~ m/_BLOCK/); - - my ($pkts, $bytes, $chain) = $line =~ m/^\s*(\d+\w?)\s+(\d+\w?)\s+(\w+)_BLOCK/; - $stats{$chain}{$table} = [ $pkts, $bytes ]; - } - - close STATS; - - unlink( '/var/tmp/iptables.txt' ); }
@@ -637,7 +382,8 @@ sub is_running #------------------------------------------------------------------------------ # sub show_running # -# Displayed when update is running +# Displayed when update is running. +# Shows a 'working' message plus some information about the IPSets. #------------------------------------------------------------------------------
sub show_running @@ -662,12 +408,7 @@ sub show_running <tr><th>$Lang::tr{'ipblacklist id'}</th><th>$Lang::tr{'ipblacklist entries'}</th></tr> END
- foreach my $name (keys %sources) - { - $stats{$name}{'size'} = ' ' if (not exists ($stats{$name}) and - exists $settings{$name} and - $settings{$name} eq 'on'); - } + get_ipset_stats();
foreach my $name (sort keys %stats) { @@ -693,13 +434,10 @@ END
sub error { - if ($errormessage) - { - Header::openbox('100%', 'left', $Lang::tr{'error messages'}); - print "<class name='base'>$errormessage\n"; - print " </class>\n"; - Header::closebox(); - } + Header::openbox('100%', 'left', $Lang::tr{'error messages'}); + print "<class name='base'>$errormessage\n"; + print " </class>\n"; + Header::closebox(); }
diff --git a/langs/en/cgi-bin/en.pl b/langs/en/cgi-bin/en.pl index b10fbdbf2..ab590cc1b 100644 --- a/langs/en/cgi-bin/en.pl +++ b/langs/en/cgi-bin/en.pl @@ -1,4 +1,4 @@ -%tr = ( +%tr = ( %tr,
'24 hours' => '24 Hours', @@ -1538,37 +1538,31 @@ 'ip alias changed' => 'External IP alias changed', 'ip alias removed' => 'External IP alias removed', 'ip info' => 'IP Information', -'ipblacklist auto list' => 'Automatic blacklist', -'ipblacklist autoblacklist enable' => 'Enable automatically updating local blacklist', -'ipblacklist autoblacklist threshold' => 'Threshold (packets/hour)', -'ipblacklist autoblacklist block time' => 'Block time (seconds)', +'ipblacklist' => 'IP Address Blacklists', 'ipblacklist blacklist settings' => 'Blacklist settings', -'ipblacklist bytes in' => 'bytes in', -'ipblacklist bytes out' => 'bytes out', -'ipblacklist check rate' => 'Update check rate (hours)', -'ipblacklist day' => 'day', -'ipblacklist disable pre' => 'Disabling', +'ipblacklist category' => 'Category', +'ipblacklist category application' => 'Application', +'ipblacklist category attacker' => 'Attacker', +'ipblacklist category c and c' => 'Malware C&C', +'ipblacklist category composite' => 'Composite', +'ipblacklist category invalid' => 'Invalid Address', +'ipblacklist category reputation' => 'Reputation', +'ipblacklist category scanner' => 'Scanner', 'ipblacklist disable mid' => 'because it is included in', -'ipblacklist disable port' => '', +'ipblacklist disable post' => '', +'ipblacklist disable pre' => 'Disabling', 'ipblacklist enable' => 'Enable', 'ipblacklist entries' => 'Entries', -'ipblacklist hour' => 'hour', +'ipblacklist hits' => 'Total number of blacklist hits for', 'ipblacklist id' => 'Blacklist', -'ipblacklist invalid block time' => 'Invalid automatic blacklist block time', -'ipblacklist invalid check rate' => 'Invalid update check rate', -'ipblacklist invalid threshold' => 'Invalid automatic blacklist threshold', +'ipblacklist input' => 'Packets Dropped In', +'ipblacklist log list' => 'Firewall log (blacklist)', 'ipblacklist log' => 'Log dropped packets', +'ipblacklist logs' => 'IP Address Blacklist Logs', 'ipblacklist name' => 'Name', -'ipblacklist pkts in' => 'pkts in', -'ipblacklist pkts out' => 'pkts out', -'ipblacklist safe note' => 'Note: safe blacklists block addresses that only generate malicious traffic and therefore will not block any wanted sites.', -'ipblacklist safe' => 'Safe', -'ipblacklist sixhour' => 'six hours', -'ipblacklist updated' => 'Last updated', +'ipblacklist output' => 'Packets Dropped Out', 'ipblacklist use ipblacklists' => 'Enable IP Blacklists', -'ipblacklist week' => 'week', 'ipblacklist working' => 'Updating IP address blacklists...', -'ipblacklist' => 'IP Address Blacklists', 'ipfire has now rebooted' => 'IPFire is rebooting now.', 'ipfire has now shutdown' => 'IPFire is shutting down now.', 'ipfire side' => 'IPFire side:', diff --git a/lfs/configroot b/lfs/configroot index ba87debb1..90b90eb3c 100644 --- a/lfs/configroot +++ b/lfs/configroot @@ -65,7 +65,7 @@ $(TARGET) : captive/settings captive/agb.txt captive/clients captive/voucher_out certs/index.txt certs/index.txt.attr ddns/config ddns/settings ddns/ipcache dhcp/settings \ dhcp/fixleases dhcp/advoptions dhcp/dhcpd.conf.local dns/settings dns/servers dnsforward/config ethernet/aliases ethernet/settings ethernet/known_nics ethernet/scanned_nics \ ethernet/wireless extrahd/scan extrahd/devices extrahd/partitions extrahd/settings firewall/settings firewall/config firewall/geoipblock firewall/input firewall/outgoing \ - fwhosts/customnetworks fwhosts/customhosts fwhosts/customgroups fwhosts/customservicegrp fwhosts/customgeoipgrp fwlogs/ipsettings fwlogs/portsettings ipblacklist/settings\ + fwhosts/customnetworks fwhosts/customhosts fwhosts/customgroups fwhosts/customservicegrp fwhosts/customgeoipgrp fwlogs/ipsettings fwlogs/portsettings ipblacklist/settings \ isdn/settings mac/settings main/hosts main/routing main/security main/settings optionsfw/settings \ ovpn/ccd.conf ovpn/ccdroute ovpn/ccdroute2 pakfire/settings portfw/config ppp/settings-1 ppp/settings-2 ppp/settings-3 ppp/settings-4 \ ppp/settings-5 ppp/settings proxy/settings proxy/squid.conf proxy/advanced/settings proxy/advanced/cre/enable remote/settings qos/settings qos/classes qos/subclasses qos/level7config qos/portconfig \ diff --git a/make.sh b/make.sh index 3d111dbab..25b8b1d59 100755 --- a/make.sh +++ b/make.sh @@ -1665,7 +1665,7 @@ buildpackages() { export LOGFILE echo "... see detailed log in _build.*.log files" >> $LOGFILE
- + # Generating list of packages used print_line "Generating packages list from logs" rm -f $BASEDIR/doc/packages-list @@ -1680,7 +1680,7 @@ buildpackages() { rm -f $BASEDIR/doc/packages-list # packages-list.txt is ready to be displayed for wiki page print_status DONE - + # Update changelog cd $BASEDIR [ -z $GIT_TAG ] || LAST_TAG=$GIT_TAG @@ -1755,7 +1755,7 @@ while [ $# -gt 0 ]; do done
# See what we're supposed to do -case "$1" in +case "$1" in build) START_TIME=$(now)
@@ -1794,7 +1794,7 @@ build)
print_build_stage "Building packages" buildpackages - + print_build_stage "Checking Logfiles for new Files"
cd $BASEDIR @@ -1859,7 +1859,7 @@ downloadsrc) FINISHED=0 cd $BASEDIR/lfs for c in `seq $MAX_RETRIES`; do - if (( FINISHED==1 )); then + if (( FINISHED==1 )); then break fi FINISHED=1 diff --git a/src/initscripts/system/firewall b/src/initscripts/system/firewall index ebb73062d..08c434440 100644 --- a/src/initscripts/system/firewall +++ b/src/initscripts/system/firewall @@ -180,17 +180,7 @@ iptables_init() { iptables -A FORWARD -j P2PBLOCK iptables -A OUTPUT -j P2PBLOCK - # IP Address Blacklist chains - iptables -N IPBLACKLISTIN - iptables -N IPBLACKLISTOUT - iptables -N IPBLACKLISTREDIN - iptables -N IPBLACKLISTREDOUT - iptables -A INPUT ! -p icmp -j IPBLACKLISTIN - iptables -A FORWARD ! -p icmp -j IPBLACKLISTIN - iptables -A FORWARD ! -p icmp -j IPBLACKLISTOUT - iptables -A OUTPUT ! -p icmp -j IPBLACKLISTOUT - - # Guardian (IPS) chains + # IPS (Guardian) chains iptables -N GUARDIAN iptables -A INPUT -j GUARDIAN iptables -A FORWARD -j GUARDIAN @@ -206,7 +196,15 @@ iptables_init() { iptables -A FORWARD -i tun+ -j OVPNBLOCK iptables -A FORWARD -o tun+ -j OVPNBLOCK
- # IPS (Suricata) chains + # IP Address Blacklist chains + iptables -N BLACKLISTIN + iptables -N BLACKLISTOUT + iptables -A INPUT ! -p icmp -j BLACKLISTIN + iptables -A FORWARD ! -p icmp -j BLACKLISTIN + iptables -A FORWARD ! -p icmp -j BLACKLISTOUT + iptables -A OUTPUT ! -p icmp -j BLACKLISTOUT + + # IPS (suricata) chains iptables -N IPS_INPUT iptables -N IPS_FORWARD iptables -N IPS_OUTPUT @@ -420,8 +418,6 @@ iptables_red_up() { iptables -F REDINPUT iptables -F REDFORWARD iptables -t nat -F REDNAT - iptables -F IPBLACKLISTIN - iptables -F IPBLACKLISTOUT
# PPPoE / PPTP Device if [ "$IFACE" != "" ]; then @@ -479,10 +475,6 @@ iptables_red_up() { iptables -t nat -A REDNAT -s "${network}" -o "${IFACE}" -j RETURN done
- # IP Address Blacklists - iptables -A IPBLACKLISTIN -i $IFACE -j IPBLACKLISTREDIN - iptables -A IPBLACKLISTOUT -o $IFACE -j IPBLACKLISTREDOUT - # Masquerade everything else iptables -t nat -A REDNAT -o $IFACE -j MASQUERADE fi diff --git a/src/misc-progs/getipsetstat.c b/src/misc-progs/getipsetstat.c index aee79542a..781bfc55b 100644 --- a/src/misc-progs/getipsetstat.c +++ b/src/misc-progs/getipsetstat.c @@ -21,8 +21,5 @@ int main(void) safe_system("/usr/sbin/ipset list -t -f /var/tmp/ipsets.txt"); safe_system("chown nobody:nobody /var/tmp/ipsets.txt");
- safe_system("/usr/sbin/ipset list AUTOBLACKLIST -q -f /var/tmp/autoblacklist.txt"); - safe_system("chown -f nobody:nobody /var/tmp/autoblacklist.txt"); - return 0; } diff --git a/src/misc-progs/ipblacklistctrl.c b/src/misc-progs/ipblacklistctrl.c index 506fa2f46..7536b1e97 100644 --- a/src/misc-progs/ipblacklistctrl.c +++ b/src/misc-progs/ipblacklistctrl.c @@ -21,7 +21,7 @@ int main(int argc, char *argv[]) { if (argc < 2) { fprintf(stderr, "\nNo argument given.\n" "ipblacklistctrl (update|restore|log-on|log-off|" - "enable|disable|autoblacklist-update|autoblacklist-clear)\n\n"); + "enable|disable)\n\n"); exit(1); }
@@ -37,14 +37,10 @@ int main(int argc, char *argv[]) { safe_system("/usr/local/bin/ipblacklist enable >/dev/null 2>&1 &"); } else if (strcmp(argv[1], "disable") == 0) { safe_system("/usr/local/bin/ipblacklist disable >/dev/null 2>&1 &"); - } else if (strcmp(argv[1], "autoblacklist-update") == 0) { - safe_system("/usr/local/bin/ipblacklist autoblacklist-update >/dev/null 2>&1 &"); - } else if (strcmp(argv[1], "autoblacklist-clear") == 0) { - safe_system("/usr/local/bin/ipblacklist autoblacklist-clear >/dev/null 2>&1 &"); } else { fprintf(stderr, "\nBad argument given.\n" "ipblacklistctrl (update|restore|log-on|log-off|" - "enable|disable|autoblacklist-update|autoblacklist-clear)\n\n"); + "enable|disable)\n\n"); exit(1); }
diff --git a/src/scripts/ipblacklist b/src/scripts/ipblacklist index b3f8048d9..6f950214c 100755 --- a/src/scripts/ipblacklist +++ b/src/scripts/ipblacklist @@ -18,25 +18,26 @@ # along with IPFire; if not, write to the Free Software # # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # # # -# Copyright (C) 2018 - 2019 The IPFire team # +# Copyright (C) 2018 - 2020 The IPFire team # # # ############################################################################ # # -# This script use a file containing blacklist details in # +# This script uses a file containing blacklist details in # # /var/ipfire/ipblacklist/sources as well as # # /var/ipfire/ipblacklistsettings containing an enable/disable flag for # # each source. # # # -# Two IPTables chains are used, IPBLACKLISTREDIN and IPBLACKLISTREDOUT, # -# which are inserted into the main INPUT, OUTPUT and FORWARD chains. # +# Two IPTables chains are used: BLACKLISTIN and BLACKLISTOUT are inserted # +# inserted into the main INPUT, OUTPUT and FORWARD chains; they capture # +# packets other than for the ICMP protocol. # # # # For each blacklist that is loaded, a chain is created to optionally log # # and then to drop matching packets. An IPSet is created containing the # # addresses or networks blocked by the blacklist, and then rules are added # -# to the IPBLACKLISTREDIN and IPBLACKLISTREDOUT chains to jump to this # -# chain if appropriate packet list matches in the set. # +# to the BLACKLISTIN and BLACKLISTOUT chains to jump to this chain if a # +# packet list matches the set. # # # -# When checking for updates, the modification time is read for each source # +# When checking for updates, the modification time is used for each source # # and if necessary the list is downloaded. The downloaded list is # # compared to the existing IPSet contents and entries created or deleted # # as necessary. # @@ -44,7 +45,7 @@ ############################################################################
use strict; -use warnings; +#use warnings;
use Carp; use Sys::Syslog qw(:standard :macros); @@ -76,23 +77,29 @@ my $lockfile = "/var/run/ipblacklist.pid"; my $proxy_settings = "${General::swroot}/proxy/settings"; my $red_setting = "/var/ipfire/red/iface"; my $detailed_log = "$tmpdir/ipblacklist_log.txt"; -my $autoblacklist = 'AUTOBLACKLIST'; +my $active = "/var/ipfire/red/active";
-my %parsers = ( 'text-with-hash-comments' => &parse_text_with_hash_comments, - 'text-with-semicolon-comments' => &parse_text_with_semicolon_comments, - 'dshield' => &parse_dshield ); +# Other configuration items + +my $margin = 30; # Scheduling allowance for run time etc in seconds +my $count = 30; # Maximum time to wait for another instance (300s) +my $max_dl_fails = 3; # Ignore check rate limit for this number of failures +my $max_size_fraction = 0.7; # Maximum fill fraction of IPSet before enlarging. +my $min_ipset_entries = 1024; # The minimum size of an IPSet. +my $max_dl_bytes = 10_485_760; # Maximum number of bytes to download. +my %parsers = ( 'ip-or-net-list' => &parse_ip_or_net_list, + 'dshield' => &parse_dshield );
############################################################################ # Default settings # Should be overwritten by reading settings files ############################################################################
-my %sources = ( ); +my %sources = ( );
-my %settings = ( 'DEBUG' => 0, - 'LOGGING' => 'on', - 'RATE' => 24, - 'ENABLE' => 'off' ); +my %settings = ( 'DEBUG' => 0, + 'LOGGING' => 'on', + 'ENABLE' => 'off' );
my %proxy_settings = ( 'UPSTREAM_PROXY' => '' ); # No Proxy in use
@@ -101,13 +108,9 @@ my %proxy_settings = ( 'UPSTREAM_PROXY' => '' ); # No Proxy in use ############################################################################
sub abort( $ ); -sub autoblacklist_update(); -sub autoblacklist_clear(); -sub create_autoblacklist(); sub create_list( $ ); sub create_ipset( $$$ ); sub debug( $$ ); -sub delete_autoblacklist(); sub delete_list( $ ); sub disable_logging(); sub disable_updates(); @@ -121,15 +124,15 @@ sub download_wget( $$$ ); sub enable_logging(); sub enable_updates(); sub get_ipsets(); +sub get_rate_seconds( $ ); sub iptables( $ ); sub ipset( $ ); sub stop_ipset(); sub is_connected(); sub log_message( $$ ); sub parse_dshield( $ ); -sub parse_text_with_hash_comments( $ ); -sub parse_text_with_semicolon_comments( $ ); -sub read_ipset( $$$ ); +sub parse_ip_or_net_list( $ ); +sub read_ipset( $$$$ ); sub update_list( $$$ );
############################################################################ @@ -144,25 +147,18 @@ my $ipset_running = 0; # Set to 1 if IPSet process is running my %status; # Status information my %checked; # Time blacklists last changed my %modified; # Time blacklists last modified -my $red_iface; # Name of red interface -my $hours = 3600; # One hour in seconds -my $margin = 600; # Allowance for run time etc -my $count = 30; # Maximum time to wait for another instance (300s) -my @wget_status = ( 'Success', 'Error', 'Parse Error', 'File I/O Error', - 'Network Error', 'SSL Verification Error', - 'Authentication Error', 'Protocol Error', 'Server Error' ); - +my $red_iface; # The name of the red interface
############################################################################ # Synchronise runs ############################################################################
# This script can be triggered either by cron or the WUI. If another -# instance is running, wait for it to finish. +# instance is running, wait for it to finish or timeout.
while (-r $lockfile and $count > 0) { - open LOCKFILE, '<', $lockfile or die "Can't open lockfile"; + open LOCKFILE, '<', $lockfile or (abort "Can't open lockfile", last); my $pid = <LOCKFILE>; close LOCKFILE;
@@ -176,7 +172,7 @@ while (-r $lockfile and $count > 0)
# Create pid file before starting main processing
-open LOCKFILE, '>', '/var/run/ipblacklist.pid' or die "Can't open PID file: $!"; +open LOCKFILE, '>', '/var/run/ipblacklist.pid' or abort "Can't open PID file: $!"; print LOCKFILE "$$\n"; close LOCKFILE;
@@ -205,16 +201,12 @@ if (-r $sources) eval qx|/bin/cat $sources|; }
-# Find out the red interface name - if (-r $red_setting) { - open IN, '<', $red_setting or die "Can't open red interface name file: $!"; - - $red_iface = <IN>; + open REDIF, '<', $red_setting or (abort "Can't open red interface file", exit); + $red_iface = <REDIF>; chomp $red_iface; - - close IN; + close REDIF; }
if (@ARGV) @@ -235,7 +227,7 @@ if (@ARGV) { # Called during system startup. # Restore saved blacklists. - # Don't do an update since that takes too long. + # Don't do an update since can take too long.
do_start() if ($settings{'ENABLE'} eq 'on'); } @@ -300,25 +292,13 @@ if (@ARGV) disable_updates(); do_delete(); } - elsif ('autoblacklist-update' =~ m/^$cmd/i) - { - # Updates AUTOBLACKLIST options - - autoblacklist_update(); - } - elsif ('autoblacklist-clear' =~ m/^$cmd/i) - { - # Clears AUTOBLACKLIST contents - - autoblacklist_clear(); - } else { - print "Usage: $0 [update|start|stop|restart|log-on|log-off|enable|disable|autoblacklist-update|autoblacklist-clear]\n"; + print "Usage: $0 [update|start|stop|restart|log-on|log-off|enable|disable]\n"; } } } -elsif ($settings{'ENABLE'} eq 'on') +elsif ($settings{'ENABLE'} eq 'on') # Default action if none specified { do_update(); } @@ -353,12 +333,9 @@ sub do_stop()
log_message LOG_NOTICE, "Stopping IP Blacklists";
- foreach my $list ( $autoblacklist, sort keys %sources ) + foreach my $list ( sort keys %sources ) { - if (exists $chains{$list}) - { - delete_list( $list ); - } + delete_list( $list ) if (exists $chains{$list}); } }
@@ -375,19 +352,17 @@ sub do_start()
foreach my $list ( sort keys %sources ) { - if (-e "$savedir/$list.conf") + delete_list( $list ) if (exists $chains{$list}); # Make sure OK to start + + if ((-e "$savedir/$list.conf") and ($red_iface)) { log_message LOG_INFO, "Restoring blacklist $list"; - system( "$ipset restore -f $savedir/$list.conf" ); + system( "$ipset restore -f $savedir/$list.conf" ); # Can't use the ipset + # function to do this
create_list( $list ); } } - - if ($settings{$autoblacklist} eq 'on') - { - create_autoblacklist(); - } }
@@ -418,10 +393,8 @@ sub do_delete() } }
- if ($settings{$autoblacklist} eq 'on') - { - delete_autoblacklist(); - } + %modified = (); + $update_status = 1; }
@@ -430,14 +403,14 @@ sub do_delete() # # Updates all the blacklists. # Creates or deletes the blacklist firewall rules as necessary and checks for -# updates to the blacklists. +# updates to the blacklists. Each blacklist has its own minimum elapsed time +# between updates, which is specified in the sources file, so the time of each +# check is stored. #------------------------------------------------------------------------------
sub do_update() { - return unless (is_connected()); - - my $type = 'hash:ip'; + return unless ($red_iface);
# Get the list of current ipsets
@@ -447,38 +420,39 @@ sub do_update()
debug 1, "Checking blacklist sources";
+ LIST: foreach my $list ( sort keys %sources ) { my @new_blacklist = (); my $name = $sources{$list}{'name'}; - my $rate = $sources{$list}{'rate'}; my $last_checked = $checked{$list} || 0; + my $failures = $checked{"${list}_failures"} || 0; my $enabled = 0;
- if (exists $modified{$list}) - { - # Limit the check rate to the minimum defined in the WUI, unless we're - # creating the list - - $rate = $settings{'RATE'} if ($settings{'RATE'} > $rate); - } - if (exists $settings{$list}) { $enabled = $settings{$list} eq 'on'; }
- debug 1, "Checking blacklist source: $name"; - - if ($enabled) + if ($enabled and is_connected()) { + debug 1, "Checking blacklist source: $name"; + + # Calculate the per list rate + + my $rate = get_rate_seconds( $sources{$list}{'rate'} ); + # Has enough time passed since the last time we checked the list? + # Ignore the limit if the last download failed
- if (($last_checked + $rate * $hours) < (time() + $margin)) + if (($last_checked + $rate) < (time() + $margin) or + ($failures > 0 and $failures < $max_dl_fails)) { + my $type = 'hash:ip'; + download_list( $list, @new_blacklist, $type );
- next unless (@new_blacklist); + next LIST unless (@new_blacklist);
if (not exists $chains{$list}) { @@ -504,76 +478,81 @@ sub do_update()
unlink "$savedir/$list.conf" if (-e "$savedir/$list.conf");
- delete $modified{$list} if (exists $modified{$list}); + delete $modified{$list} if (exists $modified{$list}); + delete $checked{"${list}_failures"} if (exists $checked{"${list}_failures"}); $update_status = 1; } }
- # Check for any deleted lists + # Check for any lists that don't exist any more
- foreach my $list (keys %sources) + foreach my $list (keys %modified) { - if (not exists $sources{$list}) - { - delete_list( $list ); + next if (exists $sources{$list});
- # Delete the save file + delete_list( $list );
- unlink "$savedir/$list.conf" if (-e "$savedir/$list.conf"); + # Delete the save file
- # Delete from the status + unlink "$savedir/$list.conf" if (-e "$savedir/$list.conf");
- delete $modified{$list} if (exists $modified{$list}); - delete $checked{$list} if (exists $checked{$list}); - $update_status = 1; - } - } + # Delete from the status
- if ($settings{$autoblacklist} eq 'on') - { - create_autoblacklist() if (not exists $chains{$autoblacklist}); + delete $modified{$list} if (exists $modified{$list}); + $update_status = 1; } - else + + foreach my $list (keys %checked) { - delete_autoblacklist() if (exists $chains{$autoblacklist}); - } + next if ($list =~ m/_failures/); + next if (exists $sources{$list});
- log_message LOG_INFO, "Completed IP Blacklist update"; + delete $checked{$list}; + delete $checked{"${list}_failures"}; + delete $settings{$list} if (exists $settings{$list}); + $update_status = 1; + } }
#------------------------------------------------------------------------------ -# sub autoblacklist_update() +# sub get_rate_seconds( text ) +# +# Converts a check rate into seconds. A sanity check is made on the coverted +# value. # -# Updates the settings for the AUTOBLACKLIST +# Parameters: +# text The value to convert in the form nnnu, where nnn is a number and u +# is either m (minutes), h (hours) or d (days). Hours is assumed if +# not specified and everything after the first letter is ignored. #------------------------------------------------------------------------------
-sub autoblacklist_update() +sub get_rate_seconds( $ ) { - # Get the list of current ipsets - - get_ipsets(); + my ($text) = @_;
- # Delete the existing AUTOBLACKLIST, if it currently exists. + my ($value, $unit) = (uc $text) =~ m/(\d+)([DHM]?)/;
- delete_autoblacklist() if (exists $chains{$autoblacklist}); - - # Re-create the AUTOBLACKLIST with the correct parameters. - - create_autoblacklist() if ($settings{$autoblacklist} eq 'on'); -} + if ($unit eq 'D') # Days + { + $value *= 60 * 60 * 24; + } + elsif ($unit eq 'M') # Minutes + { + $value *= 60; + } + else # Everything else - assume hours + { + $value *= 60 * 60; + }
+ # Sanity check - limit to range 5 min .. 1 week
-#------------------------------------------------------------------------------ -# sub autoblacklist_clear() -# -# Clears the contents of the AUTOBLACKLIST -#------------------------------------------------------------------------------ + # d h m s + $value = 5 * 60 if ($value < 5 * 60); + $value = 7 * 24 * 60 * 60 if ($value > 7 * 24 * 60 * 60);
-sub autoblacklist_clear() -{ - log_message LOG_INFO, "Flush Automatic blacklist"; - ipset( "flush $autoblacklist" ); + return $value; }
@@ -587,7 +566,7 @@ sub autoblacklist_clear()
sub is_connected() { - return (-e "${General::swroot}/red/active"); + return (-e $active); }
@@ -595,7 +574,16 @@ sub is_connected() # sub create_list( list ) # # Creates a new IPTables chain for a blacklist source. -# The set must be created before calling this function. +# The set must be created before calling this function. Two rules are added to +# the chain: +# (optional) 1 Log the packet +# 2 Drop the packet +# +# The log rule is only added when logging is enabled by the WUI. +# +# Rules are then added to the BLACKLISTIN and BLACKLISTOUT chains that check +# the packet's IP address against the IPSet and then jump to the newly created +# chain. # # Parameters: # list The name of the blacklist @@ -609,56 +597,24 @@ sub create_list( $ )
# Create new chain in filter table
- iptables( " -N ${list}_BLOCK" ) == 0 or - ( abort "Could not create IPTables chain ${list}_BLOCK", return ); + iptables( "-N ${list}_DROP" ) == 0 or + ( abort "Could not create IPTables chain ${list}_DROP", return );
# Add the logging and drop rules
if ($settings{'LOGGING'} eq 'on') { - iptables( "-A ${list}_BLOCK -j LOG -m limit --limit 10/second --log-prefix 'DROP_$list'" ) == 0 or + iptables( "-A ${list}_DROP -j LOG -m limit --limit 10/second --log-prefix 'BLKLST_$list'" ) == 0 or ( abort "Could not create IPTables chain $list LOG rule", return ); }
- iptables( "-A ${list}_BLOCK -j DROP" ) == 0 or + iptables( "-A ${list}_DROP -j DROP" ) == 0 or ( abort "Could not create IPTables chain $list drop rule", return );
# Add the rules to check against the set
- iptables( "-A IPBLACKLISTREDIN -p ALL -m set --match-set $list src -j ${list}_BLOCK" ); - iptables( "-A IPBLACKLISTREDOUT -p ALL -m set --match-set $list dst -j ${list}_BLOCK" ); -} - - -#------------------------------------------------------------------------------ -# sub create_autoblacklist() -# -# Creates a new IPTables chain for the AUTOBLACKLIST. This also creates the -# IPSet with the correct timeout. -#------------------------------------------------------------------------------ - -sub create_autoblacklist() -{ - return unless ($red_iface); # Can't add rule to policy unless this is set - - # Create the set for the AUTOBLACKLIST - - ipset( "create $autoblacklist hash:ip timeout $settings{BLOCK_PERIOD}" ); - - # Create new chain in filter table - - create_list( $autoblacklist ); - - # For the AUTOBLACKLIST there are extra rules to reset the timeout on the - # blockled addresses - - iptables( "-I ${autoblacklist}_BLOCK -m set --match-set $autoblacklist src -j SET --add-set $autoblacklist src --exist" ); - iptables( "-I ${autoblacklist}_BLOCK -m set --match-set $autoblacklist dst -j SET --add-set $autoblacklist dst --exist" ); - - # For the AUTOBLACKLIST there is an extra rule to add an entry to the list - # of blocked addresses. This is added to the input policy chain. - - iptables( "-I POLICYIN 1 -i $red_iface -m hashlimit --hashlimit-mode srcip --hashlimit-above $settings{BLOCK_THRESHOLD}/hour --hashlimit-name $autoblacklist -j SET --add-set $autoblacklist src" ); + iptables( "-A BLACKLISTIN -p ALL -i $red_iface -m set --match-set $list src -j ${list}_DROP" ); + iptables( "-A BLACKLISTOUT -p ALL -o $red_iface -m set --match-set $list dst -j ${list}_DROP" ); }
@@ -680,60 +636,25 @@ sub delete_list( $ )
# Remove the blacklist chains from the main INPUT and OUTPUT chains
- iptables( "-D IPBLACKLISTREDIN -p ALL -m set --match-set $list src -j ${list}_BLOCK" ) == 0 or - log_message LOG_ERR, "Could not remove IPSet $list from IPBLACKLISTREDIN chain"; - - iptables( "-D IPBLACKLISTREDOUT -p ALL -m set --match-set $list dst -j ${list}_BLOCK" ) == 0 or - log_message LOG_ERR, "Could not remove IPSet $list from IPBLACKLISTREDOUT chain"; + iptables( "-D BLACKLISTIN -p ALL -i $red_iface -m set --match-set $list src -j ${list}_DROP" ); + iptables( "-D BLACKLISTOUT -p ALL -o $red_iface -m set --match-set $list dst -j ${list}_DROP" );
# Flush and delete the chain
- iptables( "-F ${list}_BLOCK" ) == 0 or - log_message LOG_ERR, "Could not flush IPTables chain ${list}_BLOCK"; - - iptables( "-X ${list}_BLOCK" ) == 0 or - log_message LOG_ERR, "Could not delete IPTables chain ${list}_BLOCK"; + iptables( "-F ${list}_DROP" ); + iptables( "-X ${list}_DROP" );
# Flush and delete the set
ipset( "flush $list" ); - ipset( "destroy $list" ); }
#------------------------------------------------------------------------------ -# sub delete_autoblacklist() +# sub download_list( list, ref_list, ref_type ) # -# Deletes the autoblacklist IPTables chain when it is disabled. Also flushes -# and destroys the IPSet. -#------------------------------------------------------------------------------ - -sub delete_autoblacklist() -{ - # For the AUTOBLACKLIST there is an extra rule to remove - - unless ($red_iface) - { - iptables( "-D POLICYIN -i $red_iface -m hashlimit --hashlimit-mode srcip --hashlimit-above $settings{BLOCK_THRESHOLD}/hour --hashlimit-name $autoblacklist -j SET --add-set $autoblacklist src" ); - } - - # Now do a normal delete - - delete_list( $autoblacklist ); -} - - -#------------------------------------------------------------------------------ -# sub download_list( chain, ref_list, ref_type ) -# -# Updates the IP Addresses for a blacklist. Depending on the blacklist one of -# two methods are used: -# -# - For some lists the header is downloaded and the modification date checked. -# If newer than the existing list, the update is downloaded. -# - For other lists this is not supported,so the whole file has to be -# downloaded regardless. +# Downloads the IP Addresses for a blacklist. # # Once downloaded the list is parsed to get the IP addresses and/or networks. # @@ -747,8 +668,8 @@ sub download_list( $$$ ) { my ($list, $new_blacklist, $type) = @_;
- $checked{$list} = time(); - $update_status = 1; + $checked{$list} = time(); # Record that the list has been checked + $update_status = 1;
# Check the parser for the blacklist
@@ -758,23 +679,18 @@ sub download_list( $$$ ) return; }
- if ($sources{$list}{'method'} eq 'check-header-time') - { - download_check_header_time( $list, $new_blacklist, $type ); - } - else - { - download_wget( $list, $new_blacklist, $type ); - } + # Add alternative download mechanisms here + + download_check_header_time( $list, $new_blacklist, $type ); }
#------------------------------------------------------------------------------ -# sub download_check_header_time( chain, ref_list, ref_type ) +# sub download_check_header_time( list, ref_list, ref_type ) # -# Updates the IP Addresses for a blacklist. The header is downloaded and the -# modification date checked. If newer than the existing list, the update is -# downloaded. +# Updates the IP Addresses for a blacklist. The If-Modified-Since header is +# specified in the request so that only updated lists are downloaded (providing +# that the server supports this functionality). # # Once downloaded the list is parsed to get the IP addresses and/or networks. # @@ -782,6 +698,9 @@ sub download_list( $$$ ) # list The name of the blacklist # ref_list A reference to an array to store the downloaded blacklist # ref_type A reference to store the type of the blacklist +# +# Returns: +# The list type: 'hash:ip' or 'hash:net' #------------------------------------------------------------------------------
sub download_check_header_time( $$$ ) @@ -794,12 +713,12 @@ sub download_check_header_time( $$$ )
my $parser = $parsers{ $sources{$list}{'parser'} };
- log_message LOG_INFO, "Checking modification time for blacklist $list update with LWP"; + debug 1, "Checking for blacklist $list updates with LWP";
# Create a user agent for downloading the blacklist - # Limit the download size for safety (10 MiB) + # Limit the download size for safety
- my $ua = LWP::UserAgent->new( max_size => 10485760 ); + my $ua = LWP::UserAgent->new( max_size => $max_dl_bytes );
# Get the Proxy settings
@@ -807,55 +726,42 @@ sub download_check_header_time( $$$ ) { if ($proxy_settings{'UPSTREAM_USER'}) { - $ua->proxy("http" => "http://$proxy_settings%7B%27UPSTREAM_USER%27%7D:$proxy_settings%7B%27UPSTREA..."); - $ua->proxy("https" => "http://$proxy_settings%7B%27UPSTREAM_USER%27%7D:$proxy_settings%7B%27UPSTREA..."); + $ua->proxy( [["http", "https"] => "http://$proxy_settings%7B%27UPSTREAM_USER%27%7D:$proxy_settings%7B%27UPSTREA..."] ); } else { - $ua->proxy("http" => "http://$proxy_settings%7B%27UPSTREAM_PROXY%27%7D/"); - $ua->proxy("https" => "http://$proxy_settings%7B%27UPSTREAM_PROXY%27%7D/"); + $ua->proxy( [["http", "https"] => "http://$proxy_settings%7B%27UPSTREAM_PROXY%27%7D/"] ); } }
- # Get the blacklist modification time from the internet - - my $request = HTTP::Request->new( HEAD => $sources{$list}{'url'} ); - - my $response = $ua->request( $request ); + # Get the last modified time
- if (not $response->is_success) - { - log_message LOG_WARNING, "Failed to download $list header $sources{$list}{'url'}: ". $response->status_line; - - return; - } - - # Has the blacklist been modified since we last read it? - - if (exists $modified{$list} and $modified{$list} >= $response->last_modified) - { - # We've already got this version of the blacklist - - debug 1, "Blacklist $list not modified"; - return; - } - - debug 1, "Blacklist $list Modification times: old " . $modified{$list} . ", new " . $response->last_modified if (exists $modified{$list}); - log_message LOG_INFO, "Downloading blacklist $list with LWP"; + my $modified = gmtime( $modified{$list} || 0 );
# Download the blacklist
- $request = HTTP::Request->new( GET => $sources{$list}{'url'} ); - $response = $ua->request($request); + my $response = $ua->get( $sources{$list}{'url'}, 'If-Modified-Since' => $modified );
if (not $response->is_success) { + if ($response->code == 304) + { + # Not an error - list has not been modified + debug 1, "Blacklist $list not modified"; + + return; + } + log_message LOG_WARNING, "Failed to download $list blacklist $sources{$list}{'url'}: ". $response->status_line; + $checked{"${list}_failures"}++;
return; }
- $modified{$list} = $response->last_modified; + $modified{$list} = $response->last_modified; + $checked{"${list}_failures"} = 0; + + # Parse the downloaded list, checking if it's a list of addresses or nets
foreach my $line (split /[\r\n]+/, $response->content) { @@ -865,108 +771,12 @@ sub download_check_header_time( $$$ )
next unless ($address and $address =~ m/\d+.\d+.\d+.\d+/);
- if ($address =~ m|/\d+|) - { - $found_net = 1; - } - else + if ($address =~ m|/32|) { + $address =~ s|/32||; $found_ip = 1; } - - push @{ $new_blacklist }, $address; - } - - if ($found_net and $found_ip) - { - # Convert mixed address and network set to all network - - foreach my $address (@{ $new_blacklist }) - { - $address .= '/32' unless ($address =~ m|/\d+|); - } - - $found_ip = 0; - } - - $$type = $found_net ? 'hash:net' : 'hash:ip'; -} - - -#------------------------------------------------------------------------------ -# sub download_wget( chain, ref_list, ref_type ) -# -# Updates the IP Addresses for a blacklist. The whole file is download with -# wget and then the modification time compared with the stored modification -# time. If the update is newer then the downloaded list is parsed. -# -# Once downloaded the list is parsed to get the IP addresses and/or networks. -# -# Parameters: -# list The name of the blacklist -# ref_list A reference to an array to store the downloaded blacklist -# ref_type A reference to store the type of the blacklist -#------------------------------------------------------------------------------ - -sub download_wget( $$$ ) -{ - my ($list, $new_blacklist, $type) = @_; - my $wget_proxy = ''; - my $found_ip = 0; - my $found_net = 0; - - my $parser = $parsers{ $sources{$list}{'parser'} }; - - log_message LOG_INFO, "Downloading blacklist $list update with wget"; - - # Get the Proxy settings - - if ($proxy_settings{'UPSTREAM_PROXY'}) - { - if ($proxy_settings{'UPSTREAM_USER'}) - { - $wget_proxy = "--proxy=on --proxy-user=$proxy_settings{'UPSTREAM_USER'} --proxy-passwd=$proxy_settings{'UPSTREAM_PASSWORD'} -e http_proxy=http://$proxy_settings%7B%27UPSTREAM_PROXY%27%7D/"; - } - else - { - $wget_proxy = "--proxy=on -e http_proxy=http://$proxy_settings%7B%27UPSTREAM_PROXY%27%7D/"; - } - } - - my $retv = system( "wget $wget_proxy --no-show-progress -o $detailed_log -O $tmpdir/ipblacklist_$list $sources{$list}{'url'}" ); - - if ($retv != 0) - { - my $error = $wget_status[ $retv/256 ]; - log_message LOG_WARNING, "Failed to download $list blacklist $sources{$list}{'url'}: $error"; - return; - } - - my @file_info = stat( "$tmpdir/ipblacklist_$list" ); - - if (exists $modified{$list} and $modified{$list} >= $file_info[9]) - { - # We've already got this version of the blocklist - - debug 1, "Blacklist $list not modified"; - unlink "$tmpdir/ipblacklist_$list"; - return; - } - - open LIST, '<', "$tmpdir/ipblacklist_$list" or (abort "Can't open downloaded blacklist for $list: $!", return); - - $modified{$list} = $file_info[9]; - - foreach my $line (<LIST>) - { - chomp $line; - - my $address = &$parser( $line ); - - next unless ($address); - next unless ($address =~ m|\d+.\d+.\d+.\d+|); - - if ($address =~ m|/\d+|) + elsif ($address =~ m|/\d+|) { $found_net = 1; } @@ -978,13 +788,9 @@ sub download_wget( $$$ ) push @{ $new_blacklist }, $address; }
- close LIST; - - unlink "$tmpdir/ipblacklist_$list"; - if ($found_net and $found_ip) { - # Convert mixed address and network set to all network + # Convert mixed addresses and networks to all networks
foreach my $address (@{ $new_blacklist }) { @@ -999,38 +805,49 @@ sub download_wget( $$$ )
#------------------------------------------------------------------------------ -# sub read_ipset( list, old, type ) +# sub read_ipset( list, old, type, maxelem ) # -# Reads the existing contents of the set +# Reads the existing contents and type of the set. # # Parameters: -# chain The name of the blacklist -# old Reference to array to contain blacklist -# type Reference to type +# list The name of the blacklist +# old Reference to array to contain blacklist +# type Reference to type +# maxelem Reference to maximum number of elements #------------------------------------------------------------------------------
-sub read_ipset( $$$ ) +sub read_ipset( $$$$ ) { - my ($list, $old, $type) = @_; - my $found_net = 0; - my $found_ip = 0; + my ($list, $old, $type, $maxelem) = @_; + my $found_net = 0; + my $found_ip = 0;
debug 2, "Reading existing ipset for blacklist $list";
foreach my $line (qx/$ipset list $list/) { + if ($line =~ m|Header:.*maxelem (\d+)|) + { + $$maxelem = $1; + next; + } + next unless ($line =~ m|(\d+.\d+.\d+.\d+(?:/\d+)?)|);
my $address = $1;
- if (($address =~ m|/\d+$|) and ($address !~ m|/32$|)) + if ($address =~ m|/32|) + { + $found_ip = 1; + $address =~ s|/32$||; + } + elsif ($address =~ m|/\d+$|) { $found_net = 1; } else { $found_ip = 1; - $address =~ s|/32$||; }
$$old{$address} = 1; @@ -1038,7 +855,7 @@ sub read_ipset( $$$ )
if ($found_ip and $found_net) { - # Convert mixed address and network set to all network + # Convert mixed addresses and networks to all networks
my @ads_list = keys %{ $old };
@@ -1059,12 +876,13 @@ sub read_ipset( $$$ )
#------------------------------------------------------------------------------ -# sub update_list( chain, new, new_type ) +# sub update_list( list, new, new_type ) # # Updates the IP Addresses for a blacklist # # The new list is compared to the existing list and new entries added or old -# entries deleted as necessary. +# entries deleted as necessary. If the list type ('hash:ip' or 'hash:net') has +# changed then the IPSet is deleted and re-created with the new type. # # Parameters: # list The name of the blacklist @@ -1078,49 +896,63 @@ sub update_list( $$$ ) my %old; my $old_type; my $changes = 0; + my $maxelem = 0;
- debug 2, "Checking for $list blacklist update from $sources{$list}{'url'}"; + debug 1, "Checking for $list blacklist update from $sources{$list}{'url'}";
- log_message LOG_INFO, "Updating $list blacklist"; + if (exists $chains{$list} ) + { + my $recreate_ipset = 0;
- read_ipset( $list, %old, $old_type ); + read_ipset( $list, %old, $old_type, $maxelem );
- # Check the IPSet type hasn't changed + # Check the IPSet type hasn't changed
- if ($new_type ne $old_type) - { - # Change the IPSet type. This requires removing references to it first. - # We could delete and then create the chain, but doing it like this keeps - # the statistics. + if ($new_type ne $old_type) + { + log_message LOG_NOTICE, "Blacklist $list changed type from $old_type to $new_type"; + $recreate_ipset = 1; + }
- log_message LOG_NOTICE, "Blacklist $list changed type from $old_type to $new_type"; + if ($max_size_fraction * $maxelem < scalar @{ $new } ) + { + log_message LOG_NOTICE, "Blacklist $list changed size from $maxelem"; + $recreate_ipset = 1; + } + + if ($recreate_ipset) + { + # Change the IPSet type and/or size. This requires removing references + # to it first. We could delete and then create the chain, but doing it + # like this keeps the statistics.
- # Remove the IPSet from the IPTables chains + # Remove the IPSet from the IPTables chains
- iptables( "-D 'IPBLACKLISTREDIN' -p ALL -m set --match-set $list src -j ${list}_BLOCK" ) == 0 or - log_message LOG_ERR, "Could not remove ${list} from IPBLACKLISTREDIN chain"; + iptables( "-D 'BLACKLISTIN' -p ALL -i $red_iface -m set --match-set $list src -j ${list}_DROP" ) == 0 or + log_message LOG_ERR, "Could not remove ${list} from BLACKLISTIN chain";
- iptables( "-D 'IPBLACKLISTREDOUT' -p ALL -m set --match-set $list dst -j ${list}_BLOCK" ) == 0 or - log_message LOG_ERR, "Could not remove ${list} from IPBLACKLISTREDOUT chain"; + iptables( "-D 'BLACKLISTOUT' -p ALL -o $red_iface -m set --match-set $list dst -j ${list}_DROP" ) == 0 or + log_message LOG_ERR, "Could not remove ${list} from BLACKLISTOUT chain";
- # Flush and delete the old set + # Flush and delete the old set
- ipset( "flush $list" ); - ipset( "destroy $list" ); + ipset( "flush $list" ); + ipset( "destroy $list" );
- %old = (); + %old = (); # Since we've deleted the old set it can't have any entries.
- # Create the new ipset + # Create the new ipset
- create_ipset( $list, $new_type, scalar @{ $new } ); + create_ipset( $list, $new_type, scalar @{ $new } );
- # Add the rules to check against the set + # Add the rules to check against the set
- iptables( "-A 'IPBLACKLISTREDIN' -p ALL -m set --match-set $list src -j ${list}_BLOCK" ) == 0 or - log_message LOG_ERR, "Could not add IPSet $list to IPBLACKLISTREDIN chain"; + iptables( "-A 'BLACKLISTIN' -p ALL -i $red_iface -m set --match-set $list src -j ${list}_DROP" ) == 0 or + log_message LOG_ERR, "Could not add IPSet $list to BLACKLISTIN chain";
- iptables( "-A 'IPBLACKLISTREDOUT' -p ALL -m set --match-set $list dst -j ${list}_BLOCK" ) == 0 or - log_message LOG_ERR, "Could not add IPSet $list to IPBLACKLISTREDOUT chain"; + iptables( "-A 'BLACKLISTOUT' -p ALL -o $red_iface -m set --match-set $list dst -j ${list}_DROP" ) == 0 or + log_message LOG_ERR, "Could not add IPSet $list to BLACKLISTOUT chain"; + } }
# Process the blacklist @@ -1131,11 +963,11 @@ sub update_list( $$$ )
if (exists $old{$address}) { - delete $old{$address}; # Not new + delete $old{$address}; # Not new - don't delete from chain later } else { - ipset( "add $list $address -exist" ); + ipset( "add $list $address -exist" ); # New - add it
$changes++; } @@ -1156,14 +988,13 @@ sub update_list( $$$ ) debug 3, "Delete old net $address from blacklist $list"; }
- - log_message LOG_INFO, "Finished updating $list blacklist with $changes changes"; + log_message LOG_INFO, "Updated $list blacklist with $changes changes";
# Save the blacklist for the next reboot
mkdir "$savedir" unless (-d "$savedir" );
- ipset( "save $list -file $savedir/$list.conf" ); + ipset( "save $list -file $savedir/$list.conf" ) if ($changes > 0);
stop_ipset(); } @@ -1195,28 +1026,29 @@ sub get_ipsets( )
#------------------------------------------------------------------------------ -# sub create_ipset( name, type, size ) +# sub create_ipset( list, type, size ) # -# Creates a new IPSet. The current and maximum size of the set are determined -# by taking the next power of two greater than the numer of entries, subject to -# a minimum size. This allows for future expansion. +# Creates a new IPSet. The current size of the set is determined by taking the +# next power of two greater than the number of entries; the maximum size is set +# to double this, subject to a minimum size. This allows for future expansion. # # Parameters: -# name The name of the blacklist +# list The name of the blacklist # type The type of the blacklist (hash:ip or hash:net) # size The number of entries in the lsit #------------------------------------------------------------------------------
sub create_ipset( $$$ ) { - my ($name, $type, $size) = @_; + my ($list, $type, $size) = @_;
my $hashsize = 1; $hashsize <<= 1 while ($hashsize < $size); - my $maxsize = ($hashsize < 16384) ? 32768 : $hashsize * 2; + my $maxsize = $hashsize * 2; + $maxsize = $min_ipset_entries if ($maxsize < $min_ipset_entries);
# Create the new ipset - ipset( "create $name $type hashsize $hashsize maxelem $maxsize" ); + ipset( "create $list $type hashsize $hashsize maxelem $maxsize" ); stop_ipset(); # Need to do this to action the IPSet commands }
@@ -1225,6 +1057,7 @@ sub create_ipset( $$$ ) # sub enable_logging() # # Enable logging of packets dropped by IP Blacklist rules. +# This adds a rule to log the packet to each lists' IPTables chain. #------------------------------------------------------------------------------
sub enable_logging() @@ -1237,7 +1070,7 @@ sub enable_logging() { if (exists $chains{$list}) { - iptables( "-I ${list}_BLOCK 1 -j LOG -m limit --limit 10/second --log-prefix 'DROP_$list'" ); + iptables( "-I ${list}_DROP 1 -j LOG -m limit --limit 10/second --log-prefix 'BLKLST_$list'" ); } } } @@ -1247,6 +1080,7 @@ sub enable_logging() # sub disable_logging() # # Disable logging of packets dropped by IP Blacklist rules. +# This deletes a rule to log the packet from each lists' IPTables chain. #------------------------------------------------------------------------------
sub disable_logging() @@ -1259,7 +1093,7 @@ sub disable_logging() { if (exists $chains{$list}) { - iptables( "-D ${list}_BLOCK -j LOG -m limit --limit 10/second --log-prefix 'DROP_$list'" ); + iptables( "-D ${list}_DROP -j LOG -m limit --limit 10/second --log-prefix 'BLKLST_$list'" ); } } } @@ -1269,9 +1103,14 @@ sub disable_logging() # sub enable_updates() # # Adds a command to the fcrontab to run the update hourly. +# If there is a command already in the fcrontab to do this it will be +# uncommented, otherwise a new line is added. +# # The update is executed at an offset from the hour so that all the users don't # try to download the updates at exactly the same time - the blacklists are -# provided free, so it's good manners to spread the load on the servers. +# provided free, so it's good manners to spread the load on the servers. The +# offset is initialised to a random number that avoids running on the hour +# (when a lot of other things happen), and every fifteen minutes thereafter. #------------------------------------------------------------------------------
sub enable_updates() @@ -1285,9 +1124,9 @@ sub enable_updates() { if ($line =~ m|/usr/local/bin/ipblacklist|) { - return if ($line !~ m/^#/); # Already enabled + return if ($line !~ m/^#/); # Already enabled - do nothing
- # Found - uncomment the line + # Already in fcrontab - uncomment the line
$line =~ s/^#+//; $found = 1; @@ -1298,13 +1137,20 @@ sub enable_updates()
if (not $found) { - # Add a new entry + # Add a new entry to fcrontab
- my $start = int( rand(50) ) + 5; + my $start = int( rand(13) ) + 1; + + my $times = $start; + + for (my $offset = $times+15 ; $offset < 60 ; $offset += 15) + { + $times .= ",$offset"; + }
push @lines, "\n"; push @lines, "# IP Blacklist update\n"; - push @lines, "%hourly,nice(1),random,serial $start /usr/local/bin/ipblacklist\n"; + push @lines, "%nice(1) $times * * * * /usr/local/bin/ipblacklist\n"; log_message LOG_INFO, "Add IP Address Blacklist update to crontab"; }
@@ -1329,9 +1175,9 @@ sub disable_updates() { if ($line =~ m|/usr/local/bin/ipblacklist|) { - return if ($line =~ m/^#/); # Already disabled + return if ($line =~ m/^#/); # Already disabled - do nothing
- # Found - comment the line + # In fcrontab - comment the line
$line =~ s/^#*/#/; $found = 1; @@ -1349,9 +1195,10 @@ sub disable_updates()
#------------------------------------------------------------------------------ -# sub parse_text_with_hash_comments( line ) +# sub parse_ip_or_net_list( line ) # -# Parses an input line removing comments. +# Parses an input line, looking for lines starting with an IP Address or +# Network specification. # # Parameters: # line The line to parse @@ -1360,41 +1207,11 @@ sub disable_updates() # Either an IP Address or a null string #------------------------------------------------------------------------------
-sub parse_text_with_hash_comments( $ ) +sub parse_ip_or_net_list( $ ) { my ($line) = @_;
- return "" if ($line =~ m/^\s*#/); - - $line =~ s/#.*$//; - - $line =~ m|(\d+.\d+.\d+.\d+(?:/\d+)?)|; - - return $1; -} - - -#------------------------------------------------------------------------------ -# sub parse_text_with_semicolon_comments( line ) -# -# Parses an input line removing comments. -# -# Parameters: -# line The line to parse -# -# Returns: -# Either and IP Address or a null string -#------------------------------------------------------------------------------ - -sub parse_text_with_semicolon_comments( $ ) -{ - my ($line) = @_; - - return "" if ($line =~ m/^\s*;/); - - $line =~ s/;.*$//; - - $line =~ m|(\d+.\d+.\d+.\d+(?:/\d+)?)|; + $line =~ m|^(\d+.\d+.\d+.\d+(?:/\d+)?)|;
return $1; } @@ -1424,6 +1241,7 @@ sub parse_dshield( $ )
$line =~ s/#.*$//;
+ # |Start addrs | |End Addrs | |Mask $line =~ m|(\d+.\d+.\d+.\d+(?:/\d+)?)\s+\d+.\d+.\d+.\d+(?:/\d+)?\s+(\d+)|;
return unless ($1); @@ -1436,8 +1254,8 @@ sub parse_dshield( $ ) #------------------------------------------------------------------------------ # sub iptables( cmd ) # -# Executes an IPTables command, waiting for the lock to ensure only one change -# is made at a time. +# Executes an IPTables command, waiting for the internal lock to ensure only +# one change is made at a time. # # Parameters: # cmd The command to execute @@ -1462,6 +1280,11 @@ sub iptables( $ ) # new process for each command. The sub-process is started if it's not already # running. # +# Note that the pipe is buffered so commands are not necessarily executed +# immediately. Use ipset_stop() to force commands to be executed. This should +# be done before relying on anything that the ipset commands do, for example +# before referencing the IPSet in an IPTables command. +# # Parameters: # cmd The command to execute #------------------------------------------------------------------------------ @@ -1485,6 +1308,7 @@ sub ipset( $ ) # sub stop_ipset( ) # # Stops the ipset sub-process. +# This causes any pending ipset commands to be executed. #------------------------------------------------------------------------------
sub stop_ipset( ) @@ -1500,7 +1324,7 @@ sub stop_ipset( ) #------------------------------------------------------------------------------ # sub abort( message, parameters... ) # -# Aborts the current activity, printing out an error message. +# Used when aborting the current activity, printing out an error message. # # Parameters: # message Message to be printed
-
Michael Tremer -
Tim FitzGeorge