public inbox for development@lists.ipfire.org
 help / color / mirror / Atom feed
From: Tim FitzGeorge <ipfr@tfitzgeorge.me.uk>
To: development@lists.ipfire.org
Subject: Re: [PATCH v2 0/8] ipblacklist: IP Address Blacklists
Date: Tue, 26 May 2020 18:44:35 +0100	[thread overview]
Message-ID: <60d44f5b-36ec-c3b0-4dc1-e82e81663774@tfitzgeorge.me.uk> (raw)
In-Reply-To: <B9BF1608-6C23-43FE-A4EE-6BE8CC9AE8BD@ipfire.org>

[-- Attachment #1: Type: text/plain, Size: 101615 bytes --]

Hi Michael,

The diff between the V1 and V2 patches is at the end of this email.  I've not included
html/cgi-bin/logs.cgi/ipblacklists.dat
html/cgi-bin/logs.cgi/showrequestfromblacklist.dat
(V2 patches 0003 and 0004) as they're completely new at V2 - they're modifications of the other
similar log files.

There are a lot of lines here, but a lot of them are deletions due to moving the status etc.
into the logging.

I hope the is OK.

Tim

On 16/05/2020 10:40, Michael Tremer wrote:
> Hi Tim,
> 
> This has now been sitting in my inbox for almost a month. Nobody else has commented on it.
> 
> I tried a couple of times to review this, but I do not even know where to start. This patch is just too large.
> 
> First of all, although it is split across multiple patches, it is one large patch. It is not split by changes that belong together, but just one patch per file.
> 
> I have no idea what has changed since I last time looked at the code. It has been months since the previous patchset and I do not remember each individual line.
> 
> You Git repository also does not have any changes any more. The branch has been reset.
> 
> Are you able to send a diff with the changes since the first patchset?
> 
> How can we finally bring this into the distribution?
> 
> Best,
> -Michael
> 
>> On 27 Apr 2020, at 15:31, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
>>
>> Implements downloading of IP address blacklists and implementing
>> them as IPSets.  A separate IPSet is used for each blacklist; this
>> simplifies handling of overlaps between different lists.  Traffic
>> to or from the red0/ppp0 interface is checked against the IPSets.
>> The check is placed before the IPS check as the IPSet check is
>> much lighter on CPU use which means that overall CPU use is
>> reduced.
>>
>> The available lists are defined in a separate file.  A WUI page
>> allows the desired lists to be enabled.  A minimum update check
>> interval is defined for each blacklist in the definition file.
>>
>> Changes since Version 1:
>>
>> - Changed Dshield download URL to preferred address.
>> - Removed Abuse.ch blacklist (discontinued).
>> - Removed Talos Malicious blacklist (not for production use).
>> - Added Feodo recommended blacklist.
>> - Added blocklist.de all blacklist.
>> - Updated ignored messages in logwatch.
>> - Modified sources file 'rate' to allow unit to be specified.
>> - Updated sources file 'disable' to allow list to be specified.
>> - Removed autoblacklist.
>> - Added WUI log pages.
>> - Removed status from settings WUI page.
>>
>> Tim FitzGeorge (8):
>>  ipblacklist: Main script
>>  ipblacklist: WUI Settings page
>>  ipblacklist: WUI Log page
>>  ipblacklist: WUI Log details page
>>  ipblacklist: WUI menus, language file etc
>>  ipblacklist: Ancillary files
>>  ipblacklist: Modifications to system
>>  ipblacklist: Build infrastructure
>>
>> config/backup/backup.pl                            |    1 +
>> config/backup/include                              |    2 +
>> config/ipblacklist/sources                         |  138 ++
>> config/logwatch/ipblacklist                        |  105 ++
>> config/logwatch/ipblacklist.conf                   |   34 +
>> config/menu/50-firewall.menu                       |    5 +
>> config/menu/70-log.menu                            |    5 +
>> config/rootfiles/common/aarch64/stage2             |    1 +
>> config/rootfiles/common/configroot                 |    2 +
>> config/rootfiles/common/ipblacklist-sources        |    1 +
>> config/rootfiles/common/logwatch                   |    2 +
>> config/rootfiles/common/misc-progs                 |    2 +
>> config/rootfiles/common/stage2                     |    1 +
>> config/rootfiles/common/web-user-interface         |    3 +
>> config/rootfiles/common/x86_64/stage2              |    1 +
>> html/cgi-bin/ipblacklist.cgi                       |  463 +++++++
>> html/cgi-bin/logs.cgi/ipblacklists.dat             |  363 +++++
>> html/cgi-bin/logs.cgi/log.dat                      |    2 +
>> html/cgi-bin/logs.cgi/showrequestfromblacklist.dat |  415 ++++++
>> langs/en/cgi-bin/en.pl                             |   27 +-
>> lfs/configroot                                     |    4 +-
>> lfs/ipblacklist-sources                            |   53 +
>> lfs/logwatch                                       |    2 +
>> make.sh                                            |    1 +
>> src/initscripts/system/firewall                    |   12 +
>> src/misc-progs/Makefile                            |    2 +-
>> src/misc-progs/getipsetstat.c                      |   25 +
>> src/misc-progs/ipblacklistctrl.c                   |   48 +
>> src/scripts/ipblacklist                            | 1382 ++++++++++++++++++++
>> 29 files changed, 3098 insertions(+), 4 deletions(-)
>> create mode 100644 config/ipblacklist/sources
>> create mode 100644 config/logwatch/ipblacklist
>> create mode 100644 config/logwatch/ipblacklist.conf
>> create mode 100644 config/rootfiles/common/ipblacklist-sources
>> create mode 100644 html/cgi-bin/ipblacklist.cgi
>> create mode 100755 html/cgi-bin/logs.cgi/ipblacklists.dat
>> create mode 100755 html/cgi-bin/logs.cgi/showrequestfromblacklist.dat
>> create mode 100644 lfs/ipblacklist-sources
>> create mode 100644 src/misc-progs/getipsetstat.c
>> create mode 100644 src/misc-progs/ipblacklistctrl.c
>> create mode 100755 src/scripts/ipblacklist
>>
>> -- 
>> 2.16.4
>>
> 

diff --git a/config/firewall/firewall-policy b/config/firewall/firewall-policy
index 1198d120f..21165e933 100755
--- a/config/firewall/firewall-policy
+++ b/config/firewall/firewall-policy
@@ -22,7 +22,6 @@
 eval $(/usr/local/bin/readhash /var/ipfire/ethernet/settings)
 eval $(/usr/local/bin/readhash /var/ipfire/firewall/settings)
 eval $(/usr/local/bin/readhash /var/ipfire/optionsfw/settings)
-eval $(/usr/local/bin/readhash /var/ipfire/ipblacklist/settings)
 
 function iptables() {
 	/sbin/iptables --wait "$@"
@@ -98,10 +97,6 @@ case "${HAVE_OPENVPN},${POLICY}" in
 		;;
 esac
 
-if [ "${AUTOBLACKLIST}" = "on" ]; then
-	iptables -A POLICYIN -i ${IFACE} -m hashlimit --hashlimit-mode srcip --hashlimit-above ${BLOCK_THRESHOLD}/hour --hashlimit-name AUTOBLACKLIST -j SET --add-set AUTOBLACKLIST src
-fi
-
 case "${FWPOLICY2}" in
 	REJECT)
 		if [ "${DROPINPUT}" = "on" ]; then
diff --git a/config/ipblacklist/sources b/config/ipblacklist/sources
index ab991e12a..3cfa7f7d4 100644
--- a/config/ipblacklist/sources
+++ b/config/ipblacklist/sources
@@ -13,139 +13,126 @@
 #                                                                          #
 # The fields are:                                                          #
 #                                                                          #
-# name    The blacklist's full name                                        #
-# url     URL of the file containing the list                              #
-# info    URL giving information about the source                          #
-# parser  The parser function used to extract IP addresses from the        #
-#         downloaded list                                                  #
-# method  Method used to download updates.                                 #
-# rate    Minimum number of hours between checks for updates               #
-# safe    'yes' if the list is unlikely to contain addresses that can be   #
-#         used for legitimate traffic, or 'no' otherwise                   #
-# disable Name of another list to disable if this one is enabled.  Used    #
-#         when the other list is a subset of this one.                     #
+# name     The blacklist's full name                                       #
+# url      URL of the file containing the list                             #
+# info     URL giving information about the source                         #
+# parser   The parser function used to extract IP addresses from the       #
+#          downloaded list                                                 #
+# rate     Minimum period between checks for updates. Can be specified in  #
+#          days (d), hours (h) or minutes (m)                              #
+# category Used for documentation on the WUI.  Can be one of the following #
+#          'application'  Potentially unwanted applications                #
+#          'attacker'     Generic source of malicious packets              #
+#          'c and c'      Malware Command and Control source               #
+#          'composite'    Composite of other lists                         #
+#          'invalid'      Invalid addresses on the public internet         #
+#          'scanner'      Port scanner that is not initself malicious      #
+# disable  Name of another list to disable if this one is enabled.  Used   #
+#          when the other list is a subset of this one.                    #
 #                                                                          #
-# The info and safe fields are purely for documentation.                   #
-#                                                                          #
-# Note that the Emerging Threats blacklist is a composite list containing  #
-# addresses from some of the other lists.  It is unnecessary to enable     #
-# this list if the other lists are enabled.                                #
+# The info and category fields are purely for documentation.               #
 #                                                                          #
 ############################################################################
 
 %sources = ( 'EMERGING_FWRULE' => { 'name'     => 'Emerging Threats Blocklist',
                                     'url'      => 'https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt',
                                     'info'     => 'https://doc.emergingthreats.net/bin/view/Main/EmergingFirewallRules',
-                                    'parser'   => 'text-with-hash-comments',
-                                    'method'   => 'check-header-time',
-                                    'rate'     => 1,
-                                    'safe'     => 'no' },
+                                    'parser'   => 'ip-or-net-list',
+                                    'rate'     => '1h',
+                                    'category' => 'composite',
+                                    'disable'  => ['FEODO_RECOMMENDED', 'FEODO_IP', 'FEODO_AGGRESIVE', 'SPAMHAUS_DROP', 'DSHIELD'] },
              'EMERGING_COMPROMISED' => { 'name' => 'Emerging Threats Compromised IPs',
                                     'url'      => 'https://rules.emergingthreats.net/blockrules/compromised-ips.txt',
                                     'info'     => 'https://doc.emergingthreats.net/bin/view/Main/CompromisedHost',
-                                    'parser'   => 'text-with-hash-comments',
-                                    'method'   => 'check-header-time',
-                                    'rate'     => 1,
-                                    'safe'     => 'no' },
+                                    'parser'   => 'ip-or-net-list',
+                                    'rate'     => '1h',
+                                    'category' => 'attacker' },
              'SPAMHAUS_DROP'   => { 'name'     => "Spamhaus Don't Route or Peer List",
                                     'url'      => 'https://www.spamhaus.org/drop/drop.txt',
                                     'info'     => 'https://www.spamhaus.org/drop/',
-                                    'parser'   => 'text-with-semicolon-comments',
-                                    'method'   => 'check-header-time',
-                                    'rate'     => 12,
-                                    'safe'     => 'yes' },
+                                    'parser'   => 'ip-or-net-list',
+                                    'rate'     => '12h',
+                                    'category' => 'reputation' },
              'SPAMHAUS_EDROP'  => { 'name'     => "Spamhaus Extended Don't Route or Peer List",
                                     'url'      => 'https://www.spamhaus.org/drop/edrop.txt',
                                     'info'     => 'https://www.spamhaus.org/drop/',
-                                    'parser'   => 'text-with-semicolon-comments',
-                                    'method'   => 'check-header-time',
-                                    'rate'     => 1,
-                                    'safe'     => 'no' },
+                                    'parser'   => 'ip-or-net-list',
+                                    'rate'     => '1h',
+                                    'category' => 'reputation' },
              'DSHIELD'         => { 'name'     => 'Dshield.org Recommended Block List',
                                     'url'      => 'https://www.dshield.org/block.txt',
                                     'info'     => 'https://dshield.org/',
                                     'parser'   => 'dshield',
-                                    'method'   => 'check-header-time',
-                                    'rate'     => 2,
-                                    'safe'     => 'no' },
+                                    'rate'     => '1h',
+                                    'category' => 'attacker' },
+             'FEODO_RECOMMENDED'=> {'name'     => 'Feodo Trojan IP Blocklist (Recommended)',
+                                    'url'      => 'https://feodotracker.abuse.ch/downloads/ipblocklist_recommended.txt',
+                                    'info'     => 'https://feodotracker.abuse.ch/blocklist',
+                                    'parser'   => 'ip-or-net-list',
+                                    'rate'     => '5m',
+                                    'category' => 'c and c' },
              'FEODO_IP'        => { 'name'     => 'Feodo Trojan IP Blocklist',
                                     'url'      => 'https://feodotracker.abuse.ch/downloads/ipblocklist.txt',
                                     'info'     => 'https://feodotracker.abuse.ch/blocklist',
-                                    'parser'   => 'text-with-hash-comments',
-                                    'method'   => 'check-header-time',
-                                    'rate'     => 1,
-                                    'safe'     => 'no' },
+                                    'parser'   => 'ip-or-net-list',
+                                    'rate'     => '5m',
+                                    'category' => 'c and c',
+                                    'disable'  => 'FEODO_RECOMMENDED' },
              'FEODO_AGGRESIVE' => { 'name'     => 'Feodo Trojan IP Blocklist (Aggresive)',
                                     'url'      => 'https://feodotracker.abuse.ch/downloads/ipblocklist_aggressive.txt',
                                     'info'     => 'https://feodotracker.abuse.ch/blocklist',
-                                    'parser'   => 'text-with-hash-comments',
-                                    'method'   => 'check-header-time',
-                                    'rate'     => 1,
-                                    'safe'     => 'no',
-                                    'disable'  => 'FEODO_IP' },
-             'ABUSE_CH'        => { 'name'     => 'Abuse.ch Ransomware C&C Blocklist',
-                                    'url'      => 'https://ransomwaretracker.abuse.ch/downloads/RW_IPBL.txt',
-                                    'info'     => 'https://ransomwaretracker.abuse.ch/blocklist/',
-                                    'parser'   => 'text-with-hash-comments',
-                                    'method'   => 'check-header-time',
-                                    'rate'     => 1,
-                                    'safe'     => 'no' },
+                                    'parser'   => 'ip-or-net-list',
+                                    'rate'     => '5m',
+                                    'category' => 'c and c',
+                                    'disable'  => ['FEODO_IP', 'FEODO_RECOMMENDED'] },
              'CIARMY'          => { 'name'     => 'The CINS Army List',
                                     'url'      => 'https://cinsscore.com/list/ci-badguys.txt',
                                     'info'     => 'https://cinsscore.com/#list',
-                                    'parser'   => 'text-with-hash-comments',
-                                    'method'   => 'check-header-time',
-                                    'rate'     => 1,
-                                    'safe'     => 'no' },
+                                    'parser'   => 'ip-or-net-list',
+                                    'rate'     => '15m',
+                                    'category' => 'reputation' },
              'TOR_ALL'         => { 'name'     => 'Known TOR Nodes',
                                     'url'      => 'https://www.dan.me.uk/torlist',
                                     'info'     => 'https://www.dan.me.uk/tornodes',
-                                    'parser'   => 'text-with-hash-comments',
-                                    'method'   => 'wget',
-                                    'rate'     => 1,
-                                    'safe'     => 'no',
+                                    'parser'   => 'ip-or-net-list',
+                                    'rate'     => '1h',
+                                    'category' => 'application',
                                     'disable'  => 'TOR_EXIT' },
              'TOR_EXIT'        => { 'name'     => 'Known TOR Exit Nodes',
                                     'url'      => 'https://www.dan.me.uk/torlist/?exit',
                                     'info'     => 'https://www.dan.me.uk/tornodes',
-                                    'parser'   => 'text-with-hash-comments',
-                                    'method'   => 'wget',
-                                    'rate'     => 1,
-                                    'safe'     => 'no' },
-             'TALOS_MALICIOUS' => { 'name'     => 'Talos Malicious hosts list',
-                                    'url'      => 'https://www.talosintelligence.com/documents/ip-blacklist',
-                                    'info'     => 'https://www.talosintelligence.com/reputation',
-                                    'parser'   => 'text-with-hash-comments',
-                                    'method'   => 'wget',
-                                    'rate'     => 24,
-                                    'safe'     => 'no' },
+                                    'parser'   => 'ip-or-net-list',,
+                                    'rate'     => '1h',
+                                    'category' => 'application' },
              'ALIENVAULT'      => { 'name'     => 'AlienVault IP Reputation database',
                                     'url'      => 'https://reputation.alienvault.com/reputation.generic',
                                     'info'     => 'https://www.alienvault.com/resource-center/videos/what-is-ip-domain-reputation',
-                                    'parser'   => 'text-with-hash-comments',
-                                    'method'   => 'check-header-time',
-                                    'rate'     => 1,
-                                    'safe'     => 'no' },
-              'BOGON'          => { 'name'     => 'Bogus address list (Martian)',
+                                    'parser'   => 'ip-or-net-list',
+                                    'rate'     => '1h',
+                                    'category' => 'reputation' },
+             'BOGON'           => { 'name'     => 'Bogus address list (Martian)',
                                     'url'      => 'https://www.team-cymru.org/Services/Bogons/bogon-bn-agg.txt',
                                     'info'     => 'https://www.team-cymru.com/bogon-reference.html',
-                                    'parser'   => 'text-with-hash-comments',
-                                    'method'   => 'check-header-time',
-                                    'rate'     => 24,
-                                    'safe'     => 'yes' },
-              'BOGON_FULL'     => { 'name'     => 'Full Bogus Address List',
+                                    'parser'   => 'ip-or-net-list',
+                                    'rate'     => '1d',
+                                    'category' => 'invalid' },
+             'BOGON_FULL'      => { 'name'     => 'Full Bogus Address List',
                                     'url'      => 'https://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt',
                                     'info'     => 'https://www.team-cymru.com/bogon-reference.html',
-                                    'parser'   => 'text-with-hash-comments',
-                                    'method'   => 'check-header-time',
-                                    'rate'     => 24,
-                                    'safe'     => 'yes',
+                                    'parser'   => 'ip-or-net-list',
+                                    'rate'     => '4h',
+                                    'category' => 'invalid',
                                     'disable'  => 'BOGON' },
-              'SHODAN'         => { 'name'     => 'ISC Shodan scanner blacklist',
+             'SHODAN'          => { 'name'     => 'ISC Shodan scanner blacklist',
                                     'url'      => 'https://isc.sans.edu/api/threatlist/shodan?tab',
                                     'info'     => 'https://isc.sans.edu',
-                                    'parser'   => 'text-with-hash-comments',
-                                    'method'   => 'wget',
-                                    'rate'     => 24,
-                                    'safe'     => 'no' }
+                                    'parser'   => 'ip-or-net-list',
+                                    'rate'     => '1d',
+                                    'category' => 'scanner' },
+             'BLOCKLIST_DE'    => { 'name'     => 'Blocklist.de all attacks list',
+                                    'url'      => 'https://lists.blocklist.de/lists/all.txt',
+                                    'info'     => 'https://www.blocklist.de',
+                                    'parser'   => 'ip-or-net-list',
+                                    'rate'     => '30m',
+                                    'category' => 'attacker' }
            );
diff --git a/config/logwatch/ipblacklist b/config/logwatch/ipblacklist
index 0fadc6250..6d6c46188 100644
--- a/config/logwatch/ipblacklist
+++ b/config/logwatch/ipblacklist
@@ -49,24 +49,26 @@ while (defined(my $ThisLine = <STDIN>))
 
   my $text = $2;
 
-  if ($text =~ m/Finished updating (\w+) blacklist with (\d+) changes/)
+  if ($text =~ m/Updated (\w+) blacklist with (\d+) changes/)
   {
     $Updates{$1}{updates}++;
     $Updates{$1}{changes} += $2;
   }
-  elsif ($text !~ m/Starting IP Blacklists/               and
-         $text !~ m/Starting IP Blacklist processing/     and
-         $text !~ m/Updating \w+ blacklist/               and
-         $text !~ m/Stopping IP Blacklists/               and
-         $text !~ m/Deleting IP Blacklists/               and
-         $text !~ m/Completed IP Blacklist update/        and
-         $text !~ m/Finished IP Blacklist processing/     and
-         $text !~ m/Blacklist \w+ Modification times/     and
-         $text !~ m/Create IPTables chains for blacklist/ and
-         $text !~ m/Delete IPTables chains for blacklist/ and
-         $text !~ m/Checking modification time for blacklist/ and
-         $text !~ m/Restoring blacklist /                 and
-         $text !~ m/Downloading blacklist/ )
+  elsif ($text !~ m/Starting IP Blacklists/                        and
+         $text !~ m/Starting IP Blacklist processing/              and
+         $text !~ m/Stopping IP Blacklists/                        and
+         $text !~ m/Deleting IP Blacklists/                        and
+         $text !~ m/Finished IP Blacklist processing/              and
+         $text !~ m/Create IPTables chains for blacklist/          and
+         $text !~ m/Delete IPTables chains for blacklist/          and
+         $text !~ m/Add IP Address Blacklist update to crontab/    and
+         $text !~ m/Enable IP Address Blacklist update in crontab/ and
+         $text !~ m/Disable IP Address Blacklist updates/          and
+         $text !~ m/Restoring blacklist /                          and
+         $text !~ m/Blacklist \w+ changed type/                    and
+         $text !~ m/Blacklist \w+ changed size/                    and
+         $text !~ m/Enabling IP Blacklist logging/                 and
+         $text !~ m/Disabling IP Blacklist logging/ )
   {
     $Errors{$text}++;
   }
diff --git a/config/menu/50-firewall.menu b/config/menu/50-firewall.menu
index cd82bfaa3..3cfcde835 100644
--- a/config/menu/50-firewall.menu
+++ b/config/menu/50-firewall.menu
@@ -21,7 +21,7 @@
                                 'title' => "$Lang::tr{'intrusion detection system'}",
 				'enabled' => 1,
                                 };
-    $subfirewall->{'45.ipblacklist'} = {'caption' => $Lang::tr{'ipblacklist'},
+	$subfirewall->{'45.ipblacklist'} = {'caption' => $Lang::tr{'ipblacklist'},
 				'uri' => '/cgi-bin/ipblacklist.cgi',
 				'title' => "$Lang::tr{'ipblacklist'}",
 				'enabled' => 1,
diff --git a/config/menu/70-log.menu b/config/menu/70-log.menu
index 2fa0e426e..c597de60a 100644
--- a/config/menu/70-log.menu
+++ b/config/menu/70-log.menu
@@ -43,12 +43,16 @@
 				'title' => "$Lang::tr{'ids logs'}",
 				'enabled' => 1
 				};
+    $sublogs->{'55.ipblacklist'} = {'caption' => $Lang::tr{'ipblacklist logs'},
+				'uri' => '/cgi-bin/logs.cgi/ipblacklists.dat',
+				'title' => "$Lang::tr{'ipblacklist logs'}",
+				'enabled' => 1
+				};
     $sublogs->{'55.ovpnclients'} = {
 				'caption' => $Lang::tr{'ovpn rw connection log'},
 				'uri' => '/cgi-bin/logs.cgi/ovpnclients.dat',
 				'title' => "$Lang::tr{'ovpn rw connection log'}",
 				'enabled' => 1,
-				};
     $sublogs->{'60.urlfilter'} = {
 				'caption' => $Lang::tr{'urlfilter logs'},
 				'uri' => '/cgi-bin/logs.cgi/urlfilter.dat',
diff --git a/config/rootfiles/common/web-user-interface b/config/rootfiles/common/web-user-interface
index ea31a943a..37a62d357 100644
--- a/config/rootfiles/common/web-user-interface
+++ b/config/rootfiles/common/web-user-interface
@@ -45,9 +45,11 @@ srv/web/ipfire/cgi-bin/logs.cgi/firewalllogcountry.dat
 srv/web/ipfire/cgi-bin/logs.cgi/firewalllogip.dat
 srv/web/ipfire/cgi-bin/logs.cgi/firewalllogport.dat
 srv/web/ipfire/cgi-bin/logs.cgi/ids.dat
+srv/web/ipfire/cgi-bin/logs.cgi/ipblacklists.dat
 srv/web/ipfire/cgi-bin/logs.cgi/log.dat
 srv/web/ipfire/cgi-bin/logs.cgi/ovpnclients.dat
 srv/web/ipfire/cgi-bin/logs.cgi/proxylog.dat
+srv/web/ipfire/cgi-bin/logs.cgi/showrequestfromblacklist.dat
 srv/web/ipfire/cgi-bin/logs.cgi/showrequestfromcountry.dat
 srv/web/ipfire/cgi-bin/logs.cgi/showrequestfromip.dat
 srv/web/ipfire/cgi-bin/logs.cgi/showrequestfromport.dat
diff --git a/html/cgi-bin/ipblacklist.cgi b/html/cgi-bin/ipblacklist.cgi
index b2ccf7b3f..28b42edf2 100644
--- a/html/cgi-bin/ipblacklist.cgi
+++ b/html/cgi-bin/ipblacklist.cgi
@@ -17,13 +17,13 @@
 # You should have received a copy of the GNU General Public License           #
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.       #
 #                                                                             #
-# Copyright (C) 2018 - 2019 The IPFire Team                                   #
+# Copyright (C) 2018 - 2020 The IPFire Team                                   #
 #                                                                             #
 ###############################################################################
 
 use strict;
 use CGI qw/:standard/;
-#enable only the following on debugging purpose
+# enable the following only for debugging purposes
 #use warnings;
 #use CGI::Carp 'fatalsToBrowser';
 use Sort::Naturally;
@@ -34,41 +34,40 @@ require "${General::swroot}/lang.pl";
 require "${General::swroot}/header.pl";
 
 ###############################################################################
-# Initialize variables and hashes
+# Configuration variables
 ###############################################################################
 
 my $settings      = "${General::swroot}/ipblacklist/settings";
-my $modified      = "${General::swroot}/ipblacklist/modified";
 my $sources       = "${General::swroot}/ipblacklist/sources";
 my $getipstat     = '/usr/local/bin/getipstat';
 my $getipsetstat  = '/usr/local/bin/getipsetstat';
 my $control       = '/usr/local/bin/ipblacklistctrl';
 my $lockfile      = '/var/run/ipblacklist.pid';
-my $autoblacklist = 'AUTOBLACKLIST';
-my %cgiparams     = ('ACTION' => '', 'AUTOACTION' => '');
+my %cgiparams     = ('ACTION' => '');
+
+###############################################################################
+# Variables
+###############################################################################
+
 my $errormessage  = '';
 my $updating      = 0;
 my %mainsettings;
 my %color;
-my %modified;
 my %sources;
 my %stats;
-my %autoblock_addresses;
+
+# Default settings - normally overwritten by settings file
 
 my %settings = ( 'DEBUG'           => 0,
                  'LOGGING'         => 'on',
-                 'RATE'            => 24,
-                 'ENABLE'          => 'off',
-                 'BLOCK_THRESHOLD' => 10,
-                 'BLOCK_PERIOD'    => 3600,
-                 $autoblacklist    => 'off' );
+                 'ENABLE'          => 'off' );
+
+# Read all parameters
 
-# Read all parameters for site
 Header::getcgihash( \%cgiparams);
 General::readhash( "${General::swroot}/main/settings", \%mainsettings );
 General::readhash( "/srv/web/ipfire/html/themes/".$mainsettings{'THEME'}."/include/colors.txt", \%color );
 General::readhash( $settings, \%settings ) if (-r $settings);
-General::readhash( $modified, \%modified)  if (-r $modified);
 eval qx|/bin/cat $sources|                 if (-r $sources);
 
 # Show Headers
@@ -79,22 +78,11 @@ Header::showhttpheaders();
 
 if ($cgiparams{'ACTION'} eq "$Lang::tr{'save'}")
 {
-  #Save Button on configsite
+  # Save Button
 
   my %new_settings = ( 'ENABLE'          => 'off',
-                       'RATE'            => 24,
                        'LOGGING'         => 'off',
-                       'DEBUG'           => 0,
-                       'BLOCK_THRESHOLD' => $settings{'BLOCK_THRESHOLD'} || 10,
-                       'BLOCK_PERIOD'    => $settings{'BLOCK_PERIOD'} || 3600,
-                       $autoblacklist    => $settings{$autoblacklist} );
-
-  $errormessage .= "$Lang::tr{'ipblacklist invalid check rate'}<br>" if (($cgiparams{'RATE'} !~ m/^\d+$/) or
-                                                                         ($cgiparams{'RATE'} < 1) or
-                                                                         ($cgiparams{'RATE'} > 1000));
-
-  $new_settings{'RATE'} = $cgiparams{'RATE'};
-  delete $cgiparams{'RATE'};
+                       'DEBUG'           => 0 );
 
   foreach my $item ('LOGGING', 'ENABLE', keys %sources)
   {
@@ -107,16 +95,32 @@ if ($cgiparams{'ACTION'} eq "$Lang::tr{'save'}")
 
   foreach my $list (keys %sources)
   {
-    if (exists $settings{$list} and
-        $settings{$list} eq 'on' and
-        exists $sources{$list}{'override'} and
-        $settings{$sources{$list}{'override'}} eq 'on')
+    if (exists $new_settings{$list} and
+        $new_settings{$list} eq 'on' and
+        exists $sources{$list}{'disable'})
     {
-      $settings{$sources{$list}{'override'}} = 'off';
+      my @disable;
+
+      if ('ARRAY' eq ref $sources{$list}{'disable'})
+      {
+        @disable = @{ $sources{$list}{'disable'} };
+      }
+      else
+      {
+        @disable = ( $sources{$list}{'disable'} );
+      }
 
-      $updating      = 1;
-      $errormessage .= "$Lang::tr{'ipblacklist disable pre'} $sources{$list}{'override'} " .
-                       "$Lang::tr{'ipblacklist disable mid'} $list $Lang::tr{'ipblacklist disable post'}<br>\n";
+      foreach my $disable (@disable)
+      {
+        if ($new_settings{$disable} eq 'on')
+        {
+          $new_settings{$disable} = 'off';
+
+          $updating      = 1;
+          $errormessage .= "$Lang::tr{'ipblacklist disable pre'} $disable " .
+                            "$Lang::tr{'ipblacklist disable mid'} $list $Lang::tr{'ipblacklist disable post'}<br>\n";
+        }
+      }
     }
   }
 
@@ -140,7 +144,6 @@ if ($cgiparams{'ACTION'} eq "$Lang::tr{'save'}")
     }
     else
     {
-      $settings{$autoblacklist} = 'off';
       system( "$control disable" );
     }
 
@@ -160,51 +163,11 @@ if ($cgiparams{'ACTION'} eq "$Lang::tr{'save'}")
     if ($updating)
     {
       system( "$control update &" );
-      get_ipset_stats();
       show_running();
       exit 0;
     }
   }
 }
-elsif ($cgiparams{'AUTOACTION'} eq "$Lang::tr{'save'}")
-{
-  $updating = 1 if ($settings{$autoblacklist} eq 'on' and not exists $cgiparams{$autoblacklist});
-  $updating = 1 if ($settings{$autoblacklist} eq 'off' and exists $cgiparams{$autoblacklist});
-
-  $settings{$autoblacklist}    = (exists $cgiparams{$autoblacklist}) ? 'on' : 'off';
-  $settings{'BLOCK_THRESHOLD'} = $cgiparams{'BLOCK_THRESHOLD'};
-  $settings{'BLOCK_PERIOD'}    = $cgiparams{'BLOCK_PERIOD'} ;
-
-  if (($cgiparams{'BLOCK_THRESHOLD'} !~ m/^\d+$/) or
-      ($cgiparams{'BLOCK_THRESHOLD'} < 1) or
-      ($cgiparams{'BLOCK_THRESHOLD'} > 1000000))
-  {
-    $errormessage .= "$Lang::tr{'ipblacklist invalid threshold'}: $cgiparams{'BLOCK_THRESHOLD'}<br>";
-  }
-
-  if (($cgiparams{'BLOCK_PERIOD'} !~ m/^\d+$/) or
-      ($cgiparams{'BLOCK_PERIOD'} < 1) or
-      ($cgiparams{'BLOCK_PERIOD'} > 86400))
-  {
-    $errormessage .= "$Lang::tr{'ipblacklist invalid block time'}: $cgiparams{'BLOCK_PERIOD'}<br>";
-  }
-
-  if ($errormessage)
-  {
-    $updating = 0;
-  }
-  else
-  {
-    General::writehash($settings, \%settings);
-    system( "$control autoblacklist-update" ) if ($updating);
-  }
-}
-elsif ($cgiparams{'AUTOACTION'} eq "$Lang::tr{'unblock all'}")
-{
-  system( "$control autoblacklist-clear" );
-}
-
-get_ipset_stats();
 
 if (is_running())
 {
@@ -212,17 +175,12 @@ if (is_running())
   exit 0;
 }
 
-# Get blacklist statistics
-
-get_iptables_stats();
-
 # Show site
 
 Header::openpage($Lang::tr{'ipblacklist'}, 1, '');
 Header::openbigbox('100%', 'left');
-error();
 
-showstatus() if ($settings{ENABLE} eq 'on');
+error() if ($errormessage);
 
 configsite();
 
@@ -299,10 +257,6 @@ END
   <tr>
     <td style='width:24em'>$Lang::tr{'ipblacklist log'}</td>
     <td><input type='checkbox' name="LOGGING" id="LOGGING"$enable></td>
-    <td style='width:24em'>$Lang::tr{'ipblacklist check rate'}</td>
-    <td>
-      <input type='number' name='RATE' min='1' max='1000' maxlength='7' pattern='\\d+' value='$settings{'RATE'}'>
-    </td>
   </tr>
   </table>
   <br><br>
@@ -311,7 +265,7 @@ END
   <tr>
     <th align='left'>$Lang::tr{'ipblacklist id'}</th>
     <th align='left'>$Lang::tr{'ipblacklist name'}</th>
-    <th align='center'>$Lang::tr{'ipblacklist safe'}</th>
+    <th align='left'>$Lang::tr{'ipblacklist category'}</th>
     <th align='center'>$Lang::tr{'ipblacklist enable'}</th>
   </tr>
 END
@@ -322,10 +276,10 @@ END
 
   foreach my $list (sort keys %sources)
   {
-    my $name    = escapeHTML( $sources{$list}{'name'} );
-    my $safe    = $Lang::tr{$sources{$list}{safe}};
-    $enable     = '';
-    my $col     = ($lines++ % 2) ? "bgcolor='$color{'color20'}'" : "bgcolor='$color{'color22'}'";
+    my $name     = escapeHTML( $sources{$list}{'name'} );
+    my $category = $Lang::tr{"ipblacklist category $sources{$list}{'category'}"};
+    $enable      = '';
+    my $col      = ($lines++ % 2) ? "bgcolor='$color{'color20'}'" : "bgcolor='$color{'color22'}'";
 
     $enable = ' checked' if (exists $settings{$list} and $settings{$list} eq 'on');
 
@@ -346,7 +300,7 @@ END
     print <<END;
     </td>
     <td>$name</td>
-    <td align='center'>$safe</td>
+    <td>$category</td>
     <td align='center'><input type='checkbox' name="$list" id="$list"$enable></td>
     </tr>\n
 END
@@ -356,165 +310,18 @@ END
 
     print <<END;
     </table>
-    <p>$Lang::tr{'ipblacklist safe note'}</p>
   </div>
     <table style='width:100%;'>
     <tr>
         <td colspan='3' display:inline align='right'><input type='submit' name='ACTION' value='$Lang::tr{'save'}'></td>
     </tr>
     </table>
-  <div class='sources'>
-    <br>
-END
-
-    $enable = $settings{$autoblacklist} eq 'on' ? ' checked' : '';
-
-    print <<END;
-    <br><br>
-    <h2>$Lang::tr{'ipblacklist auto list'}</h2>
-      <table style='width:100%' border='0'>
-      <tr>
-        <td>
-          $Lang::tr{'ipblacklist autoblacklist enable'}
-        </td>
-        <td>
-          <input type='checkbox' name="$autoblacklist" id="$autoblacklist"$enable>
-        </td>
-        <td>
-          &nbsp;
-        </td>
-      </tr>
-      <tr>
-        <td>
-          $Lang::tr{'ipblacklist autoblacklist threshold'}
-        </td>
-        <td>
-          <input type='number' name='BLOCK_THRESHOLD' min='1' max='1000000' maxlength='7' pattern='\\d+' value='$settings{BLOCK_THRESHOLD}'>
-          </input>
-        </td>
-      </tr>
-      <tr>
-        <td>
-          $Lang::tr{'ipblacklist autoblacklist block time'}
-        </td>
-        <td>
-          <input type='number' name='BLOCK_PERIOD' min='1' max='86400' maxlength='7' pattern='\\d+' value='$settings{BLOCK_PERIOD}'>
-          </input>
-        </td>
-      </tr>
-      </table>
-      <table style='width:100%;'>
-      <tr>
-          <td colspan='3' display:inline align='right'><input type='submit' name='AUTOACTION' value='$Lang::tr{'save'}'></td>
-      </tr>
-      </table>
-    </form>
-  </div>
 END
 
   Header::closebox();
 }
 
 
-#------------------------------------------------------------------------------
-# sub showstatus()
-#
-# Displays current blacklist status
-#------------------------------------------------------------------------------
-
-sub showstatus
-{
-  Header::openbox('100%', 'center', $Lang::tr{'status'});
-
-  print <<END;
-  <table width='100%' cellspacing='1'>
-  <tr>
-    <th align='left'>$Lang::tr{'ipblacklist id'}</th>
-    <th align='right'>$Lang::tr{'ipblacklist entries'}</th>
-    <th align='right'>$Lang::tr{'ipblacklist pkts in'}</th>
-    <th align='right'>$Lang::tr{'ipblacklist bytes in'}</th>
-    <th align='right'>$Lang::tr{'ipblacklist pkts out'}</th>
-    <th align='right'>$Lang::tr{'ipblacklist bytes out'}</th>
-    <th align='center'>$Lang::tr{'ipblacklist updated'}</th>
-  </tr>
-END
-
-  # Iterate through the list of sources
-
-  foreach my $list ($autoblacklist, sort keys %sources)
-  {
-    next unless ($settings{$list} eq 'on');
-
-    my $size      = '&nbsp;';
-    my $pkts_in   = '&nbsp;';
-    my $bytes_in  = '&nbsp;';
-    my $pkts_out  = '&nbsp;';
-    my $bytes_out = '&nbsp;';
-    my $updated   = '&nbsp;';
-
-    if (exists $stats{$list})
-    {
-      ($pkts_in,  $bytes_in)  = @{ $stats{$list}{IPBLACKLISTREDIN} }  if (exists $stats{$list}{IPBLACKLISTREDIN});
-      ($pkts_out, $bytes_out) = @{ $stats{$list}{IPBLACKLISTREDOUT} } if (exists $stats{$list}{IPBLACKLISTREDOUT});
-      $size                   = $stats{$list}{size}                   if (exists $stats{$list}{size});
-    }
-
-    if (exists $modified{$list} and $modified{$list} > 0)
-    {
-      $updated = localtime( $modified{$list} );
-    }
-
-    print <<END;
-    <tr>
-    <td>$list</td>
-    <td align='right'>$size</td>
-    <td align='right'>$pkts_in</td>
-    <td align='right'>$bytes_in</td>
-    <td align='right'>$pkts_out</td>
-    <td align='right'>$bytes_out</td>
-    <td align='center'>$updated</td>
-    </tr>\n
-END
-
-  }
-
-  print <<END;
-  </table>
-END
-
-  if ($settings{$autoblacklist} eq 'on')
-  {
-    print <<END;
-    <br><br>
-    <h2>$Lang::tr{'ipblacklist auto list'}</h2>
-    <table width='60%' cellspacing='1'>
-    <tr>
-      <th align='left'>$Lang::tr{'ip address'}</th>
-      <th align='center'>$Lang::tr{'ipblacklist block time remaining'}</th>
-    </tr>
-END
-
-    foreach my $address (nsort keys %autoblock_addresses )
-    {
-      print "<tr><td>$address</td><td align='center'>$autoblock_addresses{$address}</td></tr>\n";
-    }
-
-    print <<END;
-    </table>
-    <table width='60%' border='0'>
-      <form method='post' action='$ENV{'SCRIPT_NAME'}'>
-        <tr>
-            <td align='right' width='15%'><input type='submit' name='ACTION' value='$Lang::tr{'unblock all'}'></td>
-        </tr>
-      </form>
-    </table>
-END
-  }
-
-  Header::closebox();
-}
-
-
 #------------------------------------------------------------------------------
 # sub get_ipset_stats()
 #
@@ -527,8 +334,6 @@ sub get_ipset_stats
 
   system( $getipsetstat );
 
-  # Get the number of entries in each IP set
-
   if (-r '/var/tmp/ipsets.txt')
   {
     open STATS, '<', '/var/tmp/ipsets.txt' or die "Can't open IP Sets stats file: $!";
@@ -551,66 +356,6 @@ sub get_ipset_stats
 
     unlink( '/var/tmp/ipsets.txt' );
   }
-
-  # Get the IP addresses in the autoblacklist
-
-  if (-r '/var/tmp/autoblacklist.txt')
-  {
-    open HASHTABLE, '<', '/var/tmp/autoblacklist.txt' or die "Can't open autoblacklist address file: $!";
-
-    # Iterate through the blocked addresses
-
-    foreach my $line (<HASHTABLE>)
-    {
-      next unless ($line =~ m/(\d+\.\d+\.\d+\.\d+) timeout (\d+)/);
-
-      $autoblock_addresses{$1} = format_time( $2 );
-    }
-
-    close HASHTABLE;
-
-    unlink( '/var/tmp/autoblacklist.txt' );
-  }
-
-}
-
-
-#------------------------------------------------------------------------------
-# sub get_iptables_stats()
-#
-# Gets information on the number of packets and bytes rejected by each
-# blacklist
-#------------------------------------------------------------------------------
-
-sub get_iptables_stats
-{
-  system( $getipstat );
-  unlink( '/var/tmp/iptablesmangle.txt' );
-  unlink( '/var/tmp/iptablesnat.txt' );
-
-  return unless (-r '/var/tmp/iptables.txt');
-
-  open STATS, '<', '/var/tmp/iptables.txt' or die "Can't open IP Tables stats file: $!";
-
-  my $table = 'Unknown';
-
-  foreach my $line (<STATS>)
-  {
-    if ($line =~ m/^Chain (\w+)/)
-    {
-      $table = $1;
-      next;
-    }
-
-    next unless ($line =~ m/_BLOCK/);
-
-    my ($pkts, $bytes, $chain) = $line =~ m/^\s*(\d+\w?)\s+(\d+\w?)\s+(\w+)_BLOCK/;
-    $stats{$chain}{$table} = [ $pkts, $bytes ];
-  }
-
-  close STATS;
-
-  unlink( '/var/tmp/iptables.txt' );
 }
 
 
@@ -637,7 +382,8 @@ sub is_running
 #------------------------------------------------------------------------------
 # sub show_running
 #
-# Displayed when update is running
+# Displayed when update is running.
+# Shows a 'working' message plus some information about the IPSets.
 #------------------------------------------------------------------------------
 
 sub show_running
@@ -662,12 +408,7 @@ sub show_running
     <tr><th>$Lang::tr{'ipblacklist id'}</th><th>$Lang::tr{'ipblacklist entries'}</th></tr>
 END
 
-  foreach my $name (keys %sources)
-  {
-    $stats{$name}{'size'} = '&nbsp;' if (not exists ($stats{$name})  and
-                                             exists $settings{$name} and
-                                             $settings{$name} eq 'on');
-  }
+  get_ipset_stats();
 
   foreach my $name (sort keys %stats)
   {
@@ -693,13 +434,10 @@ END
 
 sub error
 {
-  if ($errormessage)
-  {
-    Header::openbox('100%', 'left', $Lang::tr{'error messages'});
-    print "<class name='base'>$errormessage\n";
-    print "&nbsp;</class>\n";
-    Header::closebox();
-  }
+  Header::openbox('100%', 'left', $Lang::tr{'error messages'});
+  print "<class name='base'>$errormessage\n";
+  print "&nbsp;</class>\n";
+  Header::closebox();
 }
 
 
diff --git a/langs/en/cgi-bin/en.pl b/langs/en/cgi-bin/en.pl
index b10fbdbf2..ab590cc1b 100644
--- a/langs/en/cgi-bin/en.pl
+++ b/langs/en/cgi-bin/en.pl
@@ -1,4 +1,4 @@
-%tr = ( 
+%tr = (
 %tr,
 
 '24 hours' => '24 Hours',
@@ -1538,37 +1538,31 @@
 'ip alias changed' => 'External IP alias changed',
 'ip alias removed' => 'External IP alias removed',
 'ip info' => 'IP Information',
-'ipblacklist auto list' => 'Automatic blacklist',
-'ipblacklist autoblacklist enable' => 'Enable automatically updating local blacklist',
-'ipblacklist autoblacklist threshold' => 'Threshold (packets/hour)',
-'ipblacklist autoblacklist block time' => 'Block time (seconds)',
+'ipblacklist' => 'IP Address Blacklists',
 'ipblacklist blacklist settings' => 'Blacklist settings',
-'ipblacklist bytes in' => 'bytes in',
-'ipblacklist bytes out' => 'bytes out',
-'ipblacklist check rate' => 'Update check rate (hours)',
-'ipblacklist day' => 'day',
-'ipblacklist disable pre' => 'Disabling',
+'ipblacklist category' => 'Category',
+'ipblacklist category application' => 'Application',
+'ipblacklist category attacker' => 'Attacker',
+'ipblacklist category c and c' => 'Malware C&amp;C',
+'ipblacklist category composite' => 'Composite',
+'ipblacklist category invalid' => 'Invalid Address',
+'ipblacklist category reputation' => 'Reputation',
+'ipblacklist category scanner' => 'Scanner',
 'ipblacklist disable mid' => 'because it is included in',
-'ipblacklist disable port' => '',
+'ipblacklist disable post' => '',
+'ipblacklist disable pre' => 'Disabling',
 'ipblacklist enable' => 'Enable',
 'ipblacklist entries' => 'Entries',
-'ipblacklist hour' => 'hour',
+'ipblacklist hits' => 'Total number of blacklist hits for',
 'ipblacklist id' => 'Blacklist',
-'ipblacklist invalid block time' => 'Invalid automatic blacklist block time',
-'ipblacklist invalid check rate' => 'Invalid update check rate',
-'ipblacklist invalid threshold' => 'Invalid automatic blacklist threshold',
+'ipblacklist input' => 'Packets Dropped In',
+'ipblacklist log list' => 'Firewall log (blacklist)',
 'ipblacklist log' => 'Log dropped packets',
+'ipblacklist logs' => 'IP Address Blacklist Logs',
 'ipblacklist name' => 'Name',
-'ipblacklist pkts in' => 'pkts in',
-'ipblacklist pkts out' => 'pkts out',
-'ipblacklist safe note' => 'Note: safe blacklists block addresses that only generate malicious traffic and therefore will not block any wanted sites.',
-'ipblacklist safe' => 'Safe',
-'ipblacklist sixhour' => 'six hours',
-'ipblacklist updated' => 'Last updated',
+'ipblacklist output' => 'Packets Dropped Out',
 'ipblacklist use ipblacklists' => 'Enable IP Blacklists',
-'ipblacklist week' => 'week',
 'ipblacklist working' => 'Updating IP address blacklists...',
-'ipblacklist' => 'IP Address Blacklists',
 'ipfire has now rebooted' => 'IPFire is rebooting now.',
 'ipfire has now shutdown' => 'IPFire is shutting down now.',
 'ipfire side' => 'IPFire side:',
diff --git a/lfs/configroot b/lfs/configroot
index ba87debb1..90b90eb3c 100644
--- a/lfs/configroot
+++ b/lfs/configroot
@@ -65,7 +65,7 @@ $(TARGET) :
 	    captive/settings captive/agb.txt captive/clients captive/voucher_out certs/index.txt certs/index.txt.attr ddns/config ddns/settings ddns/ipcache dhcp/settings \
 	    dhcp/fixleases dhcp/advoptions dhcp/dhcpd.conf.local dns/settings dns/servers dnsforward/config ethernet/aliases ethernet/settings ethernet/known_nics ethernet/scanned_nics \
 	    ethernet/wireless extrahd/scan extrahd/devices extrahd/partitions extrahd/settings firewall/settings firewall/config firewall/geoipblock firewall/input firewall/outgoing \
-	    fwhosts/customnetworks fwhosts/customhosts fwhosts/customgroups fwhosts/customservicegrp fwhosts/customgeoipgrp fwlogs/ipsettings fwlogs/portsettings ipblacklist/settings\
+	    fwhosts/customnetworks fwhosts/customhosts fwhosts/customgroups fwhosts/customservicegrp fwhosts/customgeoipgrp fwlogs/ipsettings fwlogs/portsettings ipblacklist/settings \
 	    isdn/settings mac/settings main/hosts main/routing main/security main/settings optionsfw/settings \
 	    ovpn/ccd.conf ovpn/ccdroute ovpn/ccdroute2 pakfire/settings portfw/config ppp/settings-1 ppp/settings-2 ppp/settings-3 ppp/settings-4 \
 	    ppp/settings-5 ppp/settings proxy/settings proxy/squid.conf proxy/advanced/settings proxy/advanced/cre/enable remote/settings qos/settings qos/classes qos/subclasses qos/level7config qos/portconfig \
diff --git a/make.sh b/make.sh
index 3d111dbab..25b8b1d59 100755
--- a/make.sh
+++ b/make.sh
@@ -1665,7 +1665,7 @@ buildpackages() {
   export LOGFILE
   echo "... see detailed log in _build.*.log files" >> $LOGFILE
 
-
+  
   # Generating list of packages used
   print_line "Generating packages list from logs"
   rm -f $BASEDIR/doc/packages-list
@@ -1680,7 +1680,7 @@ buildpackages() {
   rm -f $BASEDIR/doc/packages-list
   # packages-list.txt is ready to be displayed for wiki page
   print_status DONE
-
+  
   # Update changelog
   cd $BASEDIR
   [ -z $GIT_TAG ]  || LAST_TAG=$GIT_TAG
@@ -1755,7 +1755,7 @@ while [ $# -gt 0 ]; do
 done
 
 # See what we're supposed to do
-case "$1" in
+case "$1" in 
 build)
 	START_TIME=$(now)
 
@@ -1794,7 +1794,7 @@ build)
 
 	print_build_stage "Building packages"
 	buildpackages
-
+	
 	print_build_stage "Checking Logfiles for new Files"
 
 	cd $BASEDIR
@@ -1859,7 +1859,7 @@ downloadsrc)
 	FINISHED=0
 	cd $BASEDIR/lfs
 	for c in `seq $MAX_RETRIES`; do
-		if (( FINISHED==1 )); then
+		if (( FINISHED==1 )); then 
 			break
 		fi
 		FINISHED=1
diff --git a/src/initscripts/system/firewall b/src/initscripts/system/firewall
index ebb73062d..08c434440 100644
--- a/src/initscripts/system/firewall
+++ b/src/initscripts/system/firewall
@@ -180,17 +180,7 @@ iptables_init() {
 	iptables -A FORWARD -j P2PBLOCK
 	iptables -A OUTPUT -j P2PBLOCK
 	
-	# IP Address Blacklist chains
-	iptables -N IPBLACKLISTIN
-	iptables -N IPBLACKLISTOUT
-	iptables -N IPBLACKLISTREDIN
-	iptables -N IPBLACKLISTREDOUT
-	iptables -A INPUT ! -p icmp -j IPBLACKLISTIN
-	iptables -A FORWARD ! -p icmp -j IPBLACKLISTIN
-	iptables -A FORWARD ! -p icmp -j IPBLACKLISTOUT
-	iptables -A OUTPUT ! -p icmp -j IPBLACKLISTOUT
-
-	# Guardian (IPS) chains
+	# IPS (Guardian) chains
 	iptables -N GUARDIAN
 	iptables -A INPUT -j GUARDIAN
 	iptables -A FORWARD -j GUARDIAN
@@ -206,7 +196,15 @@ iptables_init() {
 	iptables -A FORWARD -i tun+ -j OVPNBLOCK
 	iptables -A FORWARD -o tun+ -j OVPNBLOCK
 
-	# IPS (Suricata) chains
+	# IP Address Blacklist chains
+	iptables -N BLACKLISTIN
+	iptables -N BLACKLISTOUT
+	iptables -A INPUT ! -p icmp -j BLACKLISTIN
+	iptables -A FORWARD ! -p icmp -j BLACKLISTIN
+	iptables -A FORWARD ! -p icmp -j BLACKLISTOUT
+	iptables -A OUTPUT ! -p icmp -j BLACKLISTOUT
+
+	# IPS (suricata) chains
 	iptables -N IPS_INPUT
 	iptables -N IPS_FORWARD
 	iptables -N IPS_OUTPUT
@@ -420,8 +418,6 @@ iptables_red_up() {
 	iptables -F REDINPUT
 	iptables -F REDFORWARD
 	iptables -t nat -F REDNAT
-	iptables -F IPBLACKLISTIN
-	iptables -F IPBLACKLISTOUT
 
 	# PPPoE / PPTP Device
 	if [ "$IFACE" != "" ]; then
@@ -479,10 +475,6 @@ iptables_red_up() {
 			iptables -t nat -A REDNAT -s "${network}" -o "${IFACE}" -j RETURN
 		done
 
-		# IP Address Blacklists
-		iptables -A IPBLACKLISTIN -i $IFACE -j IPBLACKLISTREDIN
-		iptables -A IPBLACKLISTOUT -o $IFACE -j IPBLACKLISTREDOUT
-
 		# Masquerade everything else
 		iptables -t nat -A REDNAT -o $IFACE -j MASQUERADE
 	fi
diff --git a/src/misc-progs/getipsetstat.c b/src/misc-progs/getipsetstat.c
index aee79542a..781bfc55b 100644
--- a/src/misc-progs/getipsetstat.c
+++ b/src/misc-progs/getipsetstat.c
@@ -21,8 +21,5 @@ int main(void)
 	safe_system("/usr/sbin/ipset list -t -f /var/tmp/ipsets.txt");
 	safe_system("chown nobody:nobody /var/tmp/ipsets.txt");
 
-	safe_system("/usr/sbin/ipset list AUTOBLACKLIST -q -f /var/tmp/autoblacklist.txt");
-	safe_system("chown -f nobody:nobody /var/tmp/autoblacklist.txt");
-
 	return 0;
 }
diff --git a/src/misc-progs/ipblacklistctrl.c b/src/misc-progs/ipblacklistctrl.c
index 506fa2f46..7536b1e97 100644
--- a/src/misc-progs/ipblacklistctrl.c
+++ b/src/misc-progs/ipblacklistctrl.c
@@ -21,7 +21,7 @@ int main(int argc, char *argv[]) {
     if (argc < 2) {
         fprintf(stderr, "\nNo argument given.\n"
                 "ipblacklistctrl (update|restore|log-on|log-off|"
-                "enable|disable|autoblacklist-update|autoblacklist-clear)\n\n");
+                "enable|disable)\n\n");
         exit(1);
     }
 
@@ -37,14 +37,10 @@ int main(int argc, char *argv[]) {
         safe_system("/usr/local/bin/ipblacklist enable >/dev/null 2>&1 &");
     } else if (strcmp(argv[1], "disable") == 0) {
         safe_system("/usr/local/bin/ipblacklist disable >/dev/null 2>&1 &");
-    } else if (strcmp(argv[1], "autoblacklist-update") == 0) {
-        safe_system("/usr/local/bin/ipblacklist autoblacklist-update >/dev/null 2>&1 &");
-    } else if (strcmp(argv[1], "autoblacklist-clear") == 0) {
-        safe_system("/usr/local/bin/ipblacklist autoblacklist-clear >/dev/null 2>&1 &");
     } else {
         fprintf(stderr, "\nBad argument given.\n"
                 "ipblacklistctrl (update|restore|log-on|log-off|"
-                "enable|disable|autoblacklist-update|autoblacklist-clear)\n\n");
+                "enable|disable)\n\n");
         exit(1);
     }
 
diff --git a/src/scripts/ipblacklist b/src/scripts/ipblacklist
index b3f8048d9..6f950214c 100755
--- a/src/scripts/ipblacklist
+++ b/src/scripts/ipblacklist
@@ -18,25 +18,26 @@
 # along with IPFire; if not, write to the Free Software                    #
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA #
 #                                                                          #
-# Copyright (C) 2018 - 2019 The IPFire team                                #
+# Copyright (C) 2018 - 2020 The IPFire team                                #
 #                                                                          #
 ############################################################################
 #                                                                          #
-# This script use a file containing blacklist details in                   #
+# This script uses a file containing blacklist details in                  #
 # /var/ipfire/ipblacklist/sources as well as                               #
 # /var/ipfire/ipblacklistsettings containing an enable/disable flag for    #
 # each source.                                                             #
 #                                                                          #
-# Two IPTables chains are used, IPBLACKLISTREDIN and IPBLACKLISTREDOUT,    #
-# which are inserted into the main INPUT, OUTPUT and FORWARD chains.       #
+# Two IPTables chains are used: BLACKLISTIN and BLACKLISTOUT are inserted  #
+# inserted into the main INPUT, OUTPUT and FORWARD chains; they capture    #
+# packets other than for the ICMP protocol.                                #
 #                                                                          #
 # For each blacklist that is loaded, a chain is created to optionally log  #
 # and then to drop matching packets. An IPSet is created containing the    #
 # addresses or networks blocked by the blacklist, and then rules are added #
-# to the IPBLACKLISTREDIN and IPBLACKLISTREDOUT chains to jump to this     #
-# chain if appropriate packet list matches in the set.                     #
+# to the BLACKLISTIN and BLACKLISTOUT chains to jump to this chain if a    #
+# packet list matches the set.                                             #
 #                                                                          #
-# When checking for updates, the modification time is read for each source #
+# When checking for updates, the modification time is used for each source #
 # and if necessary the list is downloaded.  The downloaded list is         #
 # compared to the existing IPSet contents and entries created or deleted   #
 # as necessary.                                                            #
@@ -44,7 +45,7 @@
 ############################################################################
 
 use strict;
-use warnings;
+#use warnings;
 
 use Carp;
 use Sys::Syslog qw(:standard :macros);
@@ -76,23 +77,29 @@ my $lockfile       = "/var/run/ipblacklist.pid";
 my $proxy_settings = "${General::swroot}/proxy/settings";
 my $red_setting    = "/var/ipfire/red/iface";
 my $detailed_log   = "$tmpdir/ipblacklist_log.txt";
-my $autoblacklist  = 'AUTOBLACKLIST';
+my $active         = "/var/ipfire/red/active";
 
-my %parsers  = ( 'text-with-hash-comments'      => \&parse_text_with_hash_comments,
-                 'text-with-semicolon-comments' => \&parse_text_with_semicolon_comments,
-                 'dshield'                      => \&parse_dshield );
+# Other configuration items
+
+my $margin            = 30;         # Scheduling allowance for run time etc in seconds
+my $count             = 30;         # Maximum time to wait for another instance (300s)
+my $max_dl_fails      = 3;          # Ignore check rate limit for this number of failures
+my $max_size_fraction = 0.7;        # Maximum fill fraction of IPSet before enlarging.
+my $min_ipset_entries = 1024;       # The minimum size of an IPSet.
+my $max_dl_bytes      = 10_485_760; # Maximum number of bytes to download.
+my %parsers           = ( 'ip-or-net-list' => \&parse_ip_or_net_list,
+                          'dshield'        => \&parse_dshield );
 
 ############################################################################
 # Default settings
 # Should be overwritten by reading settings files
 ############################################################################
 
-my %sources  = ( );
+my %sources        = ( );
 
-my %settings = ( 'DEBUG'           => 0,
-                 'LOGGING'         => 'on',
-                 'RATE'            => 24,
-                 'ENABLE'          => 'off' );
+my %settings       = ( 'DEBUG'           => 0,
+                       'LOGGING'         => 'on',
+                       'ENABLE'          => 'off' );
 
 my %proxy_settings = ( 'UPSTREAM_PROXY' => '' );         # No Proxy in use
 
@@ -101,13 +108,9 @@ my %proxy_settings = ( 'UPSTREAM_PROXY' => '' );         # No Proxy in use
 ############################################################################
 
 sub abort( $ );
-sub autoblacklist_update();
-sub autoblacklist_clear();
-sub create_autoblacklist();
 sub create_list( $ );
 sub create_ipset( $$$ );
 sub debug( $$ );
-sub delete_autoblacklist();
 sub delete_list( $ );
 sub disable_logging();
 sub disable_updates();
@@ -121,15 +124,15 @@ sub download_wget( $$$ );
 sub enable_logging();
 sub enable_updates();
 sub get_ipsets();
+sub get_rate_seconds( $ );
 sub iptables( $ );
 sub ipset( $ );
 sub stop_ipset();
 sub is_connected();
 sub log_message( $$ );
 sub parse_dshield( $ );
-sub parse_text_with_hash_comments( $ );
-sub parse_text_with_semicolon_comments( $ );
-sub read_ipset( $$$ );
+sub parse_ip_or_net_list( $ );
+sub read_ipset( $$$$ );
 sub update_list( $$$ );
 
 ############################################################################
@@ -144,25 +147,18 @@ my $ipset_running = 0;     # Set to 1 if IPSet process is running
 my %status;                # Status information
 my %checked;               # Time blacklists last changed
 my %modified;              # Time blacklists last modified
-my $red_iface;             # Name of red interface
-my $hours          = 3600; # One hour in seconds
-my $margin         = 600;  # Allowance for run time etc
-my $count          = 30;   # Maximum time to wait for another instance (300s)
-my @wget_status    = ( 'Success', 'Error', 'Parse Error', 'File I/O Error',
-                       'Network Error', 'SSL Verification Error',
-                       'Authentication Error', 'Protocol Error', 'Server Error' );
-
+my $red_iface;             # The name of the red interface
 
 ############################################################################
 # Synchronise runs
 ############################################################################
 
 # This script can be triggered either by cron or the WUI.  If another
-# instance is running, wait for it to finish.
+# instance is running, wait for it to finish or timeout.
 
 while (-r $lockfile and $count > 0)
 {
-  open LOCKFILE, '<', $lockfile or die "Can't open lockfile";
+  open LOCKFILE, '<', $lockfile or (abort "Can't open lockfile", last);
   my $pid = <LOCKFILE>;
   close LOCKFILE;
 
@@ -176,7 +172,7 @@ while (-r $lockfile and $count > 0)
 
 # Create pid file before starting main processing
 
-open LOCKFILE, '>', '/var/run/ipblacklist.pid' or die "Can't open PID file: $!";
+open LOCKFILE, '>', '/var/run/ipblacklist.pid' or abort "Can't open PID file: $!";
 print LOCKFILE "$$\n";
 close LOCKFILE;
 
@@ -205,16 +201,12 @@ if (-r $sources)
   eval qx|/bin/cat $sources|;
 }
 
-# Find out the red interface name
-
 if (-r $red_setting)
 {
-  open IN, '<', $red_setting or die "Can't open red interface name file: $!";
-
-  $red_iface = <IN>;
+  open REDIF, '<', $red_setting or (abort "Can't open red interface file", exit);
+  $red_iface = <REDIF>;
   chomp $red_iface;
-
-  close IN;
+  close REDIF;
 }
 
 if (@ARGV)
@@ -235,7 +227,7 @@ if (@ARGV)
     {
       # Called during system startup.
       # Restore saved blacklists.
-      # Don't do an update since that takes too long.
+      # Don't do an update since can take too long.
 
       do_start() if ($settings{'ENABLE'} eq 'on');
     }
@@ -300,25 +292,13 @@ if (@ARGV)
       disable_updates();
       do_delete();
     }
-    elsif ('autoblacklist-update' =~ m/^$cmd/i)
-    {
-      # Updates AUTOBLACKLIST options
-
-      autoblacklist_update();
-    }
-    elsif ('autoblacklist-clear' =~ m/^$cmd/i)
-    {
-      # Clears AUTOBLACKLIST contents
-
-      autoblacklist_clear();
-    }
     else
     {
-      print "Usage: $0 [update|start|stop|restart|log-on|log-off|enable|disable|autoblacklist-update|autoblacklist-clear]\n";
+      print "Usage: $0 [update|start|stop|restart|log-on|log-off|enable|disable]\n";
     }
   }
 }
-elsif ($settings{'ENABLE'} eq 'on')
+elsif ($settings{'ENABLE'} eq 'on') # Default action if none specified
 {
   do_update();
 }
@@ -353,12 +333,9 @@ sub do_stop()
 
   log_message LOG_NOTICE, "Stopping IP Blacklists";
 
-  foreach my $list ( $autoblacklist, sort keys %sources )
+  foreach my $list ( sort keys %sources )
   {
-    if (exists $chains{$list})
-    {
-      delete_list( $list );
-    }
+    delete_list( $list ) if (exists $chains{$list});
   }
 }
 
@@ -375,19 +352,17 @@ sub do_start()
 
   foreach my $list ( sort keys %sources )
   {
-    if (-e "$savedir/$list.conf")
+    delete_list( $list ) if (exists $chains{$list});     # Make sure OK to start
+
+    if ((-e "$savedir/$list.conf") and ($red_iface))
     {
       log_message LOG_INFO, "Restoring blacklist $list";
-      system( "$ipset restore -f $savedir/$list.conf" );
+      system( "$ipset restore -f $savedir/$list.conf" ); # Can't use the ipset
+                                                         # function to do this
 
       create_list( $list );
     }
   }
-
-  if ($settings{$autoblacklist} eq 'on')
-  {
-    create_autoblacklist();
-  }
 }
 
 
@@ -418,10 +393,8 @@ sub do_delete()
     }
   }
 
-  if ($settings{$autoblacklist} eq 'on')
-  {
-    delete_autoblacklist();
-  }
+  %modified      = ();
+  $update_status = 1;
 }
 
 
@@ -430,14 +403,14 @@ sub do_delete()
 #
 # Updates all the blacklists.
 # Creates or deletes the blacklist firewall rules as necessary and checks for
-# updates to the blacklists.
+# updates to the blacklists.  Each blacklist has its own minimum elapsed time
+# between updates, which is specified in the sources file, so the time of each
+# check is stored.
 #------------------------------------------------------------------------------
 
 sub do_update()
 {
-  return unless (is_connected());
-
-  my $type = 'hash:ip';
+  return unless ($red_iface);
 
   # Get the list of current ipsets
 
@@ -447,38 +420,39 @@ sub do_update()
 
   debug 1, "Checking blacklist sources";
 
+  LIST:
   foreach my $list ( sort keys %sources )
   {
     my @new_blacklist = ();
     my $name          = $sources{$list}{'name'};
-    my $rate          = $sources{$list}{'rate'};
     my $last_checked  = $checked{$list} || 0;
+    my $failures      = $checked{"${list}_failures"} || 0;
     my $enabled       = 0;
 
-    if (exists $modified{$list})
-    {
-      # Limit the check rate to the minimum defined in the WUI, unless we're
-      # creating the list
-
-      $rate = $settings{'RATE'} if ($settings{'RATE'} > $rate);
-    }
-
     if (exists $settings{$list})
     {
       $enabled = $settings{$list} eq 'on';
     }
 
-    debug 1, "Checking blacklist source: $name";
-
-    if ($enabled)
+    if ($enabled and is_connected())
     {
+      debug 1, "Checking blacklist source: $name";
+
+      # Calculate the per list rate
+
+      my $rate = get_rate_seconds( $sources{$list}{'rate'} );
+
       # Has enough time passed since the last time we checked the list?
+      # Ignore the limit if the last download failed
 
-      if (($last_checked + $rate * $hours) < (time() + $margin))
+      if (($last_checked + $rate) < (time() + $margin) or
+          ($failures > 0 and $failures < $max_dl_fails))
       {
+        my $type = 'hash:ip';
+
         download_list( $list, \@new_blacklist, \$type );
 
-        next unless (@new_blacklist);
+        next LIST unless (@new_blacklist);
 
         if (not exists $chains{$list})
         {
@@ -504,76 +478,81 @@ sub do_update()
 
       unlink "$savedir/$list.conf" if (-e "$savedir/$list.conf");
 
-      delete $modified{$list} if (exists $modified{$list});
+      delete $modified{$list}             if (exists $modified{$list});
+      delete $checked{"${list}_failures"} if (exists $checked{"${list}_failures"});
       $update_status  = 1;
     }
   }
 
-  # Check for any deleted lists
+  # Check for any lists that don't exist any more
 
-  foreach my $list (keys %sources)
+  foreach my $list (keys %modified)
   {
-    if (not exists $sources{$list})
-    {
-      delete_list( $list );
+    next if (exists $sources{$list});
 
-      # Delete the save file
+    delete_list( $list );
 
-      unlink "$savedir/$list.conf" if (-e "$savedir/$list.conf");
+    # Delete the save file
 
-      # Delete from the status
+    unlink "$savedir/$list.conf" if (-e "$savedir/$list.conf");
 
-      delete $modified{$list} if (exists $modified{$list});
-      delete $checked{$list}  if (exists $checked{$list});
-      $update_status  = 1;
-    }
-  }
+    # Delete from the status
 
-  if ($settings{$autoblacklist} eq 'on')
-  {
-    create_autoblacklist() if (not exists $chains{$autoblacklist});
+    delete $modified{$list} if (exists $modified{$list});
+    $update_status  = 1;
   }
-  else
+
+  foreach my $list (keys %checked)
   {
-    delete_autoblacklist() if (exists $chains{$autoblacklist});
-  }
+    next if ($list =~ m/_failures/);
+    next if (exists $sources{$list});
 
-  log_message LOG_INFO, "Completed IP Blacklist update";
+    delete $checked{$list};
+    delete $checked{"${list}_failures"};
+    delete $settings{$list} if (exists $settings{$list});
+    $update_status  = 1;
+  }
 }
 
 
 #------------------------------------------------------------------------------
-# sub autoblacklist_update()
+# sub get_rate_seconds( text )
+#
+# Converts a check rate into seconds.  A sanity check is made on the coverted
+# value.
 #
-# Updates the settings for the AUTOBLACKLIST
+# Parameters:
+#   text  The value to convert in the form nnnu, where nnn is a number and u
+#         is either m (minutes), h (hours) or d (days).  Hours is assumed if
+#         not specified and everything after the first letter is ignored.
 #------------------------------------------------------------------------------
 
-sub autoblacklist_update()
+sub get_rate_seconds( $ )
 {
-  # Get the list of current ipsets
-
-  get_ipsets();
+  my ($text) = @_;
 
-  # Delete the existing AUTOBLACKLIST, if it currently exists.
+  my ($value, $unit) = (uc $text) =~ m/(\d+)([DHM]?)/;
 
-  delete_autoblacklist() if (exists $chains{$autoblacklist});
-
-  # Re-create the AUTOBLACKLIST with the correct parameters.
-
-  create_autoblacklist() if ($settings{$autoblacklist} eq 'on');
-}
+  if ($unit eq 'D')    # Days
+  {
+    $value *= 60 * 60 * 24;
+  }
+  elsif ($unit eq 'M') # Minutes
+  {
+    $value *= 60;
+  }
+  else                 # Everything else - assume hours
+  {
+    $value *= 60 * 60;
+  }
 
+  # Sanity check - limit to range 5 min .. 1 week
 
-#------------------------------------------------------------------------------
-# sub autoblacklist_clear()
-#
-# Clears the contents of the AUTOBLACKLIST
-#------------------------------------------------------------------------------
+  #        d    h    m    s
+  $value =           5 * 60 if ($value < 5 * 60);
+  $value = 7 * 24 * 60 * 60 if ($value > 7 * 24 * 60 * 60);
 
-sub autoblacklist_clear()
-{
-  log_message LOG_INFO, "Flush Automatic blacklist";
-  ipset( "flush $autoblacklist" );
+  return $value;
 }
 
 
@@ -587,7 +566,7 @@ sub autoblacklist_clear()
 
 sub is_connected()
 {
-  return (-e "${General::swroot}/red/active");
+  return (-e $active);
 }
 
 
@@ -595,7 +574,16 @@ sub is_connected()
 # sub create_list( list )
 #
 # Creates a new IPTables chain for a blacklist source.
-# The set must be created before calling this function.
+# The set must be created before calling this function.  Two rules are added to
+# the chain:
+# (optional) 1 Log the packet
+#            2 Drop the packet
+#
+# The log rule is only added when logging is enabled by the WUI.
+#
+# Rules are then added to the BLACKLISTIN and BLACKLISTOUT chains that check
+# the packet's IP address against the IPSet and then jump to the newly created
+# chain.
 #
 # Parameters:
 #   list  The name of the blacklist
@@ -609,56 +597,24 @@ sub create_list( $ )
 
   # Create new chain in filter table
 
-  iptables( " -N ${list}_BLOCK" ) == 0 or
-    ( abort "Could not create IPTables chain ${list}_BLOCK", return );
+  iptables( "-N ${list}_DROP" ) == 0 or
+    ( abort "Could not create IPTables chain ${list}_DROP", return );
 
   # Add the logging and drop rules
 
   if ($settings{'LOGGING'} eq 'on')
   {
-    iptables( "-A ${list}_BLOCK -j LOG -m limit --limit 10/second --log-prefix 'DROP_$list'" ) == 0 or
+    iptables( "-A ${list}_DROP -j LOG -m limit --limit 10/second --log-prefix 'BLKLST_$list'" ) == 0 or
       ( abort "Could not create IPTables chain $list LOG rule", return );
   }
 
-  iptables( "-A ${list}_BLOCK -j DROP" ) == 0 or
+  iptables( "-A ${list}_DROP -j DROP" ) == 0 or
     ( abort "Could not create IPTables chain $list drop rule", return );
 
   # Add the rules to check against the set
 
-  iptables( "-A IPBLACKLISTREDIN -p ALL -m set --match-set $list src -j ${list}_BLOCK" );
-  iptables( "-A IPBLACKLISTREDOUT -p ALL -m set --match-set $list dst -j ${list}_BLOCK" );
-}
-
-
-#------------------------------------------------------------------------------
-# sub create_autoblacklist()
-#
-# Creates a new IPTables chain for the AUTOBLACKLIST.  This also creates the
-# IPSet with the correct timeout.
-#------------------------------------------------------------------------------
-
-sub create_autoblacklist()
-{
-  return unless ($red_iface);  # Can't add rule to policy unless this is set
-
-  # Create the set for the AUTOBLACKLIST
-
-  ipset( "create $autoblacklist hash:ip timeout $settings{BLOCK_PERIOD}" );
-
-  # Create new chain in filter table
-
-  create_list( $autoblacklist );
-
-  # For the AUTOBLACKLIST there are extra rules to reset the timeout on the
-  # blockled addresses
-
-  iptables( "-I ${autoblacklist}_BLOCK -m set --match-set $autoblacklist src -j SET --add-set $autoblacklist src --exist" );
-  iptables( "-I ${autoblacklist}_BLOCK -m set --match-set $autoblacklist dst -j SET --add-set $autoblacklist dst --exist" );
-
-  # For the AUTOBLACKLIST there is an extra rule to add an entry to the list
-  # of blocked addresses.  This is added to the input policy chain.
-
-  iptables( "-I POLICYIN 1 -i $red_iface -m hashlimit --hashlimit-mode srcip --hashlimit-above $settings{BLOCK_THRESHOLD}/hour --hashlimit-name $autoblacklist -j SET --add-set $autoblacklist src" );
+  iptables( "-A BLACKLISTIN -p ALL -i $red_iface -m set --match-set $list src -j ${list}_DROP" );
+  iptables( "-A BLACKLISTOUT -p ALL -o $red_iface -m set --match-set $list dst -j ${list}_DROP" );
 }
 
 
@@ -680,60 +636,25 @@ sub delete_list( $ )
 
   # Remove the blacklist chains from the main INPUT and OUTPUT chains
 
-  iptables( "-D IPBLACKLISTREDIN -p ALL -m set --match-set $list src -j ${list}_BLOCK" ) == 0 or
-    log_message LOG_ERR, "Could not remove IPSet $list from IPBLACKLISTREDIN chain";
-
-  iptables( "-D IPBLACKLISTREDOUT -p ALL -m set --match-set $list dst -j ${list}_BLOCK" ) == 0 or
-    log_message LOG_ERR, "Could not remove IPSet $list from IPBLACKLISTREDOUT chain";
+  iptables( "-D BLACKLISTIN -p ALL -i $red_iface -m set --match-set $list src -j ${list}_DROP" );
+  iptables( "-D BLACKLISTOUT -p ALL -o $red_iface -m set --match-set $list dst -j ${list}_DROP" );
 
   # Flush and delete the chain
 
-  iptables( "-F ${list}_BLOCK" ) == 0 or
-    log_message LOG_ERR, "Could not flush IPTables chain ${list}_BLOCK";
-
-  iptables( "-X ${list}_BLOCK" ) == 0 or
-    log_message LOG_ERR, "Could not delete IPTables chain ${list}_BLOCK";
+  iptables( "-F ${list}_DROP" );
+  iptables( "-X ${list}_DROP" );
 
   # Flush and delete the set
 
   ipset( "flush $list" );
-
   ipset( "destroy $list" );
 }
 
 
 #------------------------------------------------------------------------------
-# sub delete_autoblacklist()
+# sub download_list( list, ref_list, ref_type )
 #
-# Deletes the autoblacklist IPTables chain when it is disabled. Also flushes
-# and destroys the IPSet.
-#------------------------------------------------------------------------------
-
-sub delete_autoblacklist()
-{
-  # For the AUTOBLACKLIST there is an extra rule to remove
-
-  unless ($red_iface)
-  {
-    iptables( "-D POLICYIN -i $red_iface -m hashlimit --hashlimit-mode srcip --hashlimit-above $settings{BLOCK_THRESHOLD}/hour --hashlimit-name $autoblacklist -j SET --add-set $autoblacklist src" );
-  }
-
-  # Now do a normal delete
-
-  delete_list( $autoblacklist );
-}
-
-
-#------------------------------------------------------------------------------
-# sub download_list( chain, ref_list, ref_type )
-#
-# Updates the IP Addresses for a blacklist.  Depending on the blacklist one of
-# two methods are used:
-#
-# - For some lists the header is downloaded and the modification date checked.
-#   If newer than the existing list, the update is downloaded.
-# - For other lists this is not supported,so the whole file has to be
-#   downloaded regardless.
+# Downloads the IP Addresses for a blacklist.
 #
 # Once downloaded the list is parsed to get the IP addresses and/or networks.
 #
@@ -747,8 +668,8 @@ sub download_list( $$$ )
 {
   my ($list, $new_blacklist, $type) = @_;
 
-  $checked{$list}            = time();
-  $update_status             = 1;
+  $checked{$list} = time();  # Record that the list has been checked
+  $update_status  = 1;
 
   # Check the parser for the blacklist
 
@@ -758,23 +679,18 @@ sub download_list( $$$ )
     return;
   }
 
-  if ($sources{$list}{'method'} eq 'check-header-time')
-  {
-    download_check_header_time( $list, $new_blacklist, $type );
-  }
-  else
-  {
-    download_wget( $list, $new_blacklist, $type );
-  }
+  # Add alternative download mechanisms here
+
+  download_check_header_time( $list, $new_blacklist, $type );
 }
 
 
 #------------------------------------------------------------------------------
-# sub download_check_header_time( chain, ref_list, ref_type )
+# sub download_check_header_time( list, ref_list, ref_type )
 #
-# Updates the IP Addresses for a blacklist.  The header is downloaded and the
-# modification date checked. If newer than the existing list, the update is
-# downloaded.
+# Updates the IP Addresses for a blacklist.  The If-Modified-Since header is
+# specified in the request so that only updated lists are downloaded (providing
+# that the server supports this functionality).
 #
 # Once downloaded the list is parsed to get the IP addresses and/or networks.
 #
@@ -782,6 +698,9 @@ sub download_list( $$$ )
 #   list      The name of the blacklist
 #   ref_list  A reference to an array to store the downloaded blacklist
 #   ref_type  A reference to store the type of the blacklist
+#
+# Returns:
+#   The list type: 'hash:ip' or 'hash:net'
 #------------------------------------------------------------------------------
 
 sub download_check_header_time( $$$ )
@@ -794,12 +713,12 @@ sub download_check_header_time( $$$ )
 
   my $parser = $parsers{ $sources{$list}{'parser'} };
 
-  log_message LOG_INFO, "Checking modification time for blacklist $list update with LWP";
+  debug 1, "Checking for blacklist $list updates with LWP";
 
   # Create a user agent for downloading the blacklist
-  # Limit the download size for safety (10 MiB)
+  # Limit the download size for safety
 
-  my $ua = LWP::UserAgent->new( max_size => 10485760 );
+  my $ua = LWP::UserAgent->new( max_size => $max_dl_bytes );
 
   # Get the Proxy settings
 
@@ -807,55 +726,42 @@ sub download_check_header_time( $$$ )
   {
     if ($proxy_settings{'UPSTREAM_USER'})
     {
-      $ua->proxy("http"  => "http://$proxy_settings{'UPSTREAM_USER'}:$proxy_settings{'UPSTREAM_PASSWORD'}\@$proxy_settings{'UPSTREAM_PROXY'}/");
-      $ua->proxy("https" => "http://$proxy_settings{'UPSTREAM_USER'}:$proxy_settings{'UPSTREAM_PASSWORD'}\@$proxy_settings{'UPSTREAM_PROXY'}/");
+      $ua->proxy( [["http", "https"] => "http://$proxy_settings{'UPSTREAM_USER'}:$proxy_settings{'UPSTREAM_PASSWORD'}\@$proxy_settings{'UPSTREAM_PROXY'}/"] );
     }
     else
     {
-      $ua->proxy("http"  => "http://$proxy_settings{'UPSTREAM_PROXY'}/");
-      $ua->proxy("https" => "http://$proxy_settings{'UPSTREAM_PROXY'}/");
+      $ua->proxy( [["http", "https"]  => "http://$proxy_settings{'UPSTREAM_PROXY'}/"] );
     }
   }
 
-  # Get the blacklist modification time from the internet
-
-  my $request  = HTTP::Request->new( HEAD => $sources{$list}{'url'} );
-
-  my $response = $ua->request( $request );
+  # Get the last modified time
 
-  if (not $response->is_success)
-  {
-    log_message LOG_WARNING, "Failed to download $list header $sources{$list}{'url'}: ". $response->status_line;
-
-    return;
-  }
-
-  # Has the blacklist been modified since we last read it?
-
-  if (exists $modified{$list} and $modified{$list} >= $response->last_modified)
-  {
-    # We've already got this version of the blacklist
-
-    debug 1, "Blacklist $list not modified";
-    return;
-  }
-
-  debug 1, "Blacklist $list Modification times: old " . $modified{$list} . ", new " . $response->last_modified if (exists $modified{$list});
-  log_message LOG_INFO, "Downloading blacklist $list with LWP";
+  my $modified = gmtime( $modified{$list} || 0 );
 
   # Download the blacklist
 
-  $request = HTTP::Request->new( GET => $sources{$list}{'url'} );
-  $response = $ua->request($request);
+  my $response = $ua->get( $sources{$list}{'url'}, 'If-Modified-Since' => $modified );
 
   if (not $response->is_success)
   {
+    if ($response->code == 304)
+    {
+      # Not an error - list has not been modified
+      debug 1, "Blacklist $list not modified";
+
+      return;
+    }
+
     log_message LOG_WARNING, "Failed to download $list blacklist $sources{$list}{'url'}: ". $response->status_line;
+    $checked{"${list}_failures"}++;
 
     return;
   }
 
-  $modified{$list} = $response->last_modified;
+  $modified{$list}             = $response->last_modified;
+  $checked{"${list}_failures"} = 0;
+
+  # Parse the downloaded list, checking if it's a list of addresses or nets
 
   foreach my $line (split /[\r\n]+/, $response->content)
   {
@@ -865,108 +771,12 @@ sub download_check_header_time( $$$ )
 
     next unless ($address and $address =~ m/\d+\.\d+\.\d+\.\d+/);
 
-    if ($address =~ m|/\d+|)
-    {
-      $found_net = 1;
-    }
-    else
+    if ($address =~ m|/32|)
     {
+      $address =~ s|/32||;
       $found_ip  = 1;
     }
-
-    push @{ $new_blacklist }, $address;
-  }
-
-  if ($found_net and $found_ip)
-  {
-    # Convert mixed address and network set to all network
-
-    foreach my $address (@{ $new_blacklist })
-    {
-      $address .= '/32' unless ($address =~ m|/\d+|);
-    }
-
-    $found_ip = 0;
-  }
-
-  $$type = $found_net ? 'hash:net' : 'hash:ip';
-}
-
-
-#------------------------------------------------------------------------------
-# sub download_wget( chain, ref_list, ref_type )
-#
-# Updates the IP Addresses for a blacklist.  The whole file is download with
-# wget and then the modification time compared with the stored modification
-# time.  If the update is newer then the downloaded list is parsed.
-#
-# Once downloaded the list is parsed to get the IP addresses and/or networks.
-#
-# Parameters:
-#   list      The name of the blacklist
-#   ref_list  A reference to an array to store the downloaded blacklist
-#   ref_type  A reference to store the type of the blacklist
-#------------------------------------------------------------------------------
-
-sub download_wget( $$$ )
-{
-  my ($list, $new_blacklist, $type) = @_;
-  my $wget_proxy                    = '';
-  my $found_ip                      = 0;
-  my $found_net                     = 0;
-
-  my $parser = $parsers{ $sources{$list}{'parser'} };
-
-  log_message LOG_INFO, "Downloading blacklist $list update with wget";
-
-  # Get the Proxy settings
-
-  if ($proxy_settings{'UPSTREAM_PROXY'})
-  {
-    if ($proxy_settings{'UPSTREAM_USER'})
-    {
-      $wget_proxy = "--proxy=on --proxy-user=$proxy_settings{'UPSTREAM_USER'} --proxy-passwd=$proxy_settings{'UPSTREAM_PASSWORD'} -e http_proxy=http://$proxy_settings{'UPSTREAM_PROXY'}/";
-    }
-    else
-    {
-      $wget_proxy = "--proxy=on -e http_proxy=http://$proxy_settings{'UPSTREAM_PROXY'}/";
-    }
-  }
-
-  my $retv = system( "wget $wget_proxy --no-show-progress -o $detailed_log -O $tmpdir/ipblacklist_$list $sources{$list}{'url'}" );
-
-  if ($retv != 0)
-  {
-    my $error = $wget_status[ $retv/256 ];
-    log_message LOG_WARNING, "Failed to download $list blacklist $sources{$list}{'url'}: $error";
-    return;
-  }
-
-  my @file_info = stat( "$tmpdir/ipblacklist_$list" );
-
-  if (exists $modified{$list} and $modified{$list} >= $file_info[9])
-  {
-    # We've already got this version of the blocklist
-
-    debug 1, "Blacklist $list not modified";
-    unlink "$tmpdir/ipblacklist_$list";
-    return;
-  }
-
-  open LIST, '<', "$tmpdir/ipblacklist_$list" or (abort "Can't open downloaded blacklist for $list: $!", return);
-
-  $modified{$list} = $file_info[9];
-
-  foreach my $line (<LIST>)
-  {
-    chomp $line;
-
-    my $address = &$parser( $line );
-
-    next unless ($address);
-    next unless ($address =~ m|\d+\.\d+\.\d+\.\d+|);
-
-    if ($address =~ m|/\d+|)
+    elsif ($address =~ m|/\d+|)
     {
       $found_net = 1;
     }
@@ -978,13 +788,9 @@ sub download_wget( $$$ )
     push @{ $new_blacklist }, $address;
   }
 
-  close LIST;
-
-  unlink "$tmpdir/ipblacklist_$list";
-
   if ($found_net and $found_ip)
   {
-    # Convert mixed address and network set to all network
+    # Convert mixed addresses and networks to all networks
 
     foreach my $address (@{ $new_blacklist })
     {
@@ -999,38 +805,49 @@ sub download_wget( $$$ )
 
 
 #------------------------------------------------------------------------------
-# sub read_ipset( list, old, type )
+# sub read_ipset( list, old, type, maxelem )
 #
-# Reads the existing contents of the set
+# Reads the existing contents and type of the set.
 #
 # Parameters:
-#   chain  The name of the blacklist
-#   old    Reference to array to contain blacklist
-#   type   Reference to type
+#   list    The name of the blacklist
+#   old     Reference to array to contain blacklist
+#   type    Reference to type
+#   maxelem Reference to maximum number of elements
 #------------------------------------------------------------------------------
 
-sub read_ipset( $$$ )
+sub read_ipset( $$$$ )
 {
-  my ($list, $old, $type) = @_;
-  my $found_net            = 0;
-  my $found_ip             = 0;
+  my ($list, $old, $type, $maxelem) = @_;
+  my $found_net                     = 0;
+  my $found_ip                      = 0;
 
   debug 2, "Reading existing ipset for blacklist $list";
 
   foreach my $line (qx/$ipset list $list/)
   {
+    if ($line =~ m|Header:.*maxelem (\d+)|)
+    {
+      $$maxelem = $1;
+      next;
+    }
+
     next unless ($line =~ m|(\d+\.\d+\.\d+\.\d+(?:/\d+)?)|);
 
     my $address = $1;
 
-    if (($address =~ m|/\d+$|) and ($address !~ m|/32$|))
+    if ($address =~ m|/32|)
+    {
+      $found_ip  = 1;
+      $address   =~ s|/32$||;
+    }
+    elsif ($address =~ m|/\d+$|)
     {
       $found_net = 1;
     }
     else
     {
       $found_ip  = 1;
-      $address   =~ s|/32$||;
     }
 
     $$old{$address} = 1;
@@ -1038,7 +855,7 @@ sub read_ipset( $$$ )
 
   if ($found_ip and $found_net)
   {
-    # Convert mixed address and network set to all network
+    # Convert mixed addresses and networks to all networks
 
     my @ads_list = keys %{ $old };
 
@@ -1059,12 +876,13 @@ sub read_ipset( $$$ )
 
 
 #------------------------------------------------------------------------------
-# sub update_list( chain, new, new_type )
+# sub update_list( list, new, new_type )
 #
 # Updates the IP Addresses for a blacklist
 #
 # The new list is compared to the existing list and new entries added or old
-# entries deleted as necessary.
+# entries deleted as necessary.  If the list type ('hash:ip' or 'hash:net') has
+# changed then the IPSet is deleted and re-created with the new type.
 #
 # Parameters:
 #   list      The name of the blacklist
@@ -1078,49 +896,63 @@ sub update_list( $$$ )
   my %old;
   my $old_type;
   my $changes = 0;
+  my $maxelem = 0;
 
-  debug 2, "Checking for $list blacklist update from $sources{$list}{'url'}";
+  debug 1, "Checking for $list blacklist update from $sources{$list}{'url'}";
 
-  log_message LOG_INFO, "Updating $list blacklist";
+  if (exists $chains{$list} )
+  {
+    my $recreate_ipset = 0;
 
-  read_ipset( $list, \%old, \$old_type );
+    read_ipset( $list, \%old, \$old_type, \$maxelem );
 
-  # Check the IPSet type hasn't changed
+    # Check the IPSet type hasn't changed
 
-  if ($new_type ne $old_type)
-  {
-    # Change the IPSet type.  This requires removing references to it first.
-    # We could delete and then create the chain, but doing it like this keeps
-    # the statistics.
+    if ($new_type ne $old_type)
+    {
+      log_message LOG_NOTICE, "Blacklist $list changed type from $old_type to $new_type";
+      $recreate_ipset = 1;
+    }
 
-    log_message LOG_NOTICE, "Blacklist $list changed type from $old_type to $new_type";
+    if ($max_size_fraction * $maxelem < scalar @{ $new } )
+    {
+      log_message LOG_NOTICE, "Blacklist $list changed size from $maxelem";
+      $recreate_ipset = 1;
+    }
+
+    if ($recreate_ipset)
+    {
+      # Change the IPSet type and/or size.  This requires removing references
+      # to it first. We could delete and then create the chain, but doing it
+      # like this keeps the statistics.
 
-    # Remove the IPSet from the IPTables chains
+      # Remove the IPSet from the IPTables chains
 
-    iptables( "-D 'IPBLACKLISTREDIN' -p ALL -m set --match-set $list src -j ${list}_BLOCK" ) == 0 or
-      log_message LOG_ERR, "Could not remove ${list} from IPBLACKLISTREDIN chain";
+      iptables( "-D 'BLACKLISTIN' -p ALL -i $red_iface -m set --match-set $list src -j ${list}_DROP" ) == 0 or
+        log_message LOG_ERR, "Could not remove ${list} from BLACKLISTIN chain";
 
-    iptables( "-D 'IPBLACKLISTREDOUT' -p ALL -m set --match-set $list dst -j ${list}_BLOCK" ) == 0 or
-      log_message LOG_ERR, "Could not remove ${list} from IPBLACKLISTREDOUT chain";
+      iptables( "-D 'BLACKLISTOUT' -p ALL -o $red_iface -m set --match-set $list dst -j ${list}_DROP" ) == 0 or
+        log_message LOG_ERR, "Could not remove ${list} from BLACKLISTOUT chain";
 
-    # Flush and delete the old set
+      # Flush and delete the old set
 
-    ipset( "flush $list" );
-    ipset( "destroy $list" );
+      ipset( "flush $list" );
+      ipset( "destroy $list" );
 
-    %old = ();
+      %old = ();  # Since we've deleted the old set it can't have any entries.
 
-    # Create the new ipset
+      # Create the new ipset
 
-    create_ipset( $list, $new_type, scalar @{ $new } );
+      create_ipset( $list, $new_type, scalar @{ $new } );
 
-    # Add the rules to check against the set
+      # Add the rules to check against the set
 
-    iptables( "-A 'IPBLACKLISTREDIN' -p ALL -m set --match-set $list src -j ${list}_BLOCK" ) == 0 or
-      log_message LOG_ERR, "Could not add IPSet $list to IPBLACKLISTREDIN chain";
+      iptables( "-A 'BLACKLISTIN' -p ALL -i $red_iface -m set --match-set $list src -j ${list}_DROP" ) == 0 or
+        log_message LOG_ERR, "Could not add IPSet $list to BLACKLISTIN chain";
 
-    iptables( "-A 'IPBLACKLISTREDOUT' -p ALL -m set --match-set $list dst -j ${list}_BLOCK" ) == 0 or
-      log_message LOG_ERR, "Could not add IPSet $list to IPBLACKLISTREDOUT chain";
+      iptables( "-A 'BLACKLISTOUT' -p ALL -o $red_iface -m set --match-set $list dst -j ${list}_DROP" ) == 0 or
+        log_message LOG_ERR, "Could not add IPSet $list to BLACKLISTOUT chain";
+    }
   }
 
   # Process the blacklist
@@ -1131,11 +963,11 @@ sub update_list( $$$ )
 
     if (exists $old{$address})
     {
-      delete $old{$address};  # Not new
+      delete $old{$address};  # Not new - don't delete from chain later
     }
     else
     {
-      ipset( "add $list $address -exist" );
+      ipset( "add $list $address -exist" ); # New - add it
 
       $changes++;
     }
@@ -1156,14 +988,13 @@ sub update_list( $$$ )
     debug 3, "Delete old net $address from blacklist $list";
   }
 
-
-  log_message LOG_INFO, "Finished updating $list blacklist with $changes changes";
+  log_message LOG_INFO, "Updated $list blacklist with $changes changes";
 
   # Save the blacklist for the next reboot
 
   mkdir "$savedir" unless (-d "$savedir" );
 
-  ipset( "save $list -file $savedir/$list.conf" );
+  ipset( "save $list -file $savedir/$list.conf" ) if ($changes > 0);
 
   stop_ipset();
 }
@@ -1195,28 +1026,29 @@ sub get_ipsets( )
 
 
 #------------------------------------------------------------------------------
-# sub create_ipset( name, type, size )
+# sub create_ipset( list, type, size )
 #
-# Creates a new IPSet.  The current and maximum size of the set are determined
-# by taking the next power of two greater than the numer of entries, subject to
-# a minimum size.  This allows for future expansion.
+# Creates a new IPSet.  The current size of the set is determined by taking the
+# next power of two greater than the number of entries; the maximum size is set
+# to double this, subject to a minimum size.  This allows for future expansion.
 #
 # Parameters:
-#   name  The name of the blacklist
+#   list  The name of the blacklist
 #   type  The type of the blacklist (hash:ip or hash:net)
 #   size  The number of entries in the lsit
 #------------------------------------------------------------------------------
 
 sub create_ipset( $$$ )
 {
-  my ($name, $type, $size) = @_;
+  my ($list, $type, $size) = @_;
 
   my $hashsize = 1;
   $hashsize  <<= 1 while ($hashsize < $size);
-  my $maxsize  = ($hashsize < 16384) ? 32768 : $hashsize * 2;
+  my $maxsize  = $hashsize * 2;
+  $maxsize     = $min_ipset_entries if ($maxsize < $min_ipset_entries);
 
   # Create the new ipset
-  ipset( "create $name $type hashsize $hashsize maxelem $maxsize" );
+  ipset( "create $list $type hashsize $hashsize maxelem $maxsize" );
   stop_ipset(); # Need to do this to action the IPSet commands
 }
 
@@ -1225,6 +1057,7 @@ sub create_ipset( $$$ )
 # sub enable_logging()
 #
 # Enable logging of packets dropped by IP Blacklist rules.
+# This adds a rule to log the packet to each lists' IPTables chain.
 #------------------------------------------------------------------------------
 
 sub enable_logging()
@@ -1237,7 +1070,7 @@ sub enable_logging()
   {
     if (exists $chains{$list})
     {
-      iptables( "-I ${list}_BLOCK 1 -j LOG -m limit --limit 10/second --log-prefix 'DROP_$list'" );
+      iptables( "-I ${list}_DROP 1 -j LOG -m limit --limit 10/second --log-prefix 'BLKLST_$list'" );
     }
   }
 }
@@ -1247,6 +1080,7 @@ sub enable_logging()
 # sub disable_logging()
 #
 # Disable logging of packets dropped by IP Blacklist rules.
+# This deletes a rule to log the packet from each lists' IPTables chain.
 #------------------------------------------------------------------------------
 
 sub disable_logging()
@@ -1259,7 +1093,7 @@ sub disable_logging()
   {
     if (exists $chains{$list})
     {
-      iptables( "-D ${list}_BLOCK -j LOG -m limit --limit 10/second --log-prefix 'DROP_$list'" );
+      iptables( "-D ${list}_DROP -j LOG -m limit --limit 10/second --log-prefix 'BLKLST_$list'" );
     }
   }
 }
@@ -1269,9 +1103,14 @@ sub disable_logging()
 # sub enable_updates()
 #
 # Adds a command to the fcrontab to run the update hourly.
+# If there is a command already in the fcrontab to do this it will be
+# uncommented, otherwise a new line is added.
+#
 # The update is executed at an offset from the hour so that all the users don't
 # try to download the updates at exactly the same time - the blacklists are
-# provided free, so it's good manners to spread the load on the servers.
+# provided free, so it's good manners to spread the load on the servers.  The
+# offset is initialised to a random number that avoids running on the hour
+# (when a lot of other things happen), and every fifteen minutes thereafter.
 #------------------------------------------------------------------------------
 
 sub enable_updates()
@@ -1285,9 +1124,9 @@ sub enable_updates()
   {
     if ($line =~ m|/usr/local/bin/ipblacklist|)
     {
-      return if ($line !~ m/^#/); # Already enabled
+      return if ($line !~ m/^#/); # Already enabled - do nothing
 
-      # Found - uncomment the line
+      # Already in fcrontab - uncomment the line
 
       $line  =~ s/^#+//;
       $found = 1;
@@ -1298,13 +1137,20 @@ sub enable_updates()
 
   if (not $found)
   {
-    # Add a new entry
+    # Add a new entry to fcrontab
 
-    my $start = int( rand(50) ) + 5;
+    my $start = int( rand(13) ) + 1;
+
+    my $times = $start;
+
+    for (my $offset = $times+15 ; $offset < 60 ; $offset += 15)
+    {
+      $times .= ",$offset";
+    }
 
     push @lines, "\n";
     push @lines, "# IP Blacklist update\n";
-    push @lines, "\%hourly,nice(1),random,serial $start /usr/local/bin/ipblacklist\n";
+    push @lines, "\%nice(1) $times * * * * /usr/local/bin/ipblacklist\n";
     log_message LOG_INFO, "Add IP Address Blacklist update to crontab";
   }
 
@@ -1329,9 +1175,9 @@ sub disable_updates()
   {
     if ($line =~ m|/usr/local/bin/ipblacklist|)
     {
-      return if ($line =~ m/^#/); # Already disabled
+      return if ($line =~ m/^#/); # Already disabled - do nothing
 
-      # Found - comment the line
+      # In fcrontab - comment the line
 
       $line  =~ s/^#*/#/;
       $found = 1;
@@ -1349,9 +1195,10 @@ sub disable_updates()
 
 
 #------------------------------------------------------------------------------
-# sub parse_text_with_hash_comments( line )
+# sub parse_ip_or_net_list( line )
 #
-# Parses an input line removing comments.
+# Parses an input line, looking for lines starting with an IP Address or
+# Network specification.
 #
 # Parameters:
 #   line  The line to parse
@@ -1360,41 +1207,11 @@ sub disable_updates()
 #   Either an IP Address or a null string
 #------------------------------------------------------------------------------
 
-sub parse_text_with_hash_comments( $ )
+sub parse_ip_or_net_list( $ )
 {
   my ($line) = @_;
 
-  return "" if ($line =~ m/^\s*#/);
-
-  $line =~ s/#.*$//;
-
-  $line =~ m|(\d+\.\d+\.\d+\.\d+(?:/\d+)?)|;
-
-  return $1;
-}
-
-
-#------------------------------------------------------------------------------
-# sub parse_text_with_semicolon_comments( line )
-#
-# Parses an input line removing comments.
-#
-# Parameters:
-#   line  The line to parse
-#
-# Returns:
-#   Either and IP Address or a null string
-#------------------------------------------------------------------------------
-
-sub parse_text_with_semicolon_comments( $ )
-{
-  my ($line) = @_;
-
-  return "" if ($line =~ m/^\s*;/);
-
-  $line =~ s/;.*$//;
-
-  $line =~ m|(\d+\.\d+\.\d+\.\d+(?:/\d+)?)|;
+  $line =~ m|^(\d+\.\d+\.\d+\.\d+(?:/\d+)?)|;
 
   return $1;
 }
@@ -1424,6 +1241,7 @@ sub parse_dshield( $ )
 
   $line =~ s/#.*$//;
 
+  #          |Start addrs                |   |End Addrs                |   |Mask
   $line =~ m|(\d+\.\d+\.\d+\.\d+(?:/\d+)?)\s+\d+\.\d+\.\d+\.\d+(?:/\d+)?\s+(\d+)|;
 
   return unless ($1);
@@ -1436,8 +1254,8 @@ sub parse_dshield( $ )
 #------------------------------------------------------------------------------
 # sub iptables( cmd )
 #
-# Executes an IPTables command, waiting for the lock to ensure only one change
-# is made at a time.
+# Executes an IPTables command, waiting for the internal lock to ensure only
+# one change is made at a time.
 #
 # Parameters:
 #   cmd  The command to execute
@@ -1462,6 +1280,11 @@ sub iptables( $ )
 # new process for each command.  The sub-process is started if it's not already
 # running.
 #
+# Note that the pipe is buffered so commands are not necessarily executed
+# immediately.  Use ipset_stop() to force commands to be executed.  This should
+# be done before relying on anything that the ipset commands do, for example
+# before referencing the IPSet in an IPTables command.
+#
 # Parameters:
 #   cmd  The command to execute
 #------------------------------------------------------------------------------
@@ -1485,6 +1308,7 @@ sub ipset( $ )
 # sub stop_ipset( )
 #
 # Stops the ipset sub-process.
+# This causes any pending ipset commands to be executed.
 #------------------------------------------------------------------------------
 
 sub stop_ipset( )
@@ -1500,7 +1324,7 @@ sub stop_ipset( )
 #------------------------------------------------------------------------------
 # sub abort( message, parameters... )
 #
-# Aborts the current activity, printing out an error message.
+# Used when aborting the current activity, printing out an error message.
 #
 # Parameters:
 #   message     Message to be printed


      reply	other threads:[~2020-05-26 17:44 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-04-27 14:31 Tim FitzGeorge
2020-04-27 14:31 ` [PATCH v2 1/8] ipblacklist: Main script Tim FitzGeorge
2020-04-27 14:31 ` [PATCH v2 2/8] ipblacklist: WUI Settings page Tim FitzGeorge
2020-04-27 14:31 ` [PATCH v2 3/8] ipblacklist: WUI Log page Tim FitzGeorge
2020-04-27 14:31 ` [PATCH v2 4/8] ipblacklist: WUI Log details page Tim FitzGeorge
2020-04-27 14:31 ` [PATCH v2 5/8] ipblacklist: WUI menus, language file etc Tim FitzGeorge
2020-04-27 14:31 ` [PATCH v2 6/8] ipblacklist: Ancillary files Tim FitzGeorge
2020-04-27 14:31 ` [PATCH v2 7/8] ipblacklist: Modifications to system Tim FitzGeorge
2020-04-27 14:31 ` [PATCH v2 8/8] ipblacklist: Build infrastructure Tim FitzGeorge
2020-05-16  9:40 ` [PATCH v2 0/8] ipblacklist: IP Address Blacklists Michael Tremer
2020-05-26 17:44   ` Tim FitzGeorge [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=60d44f5b-36ec-c3b0-4dc1-e82e81663774@tfitzgeorge.me.uk \
    --to=ipfr@tfitzgeorge.me.uk \
    --cc=development@lists.ipfire.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox