[PATCH 0/3] Add ASN-based anomaly detections to IPFire's web proxy: Proactive Fast Flux detection and detection for selectively announced networks

[PATCH 0/3] Add ASN-based anomaly detections to IPFire's web proxy: Proactive Fast Flux detection and detection for selectively announced networks

[Peter Müller]

[-- Attachment #1: Type: text/plain, Size: 1425 bytes --]

This patchset adds two new features to IPFire's web proxy, taking advantage
of the Autonomous System information we have at hand by using libloc.

The proactive Fast Flux detection is especially worth noticing, as even most
expensive (= advanced?) security suites do not provide similar protection,
especially not in a proactive manner.

By simply enumerating the distinct amount of Autonomous System Numbers a FQDN
ultimately resolves to, we are able to deny access to malware distribution
sites, phishing sites, C&C servers, and other cybercrime stuff hosted on Fast
Flux setups abusing cracked machines around the world - even before the FQDN
or any IP address involved is flagged as malicious by any security vendor.

Peter Müller (3):
  squid-asnbl: New package
  proxy.cgi: Implement proactive Fast Flux detection and detection for
    selectively announced destinations
  langs: Add English and German translations for newly added web proxy
    features

 config/rootfiles/common/squid-asnbl |  1 +
 html/cgi-bin/proxy.cgi              | 89 +++++++++++++++++++++++++++++
 langs/de/cgi-bin/de.pl              |  7 +++
 langs/en/cgi-bin/en.pl              |  7 +++
 lfs/squid-asnbl                     | 83 +++++++++++++++++++++++++++
 make.sh                             |  1 +
 6 files changed, 188 insertions(+)
 create mode 100644 config/rootfiles/common/squid-asnbl
 create mode 100644 lfs/squid-asnbl

-- 
2.26.2

[PATCH 1/3] squid-asnbl: New package

[Peter Müller]

[-- Attachment #1: Type: text/plain, Size: 4645 bytes --]

This package adds an ASNBL helper for detecting Fast Flux setups and
selectively announced networks (i. e. FQDNs resolving to IP addresses
not being announced by an Autonomous System) to the distribution.
Afterwards, the helper script is located at /usr/bin/asnbl-helper.py .

Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org>
---
 config/rootfiles/common/squid-asnbl |  1 +
 lfs/squid-asnbl                     | 83 +++++++++++++++++++++++++++++
 make.sh                             |  1 +
 3 files changed, 85 insertions(+)
 create mode 100644 config/rootfiles/common/squid-asnbl
 create mode 100644 lfs/squid-asnbl

diff --git a/config/rootfiles/common/squid-asnbl b/config/rootfiles/common/squid-asnbl
new file mode 100644
index 000000000..f129f441e
--- /dev/null
+++ b/config/rootfiles/common/squid-asnbl
@@ -0,0 +1 @@
+usr/bin/asnbl-helper.py
diff --git a/lfs/squid-asnbl b/lfs/squid-asnbl
new file mode 100644
index 000000000..f2396074d
--- /dev/null
+++ b/lfs/squid-asnbl
@@ -0,0 +1,83 @@
+###############################################################################
+#                                                                             #
+# IPFire.org - A linux based firewall                                         #
+# Copyright (C) 2007-2021  IPFire Team  <info(a)ipfire.org>                     #
+#                                                                             #
+# This program is free software: you can redistribute it and/or modify        #
+# it under the terms of the GNU General Public License as published by        #
+# the Free Software Foundation, either version 3 of the License, or           #
+# (at your option) any later version.                                         #
+#                                                                             #
+# This program is distributed in the hope that it will be useful,             #
+# but WITHOUT ANY WARRANTY; without even the implied warranty of              #
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the               #
+# GNU General Public License for more details.                                #
+#                                                                             #
+# You should have received a copy of the GNU General Public License           #
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.       #
+#                                                                             #
+###############################################################################
+
+
+###############################################################################
+# Definitions
+###############################################################################
+
+include Config
+
+VER        = 0.2.1
+
+THISAPP    = squid-asnbl-$(VER)
+DL_FILE    = $(THISAPP).tar.gz
+DL_FROM    = $(URL_IPFIRE)
+
+DIR_APP    = $(DIR_SRC)/$(THISAPP)
+
+TARGET     = $(DIR_INFO)/$(THISAPP)
+
+DEPS       = libloc squid python3
+
+###############################################################################
+# Top-level Rules
+###############################################################################
+
+objects = $(DL_FILE)
+
+$(DL_FILE) = $(DL_FROM)/$(DL_FILE)
+
+$(DL_FILE)_MD5 = 2225c88ba8e3ae25f5e5c8075f0e7ae8
+
+install : $(TARGET)
+
+check : $(patsubst %,$(DIR_CHK)/%,$(objects))
+
+download :$(patsubst %,$(DIR_DL)/%,$(objects))
+
+md5 : $(subst %,%_MD5,$(objects))
+
+###############################################################################
+# Downloading, checking, md5sum
+###############################################################################
+
+$(patsubst %,$(DIR_CHK)/%,$(objects)) :
+	@$(CHECK)
+
+$(patsubst %,$(DIR_DL)/%,$(objects)) :
+	@$(LOAD)
+
+$(subst %,%_MD5,$(objects)) :
+	@$(MD5)
+
+###############################################################################
+# Installation Details
+###############################################################################
+
+$(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
+	@$(PREBUILD)
+	@rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar zvxf $(DIR_DL)/$(DL_FILE)
+
+	# Install ASNBL helper script
+	cd $(DIR_APP) && install -o root -g root -m 0755 asnbl-helper.py /usr/bin/asnbl-helper.py
+
+	@rm -rf $(DIR_APP)
+	@$(POSTBUILD)
diff --git a/make.sh b/make.sh
index fc03ebcd5..19567769a 100755
--- a/make.sh
+++ b/make.sh
@@ -1623,6 +1623,7 @@ buildipfire() {
   lfsmake2 socat
   lfsmake2 libcdada
   lfsmake2 pmacct
+  lfsmake2 squid-asnbl
 }
 
 buildinstaller() {
-- 
2.26.2

[PATCH 2/3] proxy.cgi: Implement proactive Fast Flux detection and detection for selectively announced destinations

[Peter Müller]

[-- Attachment #1: Type: text/plain, Size: 6953 bytes --]

This patch adds two new features to IPFire's web proxy:

(a) Proactive Fast Flux detection
    FQDNs are resolved to their IP addresses, which are then resolved to
    corresponding Autonomous System Numbers using IPFire's location
    database. Most destinations will scatter across a very low number of
    ASNs (not to be confused with IP addresses!). FQDNs hosted on Fast
    Flux setups have a significantly higher ASN diversity (5 is usually
    a good threshold), so they can be proactively detected.

(b) Detection for selectively announced destinations
    Especially in targeted operations, miscreants host FQDNs for
    exfiltrating data or malware distributions on ASNs not announced
    globally, but only to the intended victim or it's upstream ISPs.

    That way, security researchers located in other parts of the
    internet have no insights into these attacks, hence not being able
    to publish listings or send take down notices for the domains used.

    While RPKI made this attack harder, it can still be observed every
    now and then.

    This feature also protects against accessing FQDNs resolving to IP
    addresses not being globally routeable, hence providing a trivial
    mitigation for so-called "rebound attacks" - which we cannot filter
    at DNS level currently.

Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org>
---
 html/cgi-bin/proxy.cgi | 89 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 89 insertions(+)

diff --git a/html/cgi-bin/proxy.cgi b/html/cgi-bin/proxy.cgi
index 78ad33ad2..b7227deaf 100644
--- a/html/cgi-bin/proxy.cgi
+++ b/html/cgi-bin/proxy.cgi
@@ -21,6 +21,7 @@
 
 use strict;
 use Apache::Htpasswd;
+use Scalar::Util qw(looks_like_number);
 
 # enable only the following on debugging purpose
 #use warnings;
@@ -225,6 +226,9 @@ $proxysettings{'THROTTLING_GREEN_TOTAL'} = 'unlimited';
 $proxysettings{'THROTTLING_GREEN_HOST'} = 'unlimited';
 $proxysettings{'THROTTLING_BLUE_TOTAL'} = 'unlimited';
 $proxysettings{'THROTTLING_BLUE_HOST'} = 'unlimited';
+$proxysettings{'ASNBL_FASTFLUX_DETECTION'} = 'off';
+$proxysettings{'ASNBL_FASTFLUX_THRESHOLD'} = '5';
+$proxysettings{'ASNBL_SELECANN_DETECTION'} = 'off';
 $proxysettings{'ENABLE_MIME_FILTER'} = 'off';
 $proxysettings{'AUTH_METHOD'} = 'none';
 $proxysettings{'AUTH_REALM'} = '';
@@ -414,6 +418,21 @@ if (($proxysettings{'ACTION'} eq $Lang::tr{'save'}) || ($proxysettings{'ACTION'}
 		$errormessage = $Lang::tr{'invalid maximum incoming size'};
 		goto ERROR;
 	}
+	if (($proxysettings{'ASNBL_FASTFLUX_DETECTION'} eq 'on') || ($proxysettings{'ASNBL_SELECANN_DETECTION'} eq 'on'))
+	{
+		if (-z $proxysettings{'ASNBL_FASTFLUX_THRESHOLD'}) {
+			$errormessage = $Lang::tr{'advproxy fastflux no threshold given'};
+			goto ERROR;
+		}
+		if (! looks_like_number($proxysettings{'ASNBL_FASTFLUX_THRESHOLD'})) {
+			$errormessage = $Lang::tr{'advproxy fastflux threshold invalid'};
+			goto ERROR;
+		}
+		if (($proxysettings{'ASNBL_FASTFLUX_THRESHOLD'} < 2) || ($proxysettings{'ASNBL_FASTFLUX_THRESHOLD'} > 10)) {
+			$errormessage = $Lang::tr{'advproxy fastflux threshold out of bounds'};
+			goto ERROR;
+		}
+	}
 	if (!($proxysettings{'AUTH_METHOD'} eq 'none'))
 	{
 		unless (($proxysettings{'AUTH_METHOD'} eq 'ident') &&
@@ -797,6 +816,14 @@ $selected{'THROTTLING_GREEN_HOST'}{$proxysettings{'THROTTLING_GREEN_HOST'}} = "s
 $selected{'THROTTLING_BLUE_TOTAL'}{$proxysettings{'THROTTLING_BLUE_TOTAL'}} = "selected='selected'";
 $selected{'THROTTLING_BLUE_HOST'}{$proxysettings{'THROTTLING_BLUE_HOST'}} = "selected='selected'";
 
+$checked{'ASNBL_FASTFLUX_DETECTION'}{'off'} = '';
+$checked{'ASNBL_FASTFLUX_DETECTION'}{'on'} = '';
+$checked{'ASNBL_FASTFLUX_DETECTION'}{$proxysettings{'ASNBL_FASTFLUX_DETECTION'}} = "checked='checked'";
+
+$checked{'ASNBL_SELECANN_DETECTION'}{'off'} = '';
+$checked{'ASNBL_SELECANN_DETECTION'}{'on'} = '';
+$checked{'ASNBL_SELECANN_DETECTION'}{$proxysettings{'ASNBL_SELECANN_DETECTION'}} = "checked='checked'";
+
 $checked{'ENABLE_MIME_FILTER'}{'off'} = '';
 $checked{'ENABLE_MIME_FILTER'}{'on'} = '';
 $checked{'ENABLE_MIME_FILTER'}{$proxysettings{'ENABLE_MIME_FILTER'}} = "checked='checked'";
@@ -1627,6 +1654,24 @@ END
 print <<END
 </table>
 
+<hr size='1'>
+
+<table width='100%'>
+<tr>
+       <td><b>$Lang::tr{'advproxy asbased anomaly detection'}</b></td>
+</tr>
+<tr>
+       <td class='base'>$Lang::tr{'advproxy fastflux detection'}:</td>
+       <td><input type='checkbox' name='ASNBL_FASTFLUX_DETECTION' $checked{'ASNBL_FASTFLUX_DETECTION'}{'on'} /></td>
+       <td class='base'>$Lang::tr{'advproxy fastflux detection threshold'}:</td>
+       <td><input type='text' name='ASNBL_FASTFLUX_THRESHOLD' value='$proxysettings{'ASNBL_FASTFLUX_THRESHOLD'}' size=2 /></td>
+</tr>
+<tr>
+       <td class='base'>$Lang::tr{'advproxy selectively announcements detection'}:</td>
+       <td colspan='3'><input type='checkbox' name='ASNBL_SELECANN_DETECTION' $checked{'ASNBL_SELECANN_DETECTION'}{'on'} /></td>
+</tr>
+</table>
+
 <hr size='1'>
 END
 ;
@@ -3507,6 +3552,50 @@ if (@ssl_ports) {
 	print FILE "http_access deny  CONNECT !SSL_ports\n";
 }
 
+	if ((($proxysettings{'ASNBL_FASTFLUX_DETECTION'} eq 'on') && (!-z $proxysettings{'ASNBL_FASTFLUX_THRESHOLD'})) || ($proxysettings{'ASNBL_SELECANN_DETECTION'} eq 'on')) {
+		print FILE "external_acl_type asnblhelper children-max=10 children-startup=2 ttl=86400 %DST /usr/bin/asnbl-helper.py /var/ipfire/proxy/asnbl-helper.conf\n";
+		print FILE "acl asnbl external asnblhelper\n";
+		print FILE "http_access deny asnbl\n\n";
+
+		# Write ASNBL helper configuration file...
+		open(ASNBLFILE, ">${General::swroot}/proxy/asnbl-helper.conf");
+		flock(ASNBLFILE, 2);
+
+		print ASNBLFILE<<END
+#
+# This file has been automatically generated. Manual changes will be overwritten.
+#
+
+[GENERAL]
+LOGLEVEL = INFO
+ASNDB_PATH = /var/lib/location/database.db
+USE_REPLYMAP = no
+END
+;
+
+		print ASNBLFILE "AS_DIVERSITY_THRESHOLD = $proxysettings{'ASNBL_FASTFLUX_THRESHOLD'}\n";
+
+		if ($proxysettings{'ASNBL_SELECANN_DETECTION'} eq 'on') {
+			print ASNBLFILE "BLOCK_SUSPECTED_SELECTIVE_ANNOUNCEMENTS = yes\n";
+		} else {
+			print ASNBLFILE "BLOCK_SUSPECTED_SELECTIVE_ANNOUNCEMENTS = no\n";
+		}
+
+		if ($proxysettings{'ASNBL_FASTFLUX_DETECTION'} eq 'on') {
+			print ASNBLFILE "BLOCK_DIVERSITY_EXCEEDING_DESTINATIONS = yes\n";
+		} else {
+			print ASNBLFILE "BLOCK_DIVERSITY_EXCEEDING_DESTINATIONS = no\n";
+		}
+
+		print ASNBLFILE<<END
+TESTDATA = (1.1.1.1, 13335) (8.8.8.8, 15169) (194.95.245.140, 680) (10.0.0.1, 0) (127.0.0.1, 0) (2001:638:d:c102::140, 680) (2606:4700:10::6814:d673, 13335) (fe80::1, 0)
+ACTIVE_ASNBLS = 
+END
+;
+
+		close ASNBLFILE;
+    }
+
 if ($proxysettings{'AUTH_METHOD'} eq 'ident')
 {
 print FILE "#Set ident ACLs\n";
-- 
2.26.2

[PATCH 3/3] langs: Add English and German translations for newly added web proxy features

[Peter Müller]

[-- Attachment #1: Type: text/plain, Size: 4144 bytes --]

Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org>
---
 langs/de/cgi-bin/de.pl | 7 +++++++
 langs/en/cgi-bin/en.pl | 7 +++++++
 2 files changed, 14 insertions(+)

diff --git a/langs/de/cgi-bin/de.pl b/langs/de/cgi-bin/de.pl
index 0d2228ede..642ff53de 100644
--- a/langs/de/cgi-bin/de.pl
+++ b/langs/de/cgi-bin/de.pl
@@ -255,6 +255,7 @@
 'advproxy advanced web proxy configuration' => 'Proxy-Konfiguration',
 'advproxy allowed subnets' => 'Erlaubte Subnetze (eins pro Zeile)',
 'advproxy allowed web browsers' => 'Zulässige Clients für Web-Zugriffe',
+'advproxy asbased anomaly detection' => 'Anomalieerkennungen auf Basis Autonomer Systeme',
 'advproxy back to main page' => 'Zurück zur Hauptseite',
 'advproxy banned ip clients' => 'Gesperrte IP-Adressen (eine pro Zeile)',
 'advproxy banned mac clients' => 'Gesperrte MAC-Adressen (eine pro Zeile)',
@@ -326,6 +327,11 @@
 'advproxy errmsg wpad invalid ip or mask' => 'WPAD: Ungültige IP oder Subnetz für ausgenommenes IP-Subnetz',
 'advproxy error design' => 'Design der Fehlermeldungen',
 'advproxy error language' => 'Sprache der Fehlermeldungen',
+'advproxy fastflux detection' => 'Verbindungen zu auf Fast Flux-Setups gehosteten Zielen verweigern',
+'advproxy fastflux no threshold given' => 'Kein Schwellwert für Fast Flux-Erkennung angegeben',
+'advproxy fastflux detection threshold' => 'Schwellwert',
+'advproxy fastflux threshold invalid' => 'Eingegebener Schwellwert für Fast Flux-Erkennung ist ungültig',
+'advproxy fastflux threshold out of bounds' => 'Eingegebener Schwellwert für Fast Flux-Erkennung befindet sich außerhalb zulässiger Grenzwerte',
 'advproxy friday' => 'Fr',
 'advproxy from' => 'Von',
 'advproxy group access control' => 'Gruppenbasierte Zugriffskontrolle',
@@ -362,6 +368,7 @@
 'advproxy reset' => 'Zurücksetzen',
 'advproxy saturday' => 'Sa',
 'advproxy save and restart' => 'Speichern und Neustart',
+'advproxy selectively announcements detection' => 'Verbindungen zu Zielen in selektiv propagierten Netzen verweigern',
 'advproxy squid version' => 'Squid Versionsnummer',
 'advproxy squidclamav' => 'SquidClamav',
 'advproxy ssadvanced proxy' => 'advanced proxy',
diff --git a/langs/en/cgi-bin/en.pl b/langs/en/cgi-bin/en.pl
index 2ba6961f3..1f82c47ca 100644
--- a/langs/en/cgi-bin/en.pl
+++ b/langs/en/cgi-bin/en.pl
@@ -252,6 +252,7 @@
 'advproxy advanced web proxy configuration' => 'Advanced web proxy configuration',
 'advproxy allowed subnets' => 'Allowed subnets (one per line)',
 'advproxy allowed web browsers' => 'Allowed clients for web access',
+'advproxy asbased anomaly detection' => 'Anomaly detections based on Autonomous Systems information',
 'advproxy back to main page' => 'Back to main page',
 'advproxy banned ip clients' => 'Banned IP addresses (one per line)',
 'advproxy banned mac clients' => 'Banned MAC addresses (one per line)',
@@ -323,6 +324,11 @@
 'advproxy errmsg wpad invalid ip or mask' => 'WPAD: Invalid IP or subnet for excluded IP subnet',
 'advproxy error design' => 'Error messages design',
 'advproxy error language' => 'Error messages language',
+'advproxy fastflux detection' => 'Deny access to destinations hosted on fast flux setups',
+'advproxy fastflux no threshold given' => 'No threshold was given for fast flux detection',
+'advproxy fastflux detection threshold' => 'Threshold',
+'advproxy fastflux threshold invalid' => 'Supplied fast flux detection threshold is invalid',
+'advproxy fastflux threshold out of bounds' => 'Supplied fast flux detection threshold is out of bounds',
 'advproxy friday' => 'Fri',
 'advproxy from' => 'From',
 'advproxy group access control' => 'Group based access control',
@@ -359,6 +365,7 @@
 'advproxy reset' => 'Reset',
 'advproxy saturday' => 'Sat',
 'advproxy save and restart' => 'Save and Restart',
+'advproxy selectively announcements detection' => 'Deny access to destinations hosted on selectively announced networks',
 'advproxy squid version' => 'Squid cache version',
 'advproxy squidclamav' => 'SquidClamav',
 'advproxy ssadvanced proxy' => 'advanced proxy',
-- 
2.26.2

Re: [PATCH 2/3] proxy.cgi: Implement proactive Fast Flux detection and detection for selectively announced destinations

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 7555 bytes --]

Hello,

> On 18 Jun 2021, at 18:24, Peter Müller <peter.mueller(a)ipfire.org> wrote:
> 
> This patch adds two new features to IPFire's web proxy:
> 
> (a) Proactive Fast Flux detection
>    FQDNs are resolved to their IP addresses, which are then resolved to
>    corresponding Autonomous System Numbers using IPFire's location
>    database. Most destinations will scatter across a very low number of
>    ASNs (not to be confused with IP addresses!). FQDNs hosted on Fast
>    Flux setups have a significantly higher ASN diversity (5 is usually
>    a good threshold), so they can be proactively detected.
> 
> (b) Detection for selectively announced destinations
>    Especially in targeted operations, miscreants host FQDNs for
>    exfiltrating data or malware distributions on ASNs not announced
>    globally, but only to the intended victim or it's upstream ISPs.
> 
>    That way, security researchers located in other parts of the
>    internet have no insights into these attacks, hence not being able
>    to publish listings or send take down notices for the domains used.
> 
>    While RPKI made this attack harder, it can still be observed every
>    now and then.
> 
>    This feature also protects against accessing FQDNs resolving to IP
>    addresses not being globally routeable, hence providing a trivial
>    mitigation for so-called "rebound attacks" - which we cannot filter
>    at DNS level currently.
> 
> Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org>
> ---
> html/cgi-bin/proxy.cgi | 89 ++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 89 insertions(+)
> 
> diff --git a/html/cgi-bin/proxy.cgi b/html/cgi-bin/proxy.cgi
> index 78ad33ad2..b7227deaf 100644
> --- a/html/cgi-bin/proxy.cgi
> +++ b/html/cgi-bin/proxy.cgi
> @@ -21,6 +21,7 @@
> 
> use strict;
> use Apache::Htpasswd;
> +use Scalar::Util qw(looks_like_number);
> 
> # enable only the following on debugging purpose
> #use warnings;
> @@ -225,6 +226,9 @@ $proxysettings{'THROTTLING_GREEN_TOTAL'} = 'unlimited';
> $proxysettings{'THROTTLING_GREEN_HOST'} = 'unlimited';
> $proxysettings{'THROTTLING_BLUE_TOTAL'} = 'unlimited';
> $proxysettings{'THROTTLING_BLUE_HOST'} = 'unlimited';
> +$proxysettings{'ASNBL_FASTFLUX_DETECTION'} = 'off';
> +$proxysettings{'ASNBL_FASTFLUX_THRESHOLD'} = '5';
> +$proxysettings{'ASNBL_SELECANN_DETECTION'} = 'off';
> $proxysettings{'ENABLE_MIME_FILTER'} = 'off';
> $proxysettings{'AUTH_METHOD'} = 'none';
> $proxysettings{'AUTH_REALM'} = '';
> @@ -414,6 +418,21 @@ if (($proxysettings{'ACTION'} eq $Lang::tr{'save'}) || ($proxysettings{'ACTION'}
> 		$errormessage = $Lang::tr{'invalid maximum incoming size'};
> 		goto ERROR;
> 	}
> +	if (($proxysettings{'ASNBL_FASTFLUX_DETECTION'} eq 'on') || ($proxysettings{'ASNBL_SELECANN_DETECTION'} eq 'on'))
> +	{
> +		if (-z $proxysettings{'ASNBL_FASTFLUX_THRESHOLD'}) {
> +			$errormessage = $Lang::tr{'advproxy fastflux no threshold given'};
> +			goto ERROR;
> +		}
> +		if (! looks_like_number($proxysettings{'ASNBL_FASTFLUX_THRESHOLD'})) {
> +			$errormessage = $Lang::tr{'advproxy fastflux threshold invalid'};
> +			goto ERROR;
> +		}
> +		if (($proxysettings{'ASNBL_FASTFLUX_THRESHOLD'} < 2) || ($proxysettings{'ASNBL_FASTFLUX_THRESHOLD'} > 10)) {
> +			$errormessage = $Lang::tr{'advproxy fastflux threshold out of bounds'};
> +			goto ERROR;
> +		}
> +	}
> 	if (!($proxysettings{'AUTH_METHOD'} eq 'none'))
> 	{
> 		unless (($proxysettings{'AUTH_METHOD'} eq 'ident') &&
> @@ -797,6 +816,14 @@ $selected{'THROTTLING_GREEN_HOST'}{$proxysettings{'THROTTLING_GREEN_HOST'}} = "s
> $selected{'THROTTLING_BLUE_TOTAL'}{$proxysettings{'THROTTLING_BLUE_TOTAL'}} = "selected='selected'";
> $selected{'THROTTLING_BLUE_HOST'}{$proxysettings{'THROTTLING_BLUE_HOST'}} = "selected='selected'";
> 
> +$checked{'ASNBL_FASTFLUX_DETECTION'}{'off'} = '';
> +$checked{'ASNBL_FASTFLUX_DETECTION'}{'on'} = '';
> +$checked{'ASNBL_FASTFLUX_DETECTION'}{$proxysettings{'ASNBL_FASTFLUX_DETECTION'}} = "checked='checked'";
> +
> +$checked{'ASNBL_SELECANN_DETECTION'}{'off'} = '';
> +$checked{'ASNBL_SELECANN_DETECTION'}{'on'} = '';
> +$checked{'ASNBL_SELECANN_DETECTION'}{$proxysettings{'ASNBL_SELECANN_DETECTION'}} = "checked='checked'";
> +
> $checked{'ENABLE_MIME_FILTER'}{'off'} = '';
> $checked{'ENABLE_MIME_FILTER'}{'on'} = '';
> $checked{'ENABLE_MIME_FILTER'}{$proxysettings{'ENABLE_MIME_FILTER'}} = "checked='checked'";
> @@ -1627,6 +1654,24 @@ END
> print <<END
> </table>
> 
> +<hr size='1'>
> +
> +<table width='100%'>
> +<tr>
> +       <td><b>$Lang::tr{'advproxy asbased anomaly detection'}</b></td>
> +</tr>
> +<tr>
> +       <td class='base'>$Lang::tr{'advproxy fastflux detection'}:</td>
> +       <td><input type='checkbox' name='ASNBL_FASTFLUX_DETECTION' $checked{'ASNBL_FASTFLUX_DETECTION'}{'on'} /></td>
> +       <td class='base'>$Lang::tr{'advproxy fastflux detection threshold'}:</td>
> +       <td><input type='text' name='ASNBL_FASTFLUX_THRESHOLD' value='$proxysettings{'ASNBL_FASTFLUX_THRESHOLD'}' size=2 /></td>
> +</tr>
> +<tr>
> +       <td class='base'>$Lang::tr{'advproxy selectively announcements detection'}:</td>
> +       <td colspan='3'><input type='checkbox' name='ASNBL_SELECANN_DETECTION' $checked{'ASNBL_SELECANN_DETECTION'}{'on'} /></td>
> +</tr>
> +</table>
> +
> <hr size='1'>
> END
> ;
> @@ -3507,6 +3552,50 @@ if (@ssl_ports) {
> 	print FILE "http_access deny  CONNECT !SSL_ports\n";
> }
> 
> +	if ((($proxysettings{'ASNBL_FASTFLUX_DETECTION'} eq 'on') && (!-z $proxysettings{'ASNBL_FASTFLUX_THRESHOLD'})) || ($proxysettings{'ASNBL_SELECANN_DETECTION'} eq 'on')) {
> +		print FILE "external_acl_type asnblhelper children-max=10 children-startup=2 ttl=86400 %DST /usr/bin/asnbl-helper.py /var/ipfire/proxy/asnbl-helper.conf\n";
> +		print FILE "acl asnbl external asnblhelper\n";
> +		print FILE "http_access deny asnbl\n\n";
> +
> +		# Write ASNBL helper configuration file...
> +		open(ASNBLFILE, ">${General::swroot}/proxy/asnbl-helper.conf");
> +		flock(ASNBLFILE, 2);
> +
> +		print ASNBLFILE<<END
> +#
> +# This file has been automatically generated. Manual changes will be overwritten.
> +#
> +
> +[GENERAL]
> +LOGLEVEL = INFO
> +ASNDB_PATH = /var/lib/location/database.db
> +USE_REPLYMAP = no
> +END
> +;
> +
> +		print ASNBLFILE "AS_DIVERSITY_THRESHOLD = $proxysettings{'ASNBL_FASTFLUX_THRESHOLD'}\n";
> +
> +		if ($proxysettings{'ASNBL_SELECANN_DETECTION'} eq 'on') {
> +			print ASNBLFILE "BLOCK_SUSPECTED_SELECTIVE_ANNOUNCEMENTS = yes\n";
> +		} else {
> +			print ASNBLFILE "BLOCK_SUSPECTED_SELECTIVE_ANNOUNCEMENTS = no\n";
> +		}
> +
> +		if ($proxysettings{'ASNBL_FASTFLUX_DETECTION'} eq 'on') {
> +			print ASNBLFILE "BLOCK_DIVERSITY_EXCEEDING_DESTINATIONS = yes\n";
> +		} else {
> +			print ASNBLFILE "BLOCK_DIVERSITY_EXCEEDING_DESTINATIONS = no\n";
> +		}
> +
> +		print ASNBLFILE<<END
> +TESTDATA = (1.1.1.1, 13335) (8.8.8.8, 15169) (194.95.245.140, 680) (10.0.0.1, 0) (127.0.0.1, 0) (2001:638:d:c102::140, 680) (2606:4700:10::6814:d673, 13335) (fe80::1, 0)

Why do we want to hard-code this here?

Does that not (if anywhere) belong into libloc? I disagree with hard-coding this, because what happens if Google moves their DNS server? It would break this feature.

-Michael

> +ACTIVE_ASNBLS = 
> +END
> +;
> +
> +		close ASNBLFILE;
> +    }
> +
> if ($proxysettings{'AUTH_METHOD'} eq 'ident')
> {
> print FILE "#Set ident ACLs\n";
> -- 
> 2.26.2

Re: [PATCH 2/3] proxy.cgi: Implement proactive Fast Flux detection and detection for selectively announced destinations

[Peter Müller]

[-- Attachment #1: Type: text/plain, Size: 8530 bytes --]

Hello Michael,

> Hello,
> 
>> On 18 Jun 2021, at 18:24, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>
>> This patch adds two new features to IPFire's web proxy:
>>
>> (a) Proactive Fast Flux detection
>>    FQDNs are resolved to their IP addresses, which are then resolved to
>>    corresponding Autonomous System Numbers using IPFire's location
>>    database. Most destinations will scatter across a very low number of
>>    ASNs (not to be confused with IP addresses!). FQDNs hosted on Fast
>>    Flux setups have a significantly higher ASN diversity (5 is usually
>>    a good threshold), so they can be proactively detected.
>>
>> (b) Detection for selectively announced destinations
>>    Especially in targeted operations, miscreants host FQDNs for
>>    exfiltrating data or malware distributions on ASNs not announced
>>    globally, but only to the intended victim or it's upstream ISPs.
>>
>>    That way, security researchers located in other parts of the
>>    internet have no insights into these attacks, hence not being able
>>    to publish listings or send take down notices for the domains used.
>>
>>    While RPKI made this attack harder, it can still be observed every
>>    now and then.
>>
>>    This feature also protects against accessing FQDNs resolving to IP
>>    addresses not being globally routeable, hence providing a trivial
>>    mitigation for so-called "rebound attacks" - which we cannot filter
>>    at DNS level currently.
>>
>> Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org>
>> ---
>> html/cgi-bin/proxy.cgi | 89 ++++++++++++++++++++++++++++++++++++++++++
>> 1 file changed, 89 insertions(+)
>>
>> diff --git a/html/cgi-bin/proxy.cgi b/html/cgi-bin/proxy.cgi
>> index 78ad33ad2..b7227deaf 100644
>> --- a/html/cgi-bin/proxy.cgi
>> +++ b/html/cgi-bin/proxy.cgi
>> @@ -21,6 +21,7 @@
>>
>> use strict;
>> use Apache::Htpasswd;
>> +use Scalar::Util qw(looks_like_number);
>>
>> # enable only the following on debugging purpose
>> #use warnings;
>> @@ -225,6 +226,9 @@ $proxysettings{'THROTTLING_GREEN_TOTAL'} = 'unlimited';
>> $proxysettings{'THROTTLING_GREEN_HOST'} = 'unlimited';
>> $proxysettings{'THROTTLING_BLUE_TOTAL'} = 'unlimited';
>> $proxysettings{'THROTTLING_BLUE_HOST'} = 'unlimited';
>> +$proxysettings{'ASNBL_FASTFLUX_DETECTION'} = 'off';
>> +$proxysettings{'ASNBL_FASTFLUX_THRESHOLD'} = '5';
>> +$proxysettings{'ASNBL_SELECANN_DETECTION'} = 'off';
>> $proxysettings{'ENABLE_MIME_FILTER'} = 'off';
>> $proxysettings{'AUTH_METHOD'} = 'none';
>> $proxysettings{'AUTH_REALM'} = '';
>> @@ -414,6 +418,21 @@ if (($proxysettings{'ACTION'} eq $Lang::tr{'save'}) || ($proxysettings{'ACTION'}
>> 		$errormessage = $Lang::tr{'invalid maximum incoming size'};
>> 		goto ERROR;
>> 	}
>> +	if (($proxysettings{'ASNBL_FASTFLUX_DETECTION'} eq 'on') || ($proxysettings{'ASNBL_SELECANN_DETECTION'} eq 'on'))
>> +	{
>> +		if (-z $proxysettings{'ASNBL_FASTFLUX_THRESHOLD'}) {
>> +			$errormessage = $Lang::tr{'advproxy fastflux no threshold given'};
>> +			goto ERROR;
>> +		}
>> +		if (! looks_like_number($proxysettings{'ASNBL_FASTFLUX_THRESHOLD'})) {
>> +			$errormessage = $Lang::tr{'advproxy fastflux threshold invalid'};
>> +			goto ERROR;
>> +		}
>> +		if (($proxysettings{'ASNBL_FASTFLUX_THRESHOLD'} < 2) || ($proxysettings{'ASNBL_FASTFLUX_THRESHOLD'} > 10)) {
>> +			$errormessage = $Lang::tr{'advproxy fastflux threshold out of bounds'};
>> +			goto ERROR;
>> +		}
>> +	}
>> 	if (!($proxysettings{'AUTH_METHOD'} eq 'none'))
>> 	{
>> 		unless (($proxysettings{'AUTH_METHOD'} eq 'ident') &&
>> @@ -797,6 +816,14 @@ $selected{'THROTTLING_GREEN_HOST'}{$proxysettings{'THROTTLING_GREEN_HOST'}} = "s
>> $selected{'THROTTLING_BLUE_TOTAL'}{$proxysettings{'THROTTLING_BLUE_TOTAL'}} = "selected='selected'";
>> $selected{'THROTTLING_BLUE_HOST'}{$proxysettings{'THROTTLING_BLUE_HOST'}} = "selected='selected'";
>>
>> +$checked{'ASNBL_FASTFLUX_DETECTION'}{'off'} = '';
>> +$checked{'ASNBL_FASTFLUX_DETECTION'}{'on'} = '';
>> +$checked{'ASNBL_FASTFLUX_DETECTION'}{$proxysettings{'ASNBL_FASTFLUX_DETECTION'}} = "checked='checked'";
>> +
>> +$checked{'ASNBL_SELECANN_DETECTION'}{'off'} = '';
>> +$checked{'ASNBL_SELECANN_DETECTION'}{'on'} = '';
>> +$checked{'ASNBL_SELECANN_DETECTION'}{$proxysettings{'ASNBL_SELECANN_DETECTION'}} = "checked='checked'";
>> +
>> $checked{'ENABLE_MIME_FILTER'}{'off'} = '';
>> $checked{'ENABLE_MIME_FILTER'}{'on'} = '';
>> $checked{'ENABLE_MIME_FILTER'}{$proxysettings{'ENABLE_MIME_FILTER'}} = "checked='checked'";
>> @@ -1627,6 +1654,24 @@ END
>> print <<END
>> </table>
>>
>> +<hr size='1'>
>> +
>> +<table width='100%'>
>> +<tr>
>> +       <td><b>$Lang::tr{'advproxy asbased anomaly detection'}</b></td>
>> +</tr>
>> +<tr>
>> +       <td class='base'>$Lang::tr{'advproxy fastflux detection'}:</td>
>> +       <td><input type='checkbox' name='ASNBL_FASTFLUX_DETECTION' $checked{'ASNBL_FASTFLUX_DETECTION'}{'on'} /></td>
>> +       <td class='base'>$Lang::tr{'advproxy fastflux detection threshold'}:</td>
>> +       <td><input type='text' name='ASNBL_FASTFLUX_THRESHOLD' value='$proxysettings{'ASNBL_FASTFLUX_THRESHOLD'}' size=2 /></td>
>> +</tr>
>> +<tr>
>> +       <td class='base'>$Lang::tr{'advproxy selectively announcements detection'}:</td>
>> +       <td colspan='3'><input type='checkbox' name='ASNBL_SELECANN_DETECTION' $checked{'ASNBL_SELECANN_DETECTION'}{'on'} /></td>
>> +</tr>
>> +</table>
>> +
>> <hr size='1'>
>> END
>> ;
>> @@ -3507,6 +3552,50 @@ if (@ssl_ports) {
>> 	print FILE "http_access deny  CONNECT !SSL_ports\n";
>> }
>>
>> +	if ((($proxysettings{'ASNBL_FASTFLUX_DETECTION'} eq 'on') && (!-z $proxysettings{'ASNBL_FASTFLUX_THRESHOLD'})) || ($proxysettings{'ASNBL_SELECANN_DETECTION'} eq 'on')) {
>> +		print FILE "external_acl_type asnblhelper children-max=10 children-startup=2 ttl=86400 %DST /usr/bin/asnbl-helper.py /var/ipfire/proxy/asnbl-helper.conf\n";
>> +		print FILE "acl asnbl external asnblhelper\n";
>> +		print FILE "http_access deny asnbl\n\n";
>> +
>> +		# Write ASNBL helper configuration file...
>> +		open(ASNBLFILE, ">${General::swroot}/proxy/asnbl-helper.conf");
>> +		flock(ASNBLFILE, 2);
>> +
>> +		print ASNBLFILE<<END
>> +#
>> +# This file has been automatically generated. Manual changes will be overwritten.
>> +#
>> +
>> +[GENERAL]
>> +LOGLEVEL = INFO
>> +ASNDB_PATH = /var/lib/location/database.db
>> +USE_REPLYMAP = no
>> +END
>> +;
>> +
>> +		print ASNBLFILE "AS_DIVERSITY_THRESHOLD = $proxysettings{'ASNBL_FASTFLUX_THRESHOLD'}\n";
>> +
>> +		if ($proxysettings{'ASNBL_SELECANN_DETECTION'} eq 'on') {
>> +			print ASNBLFILE "BLOCK_SUSPECTED_SELECTIVE_ANNOUNCEMENTS = yes\n";
>> +		} else {
>> +			print ASNBLFILE "BLOCK_SUSPECTED_SELECTIVE_ANNOUNCEMENTS = no\n";
>> +		}
>> +
>> +		if ($proxysettings{'ASNBL_FASTFLUX_DETECTION'} eq 'on') {
>> +			print ASNBLFILE "BLOCK_DIVERSITY_EXCEEDING_DESTINATIONS = yes\n";
>> +		} else {
>> +			print ASNBLFILE "BLOCK_DIVERSITY_EXCEEDING_DESTINATIONS = no\n";
>> +		}
>> +
>> +		print ASNBLFILE<<END
>> +TESTDATA = (1.1.1.1, 13335) (8.8.8.8, 15169) (194.95.245.140, 680) (10.0.0.1, 0) (127.0.0.1, 0) (2001:638:d:c102::140, 680) (2606:4700:10::6814:d673, 13335) (fe80::1, 0)
> 
> Why do we want to hard-code this here?

Because the ASNBL helper requires some test points to ensure the location database provided is actually
working. It is designed to prefer "fail close" (i. e. stopping operation in case something is rendering
its purpose useless) rather than "fail open" (i. e. logging a warning and proceed anyway, even if the location
database returns "0" for any given IP address).

> Does that not (if anywhere) belong into libloc? I disagree with hard-coding this, because what happens if Google moves their DNS server? It would break this feature.

Partly. We should have some sanity checks in libloc as well, but they ultimately boil down to the same problem:
We need a set of relatively statically propagated IP addresses, which the internet does not have by nature.

Do you have a better proposal for testing points? Like the IP address of our own NTP server, which we hard-coded
somewhere else, too?

Thanks, and best regards,
Peter Müller

> 
> -Michael
> 
>> +ACTIVE_ASNBLS = 
>> +END
>> +;
>> +
>> +		close ASNBLFILE;
>> +    }
>> +
>> if ($proxysettings{'AUTH_METHOD'} eq 'ident')
>> {
>> print FILE "#Set ident ACLs\n";
>> -- 
>> 2.26.2
> 

Re: [PATCH 0/3] Add ASN-based anomaly detections to IPFire's web proxy: Proactive Fast Flux detection and detection for selectively announced networks

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 2288 bytes --]

Hello Peter,

I love this feature. I think it is a one-of-a-kind thing and hopefully many more people will think the same.

However, it will need a lot of documentation and explaining.

I have a couple of high-level questions:

* Does it make sense to give the user the choice for the threshold?

It seems to be a difficult question because it requires exact knowledge what this feature actually does. My fears are that people just set this to something like “9” and the feature would become ineffective. What use-case is there to change this?

* Selective announcements: Should this necessarily live in the proxy? Why do we not generate a filter for the firewall?

-Michael

> On 18 Jun 2021, at 18:24, Peter Müller <peter.mueller(a)ipfire.org> wrote:
> 
> This patchset adds two new features to IPFire's web proxy, taking advantage
> of the Autonomous System information we have at hand by using libloc.
> 
> The proactive Fast Flux detection is especially worth noticing, as even most
> expensive (= advanced?) security suites do not provide similar protection,
> especially not in a proactive manner.
> 
> By simply enumerating the distinct amount of Autonomous System Numbers a FQDN
> ultimately resolves to, we are able to deny access to malware distribution
> sites, phishing sites, C&C servers, and other cybercrime stuff hosted on Fast
> Flux setups abusing cracked machines around the world - even before the FQDN
> or any IP address involved is flagged as malicious by any security vendor.
> 
> Peter Müller (3):
>  squid-asnbl: New package
>  proxy.cgi: Implement proactive Fast Flux detection and detection for
>    selectively announced destinations
>  langs: Add English and German translations for newly added web proxy
>    features
> 
> config/rootfiles/common/squid-asnbl |  1 +
> html/cgi-bin/proxy.cgi              | 89 +++++++++++++++++++++++++++++
> langs/de/cgi-bin/de.pl              |  7 +++
> langs/en/cgi-bin/en.pl              |  7 +++
> lfs/squid-asnbl                     | 83 +++++++++++++++++++++++++++
> make.sh                             |  1 +
> 6 files changed, 188 insertions(+)
> create mode 100644 config/rootfiles/common/squid-asnbl
> create mode 100644 lfs/squid-asnbl
> 
> -- 
> 2.26.2

Re: [PATCH 0/3] Add ASN-based anomaly detections to IPFire's web proxy: Proactive Fast Flux detection and detection for selectively announced networks

[Peter Müller]

[-- Attachment #1: Type: text/plain, Size: 3954 bytes --]

Hello Michael,

thank you for your reply.

> Hello Peter,
> 
> I love this feature. I think it is a one-of-a-kind thing and hopefully many more people will think the same.

Yes, I like the idea, too. Sometimes, security can be simple _and_ effective... :-)

> However, it will need a lot of documentation and explaining.

Indeed. I was thinking about a blog post for it; we probably need to explain Fast Flux in the
first place, and I am not sure if all of our users are aware of the existence of autonomous
systems.

> I have a couple of high-level questions:
> 
> * Does it make sense to give the user the choice for the threshold?
> 
> It seems to be a difficult question because it requires exact knowledge what this feature actually does. My fears are that people just set this to something like “9” and the feature would become ineffective. What use-case is there to change this?

One size never fits all, I guess.

Indeed, the range of useful threshold values is pretty small: Anything below 4 causes _way_ too
much false positives in productive environment, whereas even 7 appears to be too ineffective.

At the moment, the CGI catches values the ASNBL helper would treat itself as being invalid. Do
you think narrowing down this range to 4 to 7 makes sense? Or should we replace it by a dropdown
for adjusting sensitivity?

Either way, it is a good idea to tell users to leave the default where it is unless they truly
understand what they are doing.

> * Selective announcements: Should this necessarily live in the proxy? Why do we not generate a filter for the firewall?

We can do so as well, and I would love to see such a feature landing in IPFire.

Given our current state of libloc, I doubt this is possible: We would need a function that returns
all networks we do not have an AS for - to my knowledge, the libloc (bindings) do not support this
at the moment.

Apart from that: On a packet filter level, we lack the FQDN of a destination, which might be useful
to have for debugging or forensic reasons.

Also, the users will experience a timeout after n seconds. Having selective announcement detection
turned on, they'll get their error message straight away. I was told this improves UX... :-)

Thanks, and best regards,
Peter Müller

> 
> -Michael
> 
>> On 18 Jun 2021, at 18:24, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>
>> This patchset adds two new features to IPFire's web proxy, taking advantage
>> of the Autonomous System information we have at hand by using libloc.
>>
>> The proactive Fast Flux detection is especially worth noticing, as even most
>> expensive (= advanced?) security suites do not provide similar protection,
>> especially not in a proactive manner.
>>
>> By simply enumerating the distinct amount of Autonomous System Numbers a FQDN
>> ultimately resolves to, we are able to deny access to malware distribution
>> sites, phishing sites, C&C servers, and other cybercrime stuff hosted on Fast
>> Flux setups abusing cracked machines around the world - even before the FQDN
>> or any IP address involved is flagged as malicious by any security vendor.
>>
>> Peter Müller (3):
>>  squid-asnbl: New package
>>  proxy.cgi: Implement proactive Fast Flux detection and detection for
>>    selectively announced destinations
>>  langs: Add English and German translations for newly added web proxy
>>    features
>>
>> config/rootfiles/common/squid-asnbl |  1 +
>> html/cgi-bin/proxy.cgi              | 89 +++++++++++++++++++++++++++++
>> langs/de/cgi-bin/de.pl              |  7 +++
>> langs/en/cgi-bin/en.pl              |  7 +++
>> lfs/squid-asnbl                     | 83 +++++++++++++++++++++++++++
>> make.sh                             |  1 +
>> 6 files changed, 188 insertions(+)
>> create mode 100644 config/rootfiles/common/squid-asnbl
>> create mode 100644 lfs/squid-asnbl
>>
>> -- 
>> 2.26.2
> 

Re: [PATCH 0/3] Add ASN-based anomaly detections to IPFire's web proxy: Proactive Fast Flux detection and detection for selectively announced networks

[Peter Müller]

[-- Attachment #1: Type: text/plain, Size: 6225 bytes --]

Hello *,

by accident, I just stumbled across a false positive related to the Fast Flux detection:

> [root(a)maverick ~]# su squid -s /bin/bash
> bash-5.1$ /usr/bin/asnbl-helper.py /var/ipfire/proxy/asnbl-helper.conf
> Sep 06 18:28:21 squid-asnbl-helper[9945] WARN: No ASNBL configured. This is acceptable as long as this script is configured to do anything, you just have been warned...
> Sep 06 18:28:21 squid-asnbl-helper[9945] INFO: Configuation sanity tests passed, good, processing...
> Sep 06 18:28:21 squid-asnbl-helper[9945] INFO: Successfully loaded location database from /var/lib/location/database.db generated 'Mon Sep  6 05:52:56 2021' (UTC/GMT) by 'IPFire Project' - good
> Sep 06 18:28:21 squid-asnbl-helper[9945] INFO: Running ASN database response tests...
> Sep 06 18:28:21 squid-asnbl-helper[9945] INFO: ASN database operational - excellent. Waiting for input...
> fedoraproject.org
> Sep 06 18:28:26 squid-asnbl-helper[9945] WARN: Destination 'fedoraproject.org' exceeds ASN diversity threshold (9 > 5), possibly Fast Flux: [81, 3701, 15456, 16509, 21785, 22753, 36850, 54455, 61317]
> Sep 06 18:28:26 squid-asnbl-helper[9945] INFO: Denying access to possible Fast Flux destination 'fedoraproject.org'
> OK

Apparently, the Fedora folks think it is a good idea to use round-robin for load balancing:

> $ dig +short a fedoraproject.org
> 140.211.169.206
> 67.219.144.68
> 85.236.55.6
> 38.145.60.20
> 152.19.134.198
> 209.132.190.2
> 18.133.140.134
> 18.185.136.17
> 185.141.165.254
> 152.19.134.142
> 38.145.60.21
> 18.159.254.57

At the first glance, using the URL filter (by adding fedoraproject.org to the list of always allowed
domains) seems to be a straight-forward solution to this problem. However, it does not work, as the
ASNBL script is executed in the context of an ACL, while the URL filter comes as a redirect/wrapper.
Therefore, it is never reached if a "deny" ACL matches in the first place.

This is the only false positive I observed so far. Unfortunately, it is a rather bad one. :-/

Any thoughts on what to do now?

Thanks, and best regards,
Peter Müller


> Hello Michael,
> 
> thank you for your reply.
> 
>> Hello Peter,
>>
>> I love this feature. I think it is a one-of-a-kind thing and hopefully many more people will think the same.
> 
> Yes, I like the idea, too. Sometimes, security can be simple _and_ effective... :-)
> 
>> However, it will need a lot of documentation and explaining.
> 
> Indeed. I was thinking about a blog post for it; we probably need to explain Fast Flux in the
> first place, and I am not sure if all of our users are aware of the existence of autonomous
> systems.
> 
>> I have a couple of high-level questions:
>>
>> * Does it make sense to give the user the choice for the threshold?
>>
>> It seems to be a difficult question because it requires exact knowledge what this feature actually does. My fears are that people just set this to something like “9” and the feature would become ineffective. What use-case is there to change this?
> 
> One size never fits all, I guess.
> 
> Indeed, the range of useful threshold values is pretty small: Anything below 4 causes _way_ too
> much false positives in productive environment, whereas even 7 appears to be too ineffective.
> 
> At the moment, the CGI catches values the ASNBL helper would treat itself as being invalid. Do
> you think narrowing down this range to 4 to 7 makes sense? Or should we replace it by a dropdown
> for adjusting sensitivity?
> 
> Either way, it is a good idea to tell users to leave the default where it is unless they truly
> understand what they are doing.
> 
>> * Selective announcements: Should this necessarily live in the proxy? Why do we not generate a filter for the firewall?
> 
> We can do so as well, and I would love to see such a feature landing in IPFire.
> 
> Given our current state of libloc, I doubt this is possible: We would need a function that returns
> all networks we do not have an AS for - to my knowledge, the libloc (bindings) do not support this
> at the moment.
> 
> Apart from that: On a packet filter level, we lack the FQDN of a destination, which might be useful
> to have for debugging or forensic reasons.
> 
> Also, the users will experience a timeout after n seconds. Having selective announcement detection
> turned on, they'll get their error message straight away. I was told this improves UX... :-)
> 
> Thanks, and best regards,
> Peter Müller
> 
>>
>> -Michael
>>
>>> On 18 Jun 2021, at 18:24, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>>
>>> This patchset adds two new features to IPFire's web proxy, taking advantage
>>> of the Autonomous System information we have at hand by using libloc.
>>>
>>> The proactive Fast Flux detection is especially worth noticing, as even most
>>> expensive (= advanced?) security suites do not provide similar protection,
>>> especially not in a proactive manner.
>>>
>>> By simply enumerating the distinct amount of Autonomous System Numbers a FQDN
>>> ultimately resolves to, we are able to deny access to malware distribution
>>> sites, phishing sites, C&C servers, and other cybercrime stuff hosted on Fast
>>> Flux setups abusing cracked machines around the world - even before the FQDN
>>> or any IP address involved is flagged as malicious by any security vendor.
>>>
>>> Peter Müller (3):
>>>  squid-asnbl: New package
>>>  proxy.cgi: Implement proactive Fast Flux detection and detection for
>>>    selectively announced destinations
>>>  langs: Add English and German translations for newly added web proxy
>>>    features
>>>
>>> config/rootfiles/common/squid-asnbl |  1 +
>>> html/cgi-bin/proxy.cgi              | 89 +++++++++++++++++++++++++++++
>>> langs/de/cgi-bin/de.pl              |  7 +++
>>> langs/en/cgi-bin/en.pl              |  7 +++
>>> lfs/squid-asnbl                     | 83 +++++++++++++++++++++++++++
>>> make.sh                             |  1 +
>>> 6 files changed, 188 insertions(+)
>>> create mode 100644 config/rootfiles/common/squid-asnbl
>>> create mode 100644 lfs/squid-asnbl
>>>
>>> -- 
>>> 2.26.2
>>

Re: [PATCH 0/3] Add ASN-based anomaly detections to IPFire's web proxy: Proactive Fast Flux detection and detection for selectively announced networks

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 7004 bytes --]

Hello,

This is bad news indeed.

How about we have a whitelist that we ship with this?

If you are using ACLs, you can have squid check if the domain is on the whitelist and then skip the fast flux check.

That should be easy and have no overhead. If we are encountering too many items that cause trouble, we could make that whitelist editable for the user.

-Michael

> On 6 Sep 2021, at 17:35, Peter Müller <peter.mueller(a)ipfire.org> wrote:
> 
> Hello *,
> 
> by accident, I just stumbled across a false positive related to the Fast Flux detection:
> 
>> [root(a)maverick ~]# su squid -s /bin/bash
>> bash-5.1$ /usr/bin/asnbl-helper.py /var/ipfire/proxy/asnbl-helper.conf
>> Sep 06 18:28:21 squid-asnbl-helper[9945] WARN: No ASNBL configured. This is acceptable as long as this script is configured to do anything, you just have been warned...
>> Sep 06 18:28:21 squid-asnbl-helper[9945] INFO: Configuation sanity tests passed, good, processing...
>> Sep 06 18:28:21 squid-asnbl-helper[9945] INFO: Successfully loaded location database from /var/lib/location/database.db generated 'Mon Sep  6 05:52:56 2021' (UTC/GMT) by 'IPFire Project' - good
>> Sep 06 18:28:21 squid-asnbl-helper[9945] INFO: Running ASN database response tests...
>> Sep 06 18:28:21 squid-asnbl-helper[9945] INFO: ASN database operational - excellent. Waiting for input...
>> fedoraproject.org
>> Sep 06 18:28:26 squid-asnbl-helper[9945] WARN: Destination 'fedoraproject.org' exceeds ASN diversity threshold (9 > 5), possibly Fast Flux: [81, 3701, 15456, 16509, 21785, 22753, 36850, 54455, 61317]
>> Sep 06 18:28:26 squid-asnbl-helper[9945] INFO: Denying access to possible Fast Flux destination 'fedoraproject.org'
>> OK
> 
> Apparently, the Fedora folks think it is a good idea to use round-robin for load balancing:

It is indeed a very good idea to load-balance like this, but I do not know why they need to many locations for their website. This is a CDN gone mad.

>> $ dig +short a fedoraproject.org
>> 140.211.169.206
>> 67.219.144.68
>> 85.236.55.6
>> 38.145.60.20
>> 152.19.134.198
>> 209.132.190.2
>> 18.133.140.134
>> 18.185.136.17
>> 185.141.165.254
>> 152.19.134.142
>> 38.145.60.21
>> 18.159.254.57
> 
> At the first glance, using the URL filter (by adding fedoraproject.org to the list of always allowed
> domains) seems to be a straight-forward solution to this problem. However, it does not work, as the
> ASNBL script is executed in the context of an ACL, while the URL filter comes as a redirect/wrapper.
> Therefore, it is never reached if a "deny" ACL matches in the first place.
> 
> This is the only false positive I observed so far. Unfortunately, it is a rather bad one. :-/
> 
> Any thoughts on what to do now?
> 
> Thanks, and best regards,
> Peter Müller
> 
> 
>> Hello Michael,
>> 
>> thank you for your reply.
>> 
>>> Hello Peter,
>>> 
>>> I love this feature. I think it is a one-of-a-kind thing and hopefully many more people will think the same.
>> 
>> Yes, I like the idea, too. Sometimes, security can be simple _and_ effective... :-)
>> 
>>> However, it will need a lot of documentation and explaining.
>> 
>> Indeed. I was thinking about a blog post for it; we probably need to explain Fast Flux in the
>> first place, and I am not sure if all of our users are aware of the existence of autonomous
>> systems.
>> 
>>> I have a couple of high-level questions:
>>> 
>>> * Does it make sense to give the user the choice for the threshold?
>>> 
>>> It seems to be a difficult question because it requires exact knowledge what this feature actually does. My fears are that people just set this to something like “9” and the feature would become ineffective. What use-case is there to change this?
>> 
>> One size never fits all, I guess.
>> 
>> Indeed, the range of useful threshold values is pretty small: Anything below 4 causes _way_ too
>> much false positives in productive environment, whereas even 7 appears to be too ineffective.
>> 
>> At the moment, the CGI catches values the ASNBL helper would treat itself as being invalid. Do
>> you think narrowing down this range to 4 to 7 makes sense? Or should we replace it by a dropdown
>> for adjusting sensitivity?
>> 
>> Either way, it is a good idea to tell users to leave the default where it is unless they truly
>> understand what they are doing.
>> 
>>> * Selective announcements: Should this necessarily live in the proxy? Why do we not generate a filter for the firewall?
>> 
>> We can do so as well, and I would love to see such a feature landing in IPFire.
>> 
>> Given our current state of libloc, I doubt this is possible: We would need a function that returns
>> all networks we do not have an AS for - to my knowledge, the libloc (bindings) do not support this
>> at the moment.
>> 
>> Apart from that: On a packet filter level, we lack the FQDN of a destination, which might be useful
>> to have for debugging or forensic reasons.
>> 
>> Also, the users will experience a timeout after n seconds. Having selective announcement detection
>> turned on, they'll get their error message straight away. I was told this improves UX... :-)
>> 
>> Thanks, and best regards,
>> Peter Müller
>> 
>>> 
>>> -Michael
>>> 
>>>> On 18 Jun 2021, at 18:24, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>>> 
>>>> This patchset adds two new features to IPFire's web proxy, taking advantage
>>>> of the Autonomous System information we have at hand by using libloc.
>>>> 
>>>> The proactive Fast Flux detection is especially worth noticing, as even most
>>>> expensive (= advanced?) security suites do not provide similar protection,
>>>> especially not in a proactive manner.
>>>> 
>>>> By simply enumerating the distinct amount of Autonomous System Numbers a FQDN
>>>> ultimately resolves to, we are able to deny access to malware distribution
>>>> sites, phishing sites, C&C servers, and other cybercrime stuff hosted on Fast
>>>> Flux setups abusing cracked machines around the world - even before the FQDN
>>>> or any IP address involved is flagged as malicious by any security vendor.
>>>> 
>>>> Peter Müller (3):
>>>> squid-asnbl: New package
>>>> proxy.cgi: Implement proactive Fast Flux detection and detection for
>>>>   selectively announced destinations
>>>> langs: Add English and German translations for newly added web proxy
>>>>   features
>>>> 
>>>> config/rootfiles/common/squid-asnbl |  1 +
>>>> html/cgi-bin/proxy.cgi              | 89 +++++++++++++++++++++++++++++
>>>> langs/de/cgi-bin/de.pl              |  7 +++
>>>> langs/en/cgi-bin/en.pl              |  7 +++
>>>> lfs/squid-asnbl                     | 83 +++++++++++++++++++++++++++
>>>> make.sh                             |  1 +
>>>> 6 files changed, 188 insertions(+)
>>>> create mode 100644 config/rootfiles/common/squid-asnbl
>>>> create mode 100644 lfs/squid-asnbl
>>>> 
>>>> -- 
>>>> 2.26.2
>>>