[PATCH 0/3] Add ASN-based anomaly detections to IPFire's web proxy: Proactive Fast Flux detection and detection for selectively announced networks
This patchset adds two new features to IPFire's web proxy, taking advantage of the Autonomous System information we have at hand by using libloc.
The proactive Fast Flux detection is especially worth noticing, as even most expensive (= advanced?) security suites do not provide similar protection, especially not in a proactive manner.
By simply enumerating the distinct amount of Autonomous System Numbers a FQDN ultimately resolves to, we are able to deny access to malware distribution sites, phishing sites, C&C servers, and other cybercrime stuff hosted on Fast Flux setups abusing cracked machines around the world - even before the FQDN or any IP address involved is flagged as malicious by any security vendor.
Peter Müller (3): squid-asnbl: New package proxy.cgi: Implement proactive Fast Flux detection and detection for selectively announced destinations langs: Add English and German translations for newly added web proxy features
config/rootfiles/common/squid-asnbl | 1 + html/cgi-bin/proxy.cgi | 89 +++++++++++++++++++++++++++++ langs/de/cgi-bin/de.pl | 7 +++ langs/en/cgi-bin/en.pl | 7 +++ lfs/squid-asnbl | 83 +++++++++++++++++++++++++++ make.sh | 1 + 6 files changed, 188 insertions(+) create mode 100644 config/rootfiles/common/squid-asnbl create mode 100644 lfs/squid-asnbl
This package adds an ASNBL helper for detecting Fast Flux setups and selectively announced networks (i. e. FQDNs resolving to IP addresses not being announced by an Autonomous System) to the distribution. Afterwards, the helper script is located at /usr/bin/asnbl-helper.py .
Signed-off-by: Peter Müller peter.mueller@ipfire.org --- config/rootfiles/common/squid-asnbl | 1 + lfs/squid-asnbl | 83 +++++++++++++++++++++++++++++ make.sh | 1 + 3 files changed, 85 insertions(+) create mode 100644 config/rootfiles/common/squid-asnbl create mode 100644 lfs/squid-asnbl
diff --git a/config/rootfiles/common/squid-asnbl b/config/rootfiles/common/squid-asnbl new file mode 100644 index 000000000..f129f441e --- /dev/null +++ b/config/rootfiles/common/squid-asnbl @@ -0,0 +1 @@ +usr/bin/asnbl-helper.py diff --git a/lfs/squid-asnbl b/lfs/squid-asnbl new file mode 100644 index 000000000..f2396074d --- /dev/null +++ b/lfs/squid-asnbl @@ -0,0 +1,83 @@ +############################################################################### +# # +# IPFire.org - A linux based firewall # +# Copyright (C) 2007-2021 IPFire Team info@ipfire.org # +# # +# This program is free software: you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation, either version 3 of the License, or # +# (at your option) any later version. # +# # +# This program is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with this program. If not, see http://www.gnu.org/licenses/. # +# # +############################################################################### + + +############################################################################### +# Definitions +############################################################################### + +include Config + +VER = 0.2.1 + +THISAPP = squid-asnbl-$(VER) +DL_FILE = $(THISAPP).tar.gz +DL_FROM = $(URL_IPFIRE) + +DIR_APP = $(DIR_SRC)/$(THISAPP) + +TARGET = $(DIR_INFO)/$(THISAPP) + +DEPS = libloc squid python3 + +############################################################################### +# Top-level Rules +############################################################################### + +objects = $(DL_FILE) + +$(DL_FILE) = $(DL_FROM)/$(DL_FILE) + +$(DL_FILE)_MD5 = 2225c88ba8e3ae25f5e5c8075f0e7ae8 + +install : $(TARGET) + +check : $(patsubst %,$(DIR_CHK)/%,$(objects)) + +download :$(patsubst %,$(DIR_DL)/%,$(objects)) + +md5 : $(subst %,%_MD5,$(objects)) + +############################################################################### +# Downloading, checking, md5sum +############################################################################### + +$(patsubst %,$(DIR_CHK)/%,$(objects)) : + @$(CHECK) + +$(patsubst %,$(DIR_DL)/%,$(objects)) : + @$(LOAD) + +$(subst %,%_MD5,$(objects)) : + @$(MD5) + +############################################################################### +# Installation Details +############################################################################### + +$(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) + @$(PREBUILD) + @rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar zvxf $(DIR_DL)/$(DL_FILE) + + # Install ASNBL helper script + cd $(DIR_APP) && install -o root -g root -m 0755 asnbl-helper.py /usr/bin/asnbl-helper.py + + @rm -rf $(DIR_APP) + @$(POSTBUILD) diff --git a/make.sh b/make.sh index fc03ebcd5..19567769a 100755 --- a/make.sh +++ b/make.sh @@ -1623,6 +1623,7 @@ buildipfire() { lfsmake2 socat lfsmake2 libcdada lfsmake2 pmacct + lfsmake2 squid-asnbl }
buildinstaller() {
This patch adds two new features to IPFire's web proxy:
(a) Proactive Fast Flux detection FQDNs are resolved to their IP addresses, which are then resolved to corresponding Autonomous System Numbers using IPFire's location database. Most destinations will scatter across a very low number of ASNs (not to be confused with IP addresses!). FQDNs hosted on Fast Flux setups have a significantly higher ASN diversity (5 is usually a good threshold), so they can be proactively detected.
(b) Detection for selectively announced destinations Especially in targeted operations, miscreants host FQDNs for exfiltrating data or malware distributions on ASNs not announced globally, but only to the intended victim or it's upstream ISPs.
That way, security researchers located in other parts of the internet have no insights into these attacks, hence not being able to publish listings or send take down notices for the domains used.
While RPKI made this attack harder, it can still be observed every now and then.
This feature also protects against accessing FQDNs resolving to IP addresses not being globally routeable, hence providing a trivial mitigation for so-called "rebound attacks" - which we cannot filter at DNS level currently.
Signed-off-by: Peter Müller peter.mueller@ipfire.org --- html/cgi-bin/proxy.cgi | 89 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+)
diff --git a/html/cgi-bin/proxy.cgi b/html/cgi-bin/proxy.cgi index 78ad33ad2..b7227deaf 100644 --- a/html/cgi-bin/proxy.cgi +++ b/html/cgi-bin/proxy.cgi @@ -21,6 +21,7 @@
use strict; use Apache::Htpasswd; +use Scalar::Util qw(looks_like_number);
# enable only the following on debugging purpose #use warnings; @@ -225,6 +226,9 @@ $proxysettings{'THROTTLING_GREEN_TOTAL'} = 'unlimited'; $proxysettings{'THROTTLING_GREEN_HOST'} = 'unlimited'; $proxysettings{'THROTTLING_BLUE_TOTAL'} = 'unlimited'; $proxysettings{'THROTTLING_BLUE_HOST'} = 'unlimited'; +$proxysettings{'ASNBL_FASTFLUX_DETECTION'} = 'off'; +$proxysettings{'ASNBL_FASTFLUX_THRESHOLD'} = '5'; +$proxysettings{'ASNBL_SELECANN_DETECTION'} = 'off'; $proxysettings{'ENABLE_MIME_FILTER'} = 'off'; $proxysettings{'AUTH_METHOD'} = 'none'; $proxysettings{'AUTH_REALM'} = ''; @@ -414,6 +418,21 @@ if (($proxysettings{'ACTION'} eq $Lang::tr{'save'}) || ($proxysettings{'ACTION'} $errormessage = $Lang::tr{'invalid maximum incoming size'}; goto ERROR; } + if (($proxysettings{'ASNBL_FASTFLUX_DETECTION'} eq 'on') || ($proxysettings{'ASNBL_SELECANN_DETECTION'} eq 'on')) + { + if (-z $proxysettings{'ASNBL_FASTFLUX_THRESHOLD'}) { + $errormessage = $Lang::tr{'advproxy fastflux no threshold given'}; + goto ERROR; + } + if (! looks_like_number($proxysettings{'ASNBL_FASTFLUX_THRESHOLD'})) { + $errormessage = $Lang::tr{'advproxy fastflux threshold invalid'}; + goto ERROR; + } + if (($proxysettings{'ASNBL_FASTFLUX_THRESHOLD'} < 2) || ($proxysettings{'ASNBL_FASTFLUX_THRESHOLD'} > 10)) { + $errormessage = $Lang::tr{'advproxy fastflux threshold out of bounds'}; + goto ERROR; + } + } if (!($proxysettings{'AUTH_METHOD'} eq 'none')) { unless (($proxysettings{'AUTH_METHOD'} eq 'ident') && @@ -797,6 +816,14 @@ $selected{'THROTTLING_GREEN_HOST'}{$proxysettings{'THROTTLING_GREEN_HOST'}} = "s $selected{'THROTTLING_BLUE_TOTAL'}{$proxysettings{'THROTTLING_BLUE_TOTAL'}} = "selected='selected'"; $selected{'THROTTLING_BLUE_HOST'}{$proxysettings{'THROTTLING_BLUE_HOST'}} = "selected='selected'";
+$checked{'ASNBL_FASTFLUX_DETECTION'}{'off'} = ''; +$checked{'ASNBL_FASTFLUX_DETECTION'}{'on'} = ''; +$checked{'ASNBL_FASTFLUX_DETECTION'}{$proxysettings{'ASNBL_FASTFLUX_DETECTION'}} = "checked='checked'"; + +$checked{'ASNBL_SELECANN_DETECTION'}{'off'} = ''; +$checked{'ASNBL_SELECANN_DETECTION'}{'on'} = ''; +$checked{'ASNBL_SELECANN_DETECTION'}{$proxysettings{'ASNBL_SELECANN_DETECTION'}} = "checked='checked'"; + $checked{'ENABLE_MIME_FILTER'}{'off'} = ''; $checked{'ENABLE_MIME_FILTER'}{'on'} = ''; $checked{'ENABLE_MIME_FILTER'}{$proxysettings{'ENABLE_MIME_FILTER'}} = "checked='checked'"; @@ -1627,6 +1654,24 @@ END print <<END </table>
+<hr size='1'> + +<table width='100%'> +<tr> + <td><b>$Lang::tr{'advproxy asbased anomaly detection'}</b></td> +</tr> +<tr> + <td class='base'>$Lang::tr{'advproxy fastflux detection'}:</td> + <td><input type='checkbox' name='ASNBL_FASTFLUX_DETECTION' $checked{'ASNBL_FASTFLUX_DETECTION'}{'on'} /></td> + <td class='base'>$Lang::tr{'advproxy fastflux detection threshold'}:</td> + <td><input type='text' name='ASNBL_FASTFLUX_THRESHOLD' value='$proxysettings{'ASNBL_FASTFLUX_THRESHOLD'}' size=2 /></td> +</tr> +<tr> + <td class='base'>$Lang::tr{'advproxy selectively announcements detection'}:</td> + <td colspan='3'><input type='checkbox' name='ASNBL_SELECANN_DETECTION' $checked{'ASNBL_SELECANN_DETECTION'}{'on'} /></td> +</tr> +</table> + <hr size='1'> END ; @@ -3507,6 +3552,50 @@ if (@ssl_ports) { print FILE "http_access deny CONNECT !SSL_ports\n"; }
+ if ((($proxysettings{'ASNBL_FASTFLUX_DETECTION'} eq 'on') && (!-z $proxysettings{'ASNBL_FASTFLUX_THRESHOLD'})) || ($proxysettings{'ASNBL_SELECANN_DETECTION'} eq 'on')) { + print FILE "external_acl_type asnblhelper children-max=10 children-startup=2 ttl=86400 %DST /usr/bin/asnbl-helper.py /var/ipfire/proxy/asnbl-helper.conf\n"; + print FILE "acl asnbl external asnblhelper\n"; + print FILE "http_access deny asnbl\n\n"; + + # Write ASNBL helper configuration file... + open(ASNBLFILE, ">${General::swroot}/proxy/asnbl-helper.conf"); + flock(ASNBLFILE, 2); + + print ASNBLFILE<<END +# +# This file has been automatically generated. Manual changes will be overwritten. +# + +[GENERAL] +LOGLEVEL = INFO +ASNDB_PATH = /var/lib/location/database.db +USE_REPLYMAP = no +END +; + + print ASNBLFILE "AS_DIVERSITY_THRESHOLD = $proxysettings{'ASNBL_FASTFLUX_THRESHOLD'}\n"; + + if ($proxysettings{'ASNBL_SELECANN_DETECTION'} eq 'on') { + print ASNBLFILE "BLOCK_SUSPECTED_SELECTIVE_ANNOUNCEMENTS = yes\n"; + } else { + print ASNBLFILE "BLOCK_SUSPECTED_SELECTIVE_ANNOUNCEMENTS = no\n"; + } + + if ($proxysettings{'ASNBL_FASTFLUX_DETECTION'} eq 'on') { + print ASNBLFILE "BLOCK_DIVERSITY_EXCEEDING_DESTINATIONS = yes\n"; + } else { + print ASNBLFILE "BLOCK_DIVERSITY_EXCEEDING_DESTINATIONS = no\n"; + } + + print ASNBLFILE<<END +TESTDATA = (1.1.1.1, 13335) (8.8.8.8, 15169) (194.95.245.140, 680) (10.0.0.1, 0) (127.0.0.1, 0) (2001:638:d:c102::140, 680) (2606:4700:10::6814:d673, 13335) (fe80::1, 0) +ACTIVE_ASNBLS = +END +; + + close ASNBLFILE; + } + if ($proxysettings{'AUTH_METHOD'} eq 'ident') { print FILE "#Set ident ACLs\n";
Signed-off-by: Peter Müller peter.mueller@ipfire.org --- langs/de/cgi-bin/de.pl | 7 +++++++ langs/en/cgi-bin/en.pl | 7 +++++++ 2 files changed, 14 insertions(+)
diff --git a/langs/de/cgi-bin/de.pl b/langs/de/cgi-bin/de.pl index 0d2228ede..642ff53de 100644 --- a/langs/de/cgi-bin/de.pl +++ b/langs/de/cgi-bin/de.pl @@ -255,6 +255,7 @@ 'advproxy advanced web proxy configuration' => 'Proxy-Konfiguration', 'advproxy allowed subnets' => 'Erlaubte Subnetze (eins pro Zeile)', 'advproxy allowed web browsers' => 'Zulässige Clients für Web-Zugriffe', +'advproxy asbased anomaly detection' => 'Anomalieerkennungen auf Basis Autonomer Systeme', 'advproxy back to main page' => 'Zurück zur Hauptseite', 'advproxy banned ip clients' => 'Gesperrte IP-Adressen (eine pro Zeile)', 'advproxy banned mac clients' => 'Gesperrte MAC-Adressen (eine pro Zeile)', @@ -326,6 +327,11 @@ 'advproxy errmsg wpad invalid ip or mask' => 'WPAD: Ungültige IP oder Subnetz für ausgenommenes IP-Subnetz', 'advproxy error design' => 'Design der Fehlermeldungen', 'advproxy error language' => 'Sprache der Fehlermeldungen', +'advproxy fastflux detection' => 'Verbindungen zu auf Fast Flux-Setups gehosteten Zielen verweigern', +'advproxy fastflux no threshold given' => 'Kein Schwellwert für Fast Flux-Erkennung angegeben', +'advproxy fastflux detection threshold' => 'Schwellwert', +'advproxy fastflux threshold invalid' => 'Eingegebener Schwellwert für Fast Flux-Erkennung ist ungültig', +'advproxy fastflux threshold out of bounds' => 'Eingegebener Schwellwert für Fast Flux-Erkennung befindet sich außerhalb zulässiger Grenzwerte', 'advproxy friday' => 'Fr', 'advproxy from' => 'Von', 'advproxy group access control' => 'Gruppenbasierte Zugriffskontrolle', @@ -362,6 +368,7 @@ 'advproxy reset' => 'Zurücksetzen', 'advproxy saturday' => 'Sa', 'advproxy save and restart' => 'Speichern und Neustart', +'advproxy selectively announcements detection' => 'Verbindungen zu Zielen in selektiv propagierten Netzen verweigern', 'advproxy squid version' => 'Squid Versionsnummer', 'advproxy squidclamav' => 'SquidClamav', 'advproxy ssadvanced proxy' => 'advanced proxy', diff --git a/langs/en/cgi-bin/en.pl b/langs/en/cgi-bin/en.pl index 2ba6961f3..1f82c47ca 100644 --- a/langs/en/cgi-bin/en.pl +++ b/langs/en/cgi-bin/en.pl @@ -252,6 +252,7 @@ 'advproxy advanced web proxy configuration' => 'Advanced web proxy configuration', 'advproxy allowed subnets' => 'Allowed subnets (one per line)', 'advproxy allowed web browsers' => 'Allowed clients for web access', +'advproxy asbased anomaly detection' => 'Anomaly detections based on Autonomous Systems information', 'advproxy back to main page' => 'Back to main page', 'advproxy banned ip clients' => 'Banned IP addresses (one per line)', 'advproxy banned mac clients' => 'Banned MAC addresses (one per line)', @@ -323,6 +324,11 @@ 'advproxy errmsg wpad invalid ip or mask' => 'WPAD: Invalid IP or subnet for excluded IP subnet', 'advproxy error design' => 'Error messages design', 'advproxy error language' => 'Error messages language', +'advproxy fastflux detection' => 'Deny access to destinations hosted on fast flux setups', +'advproxy fastflux no threshold given' => 'No threshold was given for fast flux detection', +'advproxy fastflux detection threshold' => 'Threshold', +'advproxy fastflux threshold invalid' => 'Supplied fast flux detection threshold is invalid', +'advproxy fastflux threshold out of bounds' => 'Supplied fast flux detection threshold is out of bounds', 'advproxy friday' => 'Fri', 'advproxy from' => 'From', 'advproxy group access control' => 'Group based access control', @@ -359,6 +365,7 @@ 'advproxy reset' => 'Reset', 'advproxy saturday' => 'Sat', 'advproxy save and restart' => 'Save and Restart', +'advproxy selectively announcements detection' => 'Deny access to destinations hosted on selectively announced networks', 'advproxy squid version' => 'Squid cache version', 'advproxy squidclamav' => 'SquidClamav', 'advproxy ssadvanced proxy' => 'advanced proxy',
Hello Peter,
I love this feature. I think it is a one-of-a-kind thing and hopefully many more people will think the same.
However, it will need a lot of documentation and explaining.
I have a couple of high-level questions:
* Does it make sense to give the user the choice for the threshold?
It seems to be a difficult question because it requires exact knowledge what this feature actually does. My fears are that people just set this to something like “9” and the feature would become ineffective. What use-case is there to change this?
* Selective announcements: Should this necessarily live in the proxy? Why do we not generate a filter for the firewall?
-Michael
On 18 Jun 2021, at 18:24, Peter Müller peter.mueller@ipfire.org wrote:
This patchset adds two new features to IPFire's web proxy, taking advantage of the Autonomous System information we have at hand by using libloc.
The proactive Fast Flux detection is especially worth noticing, as even most expensive (= advanced?) security suites do not provide similar protection, especially not in a proactive manner.
By simply enumerating the distinct amount of Autonomous System Numbers a FQDN ultimately resolves to, we are able to deny access to malware distribution sites, phishing sites, C&C servers, and other cybercrime stuff hosted on Fast Flux setups abusing cracked machines around the world - even before the FQDN or any IP address involved is flagged as malicious by any security vendor.
Peter Müller (3): squid-asnbl: New package proxy.cgi: Implement proactive Fast Flux detection and detection for selectively announced destinations langs: Add English and German translations for newly added web proxy features
config/rootfiles/common/squid-asnbl | 1 + html/cgi-bin/proxy.cgi | 89 +++++++++++++++++++++++++++++ langs/de/cgi-bin/de.pl | 7 +++ langs/en/cgi-bin/en.pl | 7 +++ lfs/squid-asnbl | 83 +++++++++++++++++++++++++++ make.sh | 1 + 6 files changed, 188 insertions(+) create mode 100644 config/rootfiles/common/squid-asnbl create mode 100644 lfs/squid-asnbl
-- 2.26.2
Hello,
On 18 Jun 2021, at 18:24, Peter Müller peter.mueller@ipfire.org wrote:
This patch adds two new features to IPFire's web proxy:
(a) Proactive Fast Flux detection FQDNs are resolved to their IP addresses, which are then resolved to corresponding Autonomous System Numbers using IPFire's location database. Most destinations will scatter across a very low number of ASNs (not to be confused with IP addresses!). FQDNs hosted on Fast Flux setups have a significantly higher ASN diversity (5 is usually a good threshold), so they can be proactively detected.
(b) Detection for selectively announced destinations Especially in targeted operations, miscreants host FQDNs for exfiltrating data or malware distributions on ASNs not announced globally, but only to the intended victim or it's upstream ISPs.
That way, security researchers located in other parts of the internet have no insights into these attacks, hence not being able to publish listings or send take down notices for the domains used.
While RPKI made this attack harder, it can still be observed every now and then.
This feature also protects against accessing FQDNs resolving to IP addresses not being globally routeable, hence providing a trivial mitigation for so-called "rebound attacks" - which we cannot filter at DNS level currently.
Signed-off-by: Peter Müller peter.mueller@ipfire.org
html/cgi-bin/proxy.cgi | 89 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+)
diff --git a/html/cgi-bin/proxy.cgi b/html/cgi-bin/proxy.cgi index 78ad33ad2..b7227deaf 100644 --- a/html/cgi-bin/proxy.cgi +++ b/html/cgi-bin/proxy.cgi @@ -21,6 +21,7 @@
use strict; use Apache::Htpasswd; +use Scalar::Util qw(looks_like_number);
# enable only the following on debugging purpose #use warnings; @@ -225,6 +226,9 @@ $proxysettings{'THROTTLING_GREEN_TOTAL'} = 'unlimited'; $proxysettings{'THROTTLING_GREEN_HOST'} = 'unlimited'; $proxysettings{'THROTTLING_BLUE_TOTAL'} = 'unlimited'; $proxysettings{'THROTTLING_BLUE_HOST'} = 'unlimited'; +$proxysettings{'ASNBL_FASTFLUX_DETECTION'} = 'off'; +$proxysettings{'ASNBL_FASTFLUX_THRESHOLD'} = '5'; +$proxysettings{'ASNBL_SELECANN_DETECTION'} = 'off'; $proxysettings{'ENABLE_MIME_FILTER'} = 'off'; $proxysettings{'AUTH_METHOD'} = 'none'; $proxysettings{'AUTH_REALM'} = ''; @@ -414,6 +418,21 @@ if (($proxysettings{'ACTION'} eq $Lang::tr{'save'}) || ($proxysettings{'ACTION'} $errormessage = $Lang::tr{'invalid maximum incoming size'}; goto ERROR; }
- if (($proxysettings{'ASNBL_FASTFLUX_DETECTION'} eq 'on') || ($proxysettings{'ASNBL_SELECANN_DETECTION'} eq 'on'))
- {
if (-z $proxysettings{'ASNBL_FASTFLUX_THRESHOLD'}) {$errormessage = $Lang::tr{'advproxy fastflux no threshold given'};goto ERROR;}if (! looks_like_number($proxysettings{'ASNBL_FASTFLUX_THRESHOLD'})) {$errormessage = $Lang::tr{'advproxy fastflux threshold invalid'};goto ERROR;}if (($proxysettings{'ASNBL_FASTFLUX_THRESHOLD'} < 2) || ($proxysettings{'ASNBL_FASTFLUX_THRESHOLD'} > 10)) {$errormessage = $Lang::tr{'advproxy fastflux threshold out of bounds'};goto ERROR;}- } if (!($proxysettings{'AUTH_METHOD'} eq 'none')) { unless (($proxysettings{'AUTH_METHOD'} eq 'ident') &&
@@ -797,6 +816,14 @@ $selected{'THROTTLING_GREEN_HOST'}{$proxysettings{'THROTTLING_GREEN_HOST'}} = "s $selected{'THROTTLING_BLUE_TOTAL'}{$proxysettings{'THROTTLING_BLUE_TOTAL'}} = "selected='selected'"; $selected{'THROTTLING_BLUE_HOST'}{$proxysettings{'THROTTLING_BLUE_HOST'}} = "selected='selected'";
+$checked{'ASNBL_FASTFLUX_DETECTION'}{'off'} = ''; +$checked{'ASNBL_FASTFLUX_DETECTION'}{'on'} = ''; +$checked{'ASNBL_FASTFLUX_DETECTION'}{$proxysettings{'ASNBL_FASTFLUX_DETECTION'}} = "checked='checked'";
+$checked{'ASNBL_SELECANN_DETECTION'}{'off'} = ''; +$checked{'ASNBL_SELECANN_DETECTION'}{'on'} = ''; +$checked{'ASNBL_SELECANN_DETECTION'}{$proxysettings{'ASNBL_SELECANN_DETECTION'}} = "checked='checked'";
$checked{'ENABLE_MIME_FILTER'}{'off'} = ''; $checked{'ENABLE_MIME_FILTER'}{'on'} = ''; $checked{'ENABLE_MIME_FILTER'}{$proxysettings{'ENABLE_MIME_FILTER'}} = "checked='checked'"; @@ -1627,6 +1654,24 @@ END print <<END
</table>
+<hr size='1'>
+<table width='100%'> +<tr>
<td><b>$Lang::tr{'advproxy asbased anomaly detection'}</b></td>+</tr> +<tr>
<td class='base'>$Lang::tr{'advproxy fastflux detection'}:</td><td><input type='checkbox' name='ASNBL_FASTFLUX_DETECTION' $checked{'ASNBL_FASTFLUX_DETECTION'}{'on'} /></td><td class='base'>$Lang::tr{'advproxy fastflux detection threshold'}:</td><td><input type='text' name='ASNBL_FASTFLUX_THRESHOLD' value='$proxysettings{'ASNBL_FASTFLUX_THRESHOLD'}' size=2 /></td>+</tr> +<tr>
<td class='base'>$Lang::tr{'advproxy selectively announcements detection'}:</td><td colspan='3'><input type='checkbox' name='ASNBL_SELECANN_DETECTION' $checked{'ASNBL_SELECANN_DETECTION'}{'on'} /></td>+</tr> +</table>
<hr size='1'> END ; @@ -3507,6 +3552,50 @@ if (@ssl_ports) { print FILE "http_access deny CONNECT !SSL_ports\n"; }
- if ((($proxysettings{'ASNBL_FASTFLUX_DETECTION'} eq 'on') && (!-z $proxysettings{'ASNBL_FASTFLUX_THRESHOLD'})) || ($proxysettings{'ASNBL_SELECANN_DETECTION'} eq 'on')) {
print FILE "external_acl_type asnblhelper children-max=10 children-startup=2 ttl=86400 %DST /usr/bin/asnbl-helper.py /var/ipfire/proxy/asnbl-helper.conf\n";print FILE "acl asnbl external asnblhelper\n";print FILE "http_access deny asnbl\n\n";# Write ASNBL helper configuration file...open(ASNBLFILE, ">${General::swroot}/proxy/asnbl-helper.conf");flock(ASNBLFILE, 2);print ASNBLFILE<<END+# +# This file has been automatically generated. Manual changes will be overwritten. +#
+[GENERAL] +LOGLEVEL = INFO +ASNDB_PATH = /var/lib/location/database.db +USE_REPLYMAP = no +END +;
print ASNBLFILE "AS_DIVERSITY_THRESHOLD = $proxysettings{'ASNBL_FASTFLUX_THRESHOLD'}\n";if ($proxysettings{'ASNBL_SELECANN_DETECTION'} eq 'on') {print ASNBLFILE "BLOCK_SUSPECTED_SELECTIVE_ANNOUNCEMENTS = yes\n";} else {print ASNBLFILE "BLOCK_SUSPECTED_SELECTIVE_ANNOUNCEMENTS = no\n";}if ($proxysettings{'ASNBL_FASTFLUX_DETECTION'} eq 'on') {print ASNBLFILE "BLOCK_DIVERSITY_EXCEEDING_DESTINATIONS = yes\n";} else {print ASNBLFILE "BLOCK_DIVERSITY_EXCEEDING_DESTINATIONS = no\n";}print ASNBLFILE<<END+TESTDATA = (1.1.1.1, 13335) (8.8.8.8, 15169) (194.95.245.140, 680) (10.0.0.1, 0) (127.0.0.1, 0) (2001:638:d:c102::140, 680) (2606:4700:10::6814:d673, 13335) (fe80::1, 0)
Why do we want to hard-code this here?
Does that not (if anywhere) belong into libloc? I disagree with hard-coding this, because what happens if Google moves their DNS server? It would break this feature.
-Michael
+ACTIVE_ASNBLS = +END +;
close ASNBLFILE;- }
if ($proxysettings{'AUTH_METHOD'} eq 'ident') { print FILE "#Set ident ACLs\n"; -- 2.26.2
Hello Michael,
thank you for your reply.
Hello Peter,
I love this feature. I think it is a one-of-a-kind thing and hopefully many more people will think the same.
Yes, I like the idea, too. Sometimes, security can be simple _and_ effective... :-)
However, it will need a lot of documentation and explaining.
Indeed. I was thinking about a blog post for it; we probably need to explain Fast Flux in the first place, and I am not sure if all of our users are aware of the existence of autonomous systems.
I have a couple of high-level questions:
- Does it make sense to give the user the choice for the threshold?
It seems to be a difficult question because it requires exact knowledge what this feature actually does. My fears are that people just set this to something like “9” and the feature would become ineffective. What use-case is there to change this?
One size never fits all, I guess.
Indeed, the range of useful threshold values is pretty small: Anything below 4 causes _way_ too much false positives in productive environment, whereas even 7 appears to be too ineffective.
At the moment, the CGI catches values the ASNBL helper would treat itself as being invalid. Do you think narrowing down this range to 4 to 7 makes sense? Or should we replace it by a dropdown for adjusting sensitivity?
Either way, it is a good idea to tell users to leave the default where it is unless they truly understand what they are doing.
- Selective announcements: Should this necessarily live in the proxy? Why do we not generate a filter for the firewall?
We can do so as well, and I would love to see such a feature landing in IPFire.
Given our current state of libloc, I doubt this is possible: We would need a function that returns all networks we do not have an AS for - to my knowledge, the libloc (bindings) do not support this at the moment.
Apart from that: On a packet filter level, we lack the FQDN of a destination, which might be useful to have for debugging or forensic reasons.
Also, the users will experience a timeout after n seconds. Having selective announcement detection turned on, they'll get their error message straight away. I was told this improves UX... :-)
Thanks, and best regards, Peter Müller
-Michael
On 18 Jun 2021, at 18:24, Peter Müller peter.mueller@ipfire.org wrote:
This patchset adds two new features to IPFire's web proxy, taking advantage of the Autonomous System information we have at hand by using libloc.
The proactive Fast Flux detection is especially worth noticing, as even most expensive (= advanced?) security suites do not provide similar protection, especially not in a proactive manner.
By simply enumerating the distinct amount of Autonomous System Numbers a FQDN ultimately resolves to, we are able to deny access to malware distribution sites, phishing sites, C&C servers, and other cybercrime stuff hosted on Fast Flux setups abusing cracked machines around the world - even before the FQDN or any IP address involved is flagged as malicious by any security vendor.
Peter Müller (3): squid-asnbl: New package proxy.cgi: Implement proactive Fast Flux detection and detection for selectively announced destinations langs: Add English and German translations for newly added web proxy features
config/rootfiles/common/squid-asnbl | 1 + html/cgi-bin/proxy.cgi | 89 +++++++++++++++++++++++++++++ langs/de/cgi-bin/de.pl | 7 +++ langs/en/cgi-bin/en.pl | 7 +++ lfs/squid-asnbl | 83 +++++++++++++++++++++++++++ make.sh | 1 + 6 files changed, 188 insertions(+) create mode 100644 config/rootfiles/common/squid-asnbl create mode 100644 lfs/squid-asnbl
-- 2.26.2
Hello Michael,
Hello,
On 18 Jun 2021, at 18:24, Peter Müller peter.mueller@ipfire.org wrote:
This patch adds two new features to IPFire's web proxy:
(a) Proactive Fast Flux detection FQDNs are resolved to their IP addresses, which are then resolved to corresponding Autonomous System Numbers using IPFire's location database. Most destinations will scatter across a very low number of ASNs (not to be confused with IP addresses!). FQDNs hosted on Fast Flux setups have a significantly higher ASN diversity (5 is usually a good threshold), so they can be proactively detected.
(b) Detection for selectively announced destinations Especially in targeted operations, miscreants host FQDNs for exfiltrating data or malware distributions on ASNs not announced globally, but only to the intended victim or it's upstream ISPs.
That way, security researchers located in other parts of the internet have no insights into these attacks, hence not being able to publish listings or send take down notices for the domains used.
While RPKI made this attack harder, it can still be observed every now and then.
This feature also protects against accessing FQDNs resolving to IP addresses not being globally routeable, hence providing a trivial mitigation for so-called "rebound attacks" - which we cannot filter at DNS level currently.
Signed-off-by: Peter Müller peter.mueller@ipfire.org
html/cgi-bin/proxy.cgi | 89 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+)
diff --git a/html/cgi-bin/proxy.cgi b/html/cgi-bin/proxy.cgi index 78ad33ad2..b7227deaf 100644 --- a/html/cgi-bin/proxy.cgi +++ b/html/cgi-bin/proxy.cgi @@ -21,6 +21,7 @@
use strict; use Apache::Htpasswd; +use Scalar::Util qw(looks_like_number);
# enable only the following on debugging purpose #use warnings; @@ -225,6 +226,9 @@ $proxysettings{'THROTTLING_GREEN_TOTAL'} = 'unlimited'; $proxysettings{'THROTTLING_GREEN_HOST'} = 'unlimited'; $proxysettings{'THROTTLING_BLUE_TOTAL'} = 'unlimited'; $proxysettings{'THROTTLING_BLUE_HOST'} = 'unlimited'; +$proxysettings{'ASNBL_FASTFLUX_DETECTION'} = 'off'; +$proxysettings{'ASNBL_FASTFLUX_THRESHOLD'} = '5'; +$proxysettings{'ASNBL_SELECANN_DETECTION'} = 'off'; $proxysettings{'ENABLE_MIME_FILTER'} = 'off'; $proxysettings{'AUTH_METHOD'} = 'none'; $proxysettings{'AUTH_REALM'} = ''; @@ -414,6 +418,21 @@ if (($proxysettings{'ACTION'} eq $Lang::tr{'save'}) || ($proxysettings{'ACTION'} $errormessage = $Lang::tr{'invalid maximum incoming size'}; goto ERROR; }
- if (($proxysettings{'ASNBL_FASTFLUX_DETECTION'} eq 'on') || ($proxysettings{'ASNBL_SELECANN_DETECTION'} eq 'on'))
- {
if (-z $proxysettings{'ASNBL_FASTFLUX_THRESHOLD'}) {$errormessage = $Lang::tr{'advproxy fastflux no threshold given'};goto ERROR;}if (! looks_like_number($proxysettings{'ASNBL_FASTFLUX_THRESHOLD'})) {$errormessage = $Lang::tr{'advproxy fastflux threshold invalid'};goto ERROR;}if (($proxysettings{'ASNBL_FASTFLUX_THRESHOLD'} < 2) || ($proxysettings{'ASNBL_FASTFLUX_THRESHOLD'} > 10)) {$errormessage = $Lang::tr{'advproxy fastflux threshold out of bounds'};goto ERROR;}- } if (!($proxysettings{'AUTH_METHOD'} eq 'none')) { unless (($proxysettings{'AUTH_METHOD'} eq 'ident') &&
@@ -797,6 +816,14 @@ $selected{'THROTTLING_GREEN_HOST'}{$proxysettings{'THROTTLING_GREEN_HOST'}} = "s $selected{'THROTTLING_BLUE_TOTAL'}{$proxysettings{'THROTTLING_BLUE_TOTAL'}} = "selected='selected'"; $selected{'THROTTLING_BLUE_HOST'}{$proxysettings{'THROTTLING_BLUE_HOST'}} = "selected='selected'";
+$checked{'ASNBL_FASTFLUX_DETECTION'}{'off'} = ''; +$checked{'ASNBL_FASTFLUX_DETECTION'}{'on'} = ''; +$checked{'ASNBL_FASTFLUX_DETECTION'}{$proxysettings{'ASNBL_FASTFLUX_DETECTION'}} = "checked='checked'";
+$checked{'ASNBL_SELECANN_DETECTION'}{'off'} = ''; +$checked{'ASNBL_SELECANN_DETECTION'}{'on'} = ''; +$checked{'ASNBL_SELECANN_DETECTION'}{$proxysettings{'ASNBL_SELECANN_DETECTION'}} = "checked='checked'";
$checked{'ENABLE_MIME_FILTER'}{'off'} = ''; $checked{'ENABLE_MIME_FILTER'}{'on'} = ''; $checked{'ENABLE_MIME_FILTER'}{$proxysettings{'ENABLE_MIME_FILTER'}} = "checked='checked'"; @@ -1627,6 +1654,24 @@ END print <<END
</table>
+<hr size='1'>
+<table width='100%'> +<tr>
<td><b>$Lang::tr{'advproxy asbased anomaly detection'}</b></td>+</tr> +<tr>
<td class='base'>$Lang::tr{'advproxy fastflux detection'}:</td><td><input type='checkbox' name='ASNBL_FASTFLUX_DETECTION' $checked{'ASNBL_FASTFLUX_DETECTION'}{'on'} /></td><td class='base'>$Lang::tr{'advproxy fastflux detection threshold'}:</td><td><input type='text' name='ASNBL_FASTFLUX_THRESHOLD' value='$proxysettings{'ASNBL_FASTFLUX_THRESHOLD'}' size=2 /></td>+</tr> +<tr>
<td class='base'>$Lang::tr{'advproxy selectively announcements detection'}:</td><td colspan='3'><input type='checkbox' name='ASNBL_SELECANN_DETECTION' $checked{'ASNBL_SELECANN_DETECTION'}{'on'} /></td>+</tr> +</table>
<hr size='1'> END ; @@ -3507,6 +3552,50 @@ if (@ssl_ports) { print FILE "http_access deny CONNECT !SSL_ports\n"; }
- if ((($proxysettings{'ASNBL_FASTFLUX_DETECTION'} eq 'on') && (!-z $proxysettings{'ASNBL_FASTFLUX_THRESHOLD'})) || ($proxysettings{'ASNBL_SELECANN_DETECTION'} eq 'on')) {
print FILE "external_acl_type asnblhelper children-max=10 children-startup=2 ttl=86400 %DST /usr/bin/asnbl-helper.py /var/ipfire/proxy/asnbl-helper.conf\n";print FILE "acl asnbl external asnblhelper\n";print FILE "http_access deny asnbl\n\n";# Write ASNBL helper configuration file...open(ASNBLFILE, ">${General::swroot}/proxy/asnbl-helper.conf");flock(ASNBLFILE, 2);print ASNBLFILE<<END+# +# This file has been automatically generated. Manual changes will be overwritten. +#
+[GENERAL] +LOGLEVEL = INFO +ASNDB_PATH = /var/lib/location/database.db +USE_REPLYMAP = no +END +;
print ASNBLFILE "AS_DIVERSITY_THRESHOLD = $proxysettings{'ASNBL_FASTFLUX_THRESHOLD'}\n";if ($proxysettings{'ASNBL_SELECANN_DETECTION'} eq 'on') {print ASNBLFILE "BLOCK_SUSPECTED_SELECTIVE_ANNOUNCEMENTS = yes\n";} else {print ASNBLFILE "BLOCK_SUSPECTED_SELECTIVE_ANNOUNCEMENTS = no\n";}if ($proxysettings{'ASNBL_FASTFLUX_DETECTION'} eq 'on') {print ASNBLFILE "BLOCK_DIVERSITY_EXCEEDING_DESTINATIONS = yes\n";} else {print ASNBLFILE "BLOCK_DIVERSITY_EXCEEDING_DESTINATIONS = no\n";}print ASNBLFILE<<END+TESTDATA = (1.1.1.1, 13335) (8.8.8.8, 15169) (194.95.245.140, 680) (10.0.0.1, 0) (127.0.0.1, 0) (2001:638:d:c102::140, 680) (2606:4700:10::6814:d673, 13335) (fe80::1, 0)
Why do we want to hard-code this here?
Because the ASNBL helper requires some test points to ensure the location database provided is actually working. It is designed to prefer "fail close" (i. e. stopping operation in case something is rendering its purpose useless) rather than "fail open" (i. e. logging a warning and proceed anyway, even if the location database returns "0" for any given IP address).
Does that not (if anywhere) belong into libloc? I disagree with hard-coding this, because what happens if Google moves their DNS server? It would break this feature.
Partly. We should have some sanity checks in libloc as well, but they ultimately boil down to the same problem: We need a set of relatively statically propagated IP addresses, which the internet does not have by nature.
Do you have a better proposal for testing points? Like the IP address of our own NTP server, which we hard-coded somewhere else, too?
Thanks, and best regards, Peter Müller
-Michael
+ACTIVE_ASNBLS = +END +;
close ASNBLFILE;- }
if ($proxysettings{'AUTH_METHOD'} eq 'ident') { print FILE "#Set ident ACLs\n"; -- 2.26.2
Hello *,
by accident, I just stumbled across a false positive related to the Fast Flux detection:
[root@maverick ~]# su squid -s /bin/bash bash-5.1$ /usr/bin/asnbl-helper.py /var/ipfire/proxy/asnbl-helper.conf Sep 06 18:28:21 squid-asnbl-helper[9945] WARN: No ASNBL configured. This is acceptable as long as this script is configured to do anything, you just have been warned... Sep 06 18:28:21 squid-asnbl-helper[9945] INFO: Configuation sanity tests passed, good, processing... Sep 06 18:28:21 squid-asnbl-helper[9945] INFO: Successfully loaded location database from /var/lib/location/database.db generated 'Mon Sep 6 05:52:56 2021' (UTC/GMT) by 'IPFire Project' - good Sep 06 18:28:21 squid-asnbl-helper[9945] INFO: Running ASN database response tests... Sep 06 18:28:21 squid-asnbl-helper[9945] INFO: ASN database operational - excellent. Waiting for input... fedoraproject.org Sep 06 18:28:26 squid-asnbl-helper[9945] WARN: Destination 'fedoraproject.org' exceeds ASN diversity threshold (9 > 5), possibly Fast Flux: [81, 3701, 15456, 16509, 21785, 22753, 36850, 54455, 61317] Sep 06 18:28:26 squid-asnbl-helper[9945] INFO: Denying access to possible Fast Flux destination 'fedoraproject.org' OK
Apparently, the Fedora folks think it is a good idea to use round-robin for load balancing:
$ dig +short a fedoraproject.org 140.211.169.206 67.219.144.68 85.236.55.6 38.145.60.20 152.19.134.198 209.132.190.2 18.133.140.134 18.185.136.17 185.141.165.254 152.19.134.142 38.145.60.21 18.159.254.57
At the first glance, using the URL filter (by adding fedoraproject.org to the list of always allowed domains) seems to be a straight-forward solution to this problem. However, it does not work, as the ASNBL script is executed in the context of an ACL, while the URL filter comes as a redirect/wrapper. Therefore, it is never reached if a "deny" ACL matches in the first place.
This is the only false positive I observed so far. Unfortunately, it is a rather bad one. :-/
Any thoughts on what to do now?
Thanks, and best regards, Peter Müller
Hello Michael,
thank you for your reply.
Hello Peter,
I love this feature. I think it is a one-of-a-kind thing and hopefully many more people will think the same.
Yes, I like the idea, too. Sometimes, security can be simple _and_ effective... :-)
However, it will need a lot of documentation and explaining.
Indeed. I was thinking about a blog post for it; we probably need to explain Fast Flux in the first place, and I am not sure if all of our users are aware of the existence of autonomous systems.
I have a couple of high-level questions:
- Does it make sense to give the user the choice for the threshold?
It seems to be a difficult question because it requires exact knowledge what this feature actually does. My fears are that people just set this to something like “9” and the feature would become ineffective. What use-case is there to change this?
One size never fits all, I guess.
Indeed, the range of useful threshold values is pretty small: Anything below 4 causes _way_ too much false positives in productive environment, whereas even 7 appears to be too ineffective.
At the moment, the CGI catches values the ASNBL helper would treat itself as being invalid. Do you think narrowing down this range to 4 to 7 makes sense? Or should we replace it by a dropdown for adjusting sensitivity?
Either way, it is a good idea to tell users to leave the default where it is unless they truly understand what they are doing.
- Selective announcements: Should this necessarily live in the proxy? Why do we not generate a filter for the firewall?
We can do so as well, and I would love to see such a feature landing in IPFire.
Given our current state of libloc, I doubt this is possible: We would need a function that returns all networks we do not have an AS for - to my knowledge, the libloc (bindings) do not support this at the moment.
Apart from that: On a packet filter level, we lack the FQDN of a destination, which might be useful to have for debugging or forensic reasons.
Also, the users will experience a timeout after n seconds. Having selective announcement detection turned on, they'll get their error message straight away. I was told this improves UX... :-)
Thanks, and best regards, Peter Müller
-Michael
On 18 Jun 2021, at 18:24, Peter Müller peter.mueller@ipfire.org wrote:
This patchset adds two new features to IPFire's web proxy, taking advantage of the Autonomous System information we have at hand by using libloc.
The proactive Fast Flux detection is especially worth noticing, as even most expensive (= advanced?) security suites do not provide similar protection, especially not in a proactive manner.
By simply enumerating the distinct amount of Autonomous System Numbers a FQDN ultimately resolves to, we are able to deny access to malware distribution sites, phishing sites, C&C servers, and other cybercrime stuff hosted on Fast Flux setups abusing cracked machines around the world - even before the FQDN or any IP address involved is flagged as malicious by any security vendor.
Peter Müller (3): squid-asnbl: New package proxy.cgi: Implement proactive Fast Flux detection and detection for selectively announced destinations langs: Add English and German translations for newly added web proxy features
config/rootfiles/common/squid-asnbl | 1 + html/cgi-bin/proxy.cgi | 89 +++++++++++++++++++++++++++++ langs/de/cgi-bin/de.pl | 7 +++ langs/en/cgi-bin/en.pl | 7 +++ lfs/squid-asnbl | 83 +++++++++++++++++++++++++++ make.sh | 1 + 6 files changed, 188 insertions(+) create mode 100644 config/rootfiles/common/squid-asnbl create mode 100644 lfs/squid-asnbl
-- 2.26.2
Hello,
This is bad news indeed.
How about we have a whitelist that we ship with this?
If you are using ACLs, you can have squid check if the domain is on the whitelist and then skip the fast flux check.
That should be easy and have no overhead. If we are encountering too many items that cause trouble, we could make that whitelist editable for the user.
-Michael
On 6 Sep 2021, at 17:35, Peter Müller peter.mueller@ipfire.org wrote:
Hello *,
by accident, I just stumbled across a false positive related to the Fast Flux detection:
[root@maverick ~]# su squid -s /bin/bash bash-5.1$ /usr/bin/asnbl-helper.py /var/ipfire/proxy/asnbl-helper.conf Sep 06 18:28:21 squid-asnbl-helper[9945] WARN: No ASNBL configured. This is acceptable as long as this script is configured to do anything, you just have been warned... Sep 06 18:28:21 squid-asnbl-helper[9945] INFO: Configuation sanity tests passed, good, processing... Sep 06 18:28:21 squid-asnbl-helper[9945] INFO: Successfully loaded location database from /var/lib/location/database.db generated 'Mon Sep 6 05:52:56 2021' (UTC/GMT) by 'IPFire Project' - good Sep 06 18:28:21 squid-asnbl-helper[9945] INFO: Running ASN database response tests... Sep 06 18:28:21 squid-asnbl-helper[9945] INFO: ASN database operational - excellent. Waiting for input... fedoraproject.org Sep 06 18:28:26 squid-asnbl-helper[9945] WARN: Destination 'fedoraproject.org' exceeds ASN diversity threshold (9 > 5), possibly Fast Flux: [81, 3701, 15456, 16509, 21785, 22753, 36850, 54455, 61317] Sep 06 18:28:26 squid-asnbl-helper[9945] INFO: Denying access to possible Fast Flux destination 'fedoraproject.org' OK
Apparently, the Fedora folks think it is a good idea to use round-robin for load balancing:
It is indeed a very good idea to load-balance like this, but I do not know why they need to many locations for their website. This is a CDN gone mad.
$ dig +short a fedoraproject.org 140.211.169.206 67.219.144.68 85.236.55.6 38.145.60.20 152.19.134.198 209.132.190.2 18.133.140.134 18.185.136.17 185.141.165.254 152.19.134.142 38.145.60.21 18.159.254.57
At the first glance, using the URL filter (by adding fedoraproject.org to the list of always allowed domains) seems to be a straight-forward solution to this problem. However, it does not work, as the ASNBL script is executed in the context of an ACL, while the URL filter comes as a redirect/wrapper. Therefore, it is never reached if a "deny" ACL matches in the first place.
This is the only false positive I observed so far. Unfortunately, it is a rather bad one. :-/
Any thoughts on what to do now?
Thanks, and best regards, Peter Müller
Hello Michael,
thank you for your reply.
Hello Peter,
I love this feature. I think it is a one-of-a-kind thing and hopefully many more people will think the same.
Yes, I like the idea, too. Sometimes, security can be simple _and_ effective... :-)
However, it will need a lot of documentation and explaining.
Indeed. I was thinking about a blog post for it; we probably need to explain Fast Flux in the first place, and I am not sure if all of our users are aware of the existence of autonomous systems.
I have a couple of high-level questions:
- Does it make sense to give the user the choice for the threshold?
It seems to be a difficult question because it requires exact knowledge what this feature actually does. My fears are that people just set this to something like “9” and the feature would become ineffective. What use-case is there to change this?
One size never fits all, I guess.
Indeed, the range of useful threshold values is pretty small: Anything below 4 causes _way_ too much false positives in productive environment, whereas even 7 appears to be too ineffective.
At the moment, the CGI catches values the ASNBL helper would treat itself as being invalid. Do you think narrowing down this range to 4 to 7 makes sense? Or should we replace it by a dropdown for adjusting sensitivity?
Either way, it is a good idea to tell users to leave the default where it is unless they truly understand what they are doing.
- Selective announcements: Should this necessarily live in the proxy? Why do we not generate a filter for the firewall?
We can do so as well, and I would love to see such a feature landing in IPFire.
Given our current state of libloc, I doubt this is possible: We would need a function that returns all networks we do not have an AS for - to my knowledge, the libloc (bindings) do not support this at the moment.
Apart from that: On a packet filter level, we lack the FQDN of a destination, which might be useful to have for debugging or forensic reasons.
Also, the users will experience a timeout after n seconds. Having selective announcement detection turned on, they'll get their error message straight away. I was told this improves UX... :-)
Thanks, and best regards, Peter Müller
-Michael
On 18 Jun 2021, at 18:24, Peter Müller peter.mueller@ipfire.org wrote:
This patchset adds two new features to IPFire's web proxy, taking advantage of the Autonomous System information we have at hand by using libloc.
The proactive Fast Flux detection is especially worth noticing, as even most expensive (= advanced?) security suites do not provide similar protection, especially not in a proactive manner.
By simply enumerating the distinct amount of Autonomous System Numbers a FQDN ultimately resolves to, we are able to deny access to malware distribution sites, phishing sites, C&C servers, and other cybercrime stuff hosted on Fast Flux setups abusing cracked machines around the world - even before the FQDN or any IP address involved is flagged as malicious by any security vendor.
Peter Müller (3): squid-asnbl: New package proxy.cgi: Implement proactive Fast Flux detection and detection for selectively announced destinations langs: Add English and German translations for newly added web proxy features
config/rootfiles/common/squid-asnbl | 1 + html/cgi-bin/proxy.cgi | 89 +++++++++++++++++++++++++++++ langs/de/cgi-bin/de.pl | 7 +++ langs/en/cgi-bin/en.pl | 7 +++ lfs/squid-asnbl | 83 +++++++++++++++++++++++++++ make.sh | 1 + 6 files changed, 188 insertions(+) create mode 100644 config/rootfiles/common/squid-asnbl create mode 100644 lfs/squid-asnbl
-- 2.26.2
-
Michael Tremer -
Peter Müller