Re: [PATCH 2/3] proxy.cgi: Implement proactive Fast Flux detection and detection for selectively announced destinations

["Peter Müller" <peter.mueller@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 8530 bytes --]

Hello Michael,

> Hello,
> 
>> On 18 Jun 2021, at 18:24, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>
>> This patch adds two new features to IPFire's web proxy:
>>
>> (a) Proactive Fast Flux detection
>>    FQDNs are resolved to their IP addresses, which are then resolved to
>>    corresponding Autonomous System Numbers using IPFire's location
>>    database. Most destinations will scatter across a very low number of
>>    ASNs (not to be confused with IP addresses!). FQDNs hosted on Fast
>>    Flux setups have a significantly higher ASN diversity (5 is usually
>>    a good threshold), so they can be proactively detected.
>>
>> (b) Detection for selectively announced destinations
>>    Especially in targeted operations, miscreants host FQDNs for
>>    exfiltrating data or malware distributions on ASNs not announced
>>    globally, but only to the intended victim or it's upstream ISPs.
>>
>>    That way, security researchers located in other parts of the
>>    internet have no insights into these attacks, hence not being able
>>    to publish listings or send take down notices for the domains used.
>>
>>    While RPKI made this attack harder, it can still be observed every
>>    now and then.
>>
>>    This feature also protects against accessing FQDNs resolving to IP
>>    addresses not being globally routeable, hence providing a trivial
>>    mitigation for so-called "rebound attacks" - which we cannot filter
>>    at DNS level currently.
>>
>> Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org>
>> ---
>> html/cgi-bin/proxy.cgi | 89 ++++++++++++++++++++++++++++++++++++++++++
>> 1 file changed, 89 insertions(+)
>>
>> diff --git a/html/cgi-bin/proxy.cgi b/html/cgi-bin/proxy.cgi
>> index 78ad33ad2..b7227deaf 100644
>> --- a/html/cgi-bin/proxy.cgi
>> +++ b/html/cgi-bin/proxy.cgi
>> @@ -21,6 +21,7 @@
>>
>> use strict;
>> use Apache::Htpasswd;
>> +use Scalar::Util qw(looks_like_number);
>>
>> # enable only the following on debugging purpose
>> #use warnings;
>> @@ -225,6 +226,9 @@ $proxysettings{'THROTTLING_GREEN_TOTAL'} = 'unlimited';
>> $proxysettings{'THROTTLING_GREEN_HOST'} = 'unlimited';
>> $proxysettings{'THROTTLING_BLUE_TOTAL'} = 'unlimited';
>> $proxysettings{'THROTTLING_BLUE_HOST'} = 'unlimited';
>> +$proxysettings{'ASNBL_FASTFLUX_DETECTION'} = 'off';
>> +$proxysettings{'ASNBL_FASTFLUX_THRESHOLD'} = '5';
>> +$proxysettings{'ASNBL_SELECANN_DETECTION'} = 'off';
>> $proxysettings{'ENABLE_MIME_FILTER'} = 'off';
>> $proxysettings{'AUTH_METHOD'} = 'none';
>> $proxysettings{'AUTH_REALM'} = '';
>> @@ -414,6 +418,21 @@ if (($proxysettings{'ACTION'} eq $Lang::tr{'save'}) || ($proxysettings{'ACTION'}
>> 		$errormessage = $Lang::tr{'invalid maximum incoming size'};
>> 		goto ERROR;
>> 	}
>> +	if (($proxysettings{'ASNBL_FASTFLUX_DETECTION'} eq 'on') || ($proxysettings{'ASNBL_SELECANN_DETECTION'} eq 'on'))
>> +	{
>> +		if (-z $proxysettings{'ASNBL_FASTFLUX_THRESHOLD'}) {
>> +			$errormessage = $Lang::tr{'advproxy fastflux no threshold given'};
>> +			goto ERROR;
>> +		}
>> +		if (! looks_like_number($proxysettings{'ASNBL_FASTFLUX_THRESHOLD'})) {
>> +			$errormessage = $Lang::tr{'advproxy fastflux threshold invalid'};
>> +			goto ERROR;
>> +		}
>> +		if (($proxysettings{'ASNBL_FASTFLUX_THRESHOLD'} < 2) || ($proxysettings{'ASNBL_FASTFLUX_THRESHOLD'} > 10)) {
>> +			$errormessage = $Lang::tr{'advproxy fastflux threshold out of bounds'};
>> +			goto ERROR;
>> +		}
>> +	}
>> 	if (!($proxysettings{'AUTH_METHOD'} eq 'none'))
>> 	{
>> 		unless (($proxysettings{'AUTH_METHOD'} eq 'ident') &&
>> @@ -797,6 +816,14 @@ $selected{'THROTTLING_GREEN_HOST'}{$proxysettings{'THROTTLING_GREEN_HOST'}} = "s
>> $selected{'THROTTLING_BLUE_TOTAL'}{$proxysettings{'THROTTLING_BLUE_TOTAL'}} = "selected='selected'";
>> $selected{'THROTTLING_BLUE_HOST'}{$proxysettings{'THROTTLING_BLUE_HOST'}} = "selected='selected'";
>>
>> +$checked{'ASNBL_FASTFLUX_DETECTION'}{'off'} = '';
>> +$checked{'ASNBL_FASTFLUX_DETECTION'}{'on'} = '';
>> +$checked{'ASNBL_FASTFLUX_DETECTION'}{$proxysettings{'ASNBL_FASTFLUX_DETECTION'}} = "checked='checked'";
>> +
>> +$checked{'ASNBL_SELECANN_DETECTION'}{'off'} = '';
>> +$checked{'ASNBL_SELECANN_DETECTION'}{'on'} = '';
>> +$checked{'ASNBL_SELECANN_DETECTION'}{$proxysettings{'ASNBL_SELECANN_DETECTION'}} = "checked='checked'";
>> +
>> $checked{'ENABLE_MIME_FILTER'}{'off'} = '';
>> $checked{'ENABLE_MIME_FILTER'}{'on'} = '';
>> $checked{'ENABLE_MIME_FILTER'}{$proxysettings{'ENABLE_MIME_FILTER'}} = "checked='checked'";
>> @@ -1627,6 +1654,24 @@ END
>> print <<END
>> </table>
>>
>> +<hr size='1'>
>> +
>> +<table width='100%'>
>> +<tr>
>> +       <td><b>$Lang::tr{'advproxy asbased anomaly detection'}</b></td>
>> +</tr>
>> +<tr>
>> +       <td class='base'>$Lang::tr{'advproxy fastflux detection'}:</td>
>> +       <td><input type='checkbox' name='ASNBL_FASTFLUX_DETECTION' $checked{'ASNBL_FASTFLUX_DETECTION'}{'on'} /></td>
>> +       <td class='base'>$Lang::tr{'advproxy fastflux detection threshold'}:</td>
>> +       <td><input type='text' name='ASNBL_FASTFLUX_THRESHOLD' value='$proxysettings{'ASNBL_FASTFLUX_THRESHOLD'}' size=2 /></td>
>> +</tr>
>> +<tr>
>> +       <td class='base'>$Lang::tr{'advproxy selectively announcements detection'}:</td>
>> +       <td colspan='3'><input type='checkbox' name='ASNBL_SELECANN_DETECTION' $checked{'ASNBL_SELECANN_DETECTION'}{'on'} /></td>
>> +</tr>
>> +</table>
>> +
>> <hr size='1'>
>> END
>> ;
>> @@ -3507,6 +3552,50 @@ if (@ssl_ports) {
>> 	print FILE "http_access deny  CONNECT !SSL_ports\n";
>> }
>>
>> +	if ((($proxysettings{'ASNBL_FASTFLUX_DETECTION'} eq 'on') && (!-z $proxysettings{'ASNBL_FASTFLUX_THRESHOLD'})) || ($proxysettings{'ASNBL_SELECANN_DETECTION'} eq 'on')) {
>> +		print FILE "external_acl_type asnblhelper children-max=10 children-startup=2 ttl=86400 %DST /usr/bin/asnbl-helper.py /var/ipfire/proxy/asnbl-helper.conf\n";
>> +		print FILE "acl asnbl external asnblhelper\n";
>> +		print FILE "http_access deny asnbl\n\n";
>> +
>> +		# Write ASNBL helper configuration file...
>> +		open(ASNBLFILE, ">${General::swroot}/proxy/asnbl-helper.conf");
>> +		flock(ASNBLFILE, 2);
>> +
>> +		print ASNBLFILE<<END
>> +#
>> +# This file has been automatically generated. Manual changes will be overwritten.
>> +#
>> +
>> +[GENERAL]
>> +LOGLEVEL = INFO
>> +ASNDB_PATH = /var/lib/location/database.db
>> +USE_REPLYMAP = no
>> +END
>> +;
>> +
>> +		print ASNBLFILE "AS_DIVERSITY_THRESHOLD = $proxysettings{'ASNBL_FASTFLUX_THRESHOLD'}\n";
>> +
>> +		if ($proxysettings{'ASNBL_SELECANN_DETECTION'} eq 'on') {
>> +			print ASNBLFILE "BLOCK_SUSPECTED_SELECTIVE_ANNOUNCEMENTS = yes\n";
>> +		} else {
>> +			print ASNBLFILE "BLOCK_SUSPECTED_SELECTIVE_ANNOUNCEMENTS = no\n";
>> +		}
>> +
>> +		if ($proxysettings{'ASNBL_FASTFLUX_DETECTION'} eq 'on') {
>> +			print ASNBLFILE "BLOCK_DIVERSITY_EXCEEDING_DESTINATIONS = yes\n";
>> +		} else {
>> +			print ASNBLFILE "BLOCK_DIVERSITY_EXCEEDING_DESTINATIONS = no\n";
>> +		}
>> +
>> +		print ASNBLFILE<<END
>> +TESTDATA = (1.1.1.1, 13335) (8.8.8.8, 15169) (194.95.245.140, 680) (10.0.0.1, 0) (127.0.0.1, 0) (2001:638:d:c102::140, 680) (2606:4700:10::6814:d673, 13335) (fe80::1, 0)
> 
> Why do we want to hard-code this here?

Because the ASNBL helper requires some test points to ensure the location database provided is actually
working. It is designed to prefer "fail close" (i. e. stopping operation in case something is rendering
its purpose useless) rather than "fail open" (i. e. logging a warning and proceed anyway, even if the location
database returns "0" for any given IP address).

> Does that not (if anywhere) belong into libloc? I disagree with hard-coding this, because what happens if Google moves their DNS server? It would break this feature.

Partly. We should have some sanity checks in libloc as well, but they ultimately boil down to the same problem:
We need a set of relatively statically propagated IP addresses, which the internet does not have by nature.

Do you have a better proposal for testing points? Like the IP address of our own NTP server, which we hard-coded
somewhere else, too?

Thanks, and best regards,
Peter Müller

> 
> -Michael
> 
>> +ACTIVE_ASNBLS = 
>> +END
>> +;
>> +
>> +		close ASNBLFILE;
>> +    }
>> +
>> if ($proxysettings{'AUTH_METHOD'} eq 'ident')
>> {
>> print FILE "#Set ident ACLs\n";
>> -- 
>> 2.26.2
>