From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: [PATCH 0/3] Add ASN-based anomaly detections to IPFire's web proxy: Proactive Fast Flux detection and detection for selectively announced networks Date: Tue, 07 Sep 2021 15:28:03 +0100 Message-ID: In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============7760927072086735708==" List-Id: --===============7760927072086735708== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello, This is bad news indeed. How about we have a whitelist that we ship with this? If you are using ACLs, you can have squid check if the domain is on the white= list and then skip the fast flux check. That should be easy and have no overhead. If we are encountering too many ite= ms that cause trouble, we could make that whitelist editable for the user. -Michael > On 6 Sep 2021, at 17:35, Peter M=C3=BCller wro= te: >=20 > Hello *, >=20 > by accident, I just stumbled across a false positive related to the Fast Fl= ux detection: >=20 >> [root(a)maverick ~]# su squid -s /bin/bash >> bash-5.1$ /usr/bin/asnbl-helper.py /var/ipfire/proxy/asnbl-helper.conf >> Sep 06 18:28:21 squid-asnbl-helper[9945] WARN: No ASNBL configured. This i= s acceptable as long as this script is configured to do anything, you just ha= ve been warned... >> Sep 06 18:28:21 squid-asnbl-helper[9945] INFO: Configuation sanity tests p= assed, good, processing... >> Sep 06 18:28:21 squid-asnbl-helper[9945] INFO: Successfully loaded locatio= n database from /var/lib/location/database.db generated 'Mon Sep 6 05:52:56 = 2021' (UTC/GMT) by 'IPFire Project' - good >> Sep 06 18:28:21 squid-asnbl-helper[9945] INFO: Running ASN database respon= se tests... >> Sep 06 18:28:21 squid-asnbl-helper[9945] INFO: ASN database operational - = excellent. Waiting for input... >> fedoraproject.org >> Sep 06 18:28:26 squid-asnbl-helper[9945] WARN: Destination 'fedoraproject.= org' exceeds ASN diversity threshold (9 > 5), possibly Fast Flux: [81, 3701, = 15456, 16509, 21785, 22753, 36850, 54455, 61317] >> Sep 06 18:28:26 squid-asnbl-helper[9945] INFO: Denying access to possible = Fast Flux destination 'fedoraproject.org' >> OK >=20 > Apparently, the Fedora folks think it is a good idea to use round-robin for= load balancing: It is indeed a very good idea to load-balance like this, but I do not know wh= y they need to many locations for their website. This is a CDN gone mad. >> $ dig +short a fedoraproject.org >> 140.211.169.206 >> 67.219.144.68 >> 85.236.55.6 >> 38.145.60.20 >> 152.19.134.198 >> 209.132.190.2 >> 18.133.140.134 >> 18.185.136.17 >> 185.141.165.254 >> 152.19.134.142 >> 38.145.60.21 >> 18.159.254.57 >=20 > At the first glance, using the URL filter (by adding fedoraproject.org to t= he list of always allowed > domains) seems to be a straight-forward solution to this problem. However, = it does not work, as the > ASNBL script is executed in the context of an ACL, while the URL filter com= es as a redirect/wrapper. > Therefore, it is never reached if a "deny" ACL matches in the first place. >=20 > This is the only false positive I observed so far. Unfortunately, it is a r= ather bad one. :-/ >=20 > Any thoughts on what to do now? >=20 > Thanks, and best regards, > Peter M=C3=BCller >=20 >=20 >> Hello Michael, >>=20 >> thank you for your reply. >>=20 >>> Hello Peter, >>>=20 >>> I love this feature. I think it is a one-of-a-kind thing and hopefully ma= ny more people will think the same. >>=20 >> Yes, I like the idea, too. Sometimes, security can be simple _and_ effecti= ve... :-) >>=20 >>> However, it will need a lot of documentation and explaining. >>=20 >> Indeed. I was thinking about a blog post for it; we probably need to expla= in Fast Flux in the >> first place, and I am not sure if all of our users are aware of the existe= nce of autonomous >> systems. >>=20 >>> I have a couple of high-level questions: >>>=20 >>> * Does it make sense to give the user the choice for the threshold? >>>=20 >>> It seems to be a difficult question because it requires exact knowledge w= hat this feature actually does. My fears are that people just set this to som= ething like =E2=80=9C9=E2=80=9D and the feature would become ineffective. Wha= t use-case is there to change this? >>=20 >> One size never fits all, I guess. >>=20 >> Indeed, the range of useful threshold values is pretty small: Anything bel= ow 4 causes _way_ too >> much false positives in productive environment, whereas even 7 appears to = be too ineffective. >>=20 >> At the moment, the CGI catches values the ASNBL helper would treat itself = as being invalid. Do >> you think narrowing down this range to 4 to 7 makes sense? Or should we re= place it by a dropdown >> for adjusting sensitivity? >>=20 >> Either way, it is a good idea to tell users to leave the default where it = is unless they truly >> understand what they are doing. >>=20 >>> * Selective announcements: Should this necessarily live in the proxy? Why= do we not generate a filter for the firewall? >>=20 >> We can do so as well, and I would love to see such a feature landing in IP= Fire. >>=20 >> Given our current state of libloc, I doubt this is possible: We would need= a function that returns >> all networks we do not have an AS for - to my knowledge, the libloc (bindi= ngs) do not support this >> at the moment. >>=20 >> Apart from that: On a packet filter level, we lack the FQDN of a destinati= on, which might be useful >> to have for debugging or forensic reasons. >>=20 >> Also, the users will experience a timeout after n seconds. Having selectiv= e announcement detection >> turned on, they'll get their error message straight away. I was told this = improves UX... :-) >>=20 >> Thanks, and best regards, >> Peter M=C3=BCller >>=20 >>>=20 >>> -Michael >>>=20 >>>> On 18 Jun 2021, at 18:24, Peter M=C3=BCller = wrote: >>>>=20 >>>> This patchset adds two new features to IPFire's web proxy, taking advant= age >>>> of the Autonomous System information we have at hand by using libloc. >>>>=20 >>>> The proactive Fast Flux detection is especially worth noticing, as even = most >>>> expensive (=3D advanced?) security suites do not provide similar protect= ion, >>>> especially not in a proactive manner. >>>>=20 >>>> By simply enumerating the distinct amount of Autonomous System Numbers a= FQDN >>>> ultimately resolves to, we are able to deny access to malware distributi= on >>>> sites, phishing sites, C&C servers, and other cybercrime stuff hosted on= Fast >>>> Flux setups abusing cracked machines around the world - even before the = FQDN >>>> or any IP address involved is flagged as malicious by any security vendo= r. >>>>=20 >>>> Peter M=C3=BCller (3): >>>> squid-asnbl: New package >>>> proxy.cgi: Implement proactive Fast Flux detection and detection for >>>> selectively announced destinations >>>> langs: Add English and German translations for newly added web proxy >>>> features >>>>=20 >>>> config/rootfiles/common/squid-asnbl | 1 + >>>> html/cgi-bin/proxy.cgi | 89 +++++++++++++++++++++++++++++ >>>> langs/de/cgi-bin/de.pl | 7 +++ >>>> langs/en/cgi-bin/en.pl | 7 +++ >>>> lfs/squid-asnbl | 83 +++++++++++++++++++++++++++ >>>> make.sh | 1 + >>>> 6 files changed, 188 insertions(+) >>>> create mode 100644 config/rootfiles/common/squid-asnbl >>>> create mode 100644 lfs/squid-asnbl >>>>=20 >>>> --=20 >>>> 2.26.2 >>>=20 --===============7760927072086735708==--