Location June 2022

location@lists.ipfire.org

1 participants
2 discussions

[PATCH] location-importer: Only delete override data if we are sure to have a valid replacement
by Peter Müller 05 Jun '22

05 Jun '22

The current way of truncating all override data straight away leaves us with no data at all, should a source turn out to be unreachable or returning bogus files (yes, Cloudflare, I _am_ looking at you). It is therefore better to only delete data we know to have a valid replacement for, rather than just dropping the source altogether. Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- src/scripts/location-importer.in | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/src/scripts/location-importer.in b/src/scripts/location-importer.in index bee9186..bde92ce 100644 --- a/src/scripts/location-importer.in +++ b/src/scripts/location-importer.in @@ -1168,10 +1168,11 @@ class CLI(object): def handle_update_overrides(self, ns): with self.db.transaction(): - # Drop all data that we have + # Only drop manually created overrides, as we can be reasonably sure to have them, + # and preserve the rest. If appropriate, it is deleted by correspondent functions. self.db.execute(""" - TRUNCATE TABLE autnum_overrides; - TRUNCATE TABLE network_overrides; + DELETE FROM autnum_overrides WHERE source = 'manual'; + DELETE FROM network_overrides WHERE source = 'manual'; """) # Update overrides for various cloud providers big enough to publish their own IP @@ -1267,6 +1268,11 @@ class CLI(object): log.error("unable to preprocess Amazon AWS IP ranges: %s" % e) return + # At this point, we can assume the downloaded file to be valid + self.db.execute(""" + DELETE FROM network_overrides WHERE source = 'Amazon AWS IP feed'; + """) + # XXX: Set up a dictionary for mapping a region name to a country. Unfortunately, # there seems to be no machine-readable version available of this other than # https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availabil… @@ -1387,6 +1393,16 @@ class CLI(object): log.error("Unable to download Spamhaus DROP URL %s: %s" % (url, e)) return + # Conduct a very basic sanity check to rule out CDN issues causing bogus DROP + # downloads. + if len(fcontent) > 10: + self.db.execute(""" + DELETE FROM autnum_overrides WHERE source = 'Spamhaus ASN-DROP list'; + DELETE FROM network_overrides WHERE source = 'Spamhaus DROP lists'; + """) + else: + log.error("Spamhaus DROP URL %s returned likely bogus file, ignored" % url) + # Iterate through every line, filter comments and add remaining networks to # the override table in case they are valid... with self.db.transaction(): -- 2.35.3

1 0

[PATCH] location-importer: AS names starting with "DNIC" actually are valid
by Peter Müller 05 Jun '22

05 Jun '22

These are all assigned to the US Department of Defense. Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- src/scripts/location-importer.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/scripts/location-importer.in b/src/scripts/location-importer.in index bee9186..4e9dc8c 100644 --- a/src/scripts/location-importer.in +++ b/src/scripts/location-importer.in @@ -894,7 +894,7 @@ class CLI(object): continue # Skip any AS name that appears to be a placeholder for a different RIR or entity... - if re.match(r"^(ASN-BLK|)(AFCONC|AFRINIC|APNIC|ASNBLK|DNIC|LACNIC|RIPE|IANA)(?:\d?$|\-)", name): + if re.match(r"^(ASN-BLK|)(AFCONC|AFRINIC|APNIC|ASNBLK|LACNIC|RIPE|IANA)(?:\d?$|\-)", name): continue # Bail out in case the AS name contains anything we do not expect here... -- 2.35.3

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Location June 2022