Location September 2022

location@lists.ipfire.org

3 participants
6 discussions

[PATCH 01/10] importer: Store geofeed URLs from RIR data
by Michael Tremer 29 Oct '22

29 Oct '22

Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org> --- src/scripts/location-importer.in | 40 +++++++++++++++++++++++++++++++- 1 file changed, 39 insertions(+), 1 deletion(-) diff --git a/src/scripts/location-importer.in b/src/scripts/location-importer.in index 9faf23b..5bd5da3 100644 --- a/src/scripts/location-importer.in +++ b/src/scripts/location-importer.in @@ -182,6 +182,11 @@ class CLI(object): CREATE INDEX IF NOT EXISTS networks_family ON networks USING BTREE(family(network)); CREATE INDEX IF NOT EXISTS networks_search ON networks USING GIST(network inet_ops); + -- geofeeds + CREATE TABLE IF NOT EXISTS network_geofeeds(network inet, url text); + CREATE UNIQUE INDEX IF NOT EXISTS network_geofeeds_unique + ON network_geofeeds(network); + -- overrides CREATE TABLE IF NOT EXISTS autnum_overrides( number bigint NOT NULL, @@ -799,6 +804,16 @@ class CLI(object): inetnum[key].append(val) + # Parse the geofeed attribute + elif key == "geofeed": + inetnum["geofeed"] = val + + # Parse geofeed when used as a remark + elif key == "remark": + m = re.match(r"^(?:geofeed|Geofeed)\s+(https://.*)", val) + if m: + inetnum["geofeed"] = m.group(1) + # Skip empty objects if not inetnum or not "country" in inetnum: return @@ -810,7 +825,6 @@ class CLI(object): # them into the database, if _check_parsed_network() succeeded for single_network in inetnum.get("inet6num") or inetnum.get("inetnum"): if self._check_parsed_network(single_network): - # Skip objects with unknown country codes if they are valid to avoid log spam... if validcountries and invalidcountries: log.warning("Skipping network with bogus countr(y|ies) %s (original countries: %s): %s" % \ @@ -823,6 +837,30 @@ class CLI(object): "%s" % single_network, inetnum.get("country")[0], inetnum.get("country"), source_key, ) + # Update any geofeed information + geofeed = inetnum.get("geofeed", None) + + # Store/update any geofeeds + if geofeed: + self.db.execute(""" + INSERT INTO + network_geofeeds( + network, + url + ) + VALUES( + %s, %s + ) + ON CONFLICT (network) DO + UPDATE SET url = excluded.url""", + "%s" % single_network, geofeed, + ) + + # Delete any previous geofeeds + else: + self.db.execute("DELETE FROM network_geofeeds WHERE network = %s", + "%s" % single_network) + def _parse_org_block(self, block, source_key): org = {} for line in block: -- 2.30.2

2 11

[PATCH] location-importer.in: Conduct sanity checks per DROP list
by Peter Müller 04 Oct '22

04 Oct '22

Previously, the lack of distinction between different DROP lists caused only the last one to be persisted. Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- src/scripts/location-importer.in | 58 +++++++++++++++++++++----------- 1 file changed, 39 insertions(+), 19 deletions(-) diff --git a/src/scripts/location-importer.in b/src/scripts/location-importer.in index 8d47497..e4a9ab0 100644 --- a/src/scripts/location-importer.in +++ b/src/scripts/location-importer.in @@ -1427,18 +1427,21 @@ class CLI(object): def _update_overrides_for_spamhaus_drop(self): downloader = location.importer.Downloader() - ip_urls = [ - "https://www.spamhaus.org/drop/drop.txt", - "https://www.spamhaus.org/drop/edrop.txt", - "https://www.spamhaus.org/drop/dropv6.txt" + ip_lists = [ + ("Spamhaus DROP list", "https://www.spamhaus.org/drop/drop.txt"), + ("Spamhaus EDROP list", "https://www.spamhaus.org/drop/edrop.txt"), + ("Spamhaus DROPv6 list", "https://www.spamhaus.org/drop/dropv6.txt") ] - asn_urls = [ - "https://www.spamhaus.org/drop/asndrop.txt" + asn_lists = [ + ("Spamhaus ASN-DROP list", "https://www.spamhaus.org/drop/asndrop.txt") ] - for url in ip_urls: - # Fetch IP list + for list in ip_lists: + name = list[0] + url = list[1] + + # Fetch IP list from given URL f = downloader.retrieve(url) # Split into lines @@ -1448,11 +1451,11 @@ class CLI(object): # downloads. if len(fcontent) > 10: self.db.execute(""" - DELETE FROM autnum_overrides WHERE source = 'Spamhaus ASN-DROP list'; - DELETE FROM network_overrides WHERE source = 'Spamhaus DROP lists'; - """) + DELETE FROM network_overrides WHERE source = '%s'; + """ % name, + ) else: - log.error("Spamhaus DROP URL %s returned likely bogus file, ignored" % url) + log.error("%s (%s) returned likely bogus file, ignored" % (name, url)) continue # Iterate through every line, filter comments and add remaining networks to @@ -1475,8 +1478,8 @@ class CLI(object): # Sanitize parsed networks... if not self._check_parsed_network(network): - log.warning("Skipping bogus network found in Spamhaus DROP URL %s: %s" % \ - (url, network)) + log.warning("Skipping bogus network found in %s (%s): %s" % \ + (name, url, network)) continue # Conduct SQL statement... @@ -1488,14 +1491,31 @@ class CLI(object): ) VALUES (%s, %s, %s) ON CONFLICT (network) DO UPDATE SET is_drop = True""", "%s" % network, - "Spamhaus DROP lists", + name, True ) - for url in asn_urls: + for list in asn_lists: + name = list[0] + url = list[1] + # Fetch URL f = downloader.retrieve(url) + # Split into lines + fcontent = f.readlines() + + # Conduct a very basic sanity check to rule out CDN issues causing bogus DROP + # downloads. + if len(fcontent) > 10: + self.db.execute(""" + DELETE FROM autnum_overrides WHERE source = '%s'; + """ % name, + ) + else: + log.error("%s (%s) returned likely bogus file, ignored" % (name, url)) + continue + # Iterate through every line, filter comments and add remaining ASNs to # the override table in case they are valid... with self.db.transaction(): @@ -1518,8 +1538,8 @@ class CLI(object): # Filter invalid ASNs... if not self._check_parsed_asn(asn): - log.warning("Skipping bogus ASN found in Spamhaus DROP URL %s: %s" % \ - (url, asn)) + log.warning("Skipping bogus ASN found in %s (%s): %s" % \ + (name, url, asn)) continue # Conduct SQL statement... @@ -1531,7 +1551,7 @@ class CLI(object): ) VALUES (%s, %s, %s) ON CONFLICT (number) DO UPDATE SET is_drop = True""", "%s" % asn, - "Spamhaus ASN-DROP list", + name, True ) -- 2.35.3

3 9

Python location files moved from site-packages to dist-packages
by Peter Müller 02 Oct '22

02 Oct '22

Hello Michael, trying to update to libloc 0.9.15 in IPFire 2.x, I just noticed that all the Python files are now in /usr/lib/python3/dist-packages/location/, not /usr/lib/python3/site-packages/location/, where they used to be before. Thanks to that, "location" crashes with a ModuleNotFoundError. Adding > sys.path.append("/usr/lib/python3/dist-packages/") before the location import statements solves this, but is probably not a fix we strive for. :-) Am I overlooking something (such as changes to the LFS file)? Thanks, and best regards, Peter Müller

2 2

[PATCH] location-importer.in: Add new Amazon region codes
by Peter Müller 26 Sep '22

26 Sep '22

Note that the precise location of "ap-southeast-6" is currently undocumented. However, all IPv4 networks allocated are not routed at the time of writing, hence our allocation of Asia/Pacific should be acceptable for the time being. Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- src/scripts/location-importer.in | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/scripts/location-importer.in b/src/scripts/location-importer.in index 8d47497..1a37f9e 100644 --- a/src/scripts/location-importer.in +++ b/src/scripts/location-importer.in @@ -1341,8 +1341,10 @@ class CLI(object): "ap-southeast-2": "AU", "ap-southeast-3": "MY", "ap-southeast-4": "AU", + "ap-southeast-6": "AP", # XXX: Precise location not documented anywhere "ap-northeast-1": "JP", "ca-central-1": "CA", + "ca-west-1": "CA", "eu-central-1": "DE", "eu-central-2": "CH", "eu-west-1": "IE", -- 2.35.3

1 0

libloc 0.9.15 released
by Michael Tremer 26 Sep '22

26 Sep '22

Hello, This is the release announcement for libloc 0.9.15. It comes with the following changes: * Major refactoring of the code that maps the database into memory has been undertaken. This is to support any architectures that do not have a page size of four kilobytes. Instead mapping individual parts of the database, the entire file is being mmapped into memory which removes the earlier restriction. * The library is telling the kernel that it intends to randomly read data from the file, which results in a minor performance improvement on slower storage devices. * The testsuite has been extended and we now check for better return codes in case the database could not be opened. * The library compiles on Mac OS X again - but support is not yet ready for production. * The configure script now understands the —with-database-path switch which can be used to set an alternative path for the database. * The location Python module has a new function called “open” which called without an argument tries to open the default database, and if called with a path argument will open the database at the given path. Thank you to everyone who has contributed to this release. You can find the source here: https://git.ipfire.org/?p=location/libloc.git;a=shortlog;h=refs/tags/0.9.15 https://source.ipfire.org/releases/libloc/libloc-0.9.15.tar.gz Best, -Michael

1 0

[PATCH] override-a3: Remove AS48550 as anycast network
by ml+ipfire-location＠ppmathis.com 05 Sep '22

05 Sep '22

From: Pascal Mathis <dev(a)ppmathis.com> Signed-off-by: Pascal Mathis <dev(a)ppmathis.com> --- While AS48550 has once been used for providing an anycast network, it is being used as a regular AS since several years. I am responsible for operating this network and hereby submit this patch to remove this obsolete override. Additionally, the current name of AS48550 is incorrect and should instead reflect "SnapServ Ltd", but I suppose this change will be reflected automatically once the override is gone and the automatic update triggers? Thanks a lot in advance! overrides/override-a3.txt | 5 ----- 1 file changed, 5 deletions(-) diff --git a/overrides/override-a3.txt b/overrides/override-a3.txt index b36fbfe..9e7c4d8 100644 --- a/overrides/override-a3.txt +++ b/overrides/override-a3.txt @@ -197,11 +197,6 @@ remarks: Generic anycast network [high confidence, but not proofed] is-anycast: yes country: EU -aut-num: AS48550 -descr: Pascal Mathis trading as SnapServ Mathis -remarks: Generic anycast network -is-anycast: yes - aut-num: AS50069 descr: Misaka Network, Inc. remarks: Generic anycast network -- 2.37.3

2 1

2024

2023

2022

2021

2020

2019

2018

2017

Location September 2022