Location February 2024

location@lists.ipfire.org

2 participants
3 discussions

[PATCH] location-importer: Replace ARIN AS names source with one that offers human-readable names
by Peter Müller 21 Feb '24

21 Feb '24

This patch replaces our previous source for AS names in ARIN's realms with another file provided by ARIN that contains human-readable names for organizations ASNs have been allocated to. Please note that a TRUNCATE autnums; is necessary on machines previously running the old version of location-importer, in order to make use of this changed data source. Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- src/scripts/location-importer.in | 47 ++++++++++++++++++-------------- 1 file changed, 27 insertions(+), 20 deletions(-) diff --git a/src/scripts/location-importer.in b/src/scripts/location-importer.in index 28a4f6c..96b3a20 100644 --- a/src/scripts/location-importer.in +++ b/src/scripts/location-importer.in @@ -3,7 +3,7 @@ # # # libloc - A library to determine the location of someone on the Internet # # # -# Copyright (C) 2020-2022 IPFire Development Team <info(a)ipfire.org> # +# Copyright (C) 2020-2023 IPFire Development Team <info(a)ipfire.org> # # # # This library is free software; you can redistribute it and/or # # modify it under the terms of the GNU Lesser General Public # @@ -19,6 +19,7 @@ import argparse import concurrent.futures +import csv import http.client import ipaddress import json @@ -1033,36 +1034,42 @@ class CLI(object): def _import_as_names_from_arin(self): downloader = location.importer.Downloader() - # XXX: Download AS names file from ARIN (note that these names appear to be quite - # technical, not intended for human consumption, as description fields in - # organisation handles for other RIRs are - however, this is what we have got, - # and in some cases, it might be still better than nothing) - for line in downloader.request_lines("https://ftp.arin.net/info/asn.txt"): - # Valid lines start with a space, followed by the number of the Autonomous System ... - if not line.startswith(" "): + # Download AS names file from ARIN and load it into CSV parser + for line in downloader.request_lines("https://ftp.arin.net/pub/resource_registry_service/asns.csv"): + + # Valid lines start with a " ... + if not line.startswith("\""): continue # Split line and check if there is a valid ASN in it... - asn, name = line.split()[0:2] + for row in csv.reader([line]): + orgname = row[0] + orghandle = row[1] + firstasn = row[3] + lastasn = row[4] try: - asn = int(asn) + firstasn = int(firstasn.strip("\"")) + lastasn = int(lastasn.strip("\"")) except ValueError: - log.debug("Skipping ARIN AS names line not containing an integer for ASN") + log.debug("Skipping ARIN AS names line not containing valid integers for ASN") continue # Filter invalid ASNs... - if not self._check_parsed_asn(asn): + if not self._check_parsed_asn(firstasn): continue - # Skip any AS name that appears to be a placeholder for a different RIR or entity... - if re.match(r"^(ASN-BLK|)(AFCONC|AFRINIC|APNIC|ASNBLK|LACNIC|RIPE|IANA)(?:\d?$|\-)", name): + if firstasn > lastasn: + continue + + # Filter any bulk AS assignments, since these are present for other RIRs where + # we get better data from elsewhere. + if not firstasn == lastasn: continue - # Bail out in case the AS name contains anything we do not expect here... - if re.search(r"[^a-zA-Z0-9-_]", name): - log.debug("Skipping ARIN AS name for %s containing invalid characters: %s" % \ - (asn, name)) + # Skip any AS name that appears to be a placeholder for a different RIR or entity... + if re.match(r"^(AFRINIC|APNIC|LACNIC|RIPE)$", orghandle.strip("\"")): + continue # Things look good here, run INSERT statement and skip this one if we already have # a (better?) name for this Autonomous System... @@ -1073,8 +1080,8 @@ class CLI(object): source ) VALUES (%s, %s, %s) ON CONFLICT (number) DO NOTHING""", - asn, - name, + firstasn, + orgname.strip("\""), "ARIN", ) -- 2.35.3

2 2

[PATCH] location-importer: Add missing area code for AWS
by Peter Müller 17 Feb '24

17 Feb '24

It is unclear what exactly "eusc" refers to ("EU Secure Cloud", maybe?), but according to RIPE database records for IPv6 networks involved, this area code physically traces back to Germany, likely Berlin instead of Frankfurt. Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- src/scripts/location-importer.in | 1 + 1 file changed, 1 insertion(+) diff --git a/src/scripts/location-importer.in b/src/scripts/location-importer.in index 28a4f6c..2be4887 100644 --- a/src/scripts/location-importer.in +++ b/src/scripts/location-importer.in @@ -1603,6 +1603,7 @@ class CLI(object): "eu-south-2": "ES", "eu-west-3": "FR", "eu-north-1": "SE", + "eusc-de-east-1", "DE", # XXX: Undocumented, likely located in Berlin rather than Frankfurt "il-central-1": "IL", # XXX: This one is not documented anywhere except for ip-ranges.json itself "me-central-1": "AE", "me-south-1": "BH", -- 2.35.3

1 0

[PATCH v2] location-importer: Fix Spamhaus ASN-DROP parsing
by Peter Müller 17 Feb '24

17 Feb '24

The format of this list has changed, from a plain text file with a customer schema to JSON. Adjust our routines accordingly to make use of this list again. The second version of this patch incorporates Michael's feedback on the first version, and adds AS names to the autnums table in case they are not there already, which closes some gaps on rogue ASNs in the LACNIC area. Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> Tested-by: Peter Müller <peter.mueller(a)ipfire.org> --- src/scripts/location-importer.in | 46 ++++++++++++++++++++++++-------- 1 file changed, 35 insertions(+), 11 deletions(-) diff --git a/src/scripts/location-importer.in b/src/scripts/location-importer.in index 28a4f6c..ac7249d 100644 --- a/src/scripts/location-importer.in +++ b/src/scripts/location-importer.in @@ -3,7 +3,7 @@ # # # libloc - A library to determine the location of someone on the Internet # # # -# Copyright (C) 2020-2022 IPFire Development Team <info(a)ipfire.org> # +# Copyright (C) 2020-2024 IPFire Development Team <info(a)ipfire.org> # # # # This library is free software; you can redistribute it and/or # # modify it under the terms of the GNU Lesser General Public # @@ -1686,7 +1686,7 @@ class CLI(object): ] asn_lists = [ - ("SPAMHAUS-ASNDROP", "https://www.spamhaus.org/drop/asndrop.txt") + ("SPAMHAUS-ASNDROP", "https://www.spamhaus.org/drop/asndrop.json") ] for name, url in ip_lists: @@ -1759,22 +1759,32 @@ class CLI(object): # Iterate through every line, filter comments and add remaining ASNs to # the override table in case they are valid... - for sline in f.readlines(): + for sline in fcontent: # The response is assumed to be encoded in UTF-8... sline = sline.decode("utf-8") - # Comments start with a semicolon... - if sline.startswith(";"): + # Load every line as a JSON object and try to obtain an ASN from it... + try: + lineobj = json.loads(sline) + except json.decoder.JSONDecodeError: + log.error("Unable to parse line as a JSON object: %s" % sline) continue - # Throw away anything after the first space... - sline = sline.split()[0] + # Skip line contiaining file metadata + try: + type = lineobj["type"] - # ... strip the "AS" prefix from it ... - sline = sline.strip("AS") + if type == "metadata": + continue + except KeyError: + pass - # ... and convert it into an integer. Voila. - asn = int(sline) + try: + asn = lineobj["asn"] + as_name = lineobj["asname"] + except KeyError: + log.warning("Unable to extract necessary information from line: %s" % sline) + continue # Filter invalid ASNs... if not self._check_parsed_asn(asn): @@ -1795,6 +1805,20 @@ class CLI(object): True ) + # In case we do not have an name for this AS already, update + # autnums table accordingly + self.db.execute(""" + INSERT INTO autnums( + number, + name, + source + ) VALUES (%s, %s, %s) + ON CONFLICT (number) DO NOTHING""", + "%s" % asn, + as_name, + name + ) + @staticmethod def _parse_bool(block, key): val = block.get(key) -- 2.35.3

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Location February 2024