- Location - lists.ipfire.org

Reduced AS/route visibility
by Jordan Savoca 22 Mar '24

22 Mar '24

It seems there has been a significant reduction in the visibility of certain routes in the IPFire dataset recently, and/or spurious routes taking place of legitimate announcements. For example, 209.38.204.47 is originated by AS14061 and announced in 209.38.192.0/19, but IPFire erroneously sees 209.38.0.0/16 and locates no corresponding AS. Here are a few more prefixes with the same issue: +----------+------------------+-----------------+------------+ | ASN | Prefix | IPFire Prefix | IPFire ASN | +----------+------------------+-----------------+------------+ | AS64425 | 45.148.122.0/24 | 45.148.120.0/22 | None | | AS197540 | 94.16.116.0/22 | 94.16.96.0/19 | None | | AS36352 | 198.23.234.0/23 | 198.23.128.0/17 | None | | AS63949 | 172.105.192.0/19 | 172.104.0.0/15 | None | +----------+------------------+-----------------+------------+ Thanks so much for all your hard work, -- Jordan

2 6

[PATCH] location-importer: Replace ARIN AS names source with one that offers human-readable names
by Peter Müller 21 Feb '24

21 Feb '24

This patch replaces our previous source for AS names in ARIN's realms with another file provided by ARIN that contains human-readable names for organizations ASNs have been allocated to. Please note that a TRUNCATE autnums; is necessary on machines previously running the old version of location-importer, in order to make use of this changed data source. Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- src/scripts/location-importer.in | 47 ++++++++++++++++++-------------- 1 file changed, 27 insertions(+), 20 deletions(-) diff --git a/src/scripts/location-importer.in b/src/scripts/location-importer.in index 28a4f6c..96b3a20 100644 --- a/src/scripts/location-importer.in +++ b/src/scripts/location-importer.in @@ -3,7 +3,7 @@ # # # libloc - A library to determine the location of someone on the Internet # # # -# Copyright (C) 2020-2022 IPFire Development Team <info(a)ipfire.org> # +# Copyright (C) 2020-2023 IPFire Development Team <info(a)ipfire.org> # # # # This library is free software; you can redistribute it and/or # # modify it under the terms of the GNU Lesser General Public # @@ -19,6 +19,7 @@ import argparse import concurrent.futures +import csv import http.client import ipaddress import json @@ -1033,36 +1034,42 @@ class CLI(object): def _import_as_names_from_arin(self): downloader = location.importer.Downloader() - # XXX: Download AS names file from ARIN (note that these names appear to be quite - # technical, not intended for human consumption, as description fields in - # organisation handles for other RIRs are - however, this is what we have got, - # and in some cases, it might be still better than nothing) - for line in downloader.request_lines("https://ftp.arin.net/info/asn.txt"): - # Valid lines start with a space, followed by the number of the Autonomous System ... - if not line.startswith(" "): + # Download AS names file from ARIN and load it into CSV parser + for line in downloader.request_lines("https://ftp.arin.net/pub/resource_registry_service/asns.csv"): + + # Valid lines start with a " ... + if not line.startswith("\""): continue # Split line and check if there is a valid ASN in it... - asn, name = line.split()[0:2] + for row in csv.reader([line]): + orgname = row[0] + orghandle = row[1] + firstasn = row[3] + lastasn = row[4] try: - asn = int(asn) + firstasn = int(firstasn.strip("\"")) + lastasn = int(lastasn.strip("\"")) except ValueError: - log.debug("Skipping ARIN AS names line not containing an integer for ASN") + log.debug("Skipping ARIN AS names line not containing valid integers for ASN") continue # Filter invalid ASNs... - if not self._check_parsed_asn(asn): + if not self._check_parsed_asn(firstasn): continue - # Skip any AS name that appears to be a placeholder for a different RIR or entity... - if re.match(r"^(ASN-BLK|)(AFCONC|AFRINIC|APNIC|ASNBLK|LACNIC|RIPE|IANA)(?:\d?$|\-)", name): + if firstasn > lastasn: + continue + + # Filter any bulk AS assignments, since these are present for other RIRs where + # we get better data from elsewhere. + if not firstasn == lastasn: continue - # Bail out in case the AS name contains anything we do not expect here... - if re.search(r"[^a-zA-Z0-9-_]", name): - log.debug("Skipping ARIN AS name for %s containing invalid characters: %s" % \ - (asn, name)) + # Skip any AS name that appears to be a placeholder for a different RIR or entity... + if re.match(r"^(AFRINIC|APNIC|LACNIC|RIPE)$", orghandle.strip("\"")): + continue # Things look good here, run INSERT statement and skip this one if we already have # a (better?) name for this Autonomous System... @@ -1073,8 +1080,8 @@ class CLI(object): source ) VALUES (%s, %s, %s) ON CONFLICT (number) DO NOTHING""", - asn, - name, + firstasn, + orgname.strip("\""), "ARIN", ) -- 2.35.3

2 2

[PATCH] location-importer: Add missing area code for AWS
by Peter Müller 17 Feb '24

17 Feb '24

It is unclear what exactly "eusc" refers to ("EU Secure Cloud", maybe?), but according to RIPE database records for IPv6 networks involved, this area code physically traces back to Germany, likely Berlin instead of Frankfurt. Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- src/scripts/location-importer.in | 1 + 1 file changed, 1 insertion(+) diff --git a/src/scripts/location-importer.in b/src/scripts/location-importer.in index 28a4f6c..2be4887 100644 --- a/src/scripts/location-importer.in +++ b/src/scripts/location-importer.in @@ -1603,6 +1603,7 @@ class CLI(object): "eu-south-2": "ES", "eu-west-3": "FR", "eu-north-1": "SE", + "eusc-de-east-1", "DE", # XXX: Undocumented, likely located in Berlin rather than Frankfurt "il-central-1": "IL", # XXX: This one is not documented anywhere except for ip-ranges.json itself "me-central-1": "AE", "me-south-1": "BH", -- 2.35.3

1 0

[PATCH v2] location-importer: Fix Spamhaus ASN-DROP parsing
by Peter Müller 17 Feb '24

17 Feb '24

The format of this list has changed, from a plain text file with a customer schema to JSON. Adjust our routines accordingly to make use of this list again. The second version of this patch incorporates Michael's feedback on the first version, and adds AS names to the autnums table in case they are not there already, which closes some gaps on rogue ASNs in the LACNIC area. Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> Tested-by: Peter Müller <peter.mueller(a)ipfire.org> --- src/scripts/location-importer.in | 46 ++++++++++++++++++++++++-------- 1 file changed, 35 insertions(+), 11 deletions(-) diff --git a/src/scripts/location-importer.in b/src/scripts/location-importer.in index 28a4f6c..ac7249d 100644 --- a/src/scripts/location-importer.in +++ b/src/scripts/location-importer.in @@ -3,7 +3,7 @@ # # # libloc - A library to determine the location of someone on the Internet # # # -# Copyright (C) 2020-2022 IPFire Development Team <info(a)ipfire.org> # +# Copyright (C) 2020-2024 IPFire Development Team <info(a)ipfire.org> # # # # This library is free software; you can redistribute it and/or # # modify it under the terms of the GNU Lesser General Public # @@ -1686,7 +1686,7 @@ class CLI(object): ] asn_lists = [ - ("SPAMHAUS-ASNDROP", "https://www.spamhaus.org/drop/asndrop.txt") + ("SPAMHAUS-ASNDROP", "https://www.spamhaus.org/drop/asndrop.json") ] for name, url in ip_lists: @@ -1759,22 +1759,32 @@ class CLI(object): # Iterate through every line, filter comments and add remaining ASNs to # the override table in case they are valid... - for sline in f.readlines(): + for sline in fcontent: # The response is assumed to be encoded in UTF-8... sline = sline.decode("utf-8") - # Comments start with a semicolon... - if sline.startswith(";"): + # Load every line as a JSON object and try to obtain an ASN from it... + try: + lineobj = json.loads(sline) + except json.decoder.JSONDecodeError: + log.error("Unable to parse line as a JSON object: %s" % sline) continue - # Throw away anything after the first space... - sline = sline.split()[0] + # Skip line contiaining file metadata + try: + type = lineobj["type"] - # ... strip the "AS" prefix from it ... - sline = sline.strip("AS") + if type == "metadata": + continue + except KeyError: + pass - # ... and convert it into an integer. Voila. - asn = int(sline) + try: + asn = lineobj["asn"] + as_name = lineobj["asname"] + except KeyError: + log.warning("Unable to extract necessary information from line: %s" % sline) + continue # Filter invalid ASNs... if not self._check_parsed_asn(asn): @@ -1795,6 +1805,20 @@ class CLI(object): True ) + # In case we do not have an name for this AS already, update + # autnums table accordingly + self.db.execute(""" + INSERT INTO autnums( + number, + name, + source + ) VALUES (%s, %s, %s) + ON CONFLICT (number) DO NOTHING""", + "%s" % asn, + as_name, + name + ) + @staticmethod def _parse_bool(block, key): val = block.get(key) -- 2.35.3

1 0

[PATCH] location-importer: Fix Spamhaus ASN-DROP parsing
by Peter Müller 05 Nov '23

05 Nov '23

The format of this list has changed, from a plain text file with a customer schema to JSON. Adjust our routines accordingly to make use of this list again. Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> Tested-by: Peter Müller <peter.mueller(a)ipfire.org> --- src/scripts/location-importer.in | 21 +++++++-------------- 1 file changed, 7 insertions(+), 14 deletions(-) diff --git a/src/scripts/location-importer.in b/src/scripts/location-importer.in index 28a4f6c..8b0c676 100644 --- a/src/scripts/location-importer.in +++ b/src/scripts/location-importer.in @@ -3,7 +3,7 @@ # # # libloc - A library to determine the location of someone on the Internet # # # -# Copyright (C) 2020-2022 IPFire Development Team <info(a)ipfire.org> # +# Copyright (C) 2020-2023 IPFire Development Team <info(a)ipfire.org> # # # # This library is free software; you can redistribute it and/or # # modify it under the terms of the GNU Lesser General Public # @@ -1686,7 +1686,7 @@ class CLI(object): ] asn_lists = [ - ("SPAMHAUS-ASNDROP", "https://www.spamhaus.org/drop/asndrop.txt") + ("SPAMHAUS-ASNDROP", "https://www.spamhaus.org/drop/asndrop.json") ] for name, url in ip_lists: @@ -1759,23 +1759,16 @@ class CLI(object): # Iterate through every line, filter comments and add remaining ASNs to # the override table in case they are valid... - for sline in f.readlines(): + for sline in fcontent: # The response is assumed to be encoded in UTF-8... sline = sline.decode("utf-8") - # Comments start with a semicolon... - if sline.startswith(";"): + # Load every line as a JSON object and try to obtain an ASN from it... + try: + asn = json.loads(sline)["asn"] + except KeyError: continue - # Throw away anything after the first space... - sline = sline.split()[0] - - # ... strip the "AS" prefix from it ... - sline = sline.strip("AS") - - # ... and convert it into an integer. Voila. - asn = int(sline) - # Filter invalid ASNs... if not self._check_parsed_asn(asn): log.warning("Skipping bogus ASN found in %s (%s): %s" % \ -- 2.35.3

2 3

"location lookup 4.231.239.118" results in "[Errno 2] No such file or directory"
by Peter Müller 15 Aug '23

15 Aug '23

Hello Michael, just something I encountered today while checking on a bunch of IP addresses: Trying to look up 4.231.239.118 results in "[Errno 2] No such file or directory", an error I didn't expect to surface at this point, and I'm not sure what to make out of it. Behavior on the system (IPFire running Core Update 177) is as follows: [root@firewall ~]# location lookup 4.231.239.118 [Errno 2] No such file or directory [root@firewall ~]# location version Mon, 14 Aug 2023 05:21:26 GMT [root@firewall ~]# location --version location 0.9.16 However, it doesn't look like the entire location database is broken: [root@firewall ~]# location lookup 8.8.8.8 8.8.8.8: Network : 8.8.8.0/24 Country : United States of America Autonomous System : AS15169 - GOOGLE Anycast : yes So far, I have only encountered this behavior in conjunction with 4.231.239.118; another IPFire machine running the same Core Update and libloc version, but having the database generated on Thu, 10 Aug 2023 05:23:53 GMT also exhibits this behavior. Am I doing something wrong? Should this be filed as a bug? Thanks, and best regards, Peter Müller

3 2

libloc 0.9.17 released
by Michael Tremer 31 Jul '23

31 Jul '23

Hello, This is the release announcement for libloc 0.9.17. It comes with the following changes: * The importer is now parsing Geofeeds where available. This helps us to create a database with better accuracy for large ISPs or cloud providers. * The database writer is trying to compress the database harder: It will now look for any duplicate networks and merge neighbouring networks which will reduce the size of the database by about half. * The importer has been improved so that it runs more efficient SQL queries to create the database faster. * Temuri Doghonadze contributed a Georgian translation. * Hans-Christoph Steiner contributed bash-completion for the location(8) command. Thank you to everyone who has contributed to this release. You can find the source here: https://git.ipfire.org/?p=location/libloc.git;a=shortlog;h=refs/tags/0.9.17 https://source.ipfire.org/releases/libloc/libloc-0.9.17.tar.gz Best, -Michael

1 0

Segfault on d5ff39d "tree: Actually delete any deleted nodes"
by Valters Jansons 18 Jul '23

18 Jul '23

Hi, I am seeing a segfault during tests (src/test-as, src/test-country and src/test-signature) for all Ubuntu LTS versions. This only occurs with the latest commit (d5ff39d) applied. Looking at the change, the node still seems to be used after dereference, but I am not fully sure of the way you have planned out the deletion, therefore I am not offering a patch/resolution. >From gdb I am seeing: Program received signal SIGSEGV, Segmentation fault. loc_network_tree_delete_nodes (tree=0x55555555a370) at src/network.c:1045 1045 tree->root->deleted = 0; Best regards, Valters Jansons

2 3

[PATCH] debian: Include bash-completion when packaging
by Valters Jansons 13 Jul '23

13 Jul '23

dh_missing for a standard `debuild` run on latest `master` commit complains about the bash-completion file created for location package. This file should be listed as for inclusion in location package accordingly, so that auto-built Debian package picks it up properly. --- Sorry about not including this in the previously provided patch. I was looking at the 0.9.16 version and the changes that were missing from that, and overlooked that more recent changes (after the tag) also had changes in the output file structure. Thank you for the continued support on the project! debian/location.install | 1 + 1 file changed, 1 insertion(+) diff --git a/debian/location.install b/debian/location.install index cde47cd..deb6c4d 100644 --- a/debian/location.install +++ b/debian/location.install @@ -1,4 +1,5 @@ usr/bin +usr/share/bash-completion/completions/location var/lib/location/database.db var/lib/location/signing-key.pem lib/systemd/system -- 2.41.0

1 0

[PATCH] debian: Housekeep location.install and changelog
by Valters Jansons 13 Jul '23

13 Jul '23

In order to run `debuild` for an easy Debian package locally, the debian/*.install files must list all build results. For version 0.9.16 there is a new pre-built database.db file that belongs with `location` package, and the `man8` pages have been moved to `man1`. Additionally, the changelog is added for version 0.9.16 as to maintain the centralized reference (inside this upstream repository). The author/timestamp is based on the 0.9.16 tag metadata tracked in Git. --- This commit simply aims to have this repository buildable directly. If the intention is not to maintain files in debian/ and instead rely on external/downstream packaging entirely, then the debian/ files should be removed from this repository enitrely as to avoid confusion. debian/changelog | 30 ++++++++++++++++++++++++++++++ debian/location.install | 3 ++- 2 files changed, 32 insertions(+), 1 deletion(-) diff --git a/debian/changelog b/debian/changelog index 87699a3..eff5c1d 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,33 @@ +libloc (0.9.16-1) unstable; urgency=medium + + [ Peter Müller ] + * location-importer.in: Conduct sanity checks per DROP list + * location-importer.in: Add new Amazon region codes + + [ Michael Tremer ] + * importer: Fix potential SQL command injection + * configure: Fix incorrect database path + * python: Export __version__ in location module + * writer: Add an empty string to the stringpool + * export: Fix generating file names for ipset output + * database: Ship a recent default database + * tests: Drop the test database and use the distributed one + * database: Correct error code on verification + * writer: Fix typo in signature variable + * writer: Assign correct file descriptor for private keys + * database: Fix check if a signature is set + * configure: Drop superfluous bracket + * configure: Bump version to 0.9.16 + + [ Petr Písař ] + * Move location manual from section 8 to section 1 + * Remove shebangs from Python modules + * Move location manual from section 8 to section 1 in location- + update.service + * Install Perl files to Perl vendor directory + + -- Michael Tremer <michael.tremer(a)ipfire.org> Sat, 29 Oct 2022 13:25:36 +0000 + libloc (0.9.15-1) unstable; urgency=medium [ Peter Müller ] diff --git a/debian/location.install b/debian/location.install index f9cb894..cde47cd 100644 --- a/debian/location.install +++ b/debian/location.install @@ -1,4 +1,5 @@ usr/bin +var/lib/location/database.db var/lib/location/signing-key.pem lib/systemd/system -usr/share/man/man8 +usr/share/man/man1 -- 2.41.0

2 1

2024

2023

2022

2021

2020

2019

2018

2017

Location