[PATCH] importer: Filter non-global geofeed and RIR umbrella networks

[PATCH] importer: Filter non-global geofeed and RIR umbrella networks

[Spencer Norton]

Documentation and special-purpose ranges from RFC 5737 (192.0.2.0/24,
198.51.100.0/24, 203.0.113.0/24), RFC 3849 (2001:db8::/32), RFC 5180
(2001:2::/48), RFC 4843 (2001:10::/28), and RFC 3056 (2002::/16) are
currently exported with country: US, and RFC 4380 Teredo (2001::/32) is
exported with aut-num: 6939:

1. _fetch_geofeed() and handle_update_overrides() did not pass parsed
   prefixes through _check_parsed_network(), allowing non-global rows in
   unrestricted geofeeds (such as https://geofeed.constant.com) into
   geofeed_networks. Also reject multicast and reserved ranges in
   _check_parsed_network(), as Python's ipaddress.is_global returns True
   for 224.0.0.0/4 and ff00::/8.
2. handle_write() selected all rows from geofeed_networks into the
   candidate network table n without joining network_geofeeds, emitting
   258 out-of-bailiwick prefixes with no attributes into the database.
3. APNIC publishes top-level IANA pool placeholders (netname: APNIC-AP
   and APNIC-AP-ERX) with country: AU (for APNIC's Brisbane office) across
   its /8 and /12-/23 blocks, which otherwise cover 203.0.113.0/24
   (via 202.0.0.0/7) and 2001:db8::/32 (via 2001:c00::/22) as AU once the
   geofeed entries are removed. Skip APNIC-AP and APNIC-AP-ERX in
   _parse_inetnum_block() alongside the existing RIR umbrella netnames.
4. Purge IPv6 special-purpose ranges (2001::/23, 2001:db8::/32, 3fff::/20)
   from BGP announcements in handle_update_announcements() alongside the
   existing IPv4 test/documentation rules.

Fixes: #13934 - ipfire location db erroneously marks documentation ranges within rfc3849 and rfc5737 belonging to US
Signed-off-by: Spencer Norton <spencer.norton@norvi.tech>
---
 src/scripts/location-importer.in | 30 ++++++++++++++++++++++++++++--
 1 file changed, 28 insertions(+), 2 deletions(-)

diff --git a/src/scripts/location-importer.in b/src/scripts/location-importer.in
index 0298f4f..1a02996 100644
--- a/src/scripts/location-importer.in
+++ b/src/scripts/location-importer.in
@@ -426,9 +426,19 @@ class CLI(object):
 			UNION
 
 			SELECT
-				network
+				geofeed_networks.network
 			FROM
+				geofeeds
+			JOIN
+				network_geofeeds networks
+			ON
+				geofeeds.url = networks.url
+			JOIN
 				geofeed_networks
+			ON
+				geofeeds.id = geofeed_networks.geofeed_id
+			AND
+				networks.network >>= geofeed_networks.network
 		""")
 
 		# Create an index to search through networks faster
@@ -1195,6 +1205,11 @@ class CLI(object):
 			log.debug("Skipping unspecified network: %s" % network)
 			return False
 
+		# Ignore multicast and reserved networks
+		elif network.is_multicast or network.is_reserved:
+			log.debug("Skipping multicast or reserved network: %s" % network)
+			return False
+
 		# IPv6
 		if network.version == 6:
 			if network.prefixlen < 10:
@@ -1319,7 +1334,7 @@ class CLI(object):
 			# Filter any inetnum records which are only referring to IP space
 			# not managed by that specific RIR...
 			if key == "netname":
-				if re.match(r"^(ERX-NETBLOCK|(AFRINIC|ARIN|LACNIC|RIPE)-CIDR-BLOCK|IANA-NETBLOCK-\d{1,3}|NON-RIPE-NCC-MANAGED-ADDRESS-BLOCK|STUB-[\d-]{3,}SLASH\d{1,2})", val.strip()):
+				if re.match(r"^(APNIC-AP(-ERX)?$|ERX-NETBLOCK|(AFRINIC|ARIN|LACNIC|RIPE)-CIDR-BLOCK|IANA-NETBLOCK-\d{1,3}|NON-RIPE-NCC-MANAGED-ADDRESS-BLOCK|STUB-[\d-]{3,}SLASH\d{1,2})", val.strip()):
 					log.debug("Skipping record indicating historic/orphaned data: %s" % val.strip())
 					return
 
@@ -1617,6 +1632,9 @@ class CLI(object):
 				DELETE FROM announcements WHERE family(network) = 4 AND network <<= '198.18.0.0/15';
 				DELETE FROM announcements WHERE family(network) = 4 AND network <<= '198.51.100.0/24';
 				DELETE FROM announcements WHERE family(network) = 4 AND network <<= '203.0.113.0/24';
+				DELETE FROM announcements WHERE family(network) = 6 AND network <<= '2001::/23';
+				DELETE FROM announcements WHERE family(network) = 6 AND network <<= '2001:db8::/32';
+				DELETE FROM announcements WHERE family(network) = 6 AND network <<= '3fff::/20';
 
 				-- DELETE CGNAT address space (RFC 6598)
 				DELETE FROM announcements WHERE family(network) = 4 AND network <<= '100.64.0.0/10';
@@ -1944,6 +1962,10 @@ class CLI(object):
 								log.debug("Could not parse network: %s" % network)
 								continue
 
+							# Sanitize parsed networks...
+							if not self._check_parsed_network(network):
+								continue
+
 							# Strip any excess whitespace from country codes
 							country = country.strip()
 
@@ -2038,6 +2060,10 @@ class CLI(object):
 								log.warning("Skipping %s: You cannot overwrite default" % network)
 								continue
 
+							# Sanitize parsed networks...
+							if not self._check_parsed_network(network):
+								continue
+
 							self.db.execute("""
 								INSERT INTO
 									network_overrides
-- 
2.50.1

Re: [PATCH] importer: Filter non-global geofeed and RIR umbrella networks

[Michael Tremer]

Hello Spencer,

thank you very much for this patch, and for the very thorough analysis. It has
been merged into the master branch:

  https://git.ipfire.org/?p=location/libloc.git;a=commit;h=6274fb37b8d63c9820720bb613418d8bc8a265ee

I have changed one thing on top of it, which is the fourth point: 2001::/23
has been set aside for protocol assignments, but not all of them are
unroutable. Some are being announced on purpose, like 2001:4:112::/48 of
AS112, which would have lost its AS number, and also 2001:3::/32 (AMT),
2001:20::/28 (ORCHIDv2) and 2001:30::/28 (Drone Remote ID). Therefore, we now
only drop Teredo (2001::/32), the benchmarking range (2001:2::/48) and ORCHID
(2001:10::/28) out of that block:

  https://git.ipfire.org/?p=location/libloc.git;a=commit;h=b8dd752c6fb89875b057fec5c832d3cc668f8de4

The changes will be in the database once the importer has run with them, and
I will close bug #13934 after that.

Best,
-Michael

RE: [PATCH] importer: Filter non-global geofeed and RIR umbrella networks

[Spencer Norton]

[-- Attachment #1: Type: text/plain, Size: 2062 bytes --]

Hi Michael,

Thank you for reviewing and merging the patch,  good catch on narrowing the 2001::/23 announcement filter in b8dd752c.

Preserving the routed IETF protocol assignments inside 2001::/23 (especially AS112 at 2001:4:112::/48, along with AMT, ORCHIDv2, and Drone Remote ID) while dropping Teredo (2001::/32), benchmarking (2001:2::/48), and deprecated ORCHIDv1 (2001:10::/28) makes complete sense.

Got one more I’m submitting a patch for + 2 that I’ve confirmed have already been resolved

  *
#13971<https://bugzilla.ipfire.org/show_bug.cgi?id=13971> Recommend closure
  *
#13615<https://bugzilla.ipfire.org/show_bug.cgi?id=13615> Recommend closure


  *
#12400<https://bugzilla.ipfire.org/show_bug.cgi?id=12400> been diving into this one with my Mac, will comment and submit if / when I have a patch.

Please let me know if there are any other tickets / categories / queues that I can assist with,

Spencer

Best regards,
Spencer
> On Oct 20, 2026, at 13:18 UTC, Michael Tremer <michael.tremer@ipfire.org> wrote:
>
> Hello Spencer,
>
> thank you very much for this patch, and for the very thorough analysis. It has
> been merged into the master branch:
>
>   https://git.ipfire.org/?p=location/libloc.git;a=commit;h=6274fb37b8d63c9820720bb613418d8bc8a265ee
>
> I have changed one thing on top of it, which is the fourth point: 2001::/23
> has been set aside for protocol assignments, but not all of them are
> unroutable. Some are being announced on purpose, like 2001:4:112::/48 of
> AS112, which would have lost its AS number, and also 2001:3::/32 (AMT),
> 2001:20::/28 (ORCHIDv2) and 2001:30::/28 (Drone Remote ID). Therefore, we now
> only drop Teredo (2001::/32), the benchmarking range (2001:2::/48) and ORCHID
> (2001:10::/28) out of that block:
>
>   https://git.ipfire.org/?p=location/libloc.git;a=commit;h=b8dd752c6fb89875b057fec5c832d3cc668f8de4
>
> The changes will be in the database once the importer has run with them, and
> I will close bug #13934 after that.
>
> Best,
> -Michael

[-- Attachment #2: Type: text/html, Size: 6691 bytes --]