From mboxrd@z Thu Jan 1 00:00:00 1970 From: Peter =?utf-8?q?M=C3=BCller?= To: location@lists.ipfire.org Subject: Re: [PATCH] location-importer.in: process unaligned IP ranges in RIR data files correctly Date: Mon, 29 Mar 2021 22:34:06 +0200 Message-ID: In-Reply-To: <4abe40d0-7657-36bf-d0c6-6e2ac7e6eb3f@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============4800695057039971298==" List-Id: --===============4800695057039971298== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable By the way: https://patchwork.ipfire.org/patch/3620/ is still waiting for a d= ecision of yours. :-) > Hello Michael, >=20 > you're welcome. >=20 > Well, #11754 and #12594 would be the next issues on my list, but I have no = working code for them, yet. >=20 > Thanks, and best regards, > Peter M=C3=BCller >=20 >=20 >> Thank you for this. >> >> Are there any other things coming or can I go ahead and tag another versio= n to roll these changes out into production? >> >> -Michael >> >>> On 29 Mar 2021, at 21:24, Peter M=C3=BCller = wrote: >>> >>> The IP range given in an inetnum object apparently not necessarily >>> matches distinct subnet boundaries. As a result, the current attempt to >>> calculate its CIDR mask resulted in faulty subnets not covering the >>> entire IP range. >>> >>> This patch leaves the task of enumerating subnets to the ipaddress >>> module itself, which handles things much more robust. Since the output >>> may contain of several subnets, a list for the inetnum key is necessary >>> as well as a loop over them when conducting the SQL statements. >>> >>> Fixes: #12595 >>> >>> Cc: Michael Tremer >>> Signed-off-by: Peter M=C3=BCller >>> --- >>> src/python/location-importer.in | 31 +++++++++++-------------------- >>> 1 file changed, 11 insertions(+), 20 deletions(-) >>> >>> diff --git a/src/python/location-importer.in b/src/python/location-import= er.in >>> index 2506925..e2f201b 100644 >>> --- a/src/python/location-importer.in >>> +++ b/src/python/location-importer.in >>> @@ -3,7 +3,7 @@ >>> # = # >>> # libloc - A library to determine the location of someone on the Internet= # >>> # = # >>> -# Copyright (C) 2020 IPFire Development Team = # >>> +# Copyright (C) 2020-2021 IPFire Development Team = # >>> # = # >>> # This library is free software; you can redistribute it and/or = # >>> # modify it under the terms of the GNU Lesser General Public = # >>> @@ -604,18 +604,10 @@ class CLI(object): >>> log.warning("Could not parse line: %s" % line) >>> return >>> >>> - # Set prefix to default >>> - prefix =3D 32 >>> - >>> - # Count number of addresses in this subnet >>> - num_addresses =3D int(end_address) - int(start_address) >>> - if num_addresses: >>> - prefix -=3D math.log(num_addresses, 2) >>> - >>> - inetnum["inetnum"] =3D "%s/%.0f" % (start_address, prefix) >>> + inetnum["inetnum"] =3D list(ipaddress.summarize_address_range(start_= address, end_address)) >>> >>> elif key =3D=3D "inet6num": >>> - inetnum[key] =3D val >>> + inetnum[key] =3D [ipaddress.ip_network(val, strict=3DFalse)] >>> >>> elif key =3D=3D "country": >>> inetnum[key] =3D val.upper() >>> @@ -630,15 +622,14 @@ class CLI(object): >>> (inetnum.get("inet6num") or inetnum.get("inetnum"))) >>> return >>> >>> - network =3D ipaddress.ip_network(inetnum.get("inet6num") or inetnum.ge= t("inetnum"), strict=3DFalse) >>> - >>> - if not self._check_parsed_network(network): >>> - return >>> - >>> - self.db.execute("INSERT INTO _rirdata(network, country) \ >>> - VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country =3D exclud= ed.country", >>> - "%s" % network, inetnum.get("country"), >>> - ) >>> + # Iterate through all networks enumerated from above, check them for p= lausibility and insert >>> + # them into the database, if _check_parsed_network() succeeded >>> + for single_network in inetnum.get("inet6num") or inetnum.get("inetnum"= ): >>> + if self._check_parsed_network(single_network): >>> + self.db.execute("INSERT INTO _rirdata(network, country) \ >>> + VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country =3D excl= uded.country", >>> + "%s" % single_network, inetnum.get("country"), >>> + ) >>> >>> def _parse_org_block(self, block): >>> org =3D {} >>> --=20 >>> 2.26.2 >> --===============4800695057039971298==--