From mboxrd@z Thu Jan  1 00:00:00 1970
From: Peter =?utf-8?q?M=C3=BCller?= <peter.mueller@ipfire.org>
To: location@lists.ipfire.org
Subject: Re: [PATCH] location-importer.in: process unaligned IP ranges in RIR
 data files correctly
Date: Mon, 29 Mar 2021 22:32:56 +0200
Message-ID: <4abe40d0-7657-36bf-d0c6-6e2ac7e6eb3f@ipfire.org>
In-Reply-To: <55002AB9-77BA-43FF-895F-36CACEDE5368@ipfire.org>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============0219321478950258763=="
List-Id: <location.lists.ipfire.org>

--===============0219321478950258763==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

Hello Michael,

you're welcome.

Well, #11754 and #12594 would be the next issues on my list, but I have no wo=
rking code for them, yet.

Thanks, and best regards,
Peter M=C3=BCller


> Thank you for this.
>=20
> Are there any other things coming or can I go ahead and tag another version=
 to roll these changes out into production?
>=20
> -Michael
>=20
>> On 29 Mar 2021, at 21:24, Peter M=C3=BCller <peter.mueller(a)ipfire.org> w=
rote:
>>
>> The IP range given in an inetnum object apparently not necessarily
>> matches distinct subnet boundaries. As a result, the current attempt to
>> calculate its CIDR mask resulted in faulty subnets not covering the
>> entire IP range.
>>
>> This patch leaves the task of enumerating subnets to the ipaddress
>> module itself, which handles things much more robust. Since the output
>> may contain of several subnets, a list for the inetnum key is necessary
>> as well as a loop over them when conducting the SQL statements.
>>
>> Fixes: #12595
>>
>> Cc: Michael Tremer <michael.tremer(a)ipfire.org>
>> Signed-off-by: Peter M=C3=BCller <peter.mueller(a)ipfire.org>
>> ---
>> src/python/location-importer.in | 31 +++++++++++--------------------
>> 1 file changed, 11 insertions(+), 20 deletions(-)
>>
>> diff --git a/src/python/location-importer.in b/src/python/location-importe=
r.in
>> index 2506925..e2f201b 100644
>> --- a/src/python/location-importer.in
>> +++ b/src/python/location-importer.in
>> @@ -3,7 +3,7 @@
>> #                                                                         =
    #
>> # libloc - A library to determine the location of someone on the Internet =
    #
>> #                                                                         =
    #
>> -# Copyright (C) 2020 IPFire Development Team <info(a)ipfire.org>         =
       #
>> +# Copyright (C) 2020-2021 IPFire Development Team <info(a)ipfire.org>    =
       #
>> #                                                                         =
    #
>> # This library is free software; you can redistribute it and/or           =
    #
>> # modify it under the terms of the GNU Lesser General Public              =
    #
>> @@ -604,18 +604,10 @@ class CLI(object):
>> 					log.warning("Could not parse line: %s" % line)
>> 					return
>>
>> -				# Set prefix to default
>> -				prefix =3D 32
>> -
>> -				# Count number of addresses in this subnet
>> -				num_addresses =3D int(end_address) - int(start_address)
>> -				if num_addresses:
>> -					prefix -=3D math.log(num_addresses, 2)
>> -
>> -				inetnum["inetnum"] =3D "%s/%.0f" % (start_address, prefix)
>> +				inetnum["inetnum"] =3D list(ipaddress.summarize_address_range(start_a=
ddress, end_address))
>>
>> 			elif key =3D=3D "inet6num":
>> -				inetnum[key] =3D val
>> +				inetnum[key] =3D [ipaddress.ip_network(val, strict=3DFalse)]
>>
>> 			elif key =3D=3D "country":
>> 				inetnum[key] =3D val.upper()
>> @@ -630,15 +622,14 @@ class CLI(object):
>> 				(inetnum.get("inet6num") or inetnum.get("inetnum")))
>> 			return
>>
>> -		network =3D ipaddress.ip_network(inetnum.get("inet6num") or inetnum.get=
("inetnum"), strict=3DFalse)
>> -
>> -		if not self._check_parsed_network(network):
>> -			return
>> -
>> -		self.db.execute("INSERT INTO _rirdata(network, country) \
>> -			VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country =3D exclude=
d.country",
>> -			"%s" % network, inetnum.get("country"),
>> -		)
>> +		# Iterate through all networks enumerated from above, check them for pl=
ausibility and insert
>> +		# them into the database, if _check_parsed_network() succeeded
>> +		for single_network in inetnum.get("inet6num") or inetnum.get("inetnum"):
>> +			if self._check_parsed_network(single_network):
>> +				self.db.execute("INSERT INTO _rirdata(network, country) \
>> +					VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country =3D exclu=
ded.country",
>> +					"%s" % single_network, inetnum.get("country"),
>> +				)
>>
>> 	def _parse_org_block(self, block):
>> 		org =3D {}
>> --=20
>> 2.26.2
>=20

--===============0219321478950258763==--