From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: location@lists.ipfire.org Subject: Re: [PATCH] location-importer.in: process unaligned IP ranges in RIR data files correctly Date: Thu, 01 Apr 2021 10:38:39 +0100 Message-ID: <70E8778F-BA7D-434E-B67C-05D4D336C137@ipfire.org> In-Reply-To: <838ec212-fd84-c9f9-99c2-37c38409ec8e@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2819202870322198071==" List-Id: --===============2819202870322198071== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello, This patch has been merged and pushed into production and it looks like we no= w have some networks split into many smaller ones. The file size of the database afterhasn=E2=80=99t changed though. -Michael > On 30 Mar 2021, at 16:49, Peter M=C3=BCller wr= ote: >=20 > Hello Michael, >=20 > thank you for your reply. >=20 > Here you are: https://patchwork.ipfire.org/patch/4005/ >=20 > Aside from that, there are still 8 patches left on https://patchwork.ipfire= .org/project/location/list/. > Perhaps you might want to check these as well before tagging a new release. >=20 > #11754 and #12594 won't be ready that soon, so I am fine with a new libloc = version after the patches > mentioned above have been checked on whether they are ready for merging the= m. >=20 > Thanks, and best regards, > Peter M=C3=BCller >=20 >=20 >> Hello, >>=20 >> I was looking for this one, but could not find it. >>=20 >> It doesn=E2=80=99t apply. Would you like to rebase this to master and subm= it it again? >>=20 >> -Michael >>=20 >> P.S. Still unsure whether I should wait or not :) >>=20 >>> On 29 Mar 2021, at 21:34, Peter M=C3=BCller = wrote: >>>=20 >>> By the way: https://patchwork.ipfire.org/patch/3620/ is still waiting for= a decision of yours. :-) >>>=20 >>>> Hello Michael, >>>>=20 >>>> you're welcome. >>>>=20 >>>> Well, #11754 and #12594 would be the next issues on my list, but I have = no working code for them, yet. >>>>=20 >>>> Thanks, and best regards, >>>> Peter M=C3=BCller >>>>=20 >>>>=20 >>>>> Thank you for this. >>>>>=20 >>>>> Are there any other things coming or can I go ahead and tag another ver= sion to roll these changes out into production? >>>>>=20 >>>>> -Michael >>>>>=20 >>>>>> On 29 Mar 2021, at 21:24, Peter M=C3=BCller wrote: >>>>>>=20 >>>>>> The IP range given in an inetnum object apparently not necessarily >>>>>> matches distinct subnet boundaries. As a result, the current attempt to >>>>>> calculate its CIDR mask resulted in faulty subnets not covering the >>>>>> entire IP range. >>>>>>=20 >>>>>> This patch leaves the task of enumerating subnets to the ipaddress >>>>>> module itself, which handles things much more robust. Since the output >>>>>> may contain of several subnets, a list for the inetnum key is necessary >>>>>> as well as a loop over them when conducting the SQL statements. >>>>>>=20 >>>>>> Fixes: #12595 >>>>>>=20 >>>>>> Cc: Michael Tremer >>>>>> Signed-off-by: Peter M=C3=BCller >>>>>> --- >>>>>> src/python/location-importer.in | 31 +++++++++++-------------------- >>>>>> 1 file changed, 11 insertions(+), 20 deletions(-) >>>>>>=20 >>>>>> diff --git a/src/python/location-importer.in b/src/python/location-imp= orter.in >>>>>> index 2506925..e2f201b 100644 >>>>>> --- a/src/python/location-importer.in >>>>>> +++ b/src/python/location-importer.in >>>>>> @@ -3,7 +3,7 @@ >>>>>> # = # >>>>>> # libloc - A library to determine the location of someone on the Inter= net # >>>>>> # = # >>>>>> -# Copyright (C) 2020 IPFire Development Team = # >>>>>> +# Copyright (C) 2020-2021 IPFire Development Team = # >>>>>> # = # >>>>>> # This library is free software; you can redistribute it and/or = # >>>>>> # modify it under the terms of the GNU Lesser General Public = # >>>>>> @@ -604,18 +604,10 @@ class CLI(object): >>>>>> log.warning("Could not parse line: %s" % line) >>>>>> return >>>>>>=20 >>>>>> - # Set prefix to default >>>>>> - prefix =3D 32 >>>>>> - >>>>>> - # Count number of addresses in this subnet >>>>>> - num_addresses =3D int(end_address) - int(start_address) >>>>>> - if num_addresses: >>>>>> - prefix -=3D math.log(num_addresses, 2) >>>>>> - >>>>>> - inetnum["inetnum"] =3D "%s/%.0f" % (start_address, prefix) >>>>>> + inetnum["inetnum"] =3D list(ipaddress.summarize_address_range(sta= rt_address, end_address)) >>>>>>=20 >>>>>> elif key =3D=3D "inet6num": >>>>>> - inetnum[key] =3D val >>>>>> + inetnum[key] =3D [ipaddress.ip_network(val, strict=3DFalse)] >>>>>>=20 >>>>>> elif key =3D=3D "country": >>>>>> inetnum[key] =3D val.upper() >>>>>> @@ -630,15 +622,14 @@ class CLI(object): >>>>>> (inetnum.get("inet6num") or inetnum.get("inetnum"))) >>>>>> return >>>>>>=20 >>>>>> - network =3D ipaddress.ip_network(inetnum.get("inet6num") or inetnum= .get("inetnum"), strict=3DFalse) >>>>>> - >>>>>> - if not self._check_parsed_network(network): >>>>>> - return >>>>>> - >>>>>> - self.db.execute("INSERT INTO _rirdata(network, country) \ >>>>>> - VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country =3D exc= luded.country", >>>>>> - "%s" % network, inetnum.get("country"), >>>>>> - ) >>>>>> + # Iterate through all networks enumerated from above, check them fo= r plausibility and insert >>>>>> + # them into the database, if _check_parsed_network() succeeded >>>>>> + for single_network in inetnum.get("inet6num") or inetnum.get("inetn= um"): >>>>>> + if self._check_parsed_network(single_network): >>>>>> + self.db.execute("INSERT INTO _rirdata(network, country) \ >>>>>> + VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country =3D e= xcluded.country", >>>>>> + "%s" % single_network, inetnum.get("country"), >>>>>> + ) >>>>>>=20 >>>>>> def _parse_org_block(self, block): >>>>>> org =3D {} >>>>>> --=20 >>>>>> 2.26.2 >>>>>=20 >>=20 --===============2819202870322198071==--