From mboxrd@z Thu Jan 1 00:00:00 1970 From: Peter =?utf-8?q?M=C3=BCller?= To: location@lists.ipfire.org Subject: Re: [PATCH] location-importer.in: process unaligned IP ranges in RIR data files correctly Date: Tue, 30 Mar 2021 17:49:28 +0200 Message-ID: <838ec212-fd84-c9f9-99c2-37c38409ec8e@ipfire.org> In-Reply-To: <828D77D2-DD20-4A40-8D7F-F711129E48EA@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============8041175035297847569==" List-Id: --===============8041175035297847569== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello Michael, thank you for your reply. Here you are: https://patchwork.ipfire.org/patch/4005/ Aside from that, there are still 8 patches left on https://patchwork.ipfire.o= rg/project/location/list/. Perhaps you might want to check these as well before tagging a new release. #11754 and #12594 won't be ready that soon, so I am fine with a new libloc ve= rsion after the patches mentioned above have been checked on whether they are ready for merging them. Thanks, and best regards, Peter M=C3=BCller > Hello, >=20 > I was looking for this one, but could not find it. >=20 > It doesn=E2=80=99t apply. Would you like to rebase this to master and submi= t it again? >=20 > -Michael >=20 > P.S. Still unsure whether I should wait or not :) >=20 >> On 29 Mar 2021, at 21:34, Peter M=C3=BCller w= rote: >> >> By the way: https://patchwork.ipfire.org/patch/3620/ is still waiting for = a decision of yours. :-) >> >>> Hello Michael, >>> >>> you're welcome. >>> >>> Well, #11754 and #12594 would be the next issues on my list, but I have n= o working code for them, yet. >>> >>> Thanks, and best regards, >>> Peter M=C3=BCller >>> >>> >>>> Thank you for this. >>>> >>>> Are there any other things coming or can I go ahead and tag another vers= ion to roll these changes out into production? >>>> >>>> -Michael >>>> >>>>> On 29 Mar 2021, at 21:24, Peter M=C3=BCller wrote: >>>>> >>>>> The IP range given in an inetnum object apparently not necessarily >>>>> matches distinct subnet boundaries. As a result, the current attempt to >>>>> calculate its CIDR mask resulted in faulty subnets not covering the >>>>> entire IP range. >>>>> >>>>> This patch leaves the task of enumerating subnets to the ipaddress >>>>> module itself, which handles things much more robust. Since the output >>>>> may contain of several subnets, a list for the inetnum key is necessary >>>>> as well as a loop over them when conducting the SQL statements. >>>>> >>>>> Fixes: #12595 >>>>> >>>>> Cc: Michael Tremer >>>>> Signed-off-by: Peter M=C3=BCller >>>>> --- >>>>> src/python/location-importer.in | 31 +++++++++++-------------------- >>>>> 1 file changed, 11 insertions(+), 20 deletions(-) >>>>> >>>>> diff --git a/src/python/location-importer.in b/src/python/location-impo= rter.in >>>>> index 2506925..e2f201b 100644 >>>>> --- a/src/python/location-importer.in >>>>> +++ b/src/python/location-importer.in >>>>> @@ -3,7 +3,7 @@ >>>>> # = # >>>>> # libloc - A library to determine the location of someone on the Intern= et # >>>>> # = # >>>>> -# Copyright (C) 2020 IPFire Development Team = # >>>>> +# Copyright (C) 2020-2021 IPFire Development Team = # >>>>> # = # >>>>> # This library is free software; you can redistribute it and/or = # >>>>> # modify it under the terms of the GNU Lesser General Public = # >>>>> @@ -604,18 +604,10 @@ class CLI(object): >>>>> log.warning("Could not parse line: %s" % line) >>>>> return >>>>> >>>>> - # Set prefix to default >>>>> - prefix =3D 32 >>>>> - >>>>> - # Count number of addresses in this subnet >>>>> - num_addresses =3D int(end_address) - int(start_address) >>>>> - if num_addresses: >>>>> - prefix -=3D math.log(num_addresses, 2) >>>>> - >>>>> - inetnum["inetnum"] =3D "%s/%.0f" % (start_address, prefix) >>>>> + inetnum["inetnum"] =3D list(ipaddress.summarize_address_range(star= t_address, end_address)) >>>>> >>>>> elif key =3D=3D "inet6num": >>>>> - inetnum[key] =3D val >>>>> + inetnum[key] =3D [ipaddress.ip_network(val, strict=3DFalse)] >>>>> >>>>> elif key =3D=3D "country": >>>>> inetnum[key] =3D val.upper() >>>>> @@ -630,15 +622,14 @@ class CLI(object): >>>>> (inetnum.get("inet6num") or inetnum.get("inetnum"))) >>>>> return >>>>> >>>>> - network =3D ipaddress.ip_network(inetnum.get("inet6num") or inetnum.= get("inetnum"), strict=3DFalse) >>>>> - >>>>> - if not self._check_parsed_network(network): >>>>> - return >>>>> - >>>>> - self.db.execute("INSERT INTO _rirdata(network, country) \ >>>>> - VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country =3D excl= uded.country", >>>>> - "%s" % network, inetnum.get("country"), >>>>> - ) >>>>> + # Iterate through all networks enumerated from above, check them for= plausibility and insert >>>>> + # them into the database, if _check_parsed_network() succeeded >>>>> + for single_network in inetnum.get("inet6num") or inetnum.get("inetnu= m"): >>>>> + if self._check_parsed_network(single_network): >>>>> + self.db.execute("INSERT INTO _rirdata(network, country) \ >>>>> + VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country =3D ex= cluded.country", >>>>> + "%s" % single_network, inetnum.get("country"), >>>>> + ) >>>>> >>>>> def _parse_org_block(self, block): >>>>> org =3D {} >>>>> --=20 >>>>> 2.26.2 >>>> >=20 --===============8041175035297847569==--