* [PATCH] location-importer.in: process unaligned IP ranges in RIR data files correctly @ 2021-03-29 20:24 Peter Müller 2021-03-29 20:27 ` Michael Tremer 0 siblings, 1 reply; 8+ messages in thread From: Peter Müller @ 2021-03-29 20:24 UTC (permalink / raw) To: location [-- Attachment #1: Type: text/plain, Size: 3371 bytes --] The IP range given in an inetnum object apparently not necessarily matches distinct subnet boundaries. As a result, the current attempt to calculate its CIDR mask resulted in faulty subnets not covering the entire IP range. This patch leaves the task of enumerating subnets to the ipaddress module itself, which handles things much more robust. Since the output may contain of several subnets, a list for the inetnum key is necessary as well as a loop over them when conducting the SQL statements. Fixes: #12595 Cc: Michael Tremer <michael.tremer(a)ipfire.org> Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- src/python/location-importer.in | 31 +++++++++++-------------------- 1 file changed, 11 insertions(+), 20 deletions(-) diff --git a/src/python/location-importer.in b/src/python/location-importer.in index 2506925..e2f201b 100644 --- a/src/python/location-importer.in +++ b/src/python/location-importer.in @@ -3,7 +3,7 @@ # # # libloc - A library to determine the location of someone on the Internet # # # -# Copyright (C) 2020 IPFire Development Team <info(a)ipfire.org> # +# Copyright (C) 2020-2021 IPFire Development Team <info(a)ipfire.org> # # # # This library is free software; you can redistribute it and/or # # modify it under the terms of the GNU Lesser General Public # @@ -604,18 +604,10 @@ class CLI(object): log.warning("Could not parse line: %s" % line) return - # Set prefix to default - prefix = 32 - - # Count number of addresses in this subnet - num_addresses = int(end_address) - int(start_address) - if num_addresses: - prefix -= math.log(num_addresses, 2) - - inetnum["inetnum"] = "%s/%.0f" % (start_address, prefix) + inetnum["inetnum"] = list(ipaddress.summarize_address_range(start_address, end_address)) elif key == "inet6num": - inetnum[key] = val + inetnum[key] = [ipaddress.ip_network(val, strict=False)] elif key == "country": inetnum[key] = val.upper() @@ -630,15 +622,14 @@ class CLI(object): (inetnum.get("inet6num") or inetnum.get("inetnum"))) return - network = ipaddress.ip_network(inetnum.get("inet6num") or inetnum.get("inetnum"), strict=False) - - if not self._check_parsed_network(network): - return - - self.db.execute("INSERT INTO _rirdata(network, country) \ - VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country = excluded.country", - "%s" % network, inetnum.get("country"), - ) + # Iterate through all networks enumerated from above, check them for plausibility and insert + # them into the database, if _check_parsed_network() succeeded + for single_network in inetnum.get("inet6num") or inetnum.get("inetnum"): + if self._check_parsed_network(single_network): + self.db.execute("INSERT INTO _rirdata(network, country) \ + VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country = excluded.country", + "%s" % single_network, inetnum.get("country"), + ) def _parse_org_block(self, block): org = {} -- 2.26.2 ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] location-importer.in: process unaligned IP ranges in RIR data files correctly 2021-03-29 20:24 [PATCH] location-importer.in: process unaligned IP ranges in RIR data files correctly Peter Müller @ 2021-03-29 20:27 ` Michael Tremer 2021-03-29 20:32 ` Peter Müller 0 siblings, 1 reply; 8+ messages in thread From: Michael Tremer @ 2021-03-29 20:27 UTC (permalink / raw) To: location [-- Attachment #1: Type: text/plain, Size: 3747 bytes --] Thank you for this. Are there any other things coming or can I go ahead and tag another version to roll these changes out into production? -Michael > On 29 Mar 2021, at 21:24, Peter Müller <peter.mueller(a)ipfire.org> wrote: > > The IP range given in an inetnum object apparently not necessarily > matches distinct subnet boundaries. As a result, the current attempt to > calculate its CIDR mask resulted in faulty subnets not covering the > entire IP range. > > This patch leaves the task of enumerating subnets to the ipaddress > module itself, which handles things much more robust. Since the output > may contain of several subnets, a list for the inetnum key is necessary > as well as a loop over them when conducting the SQL statements. > > Fixes: #12595 > > Cc: Michael Tremer <michael.tremer(a)ipfire.org> > Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> > --- > src/python/location-importer.in | 31 +++++++++++-------------------- > 1 file changed, 11 insertions(+), 20 deletions(-) > > diff --git a/src/python/location-importer.in b/src/python/location-importer.in > index 2506925..e2f201b 100644 > --- a/src/python/location-importer.in > +++ b/src/python/location-importer.in > @@ -3,7 +3,7 @@ > # # > # libloc - A library to determine the location of someone on the Internet # > # # > -# Copyright (C) 2020 IPFire Development Team <info(a)ipfire.org> # > +# Copyright (C) 2020-2021 IPFire Development Team <info(a)ipfire.org> # > # # > # This library is free software; you can redistribute it and/or # > # modify it under the terms of the GNU Lesser General Public # > @@ -604,18 +604,10 @@ class CLI(object): > log.warning("Could not parse line: %s" % line) > return > > - # Set prefix to default > - prefix = 32 > - > - # Count number of addresses in this subnet > - num_addresses = int(end_address) - int(start_address) > - if num_addresses: > - prefix -= math.log(num_addresses, 2) > - > - inetnum["inetnum"] = "%s/%.0f" % (start_address, prefix) > + inetnum["inetnum"] = list(ipaddress.summarize_address_range(start_address, end_address)) > > elif key == "inet6num": > - inetnum[key] = val > + inetnum[key] = [ipaddress.ip_network(val, strict=False)] > > elif key == "country": > inetnum[key] = val.upper() > @@ -630,15 +622,14 @@ class CLI(object): > (inetnum.get("inet6num") or inetnum.get("inetnum"))) > return > > - network = ipaddress.ip_network(inetnum.get("inet6num") or inetnum.get("inetnum"), strict=False) > - > - if not self._check_parsed_network(network): > - return > - > - self.db.execute("INSERT INTO _rirdata(network, country) \ > - VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country = excluded.country", > - "%s" % network, inetnum.get("country"), > - ) > + # Iterate through all networks enumerated from above, check them for plausibility and insert > + # them into the database, if _check_parsed_network() succeeded > + for single_network in inetnum.get("inet6num") or inetnum.get("inetnum"): > + if self._check_parsed_network(single_network): > + self.db.execute("INSERT INTO _rirdata(network, country) \ > + VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country = excluded.country", > + "%s" % single_network, inetnum.get("country"), > + ) > > def _parse_org_block(self, block): > org = {} > -- > 2.26.2 ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] location-importer.in: process unaligned IP ranges in RIR data files correctly 2021-03-29 20:27 ` Michael Tremer @ 2021-03-29 20:32 ` Peter Müller 2021-03-29 20:34 ` Peter Müller 0 siblings, 1 reply; 8+ messages in thread From: Peter Müller @ 2021-03-29 20:32 UTC (permalink / raw) To: location [-- Attachment #1: Type: text/plain, Size: 4020 bytes --] Hello Michael, you're welcome. Well, #11754 and #12594 would be the next issues on my list, but I have no working code for them, yet. Thanks, and best regards, Peter Müller > Thank you for this. > > Are there any other things coming or can I go ahead and tag another version to roll these changes out into production? > > -Michael > >> On 29 Mar 2021, at 21:24, Peter Müller <peter.mueller(a)ipfire.org> wrote: >> >> The IP range given in an inetnum object apparently not necessarily >> matches distinct subnet boundaries. As a result, the current attempt to >> calculate its CIDR mask resulted in faulty subnets not covering the >> entire IP range. >> >> This patch leaves the task of enumerating subnets to the ipaddress >> module itself, which handles things much more robust. Since the output >> may contain of several subnets, a list for the inetnum key is necessary >> as well as a loop over them when conducting the SQL statements. >> >> Fixes: #12595 >> >> Cc: Michael Tremer <michael.tremer(a)ipfire.org> >> Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> >> --- >> src/python/location-importer.in | 31 +++++++++++-------------------- >> 1 file changed, 11 insertions(+), 20 deletions(-) >> >> diff --git a/src/python/location-importer.in b/src/python/location-importer.in >> index 2506925..e2f201b 100644 >> --- a/src/python/location-importer.in >> +++ b/src/python/location-importer.in >> @@ -3,7 +3,7 @@ >> # # >> # libloc - A library to determine the location of someone on the Internet # >> # # >> -# Copyright (C) 2020 IPFire Development Team <info(a)ipfire.org> # >> +# Copyright (C) 2020-2021 IPFire Development Team <info(a)ipfire.org> # >> # # >> # This library is free software; you can redistribute it and/or # >> # modify it under the terms of the GNU Lesser General Public # >> @@ -604,18 +604,10 @@ class CLI(object): >> log.warning("Could not parse line: %s" % line) >> return >> >> - # Set prefix to default >> - prefix = 32 >> - >> - # Count number of addresses in this subnet >> - num_addresses = int(end_address) - int(start_address) >> - if num_addresses: >> - prefix -= math.log(num_addresses, 2) >> - >> - inetnum["inetnum"] = "%s/%.0f" % (start_address, prefix) >> + inetnum["inetnum"] = list(ipaddress.summarize_address_range(start_address, end_address)) >> >> elif key == "inet6num": >> - inetnum[key] = val >> + inetnum[key] = [ipaddress.ip_network(val, strict=False)] >> >> elif key == "country": >> inetnum[key] = val.upper() >> @@ -630,15 +622,14 @@ class CLI(object): >> (inetnum.get("inet6num") or inetnum.get("inetnum"))) >> return >> >> - network = ipaddress.ip_network(inetnum.get("inet6num") or inetnum.get("inetnum"), strict=False) >> - >> - if not self._check_parsed_network(network): >> - return >> - >> - self.db.execute("INSERT INTO _rirdata(network, country) \ >> - VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country = excluded.country", >> - "%s" % network, inetnum.get("country"), >> - ) >> + # Iterate through all networks enumerated from above, check them for plausibility and insert >> + # them into the database, if _check_parsed_network() succeeded >> + for single_network in inetnum.get("inet6num") or inetnum.get("inetnum"): >> + if self._check_parsed_network(single_network): >> + self.db.execute("INSERT INTO _rirdata(network, country) \ >> + VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country = excluded.country", >> + "%s" % single_network, inetnum.get("country"), >> + ) >> >> def _parse_org_block(self, block): >> org = {} >> -- >> 2.26.2 > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] location-importer.in: process unaligned IP ranges in RIR data files correctly 2021-03-29 20:32 ` Peter Müller @ 2021-03-29 20:34 ` Peter Müller 2021-03-29 20:40 ` Michael Tremer 0 siblings, 1 reply; 8+ messages in thread From: Peter Müller @ 2021-03-29 20:34 UTC (permalink / raw) To: location [-- Attachment #1: Type: text/plain, Size: 4225 bytes --] By the way: https://patchwork.ipfire.org/patch/3620/ is still waiting for a decision of yours. :-) > Hello Michael, > > you're welcome. > > Well, #11754 and #12594 would be the next issues on my list, but I have no working code for them, yet. > > Thanks, and best regards, > Peter Müller > > >> Thank you for this. >> >> Are there any other things coming or can I go ahead and tag another version to roll these changes out into production? >> >> -Michael >> >>> On 29 Mar 2021, at 21:24, Peter Müller <peter.mueller(a)ipfire.org> wrote: >>> >>> The IP range given in an inetnum object apparently not necessarily >>> matches distinct subnet boundaries. As a result, the current attempt to >>> calculate its CIDR mask resulted in faulty subnets not covering the >>> entire IP range. >>> >>> This patch leaves the task of enumerating subnets to the ipaddress >>> module itself, which handles things much more robust. Since the output >>> may contain of several subnets, a list for the inetnum key is necessary >>> as well as a loop over them when conducting the SQL statements. >>> >>> Fixes: #12595 >>> >>> Cc: Michael Tremer <michael.tremer(a)ipfire.org> >>> Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> >>> --- >>> src/python/location-importer.in | 31 +++++++++++-------------------- >>> 1 file changed, 11 insertions(+), 20 deletions(-) >>> >>> diff --git a/src/python/location-importer.in b/src/python/location-importer.in >>> index 2506925..e2f201b 100644 >>> --- a/src/python/location-importer.in >>> +++ b/src/python/location-importer.in >>> @@ -3,7 +3,7 @@ >>> # # >>> # libloc - A library to determine the location of someone on the Internet # >>> # # >>> -# Copyright (C) 2020 IPFire Development Team <info(a)ipfire.org> # >>> +# Copyright (C) 2020-2021 IPFire Development Team <info(a)ipfire.org> # >>> # # >>> # This library is free software; you can redistribute it and/or # >>> # modify it under the terms of the GNU Lesser General Public # >>> @@ -604,18 +604,10 @@ class CLI(object): >>> log.warning("Could not parse line: %s" % line) >>> return >>> >>> - # Set prefix to default >>> - prefix = 32 >>> - >>> - # Count number of addresses in this subnet >>> - num_addresses = int(end_address) - int(start_address) >>> - if num_addresses: >>> - prefix -= math.log(num_addresses, 2) >>> - >>> - inetnum["inetnum"] = "%s/%.0f" % (start_address, prefix) >>> + inetnum["inetnum"] = list(ipaddress.summarize_address_range(start_address, end_address)) >>> >>> elif key == "inet6num": >>> - inetnum[key] = val >>> + inetnum[key] = [ipaddress.ip_network(val, strict=False)] >>> >>> elif key == "country": >>> inetnum[key] = val.upper() >>> @@ -630,15 +622,14 @@ class CLI(object): >>> (inetnum.get("inet6num") or inetnum.get("inetnum"))) >>> return >>> >>> - network = ipaddress.ip_network(inetnum.get("inet6num") or inetnum.get("inetnum"), strict=False) >>> - >>> - if not self._check_parsed_network(network): >>> - return >>> - >>> - self.db.execute("INSERT INTO _rirdata(network, country) \ >>> - VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country = excluded.country", >>> - "%s" % network, inetnum.get("country"), >>> - ) >>> + # Iterate through all networks enumerated from above, check them for plausibility and insert >>> + # them into the database, if _check_parsed_network() succeeded >>> + for single_network in inetnum.get("inet6num") or inetnum.get("inetnum"): >>> + if self._check_parsed_network(single_network): >>> + self.db.execute("INSERT INTO _rirdata(network, country) \ >>> + VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country = excluded.country", >>> + "%s" % single_network, inetnum.get("country"), >>> + ) >>> >>> def _parse_org_block(self, block): >>> org = {} >>> -- >>> 2.26.2 >> ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] location-importer.in: process unaligned IP ranges in RIR data files correctly 2021-03-29 20:34 ` Peter Müller @ 2021-03-29 20:40 ` Michael Tremer 2021-03-30 15:49 ` Peter Müller 0 siblings, 1 reply; 8+ messages in thread From: Michael Tremer @ 2021-03-29 20:40 UTC (permalink / raw) To: location [-- Attachment #1: Type: text/plain, Size: 4638 bytes --] Hello, I was looking for this one, but could not find it. It doesn’t apply. Would you like to rebase this to master and submit it again? -Michael P.S. Still unsure whether I should wait or not :) > On 29 Mar 2021, at 21:34, Peter Müller <peter.mueller(a)ipfire.org> wrote: > > By the way: https://patchwork.ipfire.org/patch/3620/ is still waiting for a decision of yours. :-) > >> Hello Michael, >> >> you're welcome. >> >> Well, #11754 and #12594 would be the next issues on my list, but I have no working code for them, yet. >> >> Thanks, and best regards, >> Peter Müller >> >> >>> Thank you for this. >>> >>> Are there any other things coming or can I go ahead and tag another version to roll these changes out into production? >>> >>> -Michael >>> >>>> On 29 Mar 2021, at 21:24, Peter Müller <peter.mueller(a)ipfire.org> wrote: >>>> >>>> The IP range given in an inetnum object apparently not necessarily >>>> matches distinct subnet boundaries. As a result, the current attempt to >>>> calculate its CIDR mask resulted in faulty subnets not covering the >>>> entire IP range. >>>> >>>> This patch leaves the task of enumerating subnets to the ipaddress >>>> module itself, which handles things much more robust. Since the output >>>> may contain of several subnets, a list for the inetnum key is necessary >>>> as well as a loop over them when conducting the SQL statements. >>>> >>>> Fixes: #12595 >>>> >>>> Cc: Michael Tremer <michael.tremer(a)ipfire.org> >>>> Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> >>>> --- >>>> src/python/location-importer.in | 31 +++++++++++-------------------- >>>> 1 file changed, 11 insertions(+), 20 deletions(-) >>>> >>>> diff --git a/src/python/location-importer.in b/src/python/location-importer.in >>>> index 2506925..e2f201b 100644 >>>> --- a/src/python/location-importer.in >>>> +++ b/src/python/location-importer.in >>>> @@ -3,7 +3,7 @@ >>>> # # >>>> # libloc - A library to determine the location of someone on the Internet # >>>> # # >>>> -# Copyright (C) 2020 IPFire Development Team <info(a)ipfire.org> # >>>> +# Copyright (C) 2020-2021 IPFire Development Team <info(a)ipfire.org> # >>>> # # >>>> # This library is free software; you can redistribute it and/or # >>>> # modify it under the terms of the GNU Lesser General Public # >>>> @@ -604,18 +604,10 @@ class CLI(object): >>>> log.warning("Could not parse line: %s" % line) >>>> return >>>> >>>> - # Set prefix to default >>>> - prefix = 32 >>>> - >>>> - # Count number of addresses in this subnet >>>> - num_addresses = int(end_address) - int(start_address) >>>> - if num_addresses: >>>> - prefix -= math.log(num_addresses, 2) >>>> - >>>> - inetnum["inetnum"] = "%s/%.0f" % (start_address, prefix) >>>> + inetnum["inetnum"] = list(ipaddress.summarize_address_range(start_address, end_address)) >>>> >>>> elif key == "inet6num": >>>> - inetnum[key] = val >>>> + inetnum[key] = [ipaddress.ip_network(val, strict=False)] >>>> >>>> elif key == "country": >>>> inetnum[key] = val.upper() >>>> @@ -630,15 +622,14 @@ class CLI(object): >>>> (inetnum.get("inet6num") or inetnum.get("inetnum"))) >>>> return >>>> >>>> - network = ipaddress.ip_network(inetnum.get("inet6num") or inetnum.get("inetnum"), strict=False) >>>> - >>>> - if not self._check_parsed_network(network): >>>> - return >>>> - >>>> - self.db.execute("INSERT INTO _rirdata(network, country) \ >>>> - VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country = excluded.country", >>>> - "%s" % network, inetnum.get("country"), >>>> - ) >>>> + # Iterate through all networks enumerated from above, check them for plausibility and insert >>>> + # them into the database, if _check_parsed_network() succeeded >>>> + for single_network in inetnum.get("inet6num") or inetnum.get("inetnum"): >>>> + if self._check_parsed_network(single_network): >>>> + self.db.execute("INSERT INTO _rirdata(network, country) \ >>>> + VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country = excluded.country", >>>> + "%s" % single_network, inetnum.get("country"), >>>> + ) >>>> >>>> def _parse_org_block(self, block): >>>> org = {} >>>> -- >>>> 2.26.2 >>> ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] location-importer.in: process unaligned IP ranges in RIR data files correctly 2021-03-29 20:40 ` Michael Tremer @ 2021-03-30 15:49 ` Peter Müller 2021-04-01 9:38 ` Michael Tremer 0 siblings, 1 reply; 8+ messages in thread From: Peter Müller @ 2021-03-30 15:49 UTC (permalink / raw) To: location [-- Attachment #1: Type: text/plain, Size: 5258 bytes --] Hello Michael, thank you for your reply. Here you are: https://patchwork.ipfire.org/patch/4005/ Aside from that, there are still 8 patches left on https://patchwork.ipfire.org/project/location/list/. Perhaps you might want to check these as well before tagging a new release. #11754 and #12594 won't be ready that soon, so I am fine with a new libloc version after the patches mentioned above have been checked on whether they are ready for merging them. Thanks, and best regards, Peter Müller > Hello, > > I was looking for this one, but could not find it. > > It doesn’t apply. Would you like to rebase this to master and submit it again? > > -Michael > > P.S. Still unsure whether I should wait or not :) > >> On 29 Mar 2021, at 21:34, Peter Müller <peter.mueller(a)ipfire.org> wrote: >> >> By the way: https://patchwork.ipfire.org/patch/3620/ is still waiting for a decision of yours. :-) >> >>> Hello Michael, >>> >>> you're welcome. >>> >>> Well, #11754 and #12594 would be the next issues on my list, but I have no working code for them, yet. >>> >>> Thanks, and best regards, >>> Peter Müller >>> >>> >>>> Thank you for this. >>>> >>>> Are there any other things coming or can I go ahead and tag another version to roll these changes out into production? >>>> >>>> -Michael >>>> >>>>> On 29 Mar 2021, at 21:24, Peter Müller <peter.mueller(a)ipfire.org> wrote: >>>>> >>>>> The IP range given in an inetnum object apparently not necessarily >>>>> matches distinct subnet boundaries. As a result, the current attempt to >>>>> calculate its CIDR mask resulted in faulty subnets not covering the >>>>> entire IP range. >>>>> >>>>> This patch leaves the task of enumerating subnets to the ipaddress >>>>> module itself, which handles things much more robust. Since the output >>>>> may contain of several subnets, a list for the inetnum key is necessary >>>>> as well as a loop over them when conducting the SQL statements. >>>>> >>>>> Fixes: #12595 >>>>> >>>>> Cc: Michael Tremer <michael.tremer(a)ipfire.org> >>>>> Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> >>>>> --- >>>>> src/python/location-importer.in | 31 +++++++++++-------------------- >>>>> 1 file changed, 11 insertions(+), 20 deletions(-) >>>>> >>>>> diff --git a/src/python/location-importer.in b/src/python/location-importer.in >>>>> index 2506925..e2f201b 100644 >>>>> --- a/src/python/location-importer.in >>>>> +++ b/src/python/location-importer.in >>>>> @@ -3,7 +3,7 @@ >>>>> # # >>>>> # libloc - A library to determine the location of someone on the Internet # >>>>> # # >>>>> -# Copyright (C) 2020 IPFire Development Team <info(a)ipfire.org> # >>>>> +# Copyright (C) 2020-2021 IPFire Development Team <info(a)ipfire.org> # >>>>> # # >>>>> # This library is free software; you can redistribute it and/or # >>>>> # modify it under the terms of the GNU Lesser General Public # >>>>> @@ -604,18 +604,10 @@ class CLI(object): >>>>> log.warning("Could not parse line: %s" % line) >>>>> return >>>>> >>>>> - # Set prefix to default >>>>> - prefix = 32 >>>>> - >>>>> - # Count number of addresses in this subnet >>>>> - num_addresses = int(end_address) - int(start_address) >>>>> - if num_addresses: >>>>> - prefix -= math.log(num_addresses, 2) >>>>> - >>>>> - inetnum["inetnum"] = "%s/%.0f" % (start_address, prefix) >>>>> + inetnum["inetnum"] = list(ipaddress.summarize_address_range(start_address, end_address)) >>>>> >>>>> elif key == "inet6num": >>>>> - inetnum[key] = val >>>>> + inetnum[key] = [ipaddress.ip_network(val, strict=False)] >>>>> >>>>> elif key == "country": >>>>> inetnum[key] = val.upper() >>>>> @@ -630,15 +622,14 @@ class CLI(object): >>>>> (inetnum.get("inet6num") or inetnum.get("inetnum"))) >>>>> return >>>>> >>>>> - network = ipaddress.ip_network(inetnum.get("inet6num") or inetnum.get("inetnum"), strict=False) >>>>> - >>>>> - if not self._check_parsed_network(network): >>>>> - return >>>>> - >>>>> - self.db.execute("INSERT INTO _rirdata(network, country) \ >>>>> - VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country = excluded.country", >>>>> - "%s" % network, inetnum.get("country"), >>>>> - ) >>>>> + # Iterate through all networks enumerated from above, check them for plausibility and insert >>>>> + # them into the database, if _check_parsed_network() succeeded >>>>> + for single_network in inetnum.get("inet6num") or inetnum.get("inetnum"): >>>>> + if self._check_parsed_network(single_network): >>>>> + self.db.execute("INSERT INTO _rirdata(network, country) \ >>>>> + VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country = excluded.country", >>>>> + "%s" % single_network, inetnum.get("country"), >>>>> + ) >>>>> >>>>> def _parse_org_block(self, block): >>>>> org = {} >>>>> -- >>>>> 2.26.2 >>>> > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] location-importer.in: process unaligned IP ranges in RIR data files correctly 2021-03-30 15:49 ` Peter Müller @ 2021-04-01 9:38 ` Michael Tremer 2021-04-01 16:35 ` Peter Müller 0 siblings, 1 reply; 8+ messages in thread From: Michael Tremer @ 2021-04-01 9:38 UTC (permalink / raw) To: location [-- Attachment #1: Type: text/plain, Size: 5724 bytes --] Hello, This patch has been merged and pushed into production and it looks like we now have some networks split into many smaller ones. The file size of the database afterhasn’t changed though. -Michael > On 30 Mar 2021, at 16:49, Peter Müller <peter.mueller(a)ipfire.org> wrote: > > Hello Michael, > > thank you for your reply. > > Here you are: https://patchwork.ipfire.org/patch/4005/ > > Aside from that, there are still 8 patches left on https://patchwork.ipfire.org/project/location/list/. > Perhaps you might want to check these as well before tagging a new release. > > #11754 and #12594 won't be ready that soon, so I am fine with a new libloc version after the patches > mentioned above have been checked on whether they are ready for merging them. > > Thanks, and best regards, > Peter Müller > > >> Hello, >> >> I was looking for this one, but could not find it. >> >> It doesn’t apply. Would you like to rebase this to master and submit it again? >> >> -Michael >> >> P.S. Still unsure whether I should wait or not :) >> >>> On 29 Mar 2021, at 21:34, Peter Müller <peter.mueller(a)ipfire.org> wrote: >>> >>> By the way: https://patchwork.ipfire.org/patch/3620/ is still waiting for a decision of yours. :-) >>> >>>> Hello Michael, >>>> >>>> you're welcome. >>>> >>>> Well, #11754 and #12594 would be the next issues on my list, but I have no working code for them, yet. >>>> >>>> Thanks, and best regards, >>>> Peter Müller >>>> >>>> >>>>> Thank you for this. >>>>> >>>>> Are there any other things coming or can I go ahead and tag another version to roll these changes out into production? >>>>> >>>>> -Michael >>>>> >>>>>> On 29 Mar 2021, at 21:24, Peter Müller <peter.mueller(a)ipfire.org> wrote: >>>>>> >>>>>> The IP range given in an inetnum object apparently not necessarily >>>>>> matches distinct subnet boundaries. As a result, the current attempt to >>>>>> calculate its CIDR mask resulted in faulty subnets not covering the >>>>>> entire IP range. >>>>>> >>>>>> This patch leaves the task of enumerating subnets to the ipaddress >>>>>> module itself, which handles things much more robust. Since the output >>>>>> may contain of several subnets, a list for the inetnum key is necessary >>>>>> as well as a loop over them when conducting the SQL statements. >>>>>> >>>>>> Fixes: #12595 >>>>>> >>>>>> Cc: Michael Tremer <michael.tremer(a)ipfire.org> >>>>>> Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> >>>>>> --- >>>>>> src/python/location-importer.in | 31 +++++++++++-------------------- >>>>>> 1 file changed, 11 insertions(+), 20 deletions(-) >>>>>> >>>>>> diff --git a/src/python/location-importer.in b/src/python/location-importer.in >>>>>> index 2506925..e2f201b 100644 >>>>>> --- a/src/python/location-importer.in >>>>>> +++ b/src/python/location-importer.in >>>>>> @@ -3,7 +3,7 @@ >>>>>> # # >>>>>> # libloc - A library to determine the location of someone on the Internet # >>>>>> # # >>>>>> -# Copyright (C) 2020 IPFire Development Team <info(a)ipfire.org> # >>>>>> +# Copyright (C) 2020-2021 IPFire Development Team <info(a)ipfire.org> # >>>>>> # # >>>>>> # This library is free software; you can redistribute it and/or # >>>>>> # modify it under the terms of the GNU Lesser General Public # >>>>>> @@ -604,18 +604,10 @@ class CLI(object): >>>>>> log.warning("Could not parse line: %s" % line) >>>>>> return >>>>>> >>>>>> - # Set prefix to default >>>>>> - prefix = 32 >>>>>> - >>>>>> - # Count number of addresses in this subnet >>>>>> - num_addresses = int(end_address) - int(start_address) >>>>>> - if num_addresses: >>>>>> - prefix -= math.log(num_addresses, 2) >>>>>> - >>>>>> - inetnum["inetnum"] = "%s/%.0f" % (start_address, prefix) >>>>>> + inetnum["inetnum"] = list(ipaddress.summarize_address_range(start_address, end_address)) >>>>>> >>>>>> elif key == "inet6num": >>>>>> - inetnum[key] = val >>>>>> + inetnum[key] = [ipaddress.ip_network(val, strict=False)] >>>>>> >>>>>> elif key == "country": >>>>>> inetnum[key] = val.upper() >>>>>> @@ -630,15 +622,14 @@ class CLI(object): >>>>>> (inetnum.get("inet6num") or inetnum.get("inetnum"))) >>>>>> return >>>>>> >>>>>> - network = ipaddress.ip_network(inetnum.get("inet6num") or inetnum.get("inetnum"), strict=False) >>>>>> - >>>>>> - if not self._check_parsed_network(network): >>>>>> - return >>>>>> - >>>>>> - self.db.execute("INSERT INTO _rirdata(network, country) \ >>>>>> - VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country = excluded.country", >>>>>> - "%s" % network, inetnum.get("country"), >>>>>> - ) >>>>>> + # Iterate through all networks enumerated from above, check them for plausibility and insert >>>>>> + # them into the database, if _check_parsed_network() succeeded >>>>>> + for single_network in inetnum.get("inet6num") or inetnum.get("inetnum"): >>>>>> + if self._check_parsed_network(single_network): >>>>>> + self.db.execute("INSERT INTO _rirdata(network, country) \ >>>>>> + VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country = excluded.country", >>>>>> + "%s" % single_network, inetnum.get("country"), >>>>>> + ) >>>>>> >>>>>> def _parse_org_block(self, block): >>>>>> org = {} >>>>>> -- >>>>>> 2.26.2 >>>>> >> ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] location-importer.in: process unaligned IP ranges in RIR data files correctly 2021-04-01 9:38 ` Michael Tremer @ 2021-04-01 16:35 ` Peter Müller 0 siblings, 0 replies; 8+ messages in thread From: Peter Müller @ 2021-04-01 16:35 UTC (permalink / raw) To: location [-- Attachment #1: Type: text/plain, Size: 6057 bytes --] Hello Michael, seems to work as designed then. :-) I will let the Tor folks know about this so they can distribute new location information with their next release. Thanks, and best regards, Peter Müller > Hello, > > This patch has been merged and pushed into production and it looks like we now have some networks split into many smaller ones. > > The file size of the database afterhasn’t changed though. > > -Michael > >> On 30 Mar 2021, at 16:49, Peter Müller <peter.mueller(a)ipfire.org> wrote: >> >> Hello Michael, >> >> thank you for your reply. >> >> Here you are: https://patchwork.ipfire.org/patch/4005/ >> >> Aside from that, there are still 8 patches left on https://patchwork.ipfire.org/project/location/list/. >> Perhaps you might want to check these as well before tagging a new release. >> >> #11754 and #12594 won't be ready that soon, so I am fine with a new libloc version after the patches >> mentioned above have been checked on whether they are ready for merging them. >> >> Thanks, and best regards, >> Peter Müller >> >> >>> Hello, >>> >>> I was looking for this one, but could not find it. >>> >>> It doesn’t apply. Would you like to rebase this to master and submit it again? >>> >>> -Michael >>> >>> P.S. Still unsure whether I should wait or not :) >>> >>>> On 29 Mar 2021, at 21:34, Peter Müller <peter.mueller(a)ipfire.org> wrote: >>>> >>>> By the way: https://patchwork.ipfire.org/patch/3620/ is still waiting for a decision of yours. :-) >>>> >>>>> Hello Michael, >>>>> >>>>> you're welcome. >>>>> >>>>> Well, #11754 and #12594 would be the next issues on my list, but I have no working code for them, yet. >>>>> >>>>> Thanks, and best regards, >>>>> Peter Müller >>>>> >>>>> >>>>>> Thank you for this. >>>>>> >>>>>> Are there any other things coming or can I go ahead and tag another version to roll these changes out into production? >>>>>> >>>>>> -Michael >>>>>> >>>>>>> On 29 Mar 2021, at 21:24, Peter Müller <peter.mueller(a)ipfire.org> wrote: >>>>>>> >>>>>>> The IP range given in an inetnum object apparently not necessarily >>>>>>> matches distinct subnet boundaries. As a result, the current attempt to >>>>>>> calculate its CIDR mask resulted in faulty subnets not covering the >>>>>>> entire IP range. >>>>>>> >>>>>>> This patch leaves the task of enumerating subnets to the ipaddress >>>>>>> module itself, which handles things much more robust. Since the output >>>>>>> may contain of several subnets, a list for the inetnum key is necessary >>>>>>> as well as a loop over them when conducting the SQL statements. >>>>>>> >>>>>>> Fixes: #12595 >>>>>>> >>>>>>> Cc: Michael Tremer <michael.tremer(a)ipfire.org> >>>>>>> Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> >>>>>>> --- >>>>>>> src/python/location-importer.in | 31 +++++++++++-------------------- >>>>>>> 1 file changed, 11 insertions(+), 20 deletions(-) >>>>>>> >>>>>>> diff --git a/src/python/location-importer.in b/src/python/location-importer.in >>>>>>> index 2506925..e2f201b 100644 >>>>>>> --- a/src/python/location-importer.in >>>>>>> +++ b/src/python/location-importer.in >>>>>>> @@ -3,7 +3,7 @@ >>>>>>> # # >>>>>>> # libloc - A library to determine the location of someone on the Internet # >>>>>>> # # >>>>>>> -# Copyright (C) 2020 IPFire Development Team <info(a)ipfire.org> # >>>>>>> +# Copyright (C) 2020-2021 IPFire Development Team <info(a)ipfire.org> # >>>>>>> # # >>>>>>> # This library is free software; you can redistribute it and/or # >>>>>>> # modify it under the terms of the GNU Lesser General Public # >>>>>>> @@ -604,18 +604,10 @@ class CLI(object): >>>>>>> log.warning("Could not parse line: %s" % line) >>>>>>> return >>>>>>> >>>>>>> - # Set prefix to default >>>>>>> - prefix = 32 >>>>>>> - >>>>>>> - # Count number of addresses in this subnet >>>>>>> - num_addresses = int(end_address) - int(start_address) >>>>>>> - if num_addresses: >>>>>>> - prefix -= math.log(num_addresses, 2) >>>>>>> - >>>>>>> - inetnum["inetnum"] = "%s/%.0f" % (start_address, prefix) >>>>>>> + inetnum["inetnum"] = list(ipaddress.summarize_address_range(start_address, end_address)) >>>>>>> >>>>>>> elif key == "inet6num": >>>>>>> - inetnum[key] = val >>>>>>> + inetnum[key] = [ipaddress.ip_network(val, strict=False)] >>>>>>> >>>>>>> elif key == "country": >>>>>>> inetnum[key] = val.upper() >>>>>>> @@ -630,15 +622,14 @@ class CLI(object): >>>>>>> (inetnum.get("inet6num") or inetnum.get("inetnum"))) >>>>>>> return >>>>>>> >>>>>>> - network = ipaddress.ip_network(inetnum.get("inet6num") or inetnum.get("inetnum"), strict=False) >>>>>>> - >>>>>>> - if not self._check_parsed_network(network): >>>>>>> - return >>>>>>> - >>>>>>> - self.db.execute("INSERT INTO _rirdata(network, country) \ >>>>>>> - VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country = excluded.country", >>>>>>> - "%s" % network, inetnum.get("country"), >>>>>>> - ) >>>>>>> + # Iterate through all networks enumerated from above, check them for plausibility and insert >>>>>>> + # them into the database, if _check_parsed_network() succeeded >>>>>>> + for single_network in inetnum.get("inet6num") or inetnum.get("inetnum"): >>>>>>> + if self._check_parsed_network(single_network): >>>>>>> + self.db.execute("INSERT INTO _rirdata(network, country) \ >>>>>>> + VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country = excluded.country", >>>>>>> + "%s" % single_network, inetnum.get("country"), >>>>>>> + ) >>>>>>> >>>>>>> def _parse_org_block(self, block): >>>>>>> org = {} >>>>>>> -- >>>>>>> 2.26.2 >>>>>> >>> > ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2021-04-01 16:35 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-03-29 20:24 [PATCH] location-importer.in: process unaligned IP ranges in RIR data files correctly Peter Müller 2021-03-29 20:27 ` Michael Tremer 2021-03-29 20:32 ` Peter Müller 2021-03-29 20:34 ` Peter Müller 2021-03-29 20:40 ` Michael Tremer 2021-03-30 15:49 ` Peter Müller 2021-04-01 9:38 ` Michael Tremer 2021-04-01 16:35 ` Peter Müller
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox