From mboxrd@z Thu Jan 1 00:00:00 1970 From: Peter =?utf-8?q?M=C3=BCller?= To: location@lists.ipfire.org Subject: Re: [PATCH] location-importer.in: Attempt to provide meaningful AS names if organisation handles are missing Date: Thu, 05 Aug 2021 20:49:55 +0200 Message-ID: In-Reply-To: <2F401D87-C58F-45CE-8E94-CEA624592E48@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============4920636776348386419==" List-Id: --===============4920636776348386419== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi Michael, > Hey, >=20 >> On 21 Jul 2021, at 22:23, Peter M=C3=BCller w= rote: >> >> Hello Michael, >> >> thanks for your reply. >> >> I would leave this up to you - some data-related issues are still on my li= st, but I >> am unfortunately really short on spare time at the moment. So I guess we s= hould get >> in production what we have at hand... :-) >=20 > Since we do not make too much use of this data, yet, I would like to leave = another release for a little bit and wait until the rest is coming in. thought about this again, I think we could apply this patch to libloc running= on location01, as we probably won't release a next version of libloc within the next few wee= ks - primarily due to me lacking spare time. :-/ Albeit it is hacky, I think it would be worth it. What do you think? > I generally like the release early and often thing. We should keep doing th= at. ACK. Thanks, and best regards, Peter M=C3=BCller >=20 >> By the way: Thank you for fixing location01, SSH login works fine for me a= gain. :-) >=20 > -Michael >=20 >> >> Thanks, and best regards, >> Peter M=C3=BCller >> >> >>> Thank you. Merged. >>> >>> Do we plan a release because of this soon? >>> >>>> On 19 Jul 2021, at 22:34, Peter M=C3=BCller = wrote: >>>> >>>> A decent amount of autnum objects - especially, but not exclusively in >>>> the APNIC sector - does not contain a link to an organisation handle. >>>> >>>> In such cases, this patch is going to use the first description line of >>>> the atunum object in question (if available) as a string for its name. >>>> The overwhelming majority of affected ASNs contains a valuable >>>> information there, so this is almost as good as having an organisation >>>> handle linked to it. >>>> >>>> Fixes: #12660 >>>> >>>> Signed-off-by: Peter M=C3=BCller >>>> --- >>>> src/python/location-importer.in | 28 +++++++++++++++++++++++++--- >>>> 1 file changed, 25 insertions(+), 3 deletions(-) >>>> >>>> diff --git a/src/python/location-importer.in b/src/python/location-impor= ter.in >>>> index d4b0ca2..bec0b62 100644 >>>> --- a/src/python/location-importer.in >>>> +++ b/src/python/location-importer.in >>>> @@ -393,11 +393,11 @@ class CLI(object): >>>> with self.db.transaction(): >>>> # Create some temporary tables to store parsed data >>>> self.db.execute(""" >>>> - CREATE TEMPORARY TABLE _autnums(number integer, organization text, = source text NOT NULL) >>>> + CREATE TEMPORARY TABLE _autnums(number integer NOT NULL, organizati= on text NOT NULL, source text NOT NULL) >>>> ON COMMIT DROP; >>>> CREATE UNIQUE INDEX _autnums_number ON _autnums(number); >>>> >>>> - CREATE TEMPORARY TABLE _organizations(handle text, name text NOT NU= LL, source text NOT NULL) >>>> + CREATE TEMPORARY TABLE _organizations(handle text NOT NULL, name te= xt NOT NULL, source text NOT NULL) >>>> ON COMMIT DROP; >>>> CREATE UNIQUE INDEX _organizations_handle ON _organizations(handle); >>>> >>>> @@ -604,10 +604,32 @@ class CLI(object): >>>> elif key =3D=3D "org": >>>> autnum[key] =3D val.upper() >>>> >>>> + elif key =3D=3D "descr": >>>> + # Save the first description line as well... >>>> + if not key in autnum: >>>> + autnum[key] =3D val >>>> + >>>> # Skip empty objects >>>> - if not autnum: >>>> + if not autnum or not "asn" in autnum: >>>> return >>>> >>>> + # Insert a dummy organisation handle into our temporary organisations >>>> + # table in case the AS does not have an organisation handle set, but >>>> + # has a description (a quirk often observed in APNIC area), so we can >>>> + # later display at least some string for this AS. >>>> + if not "org" in autnum: >>>> + if "descr" in autnum: >>>> + autnum["org"] =3D "LIBLOC-%s-ORGHANDLE" % autnum.get("asn") >>>> + >>>> + self.db.execute("INSERT INTO _organizations(handle, name, source) \ >>>> + VALUES(%s, %s, %s) ON CONFLICT (handle) DO NOTHING", >>>> + autnum.get("org"), autnum.get("descr"), source_key, >>>> + ) >>>> + else: >>>> + log.warning("ASN %s neither has an organisation handle nor a descri= ption line set, omitting" % \ >>>> + autnum.get("asn")) >>>> + return >>>> + >>>> # Insert into database >>>> self.db.execute("INSERT INTO _autnums(number, organization, source) \ >>>> VALUES(%s, %s, %s) ON CONFLICT (number) DO UPDATE SET \ >>>> --=20 >>>> 2.20.1 >>>> >>> >=20 --===============4920636776348386419==--