From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: location@lists.ipfire.org Subject: Re: [PATCH] location-importer.in: Attempt to provide meaningful AS names if organisation handles are missing Date: Thu, 05 Aug 2021 22:09:11 +0200 Message-ID: In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2796572982146830662==" List-Id: --===============2796572982146830662== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello, I would rather have it property tagged so that we can use our deployment proc= ess and can potentially roll back. -Michael > On 5 Aug 2021, at 20:49, Peter M=C3=BCller wro= te: >=20 > Hi Michael, >=20 >> Hey, >>=20 >>> On 21 Jul 2021, at 22:23, Peter M=C3=BCller = wrote: >>>=20 >>> Hello Michael, >>>=20 >>> thanks for your reply. >>>=20 >>> I would leave this up to you - some data-related issues are still on my l= ist, but I >>> am unfortunately really short on spare time at the moment. So I guess we = should get >>> in production what we have at hand... :-) >>=20 >> Since we do not make too much use of this data, yet, I would like to leave= another release for a little bit and wait until the rest is coming in. >=20 > thought about this again, I think we could apply this patch to libloc runni= ng on location01, > as we probably won't release a next version of libloc within the next few w= eeks - primarily > due to me lacking spare time. :-/ >=20 > Albeit it is hacky, I think it would be worth it. What do you think? >=20 >> I generally like the release early and often thing. We should keep doing t= hat. >=20 > ACK. >=20 > Thanks, and best regards, > Peter M=C3=BCller >=20 >>=20 >>> By the way: Thank you for fixing location01, SSH login works fine for me = again. :-) >>=20 >> -Michael >>=20 >>>=20 >>> Thanks, and best regards, >>> Peter M=C3=BCller >>>=20 >>>=20 >>>> Thank you. Merged. >>>>=20 >>>> Do we plan a release because of this soon? >>>>=20 >>>>> On 19 Jul 2021, at 22:34, Peter M=C3=BCller wrote: >>>>>=20 >>>>> A decent amount of autnum objects - especially, but not exclusively in >>>>> the APNIC sector - does not contain a link to an organisation handle. >>>>>=20 >>>>> In such cases, this patch is going to use the first description line of >>>>> the atunum object in question (if available) as a string for its name. >>>>> The overwhelming majority of affected ASNs contains a valuable >>>>> information there, so this is almost as good as having an organisation >>>>> handle linked to it. >>>>>=20 >>>>> Fixes: #12660 >>>>>=20 >>>>> Signed-off-by: Peter M=C3=BCller >>>>> --- >>>>> src/python/location-importer.in | 28 +++++++++++++++++++++++++--- >>>>> 1 file changed, 25 insertions(+), 3 deletions(-) >>>>>=20 >>>>> diff --git a/src/python/location-importer.in b/src/python/location-impo= rter.in >>>>> index d4b0ca2..bec0b62 100644 >>>>> --- a/src/python/location-importer.in >>>>> +++ b/src/python/location-importer.in >>>>> @@ -393,11 +393,11 @@ class CLI(object): >>>>> with self.db.transaction(): >>>>> # Create some temporary tables to store parsed data >>>>> self.db.execute(""" >>>>> - CREATE TEMPORARY TABLE _autnums(number integer, organization text,= source text NOT NULL) >>>>> + CREATE TEMPORARY TABLE _autnums(number integer NOT NULL, organizat= ion text NOT NULL, source text NOT NULL) >>>>> ON COMMIT DROP; >>>>> CREATE UNIQUE INDEX _autnums_number ON _autnums(number); >>>>>=20 >>>>> - CREATE TEMPORARY TABLE _organizations(handle text, name text NOT N= ULL, source text NOT NULL) >>>>> + CREATE TEMPORARY TABLE _organizations(handle text NOT NULL, name t= ext NOT NULL, source text NOT NULL) >>>>> ON COMMIT DROP; >>>>> CREATE UNIQUE INDEX _organizations_handle ON _organizations(handle); >>>>>=20 >>>>> @@ -604,10 +604,32 @@ class CLI(object): >>>>> elif key =3D=3D "org": >>>>> autnum[key] =3D val.upper() >>>>>=20 >>>>> + elif key =3D=3D "descr": >>>>> + # Save the first description line as well... >>>>> + if not key in autnum: >>>>> + autnum[key] =3D val >>>>> + >>>>> # Skip empty objects >>>>> - if not autnum: >>>>> + if not autnum or not "asn" in autnum: >>>>> return >>>>>=20 >>>>> + # Insert a dummy organisation handle into our temporary organisations >>>>> + # table in case the AS does not have an organisation handle set, but >>>>> + # has a description (a quirk often observed in APNIC area), so we can >>>>> + # later display at least some string for this AS. >>>>> + if not "org" in autnum: >>>>> + if "descr" in autnum: >>>>> + autnum["org"] =3D "LIBLOC-%s-ORGHANDLE" % autnum.get("asn") >>>>> + >>>>> + self.db.execute("INSERT INTO _organizations(handle, name, source) \ >>>>> + VALUES(%s, %s, %s) ON CONFLICT (handle) DO NOTHING", >>>>> + autnum.get("org"), autnum.get("descr"), source_key, >>>>> + ) >>>>> + else: >>>>> + log.warning("ASN %s neither has an organisation handle nor a descr= iption line set, omitting" % \ >>>>> + autnum.get("asn")) >>>>> + return >>>>> + >>>>> # Insert into database >>>>> self.db.execute("INSERT INTO _autnums(number, organization, source) \ >>>>> VALUES(%s, %s, %s) ON CONFLICT (number) DO UPDATE SET \ >>>>> --=20 >>>>> 2.20.1 --===============2796572982146830662==--