From mboxrd@z Thu Jan  1 00:00:00 1970
From: Michael Tremer <michael.tremer@ipfire.org>
To: location@lists.ipfire.org
Subject: Re: [PATCH] location-importer.in: Attempt to provide meaningful AS
 names if organisation handles are missing
Date: Fri, 23 Jul 2021 14:38:47 +0100
Message-ID: <2F401D87-C58F-45CE-8E94-CEA624592E48@ipfire.org>
In-Reply-To: <257789bc-404e-dc5c-7e6f-41a1479d6ad3@ipfire.org>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============4039369492256319087=="
List-Id: <location.lists.ipfire.org>

--===============4039369492256319087==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

Hey,

> On 21 Jul 2021, at 22:23, Peter M=C3=BCller <peter.mueller(a)ipfire.org> wr=
ote:
>=20
> Hello Michael,
>=20
> thanks for your reply.
>=20
> I would leave this up to you - some data-related issues are still on my lis=
t, but I
> am unfortunately really short on spare time at the moment. So I guess we sh=
ould get
> in production what we have at hand... :-)

Since we do not make too much use of this data, yet, I would like to leave an=
other release for a little bit and wait until the rest is coming in.

I generally like the release early and often thing. We should keep doing that.

> By the way: Thank you for fixing location01, SSH login works fine for me ag=
ain. :-)

-Michael

>=20
> Thanks, and best regards,
> Peter M=C3=BCller
>=20
>=20
>> Thank you. Merged.
>>=20
>> Do we plan a release because of this soon?
>>=20
>>> On 19 Jul 2021, at 22:34, Peter M=C3=BCller <peter.mueller(a)ipfire.org> =
wrote:
>>>=20
>>> A decent amount of autnum objects - especially, but not exclusively in
>>> the APNIC sector - does not contain a link to an organisation handle.
>>>=20
>>> In such cases, this patch is going to use the first description line of
>>> the atunum object in question (if available) as a string for its name.
>>> The overwhelming majority of affected ASNs contains a valuable
>>> information there, so this is almost as good as having an organisation
>>> handle linked to it.
>>>=20
>>> Fixes: #12660
>>>=20
>>> Signed-off-by: Peter M=C3=BCller <peter.mueller(a)ipfire.org>
>>> ---
>>> src/python/location-importer.in | 28 +++++++++++++++++++++++++---
>>> 1 file changed, 25 insertions(+), 3 deletions(-)
>>>=20
>>> diff --git a/src/python/location-importer.in b/src/python/location-import=
er.in
>>> index d4b0ca2..bec0b62 100644
>>> --- a/src/python/location-importer.in
>>> +++ b/src/python/location-importer.in
>>> @@ -393,11 +393,11 @@ class CLI(object):
>>> 		with self.db.transaction():
>>> 			# Create some temporary tables to store parsed data
>>> 			self.db.execute("""
>>> -				CREATE TEMPORARY TABLE _autnums(number integer, organization text, s=
ource text NOT NULL)
>>> +				CREATE TEMPORARY TABLE _autnums(number integer NOT NULL, organizatio=
n text NOT NULL, source text NOT NULL)
>>> 					ON COMMIT DROP;
>>> 				CREATE UNIQUE INDEX _autnums_number ON _autnums(number);
>>>=20
>>> -				CREATE TEMPORARY TABLE _organizations(handle text, name text NOT NUL=
L, source text NOT NULL)
>>> +				CREATE TEMPORARY TABLE _organizations(handle text NOT NULL, name tex=
t NOT NULL, source text NOT NULL)
>>> 					ON COMMIT DROP;
>>> 				CREATE UNIQUE INDEX _organizations_handle ON _organizations(handle);
>>>=20
>>> @@ -604,10 +604,32 @@ class CLI(object):
>>> 			elif key =3D=3D "org":
>>> 				autnum[key] =3D val.upper()
>>>=20
>>> +			elif key =3D=3D "descr":
>>> +				# Save the first description line as well...
>>> +				if not key in autnum:
>>> +					autnum[key] =3D val
>>> +
>>> 		# Skip empty objects
>>> -		if not autnum:
>>> +		if not autnum or not "asn" in autnum:
>>> 			return
>>>=20
>>> +		# Insert a dummy organisation handle into our temporary organisations
>>> +		# table in case the AS does not have an organisation handle set, but
>>> +		# has a description (a quirk often observed in APNIC area), so we can
>>> +		# later display at least some string for this AS.
>>> +		if not "org" in autnum:
>>> +			if "descr" in autnum:
>>> +				autnum["org"] =3D "LIBLOC-%s-ORGHANDLE" % autnum.get("asn")
>>> +
>>> +				self.db.execute("INSERT INTO _organizations(handle, name, source) \
>>> +					VALUES(%s, %s, %s) ON CONFLICT (handle) DO NOTHING",
>>> +					autnum.get("org"), autnum.get("descr"), source_key,
>>> +				)
>>> +			else:
>>> +				log.warning("ASN %s neither has an organisation handle nor a descrip=
tion line set, omitting" % \
>>> +						autnum.get("asn"))
>>> +				return
>>> +
>>> 		# Insert into database
>>> 		self.db.execute("INSERT INTO _autnums(number, organization, source) \
>>> 			VALUES(%s, %s, %s) ON CONFLICT (number) DO UPDATE SET \
>>> --=20
>>> 2.20.1
>>>=20
>>=20


--===============4039369492256319087==--