From mboxrd@z Thu Jan  1 00:00:00 1970
From: Michael Tremer <michael.tremer@ipfire.org>
To: location@lists.ipfire.org
Subject:
 Re: [PATCH v2] location-importer.in: Import (technical) AS names from ARIN
Date: Thu, 10 Jun 2021 09:52:18 +0100
Message-ID: <5B56B4C5-0AB7-4CAA-9827-9EF0DD984467@ipfire.org>
In-Reply-To: <20210608170307.623-1-peter.mueller@ipfire.org>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============7312423382085950542=="
List-Id: <location.lists.ipfire.org>

--===============7312423382085950542==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

Hello,

> On 8 Jun 2021, at 18:03, Peter M=C3=BCller <peter.mueller(a)ipfire.org> wro=
te:
>=20
> ARIN and LACNIC, unfortunately, do not seem to publish data containing
> human readable AS names. For the former, we at least have a list of
> tecnical names, which this patch fetches and inserts into the autnums
> table.
>=20
> While some of them do not seem to be suitable for human consumption (i.
> e. being very cryptic), providing these data might be helpful
> neverthelesss.
>=20
> The second version of this patch contains some additional remarks on
> efficient Python coding style from Michael, doing things more "pythonic".
>=20
> Signed-off-by: Peter M=C3=BCller <peter.mueller(a)ipfire.org>
> ---
> src/python/location-importer.in | 55 +++++++++++++++++++++++++++++++++
> 1 file changed, 55 insertions(+)
>=20
> diff --git a/src/python/location-importer.in b/src/python/location-importer=
.in
> index aa3b8f7..6ccee3b 100644
> --- a/src/python/location-importer.in
> +++ b/src/python/location-importer.in
> @@ -505,6 +505,9 @@ class CLI(object):
> 						for line in f:
> 							self._parse_line(line, source_key, validcountries)
>=20
> +		# Download and import (technical) AS names from ARIN
> +		self._import_as_names_from_arin()
> +
> 	def _check_parsed_network(self, network):
> 		"""
> 			Assistive function to detect and subsequently sort out parsed
> @@ -775,6 +778,58 @@ class CLI(object):
> 			"%s" % network, country, [country], source_key,
> 		)
>=20
> +	def _import_as_names_from_arin(self):
> +		downloader =3D location.importer.Downloader()
> +
> +		# XXX: Download AS names file from ARIN (note that these names appear to=
 be quite
> +		# technical, not intended for human consumption, as description fields in
> +		# organisation handles for other RIRs are - however, this is what we hav=
e got,
> +		# and in some cases, it might be still better than nothing)
> +		with downloader.request("https://ftp.arin.net/info/asn.txt", return_bloc=
ks=3DFalse) as f:
> +			for line in f:
> +				# Convert binary line to string...
> +				line =3D str(line)
> +
> +				# ... valid lines start with a space, followed by the number of the Au=
tonomous System ...
> +				if not line.startswith(" "):
> +					continue
> +
> +				# Split line and check if there is a valid ASN in it...
> +				asn, name =3D line.split()[0:2]
> +
> +				try:
> +					asn =3D int(asn)
> +				except ValueError:
> +					log.debug("Skipping ARIN AS names line not containing an integer for =
ASN")
> +					continue
> +
> +				if not ((1 <=3D asn and asn <=3D 23455) or (23457 <=3D asn and asn <=
=3D 64495) or (131072 <=3D asn and asn <=3D 4199999999)):
> +					log.debug("Skipping ARIN AS names line not containing a valid ASN: %s=
" % asn)
> +					continue
> +
> +				# Skip any AS name that appears to be a placeholder for a different RI=
R or entity...
> +				if re.match(r"^(ASN-BLK|)(AFCONC|AFRINIC|APNIC|ASNBLK|DNIC|LACNIC|RIPE=
|IANA)(\d?$|\-.*)", name):
> +					continue

This is still not entirely optimal. It doesn=E2=80=99t matter too much, so I =
will merge it, but=E2=80=A6

* You added a selection group which you do not need, so you could have writte=
n (?:=E2=80=A6) instead of (=E2=80=A6).

\-.* matches a literal dash and then anything after it. You do not care about=
 what comes after, so you could have just had \- and that is it. It would hav=
e saved a couple of CPU cycles because you don=E2=80=99t have to read the ent=
ire rest of the string.

> +
> +				# Bail out in case the AS name contains anything we do not expect here=
...
> +				if re.search(r"[^a-zA-Z0-9-_]", name):
> +					log.debug("Skipping ARIN AS name for %s containing invalid characters=
: %s" % \
> +							(asn, name))
> +
> +				# Things look good here, run INSERT statement and skip this one if we =
already have
> +				# a (better?) name for this Autonomous System...
> +				self.db.execute("""
> +					INSERT INTO autnums(
> +						number,
> +						name,
> +						source
> +					) VALUES (%s, %s, %s)
> +					ON CONFLICT (number) DO NOTHING""",
> +					asn,
> +					name,
> +					"ARIN",
> +				)
> +
> 	def handle_update_announcements(self, ns):
> 		server =3D ns.server[0]
>=20
> --=20
> 2.20.1
>=20

-Michael


--===============7312423382085950542==--