* [PATCH] location-importer.in: Attempt to provide meaningful AS names if organisation handles are missing
@ 2021-07-19 21:34 Peter Müller
2021-07-20 12:18 ` Michael Tremer
0 siblings, 1 reply; 6+ messages in thread
From: Peter Müller @ 2021-07-19 21:34 UTC (permalink / raw)
To: location
[-- Attachment #1: Type: text/plain, Size: 2869 bytes --]
A decent amount of autnum objects - especially, but not exclusively in
the APNIC sector - does not contain a link to an organisation handle.
In such cases, this patch is going to use the first description line of
the atunum object in question (if available) as a string for its name.
The overwhelming majority of affected ASNs contains a valuable
information there, so this is almost as good as having an organisation
handle linked to it.
Fixes: #12660
Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org>
---
src/python/location-importer.in | 28 +++++++++++++++++++++++++---
1 file changed, 25 insertions(+), 3 deletions(-)
diff --git a/src/python/location-importer.in b/src/python/location-importer.in
index d4b0ca2..bec0b62 100644
--- a/src/python/location-importer.in
+++ b/src/python/location-importer.in
@@ -393,11 +393,11 @@ class CLI(object):
with self.db.transaction():
# Create some temporary tables to store parsed data
self.db.execute("""
- CREATE TEMPORARY TABLE _autnums(number integer, organization text, source text NOT NULL)
+ CREATE TEMPORARY TABLE _autnums(number integer NOT NULL, organization text NOT NULL, source text NOT NULL)
ON COMMIT DROP;
CREATE UNIQUE INDEX _autnums_number ON _autnums(number);
- CREATE TEMPORARY TABLE _organizations(handle text, name text NOT NULL, source text NOT NULL)
+ CREATE TEMPORARY TABLE _organizations(handle text NOT NULL, name text NOT NULL, source text NOT NULL)
ON COMMIT DROP;
CREATE UNIQUE INDEX _organizations_handle ON _organizations(handle);
@@ -604,10 +604,32 @@ class CLI(object):
elif key == "org":
autnum[key] = val.upper()
+ elif key == "descr":
+ # Save the first description line as well...
+ if not key in autnum:
+ autnum[key] = val
+
# Skip empty objects
- if not autnum:
+ if not autnum or not "asn" in autnum:
return
+ # Insert a dummy organisation handle into our temporary organisations
+ # table in case the AS does not have an organisation handle set, but
+ # has a description (a quirk often observed in APNIC area), so we can
+ # later display at least some string for this AS.
+ if not "org" in autnum:
+ if "descr" in autnum:
+ autnum["org"] = "LIBLOC-%s-ORGHANDLE" % autnum.get("asn")
+
+ self.db.execute("INSERT INTO _organizations(handle, name, source) \
+ VALUES(%s, %s, %s) ON CONFLICT (handle) DO NOTHING",
+ autnum.get("org"), autnum.get("descr"), source_key,
+ )
+ else:
+ log.warning("ASN %s neither has an organisation handle nor a description line set, omitting" % \
+ autnum.get("asn"))
+ return
+
# Insert into database
self.db.execute("INSERT INTO _autnums(number, organization, source) \
VALUES(%s, %s, %s) ON CONFLICT (number) DO UPDATE SET \
--
2.20.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] location-importer.in: Attempt to provide meaningful AS names if organisation handles are missing
2021-07-19 21:34 [PATCH] location-importer.in: Attempt to provide meaningful AS names if organisation handles are missing Peter Müller
@ 2021-07-20 12:18 ` Michael Tremer
2021-07-21 21:23 ` Peter Müller
0 siblings, 1 reply; 6+ messages in thread
From: Michael Tremer @ 2021-07-20 12:18 UTC (permalink / raw)
To: location
[-- Attachment #1: Type: text/plain, Size: 3144 bytes --]
Thank you. Merged.
Do we plan a release because of this soon?
> On 19 Jul 2021, at 22:34, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>
> A decent amount of autnum objects - especially, but not exclusively in
> the APNIC sector - does not contain a link to an organisation handle.
>
> In such cases, this patch is going to use the first description line of
> the atunum object in question (if available) as a string for its name.
> The overwhelming majority of affected ASNs contains a valuable
> information there, so this is almost as good as having an organisation
> handle linked to it.
>
> Fixes: #12660
>
> Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org>
> ---
> src/python/location-importer.in | 28 +++++++++++++++++++++++++---
> 1 file changed, 25 insertions(+), 3 deletions(-)
>
> diff --git a/src/python/location-importer.in b/src/python/location-importer.in
> index d4b0ca2..bec0b62 100644
> --- a/src/python/location-importer.in
> +++ b/src/python/location-importer.in
> @@ -393,11 +393,11 @@ class CLI(object):
> with self.db.transaction():
> # Create some temporary tables to store parsed data
> self.db.execute("""
> - CREATE TEMPORARY TABLE _autnums(number integer, organization text, source text NOT NULL)
> + CREATE TEMPORARY TABLE _autnums(number integer NOT NULL, organization text NOT NULL, source text NOT NULL)
> ON COMMIT DROP;
> CREATE UNIQUE INDEX _autnums_number ON _autnums(number);
>
> - CREATE TEMPORARY TABLE _organizations(handle text, name text NOT NULL, source text NOT NULL)
> + CREATE TEMPORARY TABLE _organizations(handle text NOT NULL, name text NOT NULL, source text NOT NULL)
> ON COMMIT DROP;
> CREATE UNIQUE INDEX _organizations_handle ON _organizations(handle);
>
> @@ -604,10 +604,32 @@ class CLI(object):
> elif key == "org":
> autnum[key] = val.upper()
>
> + elif key == "descr":
> + # Save the first description line as well...
> + if not key in autnum:
> + autnum[key] = val
> +
> # Skip empty objects
> - if not autnum:
> + if not autnum or not "asn" in autnum:
> return
>
> + # Insert a dummy organisation handle into our temporary organisations
> + # table in case the AS does not have an organisation handle set, but
> + # has a description (a quirk often observed in APNIC area), so we can
> + # later display at least some string for this AS.
> + if not "org" in autnum:
> + if "descr" in autnum:
> + autnum["org"] = "LIBLOC-%s-ORGHANDLE" % autnum.get("asn")
> +
> + self.db.execute("INSERT INTO _organizations(handle, name, source) \
> + VALUES(%s, %s, %s) ON CONFLICT (handle) DO NOTHING",
> + autnum.get("org"), autnum.get("descr"), source_key,
> + )
> + else:
> + log.warning("ASN %s neither has an organisation handle nor a description line set, omitting" % \
> + autnum.get("asn"))
> + return
> +
> # Insert into database
> self.db.execute("INSERT INTO _autnums(number, organization, source) \
> VALUES(%s, %s, %s) ON CONFLICT (number) DO UPDATE SET \
> --
> 2.20.1
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] location-importer.in: Attempt to provide meaningful AS names if organisation handles are missing
2021-07-20 12:18 ` Michael Tremer
@ 2021-07-21 21:23 ` Peter Müller
2021-07-23 13:38 ` Michael Tremer
0 siblings, 1 reply; 6+ messages in thread
From: Peter Müller @ 2021-07-21 21:23 UTC (permalink / raw)
To: location
[-- Attachment #1: Type: text/plain, Size: 3609 bytes --]
Hello Michael,
thanks for your reply.
I would leave this up to you - some data-related issues are still on my list, but I
am unfortunately really short on spare time at the moment. So I guess we should get
in production what we have at hand... :-)
By the way: Thank you for fixing location01, SSH login works fine for me again. :-)
Thanks, and best regards,
Peter Müller
> Thank you. Merged.
>
> Do we plan a release because of this soon?
>
>> On 19 Jul 2021, at 22:34, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>
>> A decent amount of autnum objects - especially, but not exclusively in
>> the APNIC sector - does not contain a link to an organisation handle.
>>
>> In such cases, this patch is going to use the first description line of
>> the atunum object in question (if available) as a string for its name.
>> The overwhelming majority of affected ASNs contains a valuable
>> information there, so this is almost as good as having an organisation
>> handle linked to it.
>>
>> Fixes: #12660
>>
>> Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org>
>> ---
>> src/python/location-importer.in | 28 +++++++++++++++++++++++++---
>> 1 file changed, 25 insertions(+), 3 deletions(-)
>>
>> diff --git a/src/python/location-importer.in b/src/python/location-importer.in
>> index d4b0ca2..bec0b62 100644
>> --- a/src/python/location-importer.in
>> +++ b/src/python/location-importer.in
>> @@ -393,11 +393,11 @@ class CLI(object):
>> with self.db.transaction():
>> # Create some temporary tables to store parsed data
>> self.db.execute("""
>> - CREATE TEMPORARY TABLE _autnums(number integer, organization text, source text NOT NULL)
>> + CREATE TEMPORARY TABLE _autnums(number integer NOT NULL, organization text NOT NULL, source text NOT NULL)
>> ON COMMIT DROP;
>> CREATE UNIQUE INDEX _autnums_number ON _autnums(number);
>>
>> - CREATE TEMPORARY TABLE _organizations(handle text, name text NOT NULL, source text NOT NULL)
>> + CREATE TEMPORARY TABLE _organizations(handle text NOT NULL, name text NOT NULL, source text NOT NULL)
>> ON COMMIT DROP;
>> CREATE UNIQUE INDEX _organizations_handle ON _organizations(handle);
>>
>> @@ -604,10 +604,32 @@ class CLI(object):
>> elif key == "org":
>> autnum[key] = val.upper()
>>
>> + elif key == "descr":
>> + # Save the first description line as well...
>> + if not key in autnum:
>> + autnum[key] = val
>> +
>> # Skip empty objects
>> - if not autnum:
>> + if not autnum or not "asn" in autnum:
>> return
>>
>> + # Insert a dummy organisation handle into our temporary organisations
>> + # table in case the AS does not have an organisation handle set, but
>> + # has a description (a quirk often observed in APNIC area), so we can
>> + # later display at least some string for this AS.
>> + if not "org" in autnum:
>> + if "descr" in autnum:
>> + autnum["org"] = "LIBLOC-%s-ORGHANDLE" % autnum.get("asn")
>> +
>> + self.db.execute("INSERT INTO _organizations(handle, name, source) \
>> + VALUES(%s, %s, %s) ON CONFLICT (handle) DO NOTHING",
>> + autnum.get("org"), autnum.get("descr"), source_key,
>> + )
>> + else:
>> + log.warning("ASN %s neither has an organisation handle nor a description line set, omitting" % \
>> + autnum.get("asn"))
>> + return
>> +
>> # Insert into database
>> self.db.execute("INSERT INTO _autnums(number, organization, source) \
>> VALUES(%s, %s, %s) ON CONFLICT (number) DO UPDATE SET \
>> --
>> 2.20.1
>>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] location-importer.in: Attempt to provide meaningful AS names if organisation handles are missing
2021-07-21 21:23 ` Peter Müller
@ 2021-07-23 13:38 ` Michael Tremer
2021-08-05 18:49 ` Peter Müller
0 siblings, 1 reply; 6+ messages in thread
From: Michael Tremer @ 2021-07-23 13:38 UTC (permalink / raw)
To: location
[-- Attachment #1: Type: text/plain, Size: 4061 bytes --]
Hey,
> On 21 Jul 2021, at 22:23, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>
> Hello Michael,
>
> thanks for your reply.
>
> I would leave this up to you - some data-related issues are still on my list, but I
> am unfortunately really short on spare time at the moment. So I guess we should get
> in production what we have at hand... :-)
Since we do not make too much use of this data, yet, I would like to leave another release for a little bit and wait until the rest is coming in.
I generally like the release early and often thing. We should keep doing that.
> By the way: Thank you for fixing location01, SSH login works fine for me again. :-)
-Michael
>
> Thanks, and best regards,
> Peter Müller
>
>
>> Thank you. Merged.
>>
>> Do we plan a release because of this soon?
>>
>>> On 19 Jul 2021, at 22:34, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>>
>>> A decent amount of autnum objects - especially, but not exclusively in
>>> the APNIC sector - does not contain a link to an organisation handle.
>>>
>>> In such cases, this patch is going to use the first description line of
>>> the atunum object in question (if available) as a string for its name.
>>> The overwhelming majority of affected ASNs contains a valuable
>>> information there, so this is almost as good as having an organisation
>>> handle linked to it.
>>>
>>> Fixes: #12660
>>>
>>> Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org>
>>> ---
>>> src/python/location-importer.in | 28 +++++++++++++++++++++++++---
>>> 1 file changed, 25 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/src/python/location-importer.in b/src/python/location-importer.in
>>> index d4b0ca2..bec0b62 100644
>>> --- a/src/python/location-importer.in
>>> +++ b/src/python/location-importer.in
>>> @@ -393,11 +393,11 @@ class CLI(object):
>>> with self.db.transaction():
>>> # Create some temporary tables to store parsed data
>>> self.db.execute("""
>>> - CREATE TEMPORARY TABLE _autnums(number integer, organization text, source text NOT NULL)
>>> + CREATE TEMPORARY TABLE _autnums(number integer NOT NULL, organization text NOT NULL, source text NOT NULL)
>>> ON COMMIT DROP;
>>> CREATE UNIQUE INDEX _autnums_number ON _autnums(number);
>>>
>>> - CREATE TEMPORARY TABLE _organizations(handle text, name text NOT NULL, source text NOT NULL)
>>> + CREATE TEMPORARY TABLE _organizations(handle text NOT NULL, name text NOT NULL, source text NOT NULL)
>>> ON COMMIT DROP;
>>> CREATE UNIQUE INDEX _organizations_handle ON _organizations(handle);
>>>
>>> @@ -604,10 +604,32 @@ class CLI(object):
>>> elif key == "org":
>>> autnum[key] = val.upper()
>>>
>>> + elif key == "descr":
>>> + # Save the first description line as well...
>>> + if not key in autnum:
>>> + autnum[key] = val
>>> +
>>> # Skip empty objects
>>> - if not autnum:
>>> + if not autnum or not "asn" in autnum:
>>> return
>>>
>>> + # Insert a dummy organisation handle into our temporary organisations
>>> + # table in case the AS does not have an organisation handle set, but
>>> + # has a description (a quirk often observed in APNIC area), so we can
>>> + # later display at least some string for this AS.
>>> + if not "org" in autnum:
>>> + if "descr" in autnum:
>>> + autnum["org"] = "LIBLOC-%s-ORGHANDLE" % autnum.get("asn")
>>> +
>>> + self.db.execute("INSERT INTO _organizations(handle, name, source) \
>>> + VALUES(%s, %s, %s) ON CONFLICT (handle) DO NOTHING",
>>> + autnum.get("org"), autnum.get("descr"), source_key,
>>> + )
>>> + else:
>>> + log.warning("ASN %s neither has an organisation handle nor a description line set, omitting" % \
>>> + autnum.get("asn"))
>>> + return
>>> +
>>> # Insert into database
>>> self.db.execute("INSERT INTO _autnums(number, organization, source) \
>>> VALUES(%s, %s, %s) ON CONFLICT (number) DO UPDATE SET \
>>> --
>>> 2.20.1
>>>
>>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] location-importer.in: Attempt to provide meaningful AS names if organisation handles are missing
2021-07-23 13:38 ` Michael Tremer
@ 2021-08-05 18:49 ` Peter Müller
2021-08-05 20:09 ` Michael Tremer
0 siblings, 1 reply; 6+ messages in thread
From: Peter Müller @ 2021-08-05 18:49 UTC (permalink / raw)
To: location
[-- Attachment #1: Type: text/plain, Size: 4520 bytes --]
Hi Michael,
> Hey,
>
>> On 21 Jul 2021, at 22:23, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>
>> Hello Michael,
>>
>> thanks for your reply.
>>
>> I would leave this up to you - some data-related issues are still on my list, but I
>> am unfortunately really short on spare time at the moment. So I guess we should get
>> in production what we have at hand... :-)
>
> Since we do not make too much use of this data, yet, I would like to leave another release for a little bit and wait until the rest is coming in.
thought about this again, I think we could apply this patch to libloc running on location01,
as we probably won't release a next version of libloc within the next few weeks - primarily
due to me lacking spare time. :-/
Albeit it is hacky, I think it would be worth it. What do you think?
> I generally like the release early and often thing. We should keep doing that.
ACK.
Thanks, and best regards,
Peter Müller
>
>> By the way: Thank you for fixing location01, SSH login works fine for me again. :-)
>
> -Michael
>
>>
>> Thanks, and best regards,
>> Peter Müller
>>
>>
>>> Thank you. Merged.
>>>
>>> Do we plan a release because of this soon?
>>>
>>>> On 19 Jul 2021, at 22:34, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>>>
>>>> A decent amount of autnum objects - especially, but not exclusively in
>>>> the APNIC sector - does not contain a link to an organisation handle.
>>>>
>>>> In such cases, this patch is going to use the first description line of
>>>> the atunum object in question (if available) as a string for its name.
>>>> The overwhelming majority of affected ASNs contains a valuable
>>>> information there, so this is almost as good as having an organisation
>>>> handle linked to it.
>>>>
>>>> Fixes: #12660
>>>>
>>>> Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org>
>>>> ---
>>>> src/python/location-importer.in | 28 +++++++++++++++++++++++++---
>>>> 1 file changed, 25 insertions(+), 3 deletions(-)
>>>>
>>>> diff --git a/src/python/location-importer.in b/src/python/location-importer.in
>>>> index d4b0ca2..bec0b62 100644
>>>> --- a/src/python/location-importer.in
>>>> +++ b/src/python/location-importer.in
>>>> @@ -393,11 +393,11 @@ class CLI(object):
>>>> with self.db.transaction():
>>>> # Create some temporary tables to store parsed data
>>>> self.db.execute("""
>>>> - CREATE TEMPORARY TABLE _autnums(number integer, organization text, source text NOT NULL)
>>>> + CREATE TEMPORARY TABLE _autnums(number integer NOT NULL, organization text NOT NULL, source text NOT NULL)
>>>> ON COMMIT DROP;
>>>> CREATE UNIQUE INDEX _autnums_number ON _autnums(number);
>>>>
>>>> - CREATE TEMPORARY TABLE _organizations(handle text, name text NOT NULL, source text NOT NULL)
>>>> + CREATE TEMPORARY TABLE _organizations(handle text NOT NULL, name text NOT NULL, source text NOT NULL)
>>>> ON COMMIT DROP;
>>>> CREATE UNIQUE INDEX _organizations_handle ON _organizations(handle);
>>>>
>>>> @@ -604,10 +604,32 @@ class CLI(object):
>>>> elif key == "org":
>>>> autnum[key] = val.upper()
>>>>
>>>> + elif key == "descr":
>>>> + # Save the first description line as well...
>>>> + if not key in autnum:
>>>> + autnum[key] = val
>>>> +
>>>> # Skip empty objects
>>>> - if not autnum:
>>>> + if not autnum or not "asn" in autnum:
>>>> return
>>>>
>>>> + # Insert a dummy organisation handle into our temporary organisations
>>>> + # table in case the AS does not have an organisation handle set, but
>>>> + # has a description (a quirk often observed in APNIC area), so we can
>>>> + # later display at least some string for this AS.
>>>> + if not "org" in autnum:
>>>> + if "descr" in autnum:
>>>> + autnum["org"] = "LIBLOC-%s-ORGHANDLE" % autnum.get("asn")
>>>> +
>>>> + self.db.execute("INSERT INTO _organizations(handle, name, source) \
>>>> + VALUES(%s, %s, %s) ON CONFLICT (handle) DO NOTHING",
>>>> + autnum.get("org"), autnum.get("descr"), source_key,
>>>> + )
>>>> + else:
>>>> + log.warning("ASN %s neither has an organisation handle nor a description line set, omitting" % \
>>>> + autnum.get("asn"))
>>>> + return
>>>> +
>>>> # Insert into database
>>>> self.db.execute("INSERT INTO _autnums(number, organization, source) \
>>>> VALUES(%s, %s, %s) ON CONFLICT (number) DO UPDATE SET \
>>>> --
>>>> 2.20.1
>>>>
>>>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] location-importer.in: Attempt to provide meaningful AS names if organisation handles are missing
2021-08-05 18:49 ` Peter Müller
@ 2021-08-05 20:09 ` Michael Tremer
0 siblings, 0 replies; 6+ messages in thread
From: Michael Tremer @ 2021-08-05 20:09 UTC (permalink / raw)
To: location
[-- Attachment #1: Type: text/plain, Size: 4873 bytes --]
Hello,
I would rather have it property tagged so that we can use our deployment process and can potentially roll back.
-Michael
> On 5 Aug 2021, at 20:49, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>
> Hi Michael,
>
>> Hey,
>>
>>> On 21 Jul 2021, at 22:23, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>>
>>> Hello Michael,
>>>
>>> thanks for your reply.
>>>
>>> I would leave this up to you - some data-related issues are still on my list, but I
>>> am unfortunately really short on spare time at the moment. So I guess we should get
>>> in production what we have at hand... :-)
>>
>> Since we do not make too much use of this data, yet, I would like to leave another release for a little bit and wait until the rest is coming in.
>
> thought about this again, I think we could apply this patch to libloc running on location01,
> as we probably won't release a next version of libloc within the next few weeks - primarily
> due to me lacking spare time. :-/
>
> Albeit it is hacky, I think it would be worth it. What do you think?
>
>> I generally like the release early and often thing. We should keep doing that.
>
> ACK.
>
> Thanks, and best regards,
> Peter Müller
>
>>
>>> By the way: Thank you for fixing location01, SSH login works fine for me again. :-)
>>
>> -Michael
>>
>>>
>>> Thanks, and best regards,
>>> Peter Müller
>>>
>>>
>>>> Thank you. Merged.
>>>>
>>>> Do we plan a release because of this soon?
>>>>
>>>>> On 19 Jul 2021, at 22:34, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>>>>
>>>>> A decent amount of autnum objects - especially, but not exclusively in
>>>>> the APNIC sector - does not contain a link to an organisation handle.
>>>>>
>>>>> In such cases, this patch is going to use the first description line of
>>>>> the atunum object in question (if available) as a string for its name.
>>>>> The overwhelming majority of affected ASNs contains a valuable
>>>>> information there, so this is almost as good as having an organisation
>>>>> handle linked to it.
>>>>>
>>>>> Fixes: #12660
>>>>>
>>>>> Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org>
>>>>> ---
>>>>> src/python/location-importer.in | 28 +++++++++++++++++++++++++---
>>>>> 1 file changed, 25 insertions(+), 3 deletions(-)
>>>>>
>>>>> diff --git a/src/python/location-importer.in b/src/python/location-importer.in
>>>>> index d4b0ca2..bec0b62 100644
>>>>> --- a/src/python/location-importer.in
>>>>> +++ b/src/python/location-importer.in
>>>>> @@ -393,11 +393,11 @@ class CLI(object):
>>>>> with self.db.transaction():
>>>>> # Create some temporary tables to store parsed data
>>>>> self.db.execute("""
>>>>> - CREATE TEMPORARY TABLE _autnums(number integer, organization text, source text NOT NULL)
>>>>> + CREATE TEMPORARY TABLE _autnums(number integer NOT NULL, organization text NOT NULL, source text NOT NULL)
>>>>> ON COMMIT DROP;
>>>>> CREATE UNIQUE INDEX _autnums_number ON _autnums(number);
>>>>>
>>>>> - CREATE TEMPORARY TABLE _organizations(handle text, name text NOT NULL, source text NOT NULL)
>>>>> + CREATE TEMPORARY TABLE _organizations(handle text NOT NULL, name text NOT NULL, source text NOT NULL)
>>>>> ON COMMIT DROP;
>>>>> CREATE UNIQUE INDEX _organizations_handle ON _organizations(handle);
>>>>>
>>>>> @@ -604,10 +604,32 @@ class CLI(object):
>>>>> elif key == "org":
>>>>> autnum[key] = val.upper()
>>>>>
>>>>> + elif key == "descr":
>>>>> + # Save the first description line as well...
>>>>> + if not key in autnum:
>>>>> + autnum[key] = val
>>>>> +
>>>>> # Skip empty objects
>>>>> - if not autnum:
>>>>> + if not autnum or not "asn" in autnum:
>>>>> return
>>>>>
>>>>> + # Insert a dummy organisation handle into our temporary organisations
>>>>> + # table in case the AS does not have an organisation handle set, but
>>>>> + # has a description (a quirk often observed in APNIC area), so we can
>>>>> + # later display at least some string for this AS.
>>>>> + if not "org" in autnum:
>>>>> + if "descr" in autnum:
>>>>> + autnum["org"] = "LIBLOC-%s-ORGHANDLE" % autnum.get("asn")
>>>>> +
>>>>> + self.db.execute("INSERT INTO _organizations(handle, name, source) \
>>>>> + VALUES(%s, %s, %s) ON CONFLICT (handle) DO NOTHING",
>>>>> + autnum.get("org"), autnum.get("descr"), source_key,
>>>>> + )
>>>>> + else:
>>>>> + log.warning("ASN %s neither has an organisation handle nor a description line set, omitting" % \
>>>>> + autnum.get("asn"))
>>>>> + return
>>>>> +
>>>>> # Insert into database
>>>>> self.db.execute("INSERT INTO _autnums(number, organization, source) \
>>>>> VALUES(%s, %s, %s) ON CONFLICT (number) DO UPDATE SET \
>>>>> --
>>>>> 2.20.1
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-08-05 20:09 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-07-19 21:34 [PATCH] location-importer.in: Attempt to provide meaningful AS names if organisation handles are missing Peter Müller
2021-07-20 12:18 ` Michael Tremer
2021-07-21 21:23 ` Peter Müller
2021-07-23 13:38 ` Michael Tremer
2021-08-05 18:49 ` Peter Müller
2021-08-05 20:09 ` Michael Tremer
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox