Re: [PATCH] location-importer.in: treat AQ and BV as invalid countries

["Peter Müller" <peter.mueller@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 5259 bytes --]

Hello Michael,

thanks for your reply.

> Hi,
> 
>> On 12 Apr 2021, at 18:26, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>
>> Hello Michael,
>>
>> thanks for your reply.
>>
>> I believe you are mixing up two distinct scenarios here:
> 
> I am trying not to. I just added the name because I struggle to remember some country codes.
> 
>> (a) Historic ISO-3166-X abbreviations for countries no longer existing anymore, such as YU.
> 
> They technically do.
> 
> They are still reserved in the ISO 3166 database and they show up (in this case) in the RIPE database. So we will have to handle them no matter what.
> 
> YU does not have a succeeding national body (like Eastern Germany would have the Federal Republic of Germany) so we cannot automatically rewrite them.

This is what I meant by "does no longer exist anymore".

> 
>> (b) Country codes for existing nations, where we believe with a high level of confidence no
>>    IP networks are located in any way, either physically or legally. AQ and BV are the most
>>    notable examples known to me at the time of writing.
>>
>> Since it does not make sense to add historic country codes, I would suggest to leave networks
>> assigned to (a) countries as they are, manually creating overrides for them, as I did before.
>>
>> For (b), simply scrubbing out the country or region they belong to seems to be too harsh to
>> me, as we use this information for several things, and should not abuse it for our own purposes.
> 
> (I assume) we still have AS information about those networks and we should keep at least that.

ACK.

> 
>> Therefore, I rather suggest adding an additional column to countries.txt for indicating whether
>> this country code is acceptable to us or not.
> 
> Okay. What would you suggest?

Well, our "countries.txt" file's syntax currently looks like (delimited by tabulators):

[ISO-3166-1-alpha2 country code]	[regional code defined by us to know which Whois server should be queried]	[human-readable name of the country]

I would like to add two new columns to this file, one for declaring a country invalid in terms of
libloc, the other one to declare it as suspicious so we can handle those cases better in a later
step.

Personally, I wish those two columns to be placed before the country name for readability reasons.
However, this would be a breaking change to existing libloc instances using the "countries.txt" file
of our data repository.

Thanks, and best regards,
Peter Müller

> 
> -Michael
> 
>>
>> Thanks, and best regards,
>> Peter Müller
>>
>>> Hello,
>>>
>>> How about we add the country to the list and mark the continent with a “-“:
>>>
>>>  AF      AS      Afghanistan
>>>  YU      -       Yugoslavia
>>>
>>> -Michael
>>>
>>>> On 10 Apr 2021, at 13:32, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>>>
>>>> Hello Michael,
>>>>
>>>> thanks for your reply.
>>>>
>>>> Yes, having this configurable in countries.txt would be nice indeed. Do you propose a certain syntax for this?
>>>>
>>>> And yes, a tuple is a better idea here. I will wait for your reply and submit a second version of this patch then.
>>>>
>>>> Thanks, and best regards,
>>>> Peter Müller
>>>>
>>>>
>>>>> Hello,
>>>>>
>>>>> * would we not want this to be configurable in countries.txt?
>>>>>
>>>>> * The list should probably be a tuple.
>>>>>
>>>>> -Michael
>>>>>
>>>>>> On 1 Apr 2021, at 20:57, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>>>>>
>>>>>> Both the Bouvet Island (BV) and Antarctica (AQ) are unpopulated at the
>>>>>> time of writing. Network owners/operators putting these countries into
>>>>>> their RIR data objects are either completely braindead or doing so for
>>>>>> hostile reasons.
>>>>>>
>>>>>> While we might correct these locations to something useful by manually
>>>>>> creating overrides for them, the rationale behind this patch is not to
>>>>>> let these countries appear on productive systems in the first place, as
>>>>>> we know they _cannot_ be true.
>>>>>>
>>>>>> Therefore, this patch skips any network object that has either AQ or BV
>>>>>> country code set.
>>>>>>
>>>>>> See also: https://lists.ipfire.org/pipermail/location/2020-October/000199.html
>>>>>>
>>>>>> Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org>
>>>>>> ---
>>>>>> src/python/location-importer.in | 2 +-
>>>>>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>>
>>>>>> diff --git a/src/python/location-importer.in b/src/python/location-importer.in
>>>>>> index 1e08458..ac678dc 100644
>>>>>> --- a/src/python/location-importer.in
>>>>>> +++ b/src/python/location-importer.in
>>>>>> @@ -624,7 +624,7 @@ class CLI(object):
>>>>>> 			return
>>>>>>
>>>>>> 		# Skip objects with unknown country codes
>>>>>> -		if validcountries and inetnum.get("country") not in validcountries:
>>>>>> +		if validcountries and (inetnum.get("country") not in validcountries or inetnum.get("country") in ["AQ", "BV"]):
>>>>>> 			log.warning("Skipping network with bogus country '%s': %s" % \
>>>>>> 				(inetnum.get("country"), inetnum.get("inet6num") or inetnum.get("inetnum")))
>>>>>> 			return
>>>>>> -- 
>>>>>> 2.26.2
>>>>>
>>>
>