From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: location@lists.ipfire.org Subject: Re: [PATCH] location-importer.in: treat AQ and BV as invalid countries Date: Tue, 18 May 2021 11:47:28 +0100 Message-ID: In-Reply-To: <1c89134e-4f5b-4654-3443-ea19fc225e8e@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============8034916945771481314==" List-Id: --===============8034916945771481314== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable > On 14 May 2021, at 17:15, Peter M=C3=BCller wr= ote: >=20 > Hello Michael, >=20 > thanks for your reply. >=20 >> Hi, >>=20 >>> On 12 Apr 2021, at 18:26, Peter M=C3=BCller = wrote: >>>=20 >>> Hello Michael, >>>=20 >>> thanks for your reply. >>>=20 >>> I believe you are mixing up two distinct scenarios here: >>=20 >> I am trying not to. I just added the name because I struggle to remember s= ome country codes. >>=20 >>> (a) Historic ISO-3166-X abbreviations for countries no longer existing an= ymore, such as YU. >>=20 >> They technically do. >>=20 >> They are still reserved in the ISO 3166 database and they show up (in this= case) in the RIPE database. So we will have to handle them no matter what. >>=20 >> YU does not have a succeeding national body (like Eastern Germany would ha= ve the Federal Republic of Germany) so we cannot automatically rewrite them. >=20 > This is what I meant by "does no longer exist anymore". RIPE seems to think that YU exists :) As much as I would like it, a standard does not always represent a real-life = situation. >>=20 >>> (b) Country codes for existing nations, where we believe with a high leve= l of confidence no >>> IP networks are located in any way, either physically or legally. AQ an= d BV are the most >>> notable examples known to me at the time of writing. >>>=20 >>> Since it does not make sense to add historic country codes, I would sugge= st to leave networks >>> assigned to (a) countries as they are, manually creating overrides for th= em, as I did before. >>>=20 >>> For (b), simply scrubbing out the country or region they belong to seems = to be too harsh to >>> me, as we use this information for several things, and should not abuse i= t for our own purposes. >>=20 >> (I assume) we still have AS information about those networks and we should= keep at least that. >=20 > ACK. >=20 >>=20 >>> Therefore, I rather suggest adding an additional column to countries.txt = for indicating whether >>> this country code is acceptable to us or not. >>=20 >> Okay. What would you suggest? >=20 > Well, our "countries.txt" file's syntax currently looks like (delimited by = tabulators): >=20 > [ISO-3166-1-alpha2 country code] [regional code defined by us to know which= Whois server should be queried] [human-readable name of the country] >=20 > I would like to add two new columns to this file, one for declaring a count= ry invalid in terms of > libloc, the other one to declare it as suspicious so we can handle those ca= ses better in a later > step. Just for me to check if I understood this correctly: You would like to add a country like =E2=80=9CYU=E2=80=9D to the list and if = we hit it during the import, we will perform a certain action. What would tha= t action be? And what is a =E2=80=9Csuspicious country=E2=80=9D? I do not think it would g= o down well to mark an entire country as such. > Personally, I wish those two columns to be placed before the country name f= or readability reasons. It can only be before the country name because country names might have more = than one =E2=80=9Cword=E2=80=9D and we cannot split the list correctly any mo= re. > However, this would be a breaking change to existing libloc instances using= the "countries.txt" file > of our data repository. Yes. Can we not add markers or something else? Maybe throw away the CSV format and use something else that can be extended n= ext time? -Michael > Thanks, and best regards, > Peter M=C3=BCller >=20 >>=20 >> -Michael >>=20 >>>=20 >>> Thanks, and best regards, >>> Peter M=C3=BCller >>>=20 >>>> Hello, >>>>=20 >>>> How about we add the country to the list and mark the continent with a = =E2=80=9C-=E2=80=9C: >>>>=20 >>>> AF AS Afghanistan >>>> YU - Yugoslavia >>>>=20 >>>> -Michael >>>>=20 >>>>> On 10 Apr 2021, at 13:32, Peter M=C3=BCller wrote: >>>>>=20 >>>>> Hello Michael, >>>>>=20 >>>>> thanks for your reply. >>>>>=20 >>>>> Yes, having this configurable in countries.txt would be nice indeed. Do= you propose a certain syntax for this? >>>>>=20 >>>>> And yes, a tuple is a better idea here. I will wait for your reply and = submit a second version of this patch then. >>>>>=20 >>>>> Thanks, and best regards, >>>>> Peter M=C3=BCller >>>>>=20 >>>>>=20 >>>>>> Hello, >>>>>>=20 >>>>>> * would we not want this to be configurable in countries.txt? >>>>>>=20 >>>>>> * The list should probably be a tuple. >>>>>>=20 >>>>>> -Michael >>>>>>=20 >>>>>>> On 1 Apr 2021, at 20:57, Peter M=C3=BCller wrote: >>>>>>>=20 >>>>>>> Both the Bouvet Island (BV) and Antarctica (AQ) are unpopulated at the >>>>>>> time of writing. Network owners/operators putting these countries into >>>>>>> their RIR data objects are either completely braindead or doing so for >>>>>>> hostile reasons. >>>>>>>=20 >>>>>>> While we might correct these locations to something useful by manually >>>>>>> creating overrides for them, the rationale behind this patch is not to >>>>>>> let these countries appear on productive systems in the first place, = as >>>>>>> we know they _cannot_ be true. >>>>>>>=20 >>>>>>> Therefore, this patch skips any network object that has either AQ or = BV >>>>>>> country code set. >>>>>>>=20 >>>>>>> See also: https://lists.ipfire.org/pipermail/location/2020-October/00= 0199.html >>>>>>>=20 >>>>>>> Signed-off-by: Peter M=C3=BCller >>>>>>> --- >>>>>>> src/python/location-importer.in | 2 +- >>>>>>> 1 file changed, 1 insertion(+), 1 deletion(-) >>>>>>>=20 >>>>>>> diff --git a/src/python/location-importer.in b/src/python/location-im= porter.in >>>>>>> index 1e08458..ac678dc 100644 >>>>>>> --- a/src/python/location-importer.in >>>>>>> +++ b/src/python/location-importer.in >>>>>>> @@ -624,7 +624,7 @@ class CLI(object): >>>>>>> return >>>>>>>=20 >>>>>>> # Skip objects with unknown country codes >>>>>>> - if validcountries and inetnum.get("country") not in validcountries: >>>>>>> + if validcountries and (inetnum.get("country") not in validcountrie= s or inetnum.get("country") in ["AQ", "BV"]): >>>>>>> log.warning("Skipping network with bogus country '%s': %s" % \ >>>>>>> (inetnum.get("country"), inetnum.get("inet6num") or inetnum.get("= inetnum"))) >>>>>>> return >>>>>>> --=20 >>>>>>> 2.26.2 --===============8034916945771481314==--