Location November 2020

location@lists.ipfire.org

3 participants
7 discussions

[PATCH] location-importer.in: skip networks with unknown country codes
by Peter Müller 04 Feb '21

04 Feb '21

There is no sense in parsing and storting networks whose country codes cannot be found in the ISO-3166-x country code table. This avoids side effects in applications using the location database, and introduces another sanity check to compensate bogus RIR data. On location02, this affects some networks from APNIC (country code: ZZ) as well as a bunch of smaller allocations within the RIPE region still tagged to CS or YU (Yugoslavia). To my surprise, no network tagged as SU (Soviet Union) was found - while the NIC for .su TLD is still operational. :-) Fixes: #12510 Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- src/python/location-importer.in | 42 ++++++++++++++++++++++----------- 1 file changed, 28 insertions(+), 14 deletions(-) diff --git a/src/python/location-importer.in b/src/python/location-importer.in index 864eab1..89b556a 100644 --- a/src/python/location-importer.in +++ b/src/python/location-importer.in @@ -388,10 +388,17 @@ class CLI(object): TRUNCATE TABLE networks; """) + # Fetch all valid country codes to check parsed networks aganist... + rows = self.db.query("SELECT * FROM countries ORDER BY country_code") + validcountries = [] + + for row in rows: + validcountries.append(row.country_code) + for source in location.importer.WHOIS_SOURCES: with downloader.request(source, return_blocks=True) as f: for block in f: - self._parse_block(block) + self._parse_block(block, validcountries) # Process all parsed networks from every RIR we happen to have access to, # insert the largest network chunks into the networks table immediately... @@ -467,7 +474,7 @@ class CLI(object): # Download data with downloader.request(source) as f: for line in f: - self._parse_line(line) + self._parse_line(line, validcountries) def _check_parsed_network(self, network): """ @@ -532,7 +539,7 @@ class CLI(object): # be suitable for libloc consumption... return True - def _parse_block(self, block): + def _parse_block(self, block, validcountries = None): # Get first line to find out what type of block this is line = block[0] @@ -542,7 +549,7 @@ class CLI(object): # inetnum if line.startswith("inet6num:") or line.startswith("inetnum:"): - return self._parse_inetnum_block(block) + return self._parse_inetnum_block(block, validcountries) # organisation elif line.startswith("organisation:"): @@ -573,7 +580,7 @@ class CLI(object): autnum.get("asn"), autnum.get("org"), ) - def _parse_inetnum_block(self, block): + def _parse_inetnum_block(self, block, validcountries = None): log.debug("Parsing inetnum block:") inetnum = {} @@ -624,17 +631,17 @@ class CLI(object): if not inetnum or not "country" in inetnum: return - # Skip objects with bogus country code 'ZZ' - if inetnum.get("country") == "ZZ": - log.warning("Skipping network with bogus country 'ZZ': %s" % \ - (inetnum.get("inet6num") or inetnum.get("inetnum"))) - return - network = ipaddress.ip_network(inetnum.get("inet6num") or inetnum.get("inetnum"), strict=False) if not self._check_parsed_network(network): return + # Skip objects with unknown country codes + if validcountries and inetnum.get("country") not in validcountries: + log.warning("Skipping network with bogus country '%s': %s" % \ + (inetnum.get("country"), inetnum.get("inet6num") or inetnum.get("inetnum"))) + return + self.db.execute("INSERT INTO _rirdata(network, country) \ VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country = excluded.country", "%s" % network, inetnum.get("country"), @@ -659,7 +666,7 @@ class CLI(object): org.get("organisation"), org.get("org-name"), ) - def _parse_line(self, line): + def _parse_line(self, line, validcountries = None): # Skip version line if line.startswith("2"): return @@ -674,8 +681,15 @@ class CLI(object): log.warning("Could not parse line: %s" % line) return - # Skip any lines that are for stats only - if country_code == "*": + # Skip any lines that are for stats only or do not have a country + # code at all (avoids log spam below) + if not country_code or country_code == '*': + return + + # Skip objects with unknown country codes + if validcountries and country_code not in validcountries: + log.warning("Skipping line with bogus country '%s': %s" % \ + (country_code, line)) return if type in ("ipv6", "ipv4"): -- 2.20.1

1 2

Next steps of IPFire Location
by Michael Tremer 28 Nov '20

28 Nov '20

Hello, Since there are now so many people involved in this project, I would like to send a little email to report what has happened and what the next steps are going to be… I have spent (unfortunately) a lot of time to add the ability that we can “flatten the tree” when we export it. That means that no network will overlap with another one and we do not need to care about any orders of iptables rules being added. That was quite a slow process in Python and since IPFire has to run on small hardware it needed to be ported into C. The tree can now be exported in about 10 seconds on my development box which is running an Intel Xeon E5-2630 processor. I hope that we can reach acceptable times of only a few minutes on the slowest systems and way under a minute on an average system. To find out about that, I need you help: I just pushed all changes into the next tree and would everyone who can to install that and report how long this command runs: mkdir tmp time location export --directory=tmp --format=xt_geoip It would also be great if you could all check the output if this is still correct. I have done what I could, but I want to have an extra pair of eyes if possible. If this works well, and no more regressions are being found, I will tag this release and we will close Core Update 153. We will then roll out all recent changes that make the database better, but had to be held back because the tree needs to be flattened for most use-cases now. All older clients will then no longer receive an updated database any more. Happy testing. Best, -Michael

3 8

Re: Next steps of IPFire Location
by Michael Tremer 26 Nov '20

26 Nov '20

Hello Arne, Thank you for spotting this. I have fixed the issue with some of the networks not being sorted correctly. Since this has affected performance slightly, the code has now become slightly faster again. Regarding the 32 bit issue: This is a pain to debug now because I do not have a working 32 bit build environment with a debugger. I am compiling this now and I will work on it tomorrow. Your export of A1 was incorrect because it wasn’t exported at all. That must have been old data because I forgot to copy the flags, too. Best, -Michael > On 26 Nov 2020, at 07:51, Arne Fitzenreiter <arne_f(a)ipfire.org> wrote: > > Bad news with current nightly. The export segfault on 32bit > > Also the exported data on RPi3 was still not in > the correct order. In my export the fourth entry of A1.ipv4 is lower than the third. > > Arne > > Am 2020-11-25 21:24, schrieb Michael Tremer: >> Ladies and gentlemen, >> After days and days of looking into this, and neglecting all the more >> important things I should be doing, I have updated libloc in the >> testing tree. >> The good news is that I think that I have fixed the issue. I would >> like you all to check this for me if you could :) >> The bad news is that the reason the output was incorrect was that the >> more complicated code path was not being used when it was required. >> This has been fixed now and it is actually fast enough I think. That >> again has to be tested. >> If you all would repeat the last test with the updated version as soon >> as it finishes its build. >> It will run for many many minutes, if not an hour. I want to know that >> figure. We will bring it down. >> Best, >> -Michael >>> On 23 Nov 2020, at 11:16, Michael Tremer <michael.tremer(a)ipfire.org> wrote: >>> Hi, >>> Yes, I can confirm this. The subnets returned are not unique. >>> -Michael >>>> On 22 Nov 2020, at 11:59, Peter Müller <peter.mueller(a)ipfire.org> wrote: >>>> Hello *, >>>> I just stumbled across a strange issue on my testing machine strongly reminding me of a problem with the xt_geoip module if we printed the location database contents non-planar: >>>> For debugging and monitoring reasons, a NRPE job pinging ping.ipfire.org (which resolves to 81.3.27.38) is executed periodically. Since a few hours, this job fails due to ICMP packets to this IP address being dropped by the outgoing firewall engine. >>>> Its policy is set to "drop", with a rule allowing ICMP traffic to DE. While "location lookup 81.3.27.38" correctly returns DE for the prefix 31.8.0.0/18, it seems as the xt_geoip module did not get that and misclassifies this IP address. >>>> Is somebody able to verify or falsify this issue on Core Development Build 153? >>>> Thanks, and best regards, >>>> Peter Müller

1 0

Re: Next steps of IPFire Location
by Michael Tremer 23 Nov '20

23 Nov '20

Thank you for all your feedback. It helped a lot. Or rather… It would have if Arne didn’t find that this whole algorithm doesn’t work :) It simply does not do what we expect it to do and after my fix this takes a long time again. I hope that I will find some time this week to improve the implementation further and will get back to you all very soon :) Best, -Michael > On 22 Nov 2020, at 20:06, Adolf Belka <ahb.ipfire(a)gmail.com> wrote: > > Dear All, > > > I have the results now for the location testing on my Intel(R) Celeron(R) CPU J1900 @ 1.99GHz production box. > > Stable > real 0m37.240s > user 0m37.086s > sys 0m0.070s > > > Unstable > real 0m25.827s > user 0m25.568s > sys 0m0.123s > > > Regards, > > Adolf. > > On 19/11/2020 18:48, Adolf Belka wrote: >> I have tested it out on my IPFire virtual machine test bed. >> >> I got between 4.8 and 5 secs. >> >> The computer is an i5-8400 CPU @ 2.80GHz >> >> I will also be testing it out on my production box which is a Intel(R) Celeron(R) CPU J1900 @ 1.99GHz but I have to prepare for that as I will need to re-install Core152 after carrying out the test. Will email the results when I have them. >> >> Regards, >> Adolf. >> >> >> On 19/11/2020 14:10, Michael Tremer wrote: >>> I managed to export the database on my IPFire Mini Appliance in about one minute: >>> >>> [root@fw01 libloc]# time location export --directory=tmp --format=xt_geoip >>> >>> real 1m3.423s >>> user 1m1.470s >>> sys 0m1.009s >>> >>> The system was under a little bit of load due to some network activity, but I am generally quite happy with this. >>> >>> I added some more fine-tuning and could decrease the export time on my Debian test machine by another 40% to around six seconds. >>> >>> -Michael >>> >>>> On 18 Nov 2020, at 19:26, Michael Tremer <michael.tremer(a)ipfire.org> wrote: >>>> >>>> Hello, >>>> >>>> Since there are now so many people involved in this project, I would like to send a little email to report what has happened and what the next steps are going to be… >>>> >>>> I have spent (unfortunately) a lot of time to add the ability that we can “flatten the tree” when we export it. That means that no network will overlap with another one and we do not need to care about any orders of iptables rules being added. >>>> >>>> That was quite a slow process in Python and since IPFire has to run on small hardware it needed to be ported into C. The tree can now be exported in about 10 seconds on my development box which is running an Intel Xeon E5-2630 processor. >>>> >>>> I hope that we can reach acceptable times of only a few minutes on the slowest systems and way under a minute on an average system. To find out about that, I need you help: >>>> >>>> I just pushed all changes into the next tree and would everyone who can to install that and report how long this command runs: >>>> >>>> mkdir tmp >>>> time location export --directory=tmp --format=xt_geoip >>>> >>>> It would also be great if you could all check the output if this is still correct. I have done what I could, but I want to have an extra pair of eyes if possible. >>>> >>>> If this works well, and no more regressions are being found, I will tag this release and we will close Core Update 153. >>>> >>>> We will then roll out all recent changes that make the database better, but had to be held back because the tree needs to be flattened for most use-cases now. All older clients will then no longer receive an updated database any more. >>>> >>>> Happy testing. >>>> >>>> Best, >>>> -Michael >>>

1 0

[PATCH] countries: treat Cyprus as being part of the European area
by Peter Müller 16 Nov '20

16 Nov '20

While this country is politically split, it's IP space is maintained by RIPE, which is why we have to tag it as "EU" rather than "AS" to ensure ipinfo.cgi asking the right WHOIS servers. Fixes: #12524 Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- countries.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/countries.txt b/countries.txt index ced8aa9..faaa902 100644 --- a/countries.txt +++ b/countries.txt @@ -54,7 +54,7 @@ CU NA Cuba CV AF Cabo Verde CW SA Curaçao CX AS Christmas Island -CY AS Cyprus +CY EU Cyprus CZ EU Czechia DE EU Germany DJ AF Djibouti -- 2.26.2

1 0

[PATCH] override-{a[1-3],other}: regular batch of various overrides
by Peter Müller 16 Nov '20

16 Nov '20

Since the "Asline" IP hijacking gang tampers with RIR data, probably to evade location based firewall rules, their Autonomous Systems were pinned to the AP region (the given Hong Kong contact address seems to be bogus for at least one /16 stolen AFRINIC chunk) for safety reasons. Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- overrides/override-a1.txt | 25 +++++++++++++++---------- overrides/override-a2.txt | 12 ++++++++++++ overrides/override-a3.txt | 5 +++++ overrides/override-other.txt | 35 +++++++++++++++++++++++++++++++++++ 4 files changed, 67 insertions(+), 10 deletions(-) diff --git a/overrides/override-a1.txt b/overrides/override-a1.txt index e81d6c2..7aca339 100644 --- a/overrides/override-a1.txt +++ b/overrides/override-a1.txt @@ -596,16 +596,6 @@ descr: ThinkTech Technology Industrial CO. Limited remarks: VPN provider is-anonymous-proxy: yes -net: 94.199.160.0/23 -descr: MIK Telecom VPN pool -remarks: VPN provider -is-anonymous-proxy: yes - -net: 95.129.56.0/21 -descr: Azimut-R VPN Service -remarks: VPN provider -is-anonymous-proxy: yes - net: 91.193.75.0/24 descr: KGB Hosting d.o.o. / David Craig remarks: (Rogue) VPN provider @@ -616,6 +606,21 @@ descr: Privax LTD remarks: VPN provider is-anonymous-proxy: yes +net: 92.118.39.0/24 +descr: CloudMine NET +remarks: VPN provider [high confidence, but not proofed] +is-anonymous-proxy: yes + +net: 94.199.160.0/23 +descr: MIK Telecom VPN pool +remarks: VPN provider +is-anonymous-proxy: yes + +net: 95.129.56.0/21 +descr: Azimut-R VPN Service +remarks: VPN provider +is-anonymous-proxy: yes + net: 95.154.64.0/18 descr: Octopusnet VPN remarks: VPN provider diff --git a/overrides/override-a2.txt b/overrides/override-a2.txt index 8f03159..a55c940 100644 --- a/overrides/override-a2.txt +++ b/overrides/override-a2.txt @@ -152,6 +152,12 @@ descr: Arab Satellite Communications Organization remarks: Satellite Internet provider is-satellite-provider: yes +aut-num: AS42962 +descr: CoreLink Communications +remarks: Chinese satellite Internet provider [high confidence, but not proofed] +is-satellite-provider: yes +country: AP + aut-num: AS43905 descr: Telenor Satellite AS remarks: Satellite Internet provider @@ -1616,3 +1622,9 @@ net: 2a04:2880::/30 descr: Satellite Solutions Worldwide Ltd remarks: Satellite Internet provider is-satellite-provider: yes + +net: 2a0a:2840::/29 +descr: CoreLink Communications +remarks: Chinese satellite Internet provider [high confidence, but not proofed] +is-satellite-provider: yes +country: AP diff --git a/overrides/override-a3.txt b/overrides/override-a3.txt index 924c859..07b2621 100644 --- a/overrides/override-a3.txt +++ b/overrides/override-a3.txt @@ -1527,6 +1527,11 @@ descr: marbis GmbH remarks: Generic anycast network [high confidence, but not proofed] is-anycast: yes +net: 2a05:7f00::/29 +descr: nic.at GmbH and friends +remarks: TLD operator's anycast network +is-anycast: yes + net: 2a06:e881:4001::/48 descr: Thomas Harwood remarks: Public anycast DNS resolver diff --git a/overrides/override-other.txt b/overrides/override-other.txt index d4c3f5b..98ea79b 100644 --- a/overrides/override-other.txt +++ b/overrides/override-other.txt @@ -18,6 +18,16 @@ descr: Iron Mountain Data Center remarks: ISP located in US, but some RIR data for announced prefixes contain garbage country: US +aut-num: AS18013 +descr: ASLINE LIMITED +remarks: IP hijacker, traces back to AP region +country: AP + +aut-num: AS18254 +descr: KLAYER LLC +remarks: part of the "Asline" IP hijacking gang, traces back to AP region +country: AP + aut-num: AS24700 descr: Yes Networks Unlimited Ltd remarks: traces to UA, but some RIR entries seem to contain garbage (VG) @@ -33,6 +43,11 @@ descr: IP Interactive UG (haftungsbeschraenkt) remarks: ISP located in BG, but RIR data for announced prefixes contain garbage country: BG +aut-num: AS35478 +descr: Buena Telecom SRL +remarks: ISP located in RO, but RIR data for announced prefixes contain garbage +country: RO + aut-num: AS37518 descr: Fiber Grid Inc. remarks: tampers with RIR data, traces back to SE @@ -73,6 +88,11 @@ descr: PPTECHNOLOGY LIMITED remarks: bulletproof ISP (related to AS204655) located in NL country: NL +aut-num: AS49466 +descr: KLAYER LLC +remarks: part of the "Asline" IP hijacking gang, traces back to AP region +country: AP + aut-num: AS49505 descr: Selectel remarks: ISP located in RU, but some RIR data for announced prefixes contain garbage @@ -108,6 +128,11 @@ descr: DXTL Tseung Kwan O Service remarks: tampers with RIR data, traces back to AP region country: AP +aut-num: AS137951 +descr: Clayer Limited +remarks: part of the "Asline" IP hijacking gang, tampers with RIR data, traces back to AP region +country: AP + aut-num: AS201133 descr: Verdina Ltd. remarks: ISP located in BG, but RIR data for announced prefixes contain garbage @@ -138,6 +163,11 @@ descr: Altrosky Technology Ltd. remarks: fake offshore location (SC), traces back to CZ and NL country: EU +aut-num: AS208046 +descr: Maximilian Kutzner trading as HostSlick +remarks: traces back to NL, but some RIR data for announced prefixes contain garbage +country: NL + aut-num: AS209132 descr: Alviva Holding Limited remarks: ISP located in BG, but RIR data for announced prefixes contain garbage @@ -158,6 +188,11 @@ descr: IP Connect Inc. remarks: fake offshore location (SC), traces back to NL country: NL +aut-num: AS398478 +descr: PEG TECH INC +remarks: ISP located in HK, tampers with RIR data +country: HK + net: 5.252.32.0/22 descr: StormWall s.r.o. remarks: claims to be located in DE, but traces back to somewhere else in central Europe -- 2.26.2

1 0

[PATCH] location-importer.in: always convert organisation handles into upper cases
by Peter Müller 03 Nov '20

03 Nov '20

Fixes: #12523 Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- src/python/location-importer.in | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/python/location-importer.in b/src/python/location-importer.in index 864eab1..2dec89e 100644 --- a/src/python/location-importer.in +++ b/src/python/location-importer.in @@ -560,7 +560,7 @@ class CLI(object): autnum["asn"] = m.group(2) elif key == "org": - autnum[key] = val + autnum[key] = val.upper() # Skip empty objects if not autnum: @@ -646,7 +646,9 @@ class CLI(object): # Split line key, val = split_line(line) - if key in ("organisation", "org-name"): + if key == "organisation": + org[key] = val.upper() + elif key == "org-name": org[key] = val # Skip empty objects -- 2.20.1

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Location November 2020