Location October 2020

location@lists.ipfire.org

3 participants
14 discussions

What generates database.db from database.txt?
by Gisle Vanem 21 Oct '20

21 Oct '20

Hello list, my ports post. I'm on Windows and have little interest in installing IPFire or do any Linux related. But the 'libloc' library caught my interest. So I did a port to MSVC, MinGW and clang-cl (x86 only) in a hurry. Running all the 'test-*' programs, it worked seemingly well. Until I built the Python3 module and played with e.g. 'location dump' and 'verify'. Nothing worked :-( So I reckon the .db file is hosed somehow. AFAICS, it's not an issue with structure packing or 'fopen(path, "r")'. Anyway, I want to regenerate 'database.db' from 'database.txt' 'countries.txt' and the 'override' files myself. But how? According to the git log at: git://git.ipfire.org/location/location-database.git there once was a 'compile-database' script. Not now. -- --gv

2 7

[PATCH] Add overrides for RFC1918
by Michael Tremer 15 Oct '20

15 Oct '20

Because of RIPE (or somebody else) adding 192.168.0.0/15, we are classifying those addresses as EU. This renders whole networks inaccessible and therefore we are adding those networks with a faked country until we find a better solution. Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org> --- overrides/override-rfc1918.txt | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 overrides/override-rfc1918.txt diff --git a/overrides/override-rfc1918.txt b/overrides/override-rfc1918.txt new file mode 100644 index 0000000..768ea92 --- /dev/null +++ b/overrides/override-rfc1918.txt @@ -0,0 +1,18 @@ +# +# override-rfc1918 [.txt] +# +# This files contains networks from RFC1918 (and other similar standards) +# which are added to the database with special attributes +# + +net: 10.0.0.0/8 +descr: Private address space from RFC1918 +country: XX + +net: 172.16.0.0/12 +descr: Private address space from RFC1918 +country: XX + +net: 192.168.0.0/16 +descr: Private address space from RFC1918 +country: XX -- 2.20.1

1 1

[PATCH 1/2] Revert "Revert "importer: Import raw sources for inetnum's again""
by Peter Müller 12 Oct '20

12 Oct '20

This reverts commit 92f6abf4e272672bb0a71cfe991261b95ebe2fef. --- src/python/importer.py | 14 ++++---- src/python/location-importer.in | 63 +++++++++++++++++++++++++++++++++ 2 files changed, 70 insertions(+), 7 deletions(-) diff --git a/src/python/importer.py b/src/python/importer.py index de20f37..f19db4b 100644 --- a/src/python/importer.py +++ b/src/python/importer.py @@ -30,8 +30,8 @@ WHOIS_SOURCES = ( "https://ftp.afrinic.net/pub/pub/dbase/afrinic.db.gz", # Asia Pacific Network Information Centre - #"https://ftp.apnic.net/apnic/whois/apnic.db.inet6num.gz", - #"https://ftp.apnic.net/apnic/whois/apnic.db.inetnum.gz", + "https://ftp.apnic.net/apnic/whois/apnic.db.inet6num.gz", + "https://ftp.apnic.net/apnic/whois/apnic.db.inetnum.gz", #"https://ftp.apnic.net/apnic/whois/apnic.db.route6.gz", #"https://ftp.apnic.net/apnic/whois/apnic.db.route.gz", "https://ftp.apnic.net/apnic/whois/apnic.db.aut-num.gz", @@ -45,8 +45,8 @@ WHOIS_SOURCES = ( # XXX ??? # Réseaux IP Européens - #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.inet6num.gz", - #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.inetnum.gz", + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inet6num.gz", + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inetnum.gz", #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route6.gz", #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route.gz", "https://ftp.ripe.net/ripe/dbase/split/ripe.db.aut-num.gz", @@ -55,10 +55,10 @@ WHOIS_SOURCES = ( EXTENDED_SOURCES = ( # African Network Information Centre - "https://ftp.afrinic.net/pub/stats/afrinic/delegated-afrinic-extended-latest", + #"https://ftp.afrinic.net/pub/stats/afrinic/delegated-afrinic-extended-latest", # Asia Pacific Network Information Centre - "https://ftp.apnic.net/apnic/stats/apnic/delegated-apnic-extended-latest", + #"https://ftp.apnic.net/apnic/stats/apnic/delegated-apnic-extended-latest", # American Registry for Internet Numbers "https://ftp.arin.net/pub/stats/arin/delegated-arin-extended-latest", @@ -67,7 +67,7 @@ EXTENDED_SOURCES = ( "http://ftp.lacnic.net/pub/stats/lacnic/delegated-lacnic-extended-latest", # Réseaux IP Européens - "https://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest", + #"https://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest", ) class Downloader(object): diff --git a/src/python/location-importer.in b/src/python/location-importer.in index 77952f2..e3a07a0 100644 --- a/src/python/location-importer.in +++ b/src/python/location-importer.in @@ -393,6 +393,10 @@ class CLI(object): if line.startswith("aut-num:"): return self._parse_autnum_block(block) + # inetnum + if line.startswith("inet6num:") or line.startswith("inetnum:"): + return self._parse_inetnum_block(block) + # organisation elif line.startswith("organisation:"): return self._parse_org_block(block) @@ -422,6 +426,65 @@ class CLI(object): autnum.get("asn"), autnum.get("org"), ) + def _parse_inetnum_block(self, block): + logging.debug("Parsing inetnum block:") + + inetnum = {} + for line in block: + logging.debug(line) + + # Split line + key, val = split_line(line) + + if key == "inetnum": + start_address, delim, end_address = val.partition("-") + + # Strip any excess space + start_address, end_address = start_address.rstrip(), end_address.strip() + + # Convert to IP address + try: + start_address = ipaddress.ip_address(start_address) + end_address = ipaddress.ip_address(end_address) + except ValueError: + logging.warning("Could not parse line: %s" % line) + return + + # Set prefix to default + prefix = 32 + + # Count number of addresses in this subnet + num_addresses = int(end_address) - int(start_address) + if num_addresses: + prefix -= math.log(num_addresses, 2) + + inetnum["inetnum"] = "%s/%.0f" % (start_address, prefix) + + elif key == "inet6num": + inetnum[key] = val + + elif key == "country": + if val == "UNITED STATES": + val = "US" + + inetnum[key] = val.upper() + + # Skip empty objects + if not inetnum: + return + + network = ipaddress.ip_network(inetnum.get("inet6num") or inetnum.get("inetnum"), strict=False) + + # Bail out in case we have processed a non-public IP network + if network.is_private: + logging.warning("Skipping non-globally routable network: %s" % network) + return + + self.db.execute("INSERT INTO networks(network, country) \ + VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country = excluded.country", + "%s" % network, inetnum.get("country"), + ) + def _parse_org_block(self, block): org = {} for line in block: -- 2.20.1

1 1

[PATCH] overrides/override-{a[1-3],other}.txt: weekly batch of override updates
by Peter Müller 05 Oct '20

05 Oct '20

Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- overrides/override-a1.txt | 52 +++++++++++++++++------------------- overrides/override-a2.txt | 10 +++++++ overrides/override-a3.txt | 10 +++++++ overrides/override-other.txt | 40 +++++++++++++++++++++++++++ 4 files changed, 84 insertions(+), 28 deletions(-) diff --git a/overrides/override-a1.txt b/overrides/override-a1.txt index 9f64dd8..d4fe379 100644 --- a/overrides/override-a1.txt +++ b/overrides/override-a1.txt @@ -38,6 +38,11 @@ descr: VPN de Mexico, S.A. de C.V. remarks: VPN provider is-anonymous-proxy: yes +aut-num: AS34962 +descr: Epik Network +remarks: VPN provider +is-anonymous-proxy: yes + aut-num: AS35029 descr: WebLine LTD remarks: (Rogue) VPN provider @@ -55,6 +60,12 @@ descr: Asiamax Ltd. VPN remarks: VPN provider is-anonymous-proxy: yes +aut-num: AS43233 +descr: VPS 404 Ltd. +remarks: VPN provider [high confidence, but not proofed] located in ES +is-anonymous-proxy: yes +country: ES + aut-num: AS45792 descr: Layer 3 VPN ASN remarks: VPN provider @@ -132,6 +143,11 @@ descr: Privex Inc. remarks: VPN and Tor relay provider is-anonymous-proxy: yes +aut-num: AS212987 +descr: NekoCloud Solutions Limited +remarks: VPN provider [high confidence, but not proofed] +is-anonymous-proxy: yes + aut-num: AS394087 descr: Secure Internet LLC / PureVPN remarks: VPN provider @@ -322,6 +338,11 @@ descr: GZ Systems Limited / PureVPN remarks: VPN provider is-anonymous-proxy: yes +net: 45.9.12.0/22 +descr: VPNHost SIA +remarks: VPN provider +is-anonymous-proxy: yes + net: 45.12.144.0/22 descr: VPNHost SIA remarks: VPN provider @@ -1123,29 +1144,9 @@ descr: Canar VPN Range remarks: VPN provider is-anonymous-proxy: yes -net: 196.52.50.0/24 -descr: BGRVPN -remarks: VPN provider -is-anonymous-proxy: yes - -net: 196.53.164.0/22 -descr: BGRVPN -remarks: VPN provider -is-anonymous-proxy: yes - -net: 196.53.82.0/24 -descr: VPNetworks -remarks: VPN provider -is-anonymous-proxy: yes - -net: 196.53.83.0/24 -descr: VPNetworks -remarks: VPN provider -is-anonymous-proxy: yes - -net: 196.54.53.0/24 -descr: BGRVPN -remarks: VPN provider +net: 196.52.0.0/14 +descr: LogicWeb Inc. / BGRVPN / Private Internet Access / VPNetworks / CookieProxy / etc. pp. +remarks: large IP chunk mostly used by VPN providers is-anonymous-proxy: yes net: 197.221.161.0/24 @@ -1288,11 +1289,6 @@ descr: OVPN Integritet AB remarks: VPN provider is-anonymous-proxy: yes -net: 217.115.10.128/28 -descr: Chaos Computer Club e.V. -remarks: Tor relay provider -is-anonymous-proxy: yes - net: 217.148.75.0/24 descr: ITNow S.A. VPN pool / CAXIA remarks: VPN provider diff --git a/overrides/override-a2.txt b/overrides/override-a2.txt index 88783d8..8f03159 100644 --- a/overrides/override-a2.txt +++ b/overrides/override-a2.txt @@ -227,6 +227,11 @@ descr: Alpha Sattelite Network remarks: Satellite Internet provider is-satellite-provider: yes +aut-num: AS203770 +descr: Matooma SAS +remarks: Satellite Internet provider +is-satellite-provider: yes + aut-num: AS393582 descr: VIDEO DIRECT SATELLITE & ENTERTAINMENT / WisperISP remarks: Satellite Internet provider @@ -927,6 +932,11 @@ descr: Satellite Solutions Worldwide Ltd remarks: Satellite Internet provider is-satellite-provider: yes +net: 145.249.96.0/21 +descr: Media Broadcast Satellite GmbH +remarks: Satellite Internet provider +is-satellite-provider: yes + net: 154.73.28.0/22 descr: RAWAFED Libya remarks: Satellite Internet provider diff --git a/overrides/override-a3.txt b/overrides/override-a3.txt index e692f4c..26acb64 100644 --- a/overrides/override-a3.txt +++ b/overrides/override-a3.txt @@ -184,6 +184,11 @@ descr: subspace remarks: Generic anycast network [high confidence, but not proofed] is-anycast: yes +net: 185.89.134.0/24 +descr: Isset +remarks: Generic anycast network +is-anycast: yes + net: 185.143.232.0/22 descr: Noyan Abr Arvan Co. remarks: Generic anycast network @@ -214,6 +219,11 @@ descr: xTom / dns.sb remarks: Public anycast DNS resolver is-anycast: yes +net: 196.55.0.0/24 +descr: Epik Network +remarks: Generic anycast network +is-anycast: yes + net: 196.216.168.0/24 descr: AfriNIC Anycast Services remarks: Generic anycast network diff --git a/overrides/override-other.txt b/overrides/override-other.txt index f8e6799..67b7024 100644 --- a/overrides/override-other.txt +++ b/overrides/override-other.txt @@ -13,6 +13,11 @@ # Please keep this file sorted. # +aut-num: AS12025 +descr: Iron Mountain Data Center +remarks: ISP located in US, but some RIR data for announced prefixes contain garbage +country: US + aut-num: AS24700 descr: Yes Networks Unlimited Ltd remarks: traces to UA, but some RIR entries seem to contain garbage (VG) @@ -33,11 +38,26 @@ descr: Bunea TELECOM SRL remarks: ISP located in RO, but some RIR data for announced prefixes contain garbage country: RO +aut-num: AS42745 +descr: Safe Value Limited +remarks: tampers with RIR data, traces back to somewhere in central Europe +country: EU + +aut-num: AS42831 +descr: UKSERVERS +remarks: ISP located in UK, but some RIR data for announced prefixes contain garbage +country: UK + aut-num: AS48090 descr: PPTECHNOLOGY LIMITED remarks: bulletproof ISP (related to AS204655) located in NL country: NL +aut-num: AS51558 +descr: Smart Telecom S.A.R.L +remarks: tampers with RIR data, traces back to RU +country: RU + aut-num: AS56611 descr: REBA Communications BV remarks: bulletproof ISP (related to AS202425) located in NL @@ -103,6 +123,16 @@ descr: Private Internet Hosting LTD remarks: bulletproof ISP located in RU country: RU +aut-num: AS213373 +descr: IP Connect Inc. +remarks: fake offshore location (SC), traces back to NL +country: NL + +net: 5.252.32.0/22 +descr: StormWall s.r.o. +remarks: claims to be located in DE, but traces back to somewhere in central Europe +country: EU + net: 31.220.0.0/22 descr: Amarutu Technology Ltd. / KoDDoS / ESecurity remarks: fake offshore location (BZ), traces back to NL @@ -143,6 +173,11 @@ descr: QWARTA LLC remarks: fake location (US), WHOIS contact and traceroutes point to RU country: RU +net: 103.197.148.0/22 +descr: I.C.S. Trabia-Network S.R.L. +remarks: fake offshore location (HK), traces back to MD +country: MD + net: 178.239.20.0/24 descr: Anthony Marshall / Game Hosting Net / FlokiNET Ltd. remarks: fake location (BA), traces back to RO @@ -187,3 +222,8 @@ net: 2a02:e00:ffe9::/48 descr: Bradler & Krantz GmbH & Co. KG remarks: allocated to DE, but actually announced from GB country: GB + +net: 2a05:9f06::/31 +descr: Sibyl System LTD +remarks: RIR data contain garbage +country: NL -- 2.26.2

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Location October 2020