- Location - lists.ipfire.org

[PATCH] overrides-*: weekly batch of various overrides
by Peter Müller 10 Jun '21

10 Jun '21

Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- overrides/override-a1.txt | 36 +++++++++ overrides/override-a2.txt | 10 +++ overrides/override-a3.txt | 88 +++++++++++++++++++- overrides/override-other.txt | 152 ++++++++++++++++++++++++++++++++--- 4 files changed, 274 insertions(+), 12 deletions(-) diff --git a/overrides/override-a1.txt b/overrides/override-a1.txt index 921dd0c..3734c56 100644 --- a/overrides/override-a1.txt +++ b/overrides/override-a1.txt @@ -77,6 +77,12 @@ remarks: VPN provider [high confidence, but not proofed] located in ES is-anonymous-proxy: yes country: ES +aut-num: AS44571 +descr: Netvillage Ltd. +remarks: VPN provider [high confidence, but not proofed] located in or near RU +is-anonymous-proxy: yes +country: RU + aut-num: AS45792 descr: Layer 3 VPN ASN remarks: VPN provider @@ -210,6 +216,11 @@ descr: BOET NOTIFY LTD. remarks: VPN provider [high confidence, but not proofed] is-anonymous-proxy: yes +aut-num: AS212371 +descr: Evolve Proxies Ltd. +remarks: VPN provider [high confidence, but not proofed] +is-anonymous-proxy: yes + aut-num: AS212786 descr: Pooky L&W SDN BHD remarks: VPN provider [high confidence, but not proofed] @@ -441,6 +452,11 @@ descr: VPNHost SIA remarks: VPN provider is-anonymous-proxy: yes +net: 45.14.80.0/22 +descr: ALTUS COMMUNICATIONS, INC. +remarks: VPN and/or open proxy provider [high confidence, but not proofed] +is-anonymous-proxy: yes + net: 45.41.132.0/24 descr: VPN Consumer Network remarks: VPN provider @@ -512,6 +528,16 @@ descr: Low budget VPN service remarks: VPN provider is-anonymous-proxy: yes +net: 45.221.64.0/21 +descr: Secure Internet Limited +remarks: VPN provider +is-anonymous-proxy: yes + +net: 45.231.206.0/23 +descr: SENTRIGLOBAL LTD +remarks: VPN provider [high confidence, but not proofed] +is-anonymous-proxy: yes + net: 46.36.200.0/22 descr: IAPS Security Services, L.L.C. remarks: VPN provider @@ -698,6 +724,11 @@ descr: VPN-Services / KDDI Corporation remarks: VPN provider is-anonymous-proxy: yes +net: 85.208.112.0/22 +descr: VPNHOST SIA +remarks: VPN provider +is-anonymous-proxy: yes + net: 88.81.248.0/24 descr: TopNet ISP VPN remarks: VPN provider @@ -954,6 +985,11 @@ descr: Cloud Innovation Ltd. remarks: hijacked AFRINIC IP chunk, owned by suspicous offshore company, scattered across dirty networks worldwide - not a safe place to go is-anonymous-proxy: yes +net: 156.0.200.0/22 +descr: xTom Limited +remarks: ... network operator thinks messing with countries and having an offshore company for it is funny :-/ +is-anonymous-proxy: yes + net: 161.129.60.0/24 descr: 10VPN Hosting remarks: VPN provider diff --git a/overrides/override-a2.txt b/overrides/override-a2.txt index b8b3b08..24db2a8 100644 --- a/overrides/override-a2.txt +++ b/overrides/override-a2.txt @@ -197,6 +197,11 @@ descr: Shaw Satellite Services, Inc. remarks: Satellite Internet provider [high confidence, but not proofed] is-satellite-provider: yes +aut-num: AS32806 +descr: Emerging Markets Communications +remarks: Satellite Internet provider +is-satellite-provider: yes + aut-num: AS35410 descr: LVS Transnational Corporation (tm) OOO Nord-telecom remarks: Satellite Internet provider [high confidence, but not proofed] @@ -293,6 +298,11 @@ remarks: Chinese satellite Internet provider [high confidence, but not proofe is-satellite-provider: yes country: AP +aut-num: AS42705 +descr: Talia +remarks: Satellite Internet provider +is-satellite-provider: yes + aut-num: AS42726 descr: BusinessCom CZ spol. s r.o. remarks: Satellite Internet provider diff --git a/overrides/override-a3.txt b/overrides/override-a3.txt index daacc95..9200b78 100644 --- a/overrides/override-a3.txt +++ b/overrides/override-a3.txt @@ -25,6 +25,11 @@ descr: DirectNIC, Ltd. remarks: Generic anycast network [high confidence, but not proofed] is-anycast: yes +aut-num: AS13335 +descr: Cloudflare, Inc. +remarks: Worldwide CDN, does not make sense to assign their networks to a specific country +is-anycast: yes + aut-num: AS16625 descr: Akamai Technologies, Inc. remarks: Worldwide CDN, does not make sense to assign their networks to a specific country @@ -37,14 +42,29 @@ is-anycast: yes aut-num: AS20940 descr: Akamai International BV -remarks: Worldwide CDN, does not make sense to assign their networks to a specific country +remarks: Worldwide CDN, does not make sense to assign their networks to a specific country is-anycast: yes +aut-num: AS21342 +descr: Akamai International BV +remarks: Worldwide CDN, does not make sense to assign their networks to a specific country +is-anycast: yes + +aut-num: AS24319 +descr: Akamai International BV +remarks: Worldwide CDN, does not make sense to assign their networks to a specific country +is-anycast: yesx + aut-num: AS28064 descr: Smart Data Center S.A. remarks: Generic anycast network [high confidence, but not proofed] is-anycast: yes +aut-num: AS29997 +descr: Vitalwerks Internet Solutions, LLC +remarks: Public anycast DNS resolver network [high confidence, but not proofed] +is-anycast: yes + aut-num: AS31529 descr: DENIC eG remarks: TLD operator's anycast network @@ -55,6 +75,11 @@ descr: Citigroup remarks: Public anycast DNS resolver network [high confidence, but not proofed] is-anycast: yes +aut-num: AS34164 +descr: Akamai International BV +remarks: Worldwide CDN, does not make sense to assign their networks to a specific country +is-anycast: yes + aut-num: AS34738 descr: WEBHOST LIMITED remarks: Generic anycast network [high confidence, but not proofed] @@ -80,6 +105,12 @@ descr: ANEXIA Internetdienstleistungs GmbH remarks: Public anycast DNS nameserver network [high confidence, but not proofed] is-anycast: yes +aut-num: AS49287 +descr: Melbikomas UAB +remarks: Generic anycast network [high confidence, but not proofed] +is-anycast: yes +country: EU + aut-num: AS48550 descr: Pascal Mathis trading as SnapServ Mathis remarks: Generic anycast network @@ -215,6 +246,11 @@ descr: Cloudflare, Inc. remarks: Public anycast DNS resolver is-anycast: yes +net: 2.16.0.0/13 +descr: Akamai International B.V. +remarks: IP chunk in use for providing CDN services worldwide, subnets scatter across various Autonomous Systems +is-anycast: yes + net: 2.58.58.0/24 descr: Hybula B.V. remarks: Generic anycast network @@ -250,6 +286,21 @@ descr: Quad9 remarks: Public anycast DNS resolver is-anycast: yes +net: 23.32.0.0/11 +descr: Akamai International B.V. +remarks: IP chunk in use for providing CDN services worldwide, subnets scatter across various Autonomous Systems +is-anycast: yes + +net: 23.72.0.0/13 +descr: Akamai International B.V. +remarks: IP chunk in use for providing CDN services worldwide, subnets scatter across various Autonomous Systems +is-anycast: yes + +net: 23.192.0.0/11 +descr: Akamai International B.V. +remarks: IP chunk in use for providing CDN services worldwide, subnets scatter across various Autonomous Systems +is-anycast: yes + net: 37.72.151.0/24 descr: LWLcom GmbH remarks: Generic anycast network @@ -575,6 +626,11 @@ descr: Thomas Steen Rasmussen / UncensoredDNS / censurfridns.dk remarks: Public anycast DNS resolver is-anycast: yes +net: 92.122.0.0/15 +descr: Akamai International B.V. +remarks: IP chunk in use for providing CDN services worldwide, subnets scatter across various Autonomous Systems +is-anycast: yes + net: 92.118.229.0/24 descr: Epik remarks: Generic anycast network @@ -650,6 +706,11 @@ descr: Vision247 remarks: Generic anycast network is-anycast: yes +net: 95.100.0.0/15 +descr: Akamai International B.V. +remarks: IP chunk in use for providing CDN services worldwide, subnets scatter across various Autonomous Systems +is-anycast: yes + net: 95.177.2.0/24 descr: Griffin Internet Infrastructure Anycast remarks: Generic anycast network @@ -660,6 +721,11 @@ descr: VikingHost remarks: Generic anycast network is-anycast: yes +net: 104.64.0.0/10 +descr: Akamai International B.V. +remarks: IP chunk in use for providing CDN services worldwide, subnets scatter across various Autonomous Systems +is-anycast: yes + net: 109.72.143.0/24 descr: GPC Services AG remarks: Generic anycast network @@ -1462,6 +1528,11 @@ descr: SafeDNS, Inc. remarks: Generic anycast network is-anycast: yes +net: 195.47.253.0/24 +descr: LLC "Hostmaster" +remarks: TLD operator's anycast network +is-anycast: yes + net: 195.80.153.0/24 descr: SAFECLOUDBOX remarks: Generic anycast network @@ -1587,6 +1658,11 @@ descr: Thomas Steen Rasmussen / UncensoredDNS / censurfridns.dk remarks: Public anycast DNS resolver is-anycast: yes +net: 2001:678:a80::/48 +descr: Informatics Solutions Limited +remarks: Generic anycast network +is-anycast: yes + net: 2606:4700:4700::/48 descr: Cloudflare, Inc. remarks: Public anycast DNS resolver @@ -1612,6 +1688,11 @@ descr: Thomas Harwood remarks: Public anycast DNS resolver is-anycast: yes +net: 2a06:e881:4300::/44 +descr: RISHIKESHAN LAVAKUMAR +remarks: Generic anycast network +is-anycast: yes + net: 2a07:59c6:d006::/48 descr: NetShelter remarks: Generic anycast network @@ -1627,6 +1708,11 @@ descr: Zeimudo Networks remarks: Generic anycast network is-anycast: yes +net: 2a0b:6b83::/48 +descr: Roelf Wichertjes +remarks: Generic anycast network +is-anycast: yes + net: 2a0c:3800:ff::/48 descr: Jean-Remy Buchs remarks: Generic anycast network diff --git a/overrides/override-other.txt b/overrides/override-other.txt index e387dbd..ce5bdb5 100644 --- a/overrides/override-other.txt +++ b/overrides/override-other.txt @@ -23,10 +23,15 @@ descr: PJSC "Vimpelcom" remarks: ISP located in RU, but some RIR data for announced prefixes contain garbage country: RU +aut-num: AS3741 +descr: Dimension Data +remarks: ISP (?) located in ZA, but some RIR data for announced prefixes contain garbage +country: ZA + aut-num: AS6134 descr: XNNET LLC remarks: traces back to an unknown oversea location (HK?), seems to tamper with RIR data -country: HK +country: AP aut-num: AS7203 descr: Leaseweb USA, Inc. @@ -53,6 +58,11 @@ descr: KLAYER LLC remarks: part of the "Asline" IP hijacking gang, traces back to AP region country: AP +aut-num: AS21100 +descr: ITL LLC +remarks: ISP headquatered in BG and/or UA, physically located in NL, some RIR data for announced prefixes contain inaccurate data +country: NL + aut-num: AS22769 descr: DDOSING NETWORK remarks: IP hijacker located somewhere in AP, massively tampers with RIR data @@ -78,6 +88,11 @@ descr: Netcalibre Ltd. remarks: ISP located in GB, but some RIR data for announced prefixes contain garbage country: GB +aut-num: AS26021 +descr: Unicycle, LLC +remarks: traces back to NL +country: NL + aut-num: AS27411 descr: Leaseweb USA, Inc. remarks: ISP located in US, but some RIR data for announced prefixes contain garbage @@ -128,6 +143,11 @@ descr: Neterra Ltd. remarks: ISP located in RU, but some RIR data for announced prefixes contain garbage country: RU +aut-num: AS39182 +descr: Netex Limited +remarks: ISP located in RU, but some RIR data for announced prefixes contain garbage +country: RU + aut-num: AS39287 descr: ab stract / Peter Kolmisoppi remarks: tampers with RIR data, traces back to SE @@ -138,6 +158,11 @@ descr: Confluence Networks Inc. remarks: fake offshore location (VG), traces back to Austin, TX, US country: US +aut-num: AS40065 +descr: CNSERVERS LLC +remarks: Shady ISP located in US, tampers with RIR data +country: US + aut-num: AS41466 descr: Treidinvest LLC remarks: another shady customer of "Tamatiya EOOD / 4Vendeta", located in BG, tampers with RIR data @@ -146,7 +171,12 @@ country: BG aut-num: AS41564 descr: Packet Exchange Limited remarks: shady uplink for a bunch of dirty ISPs in SE (and likely elsewhere in EU), routing stolen AfriNIC networks, RIR data of prefixes announced by this AS cannot be trusted -country: EU +country: SE + +aut-num: AS41608 +descr: NextGenWebs, S.L. +remarks: traces back to NL +country: NL aut-num: AS42397 descr: Bunea TELECOM SRL @@ -165,14 +195,29 @@ country: GB aut-num: AS42960 descr: Cloud Management LLC -remarks: tampers with RIR data, traces back to HK -country: HK +remarks: tampers with RIR data, traces back to AP area +country: AP + +aut-num: AS43260 +descr: DGN TEKNOLOJI A.S. +remarks: ISP located in TR, but many RIR data for announced prefixes contain garbage +country: TR aut-num: AS43310 descr: TOV "LVS" remarks: ISP located in UA, but some RIR data for announced prefixes contain garbage country: UA +aut-num: AS43350 +descr: NForce Entertainment B.V. +remarks: ISP located in NL, but some RIR data for announced prefixes contain garbage +country: NL + +aut-num: AS43624 +descr: PQ HOSTING S.R.L. +remarks: tampers with RIR data sometimes, traces back to NL +country: NL + aut-num: AS44015 descr: Landgard Management Inc. remarks: bulletproof ISP with strong links to RU @@ -188,16 +233,16 @@ descr: PPTECHNOLOGY LIMITED remarks: bulletproof ISP (related to AS204655) located in NL country: NL -aut-num: AS41564 -descr: Global Colocation Limited -remarks: part of a dirty ISP conglomerate most likely operating out of SE -country: SE - aut-num: AS48721 descr: ADM Service Ltd. remarks: traces back to Vilnius, LT country: LT +aut-num: AS49392 +descr: LLC Baxet +remarks: tampers with RIR data, traces back to RU +country: RU + aut-num: AS49466 descr: KLAYER LLC remarks: part of the "Asline" IP hijacking gang, traces back to AP region @@ -218,11 +263,21 @@ descr: Tamatiya EOOD / 4Vendeta remarks: Questionable (at best) ISP located in BG, clients massively tamper with RIR data country: BG +aut-num: AS50673 +descr: Serverius Holding B.V. +remarks: ISP located in NL, but some RIR data for announced prefixes contain garbage +country: NL + aut-num: AS51558 descr: Smart Telecom S.A.R.L remarks: tampers with RIR data, traces back to RU country: RU +aut-num: AS51659 +descr: LLC Baxet +remarks: tampers with RIR data, traces back to RU +country: RU + aut-num: AS55933 descr: Cloudie Limited remarks: part of the "Asline" IP hijacking gang, tampers with RIR data, traces back to AP region @@ -238,6 +293,11 @@ descr: PE Skurykhin Mukola Volodumurovuch remarks: tampers with RIR data, traces back to UA country: UA +aut-num: AS57043 +descr: Hostkey B.V. +remarks: ISP located in NL, but some RIR data for announced prefixes contain garbage +country: NL + aut-num: AS57717 descr: FiberXpress BV remarks: bulletproof ISP (related to AS202425) located in NL @@ -248,11 +308,26 @@ descr: DDOS-GUARD LTD remarks: shady ISP, customers massively tamper with RIR data, we cannot trust this network country: RU +aut-num: AS57756 +descr: Telefonica LLC +remarks: ISP located in RU, but some RIR data for announced prefixes contain garbage +country: RU + aut-num: AS57858 descr: Inter Connects Inc. remarks: part of a dirty ISP conglomerate operating most likely out of SE, hijacking stolen AfriNIC networks, massively tampers with RIR data country: SE +aut-num: AS57972 +descr: Inter Connects Inc. +remarks: part of a dirty ISP conglomerate operating most likely out of SE, hijacking stolen AfriNIC networks, massively tampers with RIR data +country: SE + +aut-num: AS58061 +descr: Scalaxy B.V. +remarks: ISP located in NL, but some RIR data for announced prefixes contain garbage +country: NL + aut-num: AS58073 descr: YISP BV remarks: ISP located in NL, but some RIR data for announced prefixes contain garbage @@ -278,6 +353,11 @@ descr: Batterflyai Media Ltd. remarks: ISP located in RU, but some RIR data for announced prefixes contain garbage country: RU +aut-num: AS60404 +descr: The Infrastructure Group B.V. +remarks: ISP located in NL, but some RIR data for announced prefixes contain garbage +country: NL + aut-num: AS60412 descr: DignusData LLC remarks: ISP located in PL, but _all_ RIR data for announced prefixes contain garbage @@ -313,6 +393,11 @@ descr: VpsQuan L.L.C. remarks: claims to be located in US, but traces to HK country: HK +aut-num: AS62900 +descr: ColoMX LLC +remarks: ISP located in US, but many RIR data for announced prefixes contain garbage +country: US + aut-num: AS63119 descr: Inter Connects Inc. remarks: part of a dirty ISP conglomerate, traces back to US this time @@ -338,6 +423,11 @@ descr: Leaseweb Asia Pacific pte. ltd. remarks: ISP located in HK, some RIR data for announced prefixes contain garbage country: HK +aut-num: AS134196 +descr: ULan Network Limited +remarks: part of the "Asline" IP hijacking gang, tampers with RIR data, traces back to AP region +country: AP + aut-num: AS134351 descr: Leaseweb Japan K.K. remarks: ISP located in JP, some RIR data for announced prefixes contain garbage @@ -383,6 +473,11 @@ descr: ANLIAN NETWORK TECHNOLOGY CO., LIMITED remarks: ISP located in HK, tampers with RIR data country: HK +aut-num: AS139879 +descr: Galaxy Broadband +remarks: ISP located in PK, but announces 204.137.128.0/18, which is ARIN space, assigned to "AGIS" / Cogent - odd... +country: PK + aut-num: AS140733 descr: Wujidun Network Limited remarks: part of the "Asline" IP hijacking gang, tampers with RIR data, traces back to AP region @@ -440,8 +535,8 @@ country: NL aut-num: AS202492 descr: SILVERHILL GROUP HOLDING LTD / SAKIS POLUNIGIS -remarks: fake offshore location (SC), traces back to NL -country: NL +remarks: fake offshore location (SC), traces back to RU +country: RU aut-num: AS202769 descr: Cooperative Investments LLC @@ -458,6 +553,11 @@ descr: Kevin Holly trading as Silent Ghost e.U. remarks: AS run by someone who thinks allocating IP networks to AQ is funny (it is not, kid) :-/ country: NL +aut-num: AS204307 +descr: Rishikeshan Lavakumar +remarks: Network operator thinks messing with RIR data is funny... :-/ +country: EU + aut-num: AS204353 descr: Global Offshore Limited remarks: part of a dirty ISP conglomerate with links to SE, RIR data of prefixes announced by this AS cannot be trusted @@ -513,6 +613,11 @@ descr: Maximilian Kutzner trading as HostSlick remarks: traces back to NL, but some RIR data for announced prefixes contain garbage country: NL +aut-num: AS208258 +descr: Access2.IT Group B.V. +remarks: ISP located in NL, but some RIR data for announced prefixes contain garbage +country: NL + aut-num: AS208410 descr: Internet Hosting Ltd. remarks: another shady customer of "Tamatiya EOOD / 4Vendeta", located in BG, tampers with RIR data @@ -553,6 +658,11 @@ descr: Harry Dowd remarks: ISP located in GB, but RIR data for announced prefixes contain garbage country: GB +aut-num: AS212477 +descr: RoyaleHosting B.V. +remarks: ISP located in NL, but RIR data for announced prefixes contain garbage +country: NL + aut-num: AS212962 descr: Quality Area Ltd. remarks: traceroutes dead-end somewhere near Crimera, UA @@ -798,6 +908,11 @@ descr: NetConn Services Ltd remarks: APNIC chunk owned by a HK-based company, routed to AP region, but assigned to SC country: AP +net: 193.186.196.0/22 +descr: QUIKA LTD +remarks: claims to be located in DE, traces back to GB +country: GB + net: 195.66.165.0/24 descr: Posta Crne Gore remarks: Orphaned RIR data, see: https://lists.ipfire.org/pipermail/location/2021-April/000267.html @@ -848,7 +963,22 @@ descr: Datashield, Inc. remarks: fake offshore location (SC), traces back to NL country: NL +net: 2a09:4c0:3000::/40 +descr: Securebit AG +remarks: ... who thinks assigning networks to unpopulated Bouvet Island (BV) is funny :-/ +country: CH + +net: 2a0e:b101:3000::/40 +descr: 4b42 UG (haftungsbeschränkt) +remarks: ... who thinks assigning networks to unpopulated Bouvet Island (BV) is funny :-/ +country: DE + net: 2a0f:7a80::/29 descr: ASLINE Limited remarks: APNIC chunk owned by a HK-based company, but assigned to DE country: AP + +net: 2a0f:e400:3000::/40 +descr: Kevin Buehl +remarks: ... who thinks assigning networks to unpopulated Bouvet Island (BV) is funny :-/ +country: CH -- 2.26.2

1 0

[PATCH] location-importer.in: import additional IP information for Amazon AWS IP networks
by Peter Müller 10 Jun '21

10 Jun '21

Amazon publishes information regarding some of their IP networks primarily used for AWS cloud services in a machine-readable format. To improve libloc lookup results for these, we have little choice other than importing and parsing them. Unfortunately, there seems to be no machine-readable list of the locations of their data centers or availability zones available. If there _is_ any, please let the author know. Fixes: #12594 Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- src/python/location-importer.in | 110 ++++++++++++++++++++++++++++++++ 1 file changed, 110 insertions(+) diff --git a/src/python/location-importer.in b/src/python/location-importer.in index 1e08458..5be1d61 100644 --- a/src/python/location-importer.in +++ b/src/python/location-importer.in @@ -19,6 +19,7 @@ import argparse import ipaddress +import json import logging import math import re @@ -931,6 +932,10 @@ class CLI(object): TRUNCATE TABLE network_overrides; """) + # Update overrides for various cloud providers big enough to publish their own IP + # network allocation lists in a machine-readable format... + self._update_overrides_for_aws() + for file in ns.files: log.info("Reading %s..." % file) @@ -998,6 +1003,111 @@ class CLI(object): else: log.warning("Unsupported type: %s" % type) + def _update_overrides_for_aws(self): + # Download Amazon AWS IP allocation file to create overrides... + downloader = location.importer.Downloader() + + try: + with downloader.request("https://ip-ranges.amazonaws.com/ip-ranges.json", return_blocks=False) as f: + aws_ip_dump = json.load(f.body) + except Exception as e: + log.error("unable to preprocess Amazon AWS IP ranges: %s" % e) + return + + # XXX: Set up a dictionary for mapping a region name to a country. Unfortunately, + # there seems to be no machine-readable version available of this other than + # https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availabil… + # (worse, it seems to be incomplete :-/ ); https://www.cloudping.cloud/endpoints + # was helpful here as well. + aws_region_country_map = { + "af-south-1": "ZA", + "ap-east-1": "HK", + "ap-south-1": "IN", + "ap-south-2": "IN", + "ap-northeast-3": "JP", + "ap-northeast-2": "KR", + "ap-southeast-1": "SG", + "ap-southeast-2": "AU", + "ap-southeast-3": "MY", + "ap-northeast-1": "JP", + "ca-central-1": "CA", + "eu-central-1": "DE", + "eu-central-2": "CH", + "eu-west-1": "IE", + "eu-west-2": "GB", + "eu-south-1": "IT", + "eu-south-2": "ES", + "eu-west-3": "FR", + "eu-north-1": "SE", + "me-south-1": "BH", + "sa-east-1": "BR" + } + + # Fetch all valid country codes to check parsed networks aganist... + rows = self.db.query("SELECT * FROM countries ORDER BY country_code") + validcountries = [] + + for row in rows: + validcountries.append(row.country_code) + + with self.db.transaction(): + for snetwork in aws_ip_dump["prefixes"] + aws_ip_dump["ipv6_prefixes"]: + try: + network = ipaddress.ip_network(snetwork.get("ip_prefix") or snetwork.get("ipv6_prefix"), strict=False) + except ValueError: + log.warning("Unable to parse line: %s" % snetwork) + continue + + # Sanitize parsed networks... + if not self._check_parsed_network(network): + continue + + # Determine region of this network... + region = snetwork["region"] + cc = None + is_anycast = False + + # Any region name starting with "us-" will get "US" country code assigned straight away... + if region.startswith("us-"): + cc = "US" + elif region.startswith("cn-"): + # ... same goes for China ... + cc = "CN" + elif region == "GLOBAL": + # ... funny region name for anycast-like networks ... + is_anycast = True + elif region in aws_region_country_map: + # ... assign looked up country code otherwise ... + cc = aws_region_country_map[region] + else: + # ... and bail out if we are missing something here + log.warning("Unable to determine country code for line: %s" % snetwork) + continue + + # Skip networks with unknown country codes + if not is_anycast and validcountries and cc not in validcountries: + log.warning("Skipping Amazon AWS network with bogus country '%s': %s" % \ + (cc, network)) + return + + # Conduct SQL statement... + self.db.execute(""" + INSERT INTO network_overrides( + network, + country, + is_anonymous_proxy, + is_satellite_provider, + is_anycast + ) VALUES (%s, %s, %s, %s, %s) + ON CONFLICT (network) DO NOTHING""", + "%s" % network, + cc, + None, + None, + is_anycast, + ) + + @staticmethod def _parse_bool(block, key): val = block.get(key) -- 2.26.2

2 11

[PATCH v2] location-importer.in: Import (technical) AS names from ARIN
by Peter Müller 10 Jun '21

10 Jun '21

ARIN and LACNIC, unfortunately, do not seem to publish data containing human readable AS names. For the former, we at least have a list of tecnical names, which this patch fetches and inserts into the autnums table. While some of them do not seem to be suitable for human consumption (i. e. being very cryptic), providing these data might be helpful neverthelesss. The second version of this patch contains some additional remarks on efficient Python coding style from Michael, doing things more "pythonic". Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- src/python/location-importer.in | 55 +++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/src/python/location-importer.in b/src/python/location-importer.in index aa3b8f7..6ccee3b 100644 --- a/src/python/location-importer.in +++ b/src/python/location-importer.in @@ -505,6 +505,9 @@ class CLI(object): for line in f: self._parse_line(line, source_key, validcountries) + # Download and import (technical) AS names from ARIN + self._import_as_names_from_arin() + def _check_parsed_network(self, network): """ Assistive function to detect and subsequently sort out parsed @@ -775,6 +778,58 @@ class CLI(object): "%s" % network, country, [country], source_key, ) + def _import_as_names_from_arin(self): + downloader = location.importer.Downloader() + + # XXX: Download AS names file from ARIN (note that these names appear to be quite + # technical, not intended for human consumption, as description fields in + # organisation handles for other RIRs are - however, this is what we have got, + # and in some cases, it might be still better than nothing) + with downloader.request("https://ftp.arin.net/info/asn.txt", return_blocks=False) as f: + for line in f: + # Convert binary line to string... + line = str(line) + + # ... valid lines start with a space, followed by the number of the Autonomous System ... + if not line.startswith(" "): + continue + + # Split line and check if there is a valid ASN in it... + asn, name = line.split()[0:2] + + try: + asn = int(asn) + except ValueError: + log.debug("Skipping ARIN AS names line not containing an integer for ASN") + continue + + if not ((1 <= asn and asn <= 23455) or (23457 <= asn and asn <= 64495) or (131072 <= asn and asn <= 4199999999)): + log.debug("Skipping ARIN AS names line not containing a valid ASN: %s" % asn) + continue + + # Skip any AS name that appears to be a placeholder for a different RIR or entity... + if re.match(r"^(ASN-BLK|)(AFCONC|AFRINIC|APNIC|ASNBLK|DNIC|LACNIC|RIPE|IANA)(\d?$|\-.*)", name): + continue + + # Bail out in case the AS name contains anything we do not expect here... + if re.search(r"[^a-zA-Z0-9-_]", name): + log.debug("Skipping ARIN AS name for %s containing invalid characters: %s" % \ + (asn, name)) + + # Things look good here, run INSERT statement and skip this one if we already have + # a (better?) name for this Autonomous System... + self.db.execute(""" + INSERT INTO autnums( + number, + name, + source + ) VALUES (%s, %s, %s) + ON CONFLICT (number) DO NOTHING""", + asn, + name, + "ARIN", + ) + def handle_update_announcements(self, ns): server = ns.server[0] -- 2.20.1

2 1

[PATCH] location-importer.in: Import (technical) AS names from ARIN
by Peter Müller 08 Jun '21

08 Jun '21

ARIN and LACNIC, unfortunately, do not seem to publish data containing human readable AS names. For the former, we at least have a list of tecnical names, which this patch fetches and inserts into the autnums table. While some of them do not seem to be suitable for human consumption (i. e. being very cryptic), providing these data might be helpful neverthelesss. Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- src/python/location-importer.in | 61 +++++++++++++++++++++++++++++++++ 1 file changed, 61 insertions(+) diff --git a/src/python/location-importer.in b/src/python/location-importer.in index aa3b8f7..2a9bf33 100644 --- a/src/python/location-importer.in +++ b/src/python/location-importer.in @@ -505,6 +505,9 @@ class CLI(object): for line in f: self._parse_line(line, source_key, validcountries) + # Download and import (technical) AS names from ARIN + self._import_as_names_from_arin() + def _check_parsed_network(self, network): """ Assistive function to detect and subsequently sort out parsed @@ -775,6 +778,64 @@ class CLI(object): "%s" % network, country, [country], source_key, ) + def _import_as_names_from_arin(self): + downloader = location.importer.Downloader() + + # XXX: Download AS names file from ARIN (note that these names appear to be quite + # technical, not intended for human consumption, as description fields in + # organisation handles for other RIRs are - however, this is what we have got, + # and in some cases, it might be still better than nothing) + try: + with downloader.request("https://ftp.arin.net/info/asn.txt", return_blocks=False) as f: + arin_as_names_file = f.body + except Exception as e: + log.error("failed to download and preprocess AS name file from ARIN: %s" % e) + return + + # Split downloaded body into lines and parse each of them... + for sline in arin_as_names_file.readlines(): + + # ... valid lines start with a space, followed by the number of the Autonomous System ... + if not sline.startswith(b" "): + continue + + # Split line and check if there is a valid ASN in it... + scontents = sline.split() + try: + asn = int(scontents[0]) + except ValueError: + log.debug("Skipping ARIN AS names line not containing an integer for ASN") + continue + + if not ((1 <= asn and asn <= 23455) or (23457 <= asn and asn <= 64495) or (131072 <= asn and asn <= 4199999999)): + log.debug("Skipping ARIN AS names line not containing a valid ASN: %s" % asn) + continue + + # Skip any AS name that appears to be a placeholder for a different RIR or entity... + as_name = scontents[1].decode("ascii") + + if re.match(r"^(ASN-BLK|)(AFCONC|AFRINIC|APNIC|ASNBLK|DNIC|LACNIC|RIPE|IANA)\d{0,1}-*", as_name): + continue + + # Bail out in case the AS name contains anything we do not expect here... + if re.search(r"[^a-zA-Z0-9-_]", as_name): + log.debug("Skipping ARIN AS name for %s containing invalid characters: %s" % \ + (asn, as_name)) + + # Things look good here, run INSERT statement and skip this one if we already have + # a (better?) name for this Autonomous System... + self.db.execute(""" + INSERT INTO autnums( + number, + name, + source + ) VALUES (%s, %s, %s) + ON CONFLICT (number) DO NOTHING""", + asn, + as_name, + "ARIN", + ) + def handle_update_announcements(self, ns): server = ns.server[0] -- 2.20.1

2 3

[PATCH v2 1/2] location-importer.in: add source column for overrides as well
by Peter Müller 08 Jun '21

08 Jun '21

This allows us to track changes introduced by IP feeds from 3rd parties, such as Amazon AWS, on the SQL server side. In order not to break existing tables (which would required TRUNCATE), there currently is no constraint set for the new column, but "NOT NULL" is planned in the future. Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- src/python/location-importer.in | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/src/python/location-importer.in b/src/python/location-importer.in index aa3b8f7..78bfd55 100644 --- a/src/python/location-importer.in +++ b/src/python/location-importer.in @@ -183,6 +183,7 @@ class CLI(object): ); CREATE UNIQUE INDEX IF NOT EXISTS autnum_overrides_number ON autnum_overrides(number); + ALTER TABLE autnum_overrides ADD COLUMN IF NOT EXISTS source text; ALTER TABLE autnum_overrides ADD COLUMN IF NOT EXISTS is_drop boolean; CREATE TABLE IF NOT EXISTS network_overrides( @@ -196,6 +197,7 @@ class CLI(object): ON network_overrides(network); CREATE INDEX IF NOT EXISTS network_overrides_search ON network_overrides USING GIST(network inet_ops); + ALTER TABLE network_overrides ADD COLUMN IF NOT EXISTS source text; ALTER TABLE network_overrides ADD COLUMN IF NOT EXISTS is_drop boolean; """) @@ -997,14 +999,16 @@ class CLI(object): INSERT INTO network_overrides( network, country, + source, is_anonymous_proxy, is_satellite_provider, is_anycast, is_drop - ) VALUES (%s, %s, %s, %s, %s, %s) + ) VALUES (%s, %s, %s, %s, %s, %s, %s) ON CONFLICT (network) DO NOTHING""", "%s" % network, block.get("country"), + "manual", self._parse_bool(block, "is-anonymous-proxy"), self._parse_bool(block, "is-satellite-provider"), self._parse_bool(block, "is-anycast"), @@ -1027,15 +1031,17 @@ class CLI(object): number, name, country, + source, is_anonymous_proxy, is_satellite_provider, is_anycast, is_drop - ) VALUES(%s, %s, %s, %s, %s, %s, %s) + ) VALUES(%s, %s, %s, %s, %s, %s, %s, %s) ON CONFLICT DO NOTHING""", autnum, block.get("name"), block.get("country"), + "manual", self._parse_bool(block, "is-anonymous-proxy"), self._parse_bool(block, "is-satellite-provider"), self._parse_bool(block, "is-anycast"), -- 2.20.1

1 1

[PATCH] Implement an additional flag for hostile networks safe to drop
by Peter Müller 07 Jun '21

07 Jun '21

This patch implements an additional flag intended for networks and Autonomous Systems being considered hostile. While libloc does not and should not be an opinionated database, reality shows it is being used this way. Hereby, we assign "XD" (drop) as a custom country code for networks being flagged this way. According to ISO, "XA" to "XZ" are reserved for "user-assgined codes" (https://www.iso.org/glossary-for-iso-3166.html), so this is a safe thing to do. This patch does not interfere with "A1" to "A3", which we currently assign outside standardised country code ranges for historical reasons. Neither does it specify any policy or source for tagging networks with a "drop" flag. Doing so is beyond the scope of this - technical - approach. To avoid confusions with the SQL "DROP" command, "is_drop" will be used as a column name for database operations. Thanks to Michael for his remarks and ideas during the run-up. Cc: Michael Tremer <michael.tremer(a)ipfire.org> Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- man/location.txt | 4 ++-- po/de.po | 3 +++ src/loc/network.h | 3 ++- src/perl/Location.xs | 2 ++ src/python/export.py | 3 ++- src/python/location-importer.in | 32 +++++++++++++++++++++++++++----- src/python/location.in | 19 ++++++++++++++++--- src/python/locationmodule.c | 5 ++++- 8 files changed, 58 insertions(+), 13 deletions(-) diff --git a/man/location.txt b/man/location.txt index df1da53..b38f21c 100644 --- a/man/location.txt +++ b/man/location.txt @@ -10,7 +10,7 @@ location - Query the location database `location list-countries [--show-name] [--show-continent]` `location list-networks-by-as ASN` `location list-networks-by-cc COUNTRY_CODE` -`location list-networks-by-flags [--anonymous-proxy|--satellite-provider|--anycast]` +`location list-networks-by-flags [--anonymous-proxy|--satellite-provider|--anycast|--drop]` `location lookup ADDRESS [ADDRESS...]` `location search-as STRING` `location update [--cron=daily|weekly|monthly]` @@ -73,7 +73,7 @@ or countries. + See above for usage of the '--family' and '--format' parameters. -'list-networks-by-flags [--family=[ipv6|ipv4]] [--format=FORMAT] [--anonymous-proxy|--satellite-provider|--anycast]':: +'list-networks-by-flags [--family=[ipv6|ipv4]] [--format=FORMAT] [--anonymous-proxy|--satellite-provider|--anycast|--drop]':: Lists all networks that have a certain flag. + See above for usage of the '--family' and '--format' parameters. diff --git a/po/de.po b/po/de.po index 3b073d6..3cbcdd7 100644 --- a/po/de.po +++ b/po/de.po @@ -152,6 +152,9 @@ msgstr "" msgid "Anycasts" msgstr "" +msgid "Hostile Networks safe to drop" +msgstr "" + msgid "Lists all countries" msgstr "" diff --git a/src/loc/network.h b/src/loc/network.h index af3dafd..a30f653 100644 --- a/src/loc/network.h +++ b/src/loc/network.h @@ -1,7 +1,7 @@ /* libloc - A library to determine the location of someone on the Internet - Copyright (C) 2017 IPFire Development Team <info(a)ipfire.org> + Copyright (C) 2017-2021 IPFire Development Team <info(a)ipfire.org> This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public @@ -27,6 +27,7 @@ enum loc_network_flags { LOC_NETWORK_FLAG_ANONYMOUS_PROXY = (1 << 0), // A1 LOC_NETWORK_FLAG_SATELLITE_PROVIDER = (1 << 1), // A2 LOC_NETWORK_FLAG_ANYCAST = (1 << 2), // A3 + LOC_NETWORK_FLAG_DROP = (1 << 3), // XD }; struct loc_network; diff --git a/src/perl/Location.xs b/src/perl/Location.xs index b7676d2..73f85b4 100644 --- a/src/perl/Location.xs +++ b/src/perl/Location.xs @@ -198,6 +198,8 @@ lookup_network_has_flag(db, address, flag) iv |= LOC_NETWORK_FLAG_SATELLITE_PROVIDER; else if (strcmp("LOC_NETWORK_FLAG_ANYCAST", flag) == 0) iv |= LOC_NETWORK_FLAG_ANYCAST; + else if (strcmp("LOC_NETWORK_FLAG_DROP", flag) == 0) + iv |= LOC_NETWORK_FLAG_DROP; else croak("Invalid flag"); diff --git a/src/python/export.py b/src/python/export.py index f0eae26..3b9e1e0 100644 --- a/src/python/export.py +++ b/src/python/export.py @@ -3,7 +3,7 @@ # # # libloc - A library to determine the location of someone on the Internet # # # -# Copyright (C) 2020 IPFire Development Team <info(a)ipfire.org> # +# Copyright (C) 2020-2021 IPFire Development Team <info(a)ipfire.org> # # # # This library is free software; you can redistribute it and/or # # modify it under the terms of the GNU Lesser General Public # @@ -33,6 +33,7 @@ FLAGS = { _location.NETWORK_FLAG_ANONYMOUS_PROXY : "A1", _location.NETWORK_FLAG_SATELLITE_PROVIDER : "A2", _location.NETWORK_FLAG_ANYCAST : "A3", + _location.NETWORK_FLAG_DROP : "XD", } class OutputWriter(object): diff --git a/src/python/location-importer.in b/src/python/location-importer.in index f796652..ea1e8f5 100644 --- a/src/python/location-importer.in +++ b/src/python/location-importer.in @@ -182,6 +182,7 @@ class CLI(object): ); CREATE UNIQUE INDEX IF NOT EXISTS autnum_overrides_number ON autnum_overrides(number); + ALTER TABLE autnum_overrides ADD COLUMN IF NOT EXISTS is_drop boolean; CREATE TABLE IF NOT EXISTS network_overrides( network inet NOT NULL, @@ -194,6 +195,7 @@ class CLI(object): ON network_overrides(network); CREATE INDEX IF NOT EXISTS network_overrides_search ON network_overrides USING GIST(network inet_ops); + ALTER TABLE network_overrides ADD COLUMN IF NOT EXISTS is_drop boolean; """) return db @@ -301,7 +303,20 @@ class CLI(object): WHERE networks.autnum = overrides.number ), FALSE - ) AS is_anycast + ) AS is_anycast, + COALESCE( + ( + SELECT is_drop FROM network_overrides overrides + WHERE networks.network <<= overrides.network + ORDER BY masklen(overrides.network) DESC + LIMIT 1 + ), + ( + SELECT is_drop FROM autnum_overrides overrides + WHERE networks.autnum = overrides.number + ), + FALSE + ) AS is_drop FROM ( SELECT known_networks.network AS network, @@ -350,6 +365,9 @@ class CLI(object): if row.is_anycast: network.set_flag(location.NETWORK_FLAG_ANYCAST) + if row.is_drop: + network.set_flag(location.NETWORK_FLAG_DROP) + # Add all countries log.info("Writing countries...") rows = self.db.query("SELECT * FROM countries ORDER BY country_code") @@ -966,14 +984,16 @@ class CLI(object): country, is_anonymous_proxy, is_satellite_provider, - is_anycast - ) VALUES (%s, %s, %s, %s, %s) + is_anycast, + is_drop + ) VALUES (%s, %s, %s, %s, %s, %s) ON CONFLICT (network) DO NOTHING""", "%s" % network, block.get("country"), self._parse_bool(block, "is-anonymous-proxy"), self._parse_bool(block, "is-satellite-provider"), self._parse_bool(block, "is-anycast"), + self._parse_bool(block, "drop"), ) elif type == "aut-num": @@ -994,8 +1014,9 @@ class CLI(object): country, is_anonymous_proxy, is_satellite_provider, - is_anycast - ) VALUES(%s, %s, %s, %s, %s, %s) + is_anycast, + is_drop + ) VALUES(%s, %s, %s, %s, %s, %s, %s) ON CONFLICT DO NOTHING""", autnum, block.get("name"), @@ -1003,6 +1024,7 @@ class CLI(object): self._parse_bool(block, "is-anonymous-proxy"), self._parse_bool(block, "is-satellite-provider"), self._parse_bool(block, "is-anycast"), + self._parse_bool(block, "drop"), ) else: diff --git a/src/python/location.in b/src/python/location.in index e02b4e8..0c89d75 100644 --- a/src/python/location.in +++ b/src/python/location.in @@ -3,7 +3,7 @@ # # # libloc - A library to determine the location of someone on the Internet # # # -# Copyright (C) 2017 IPFire Development Team <info(a)ipfire.org> # +# Copyright (C) 2017-2021 IPFire Development Team <info(a)ipfire.org> # # # # This library is free software; you can redistribute it and/or # # modify it under the terms of the GNU Lesser General Public # @@ -146,6 +146,9 @@ class CLI(object): list_networks_by_flags.add_argument("--anycast", action="store_true", help=_("Anycasts"), ) + list_networks_by_flags.add_argument("--drop", + action="store_true", help=_("Hostile Networks safe to drop"), + ) list_networks_by_flags.add_argument("--family", choices=("ipv6", "ipv4")) list_networks_by_flags.add_argument("--format", choices=location.export.formats.keys(), default="list") @@ -305,6 +308,12 @@ class CLI(object): _("Anycast"), _("yes"), )) + # Hostile Network + if network.has_flag(location.NETWORK_FLAG_DROP): + print(format % ( + _("Hostile Network safe to drop"), _("yes"), + )) + return ret def handle_dump(self, db, ns): @@ -346,6 +355,7 @@ class CLI(object): location.NETWORK_FLAG_ANONYMOUS_PROXY : "is-anonymous-proxy:", location.NETWORK_FLAG_SATELLITE_PROVIDER : "is-satellite-provider:", location.NETWORK_FLAG_ANYCAST : "is-anycast:", + location.NETWORK_FLAG_DROP : "drop:", } # Iterate over all networks @@ -523,6 +533,9 @@ class CLI(object): if ns.anycast: flags |= location.NETWORK_FLAG_ANYCAST + if ns.drop: + flags |= location.NETWORK_FLAG_DROP + if not flags: raise ValueError(_("You must at least pass one flag")) @@ -551,7 +564,7 @@ class CLI(object): asns.append(object) elif location.country_code_is_valid(object) \ - or object in ("A1", "A2", "A3"): + or object in ("A1", "A2", "A3", "XD"): countries.append(object) else: @@ -560,7 +573,7 @@ class CLI(object): # Default to exporting all countries if not countries and not asns: - countries = ["A1", "A2", "A3"] + [country.code for country in db.countries] + countries = ["A1", "A2", "A3", "XD"] + [country.code for country in db.countries] # Select the output format writer = self.__get_output_formatter(ns) diff --git a/src/python/locationmodule.c b/src/python/locationmodule.c index 5b72be9..5dd4ec6 100644 --- a/src/python/locationmodule.c +++ b/src/python/locationmodule.c @@ -1,7 +1,7 @@ /* libloc - A library to determine the location of someone on the Internet - Copyright (C) 2017 IPFire Development Team <info(a)ipfire.org> + Copyright (C) 2017-2021 IPFire Development Team <info(a)ipfire.org> This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public @@ -169,6 +169,9 @@ PyMODINIT_FUNC PyInit__location(void) { if (PyModule_AddIntConstant(m, "NETWORK_FLAG_ANYCAST", LOC_NETWORK_FLAG_ANYCAST)) return NULL; + if (PyModule_AddIntConstant(m, "NETWORK_FLAG_DROP", LOC_NETWORK_FLAG_DROP)) + return NULL; + // Add latest database version if (PyModule_AddIntConstant(m, "DATABASE_VERSION_LATEST", LOC_DATABASE_VERSION_LATEST)) return NULL; -- 2.26.2

2 1

[PATCH] overrides-{a{1,3},other}: regular batch of various overrides
by Peter Müller 03 Jun '21

03 Jun '21

Including location pinning for various LeaseWeb AS, as their customers seem to tamper with RIR data a lot. Fortunately for use, they use one AS per PoP, so we can trace back locations quite easy. :-) AS209242 is especially - um - interesting. Given Cloudflare's nature, it is impossible to tell where these shady prefixes announced by it are located. Most of them point to letterbox companies, hosting questionable services at best. Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- overrides/override-a1.txt | 32 +++++++++- overrides/override-a3.txt | 30 ++++++++++ overrides/override-other.txt | 112 ++++++++++++++++++++++++++++++++++- 3 files changed, 172 insertions(+), 2 deletions(-) diff --git a/overrides/override-a1.txt b/overrides/override-a1.txt index 3a65232..b884c5d 100644 --- a/overrides/override-a1.txt +++ b/overrides/override-a1.txt @@ -44,6 +44,11 @@ descr: VPN de Mexico, S.A. de C.V. remarks: VPN provider is-anonymous-proxy: yes +aut-num: AS32781 +descr: Defender cloud international LLC +remarks: VPN provider [high confidence, but not proofed] +is-anonymous-proxy: yes + aut-num: AS34962 descr: Epik Network remarks: Shady ISP and registrar, many prefixes announced refer to "anonymize" infrastructure @@ -236,6 +241,11 @@ descr: Business VPN LLC remarks: VPN provider is-anonymous-proxy: yes +aut-num: AS398271 +descr: HardenedVPN[.]com LLC +remarks: VPN provider +is-anonymous-proxy: yes + net: 2.57.171.0/24 descr: VPN Consumer Network remarks: VPN provider @@ -476,6 +486,11 @@ descr: Shtrauh Andrey remarks: VPN provider [high confidence, but not proofed] is-anonymous-proxy: yes +net: 45.147.56.0/23 +descr: OLMERA GROUP LTD +remarks: VPN provider [high confidence, but not proofed] +is-anonymous-proxy: yes + net: 45.151.115.0/24 descr: ikoProxies [high confidence, but not proofed] remarks: VPN provider located in NL @@ -489,7 +504,7 @@ is-anonymous-proxy: yes net: 45.192.0.0/12 descr: Cloud Innovation Ltd. -remarks: hijacked AFRINIC IP chunk owned by an offshore company, routed to several dirty networks worldwide, cannot tell what is going on here +remarks: hijacked (?) AFRINIC IP chunk owned by an offshore company, routed to several dirty networks worldwide, cannot tell what is going on here is-anonymous-proxy: yes net: 45.220.72.0/22 @@ -1131,6 +1146,11 @@ remarks: (Rogue) VPN provider is-anonymous-proxy: yes country: EU +net: 185.164.59.0/24 +descr: Buyproxies / Yuli Azarch +remarks: VPN provider [high confidence, but not proofed] +is-anonymous-proxy: yes + net: 185.165.153.0/24 descr: Privacy Matters VPN service / nVPN / David Craig remarks: (Rogue) VPN provider @@ -1384,6 +1404,11 @@ descr: VPNClientPublics remarks: VPN provider is-anonymous-proxy: yes +net: 198.44.224.0/19 +descr: Defender cloud international LLC +remarks: VPN provider [high confidence, but not proofed] +is-anonymous-proxy: yes + net: 199.249.223.0/24 descr: Quintex Alliance Consulting remarks: Tor relay provider @@ -1454,6 +1479,11 @@ descr: SecuredConnectivity remarks: VPN provider is-anonymous-proxy: yes +net: 206.123.128.0/19 +descr: Secure Internet LLC +remarks: VPN provider +is-anonymous-proxy: yes + net: 209.107.176.0/20 descr: Google VPN remarks: VPN provider diff --git a/overrides/override-a3.txt b/overrides/override-a3.txt index ec246c1..daacc95 100644 --- a/overrides/override-a3.txt +++ b/overrides/override-a3.txt @@ -30,11 +30,21 @@ descr: Akamai Technologies, Inc. remarks: Worldwide CDN, does not make sense to assign their networks to a specific country is-anycast: yes +aut-num: AS20577 +descr: TRADEWEB EUROPE LIMITED +remarks: Generic anycast network [high confidence, but not proofed] +is-anycast: yes + aut-num: AS20940 descr: Akamai International BV remarks: Worldwide CDN, does not make sense to assign their networks to a specific country is-anycast: yes +aut-num: AS28064 +descr: Smart Data Center S.A. +remarks: Generic anycast network [high confidence, but not proofed] +is-anycast: yes + aut-num: AS31529 descr: DENIC eG remarks: TLD operator's anycast network @@ -45,6 +55,11 @@ descr: Citigroup remarks: Public anycast DNS resolver network [high confidence, but not proofed] is-anycast: yes +aut-num: AS34738 +descr: WEBHOST LIMITED +remarks: Generic anycast network [high confidence, but not proofed] +is-anycast: yes + aut-num: AS34868 descr: wasted.io Ltd. remarks: Generic anycast network @@ -80,6 +95,11 @@ descr: coreIT.pl remarks: Generic anycast network is-anycast: yes +aut-num: AS60626 +descr: Leaseweb CDN B.V. +remarks: Generic anycast network +is-anycast: yes + aut-num: AS60890 descr: CentralNic Ltd remarks: Generic anycast network @@ -95,6 +115,11 @@ descr: OpenTLD BV remarks: TLD operator's anycast network is-anycast: yes +aut-num: AS61072 +descr: EZNet LIMITED +remarks: Generic anycast network +is-anycast: yes + aut-num: AS62766 descr: SJB Communications remarks: Generic anycast network @@ -160,6 +185,11 @@ descr: ipcom GmbH remarks: Generic anycast network is-anycast: yes +aut-num: AS209242 +descr: Cloudflare London, LLC / Cloudflare Spectrum +remarks: CFs' "bring your own IP networks and we will announce it" AS, lots of shady announcements originated from there, clients are located worldwide as single peer is CF's main AS +is-anycast: yes + aut-num: AS209813 descr: Fast Content Delivery Ltd. remarks: Generic anycast network diff --git a/overrides/override-other.txt b/overrides/override-other.txt index 0b521d1..e387dbd 100644 --- a/overrides/override-other.txt +++ b/overrides/override-other.txt @@ -13,6 +13,11 @@ # Please keep this file sorted. # +aut-num: AS1820 +descr: WNET TELECOM USA Corp. +remarks: traces back to various locations in UA, seems to tamper with RIR data +country: UA + aut-num: AS3216 descr: PJSC "Vimpelcom" remarks: ISP located in RU, but some RIR data for announced prefixes contain garbage @@ -23,6 +28,11 @@ descr: XNNET LLC remarks: traces back to an unknown oversea location (HK?), seems to tamper with RIR data country: HK +aut-num: AS7203 +descr: Leaseweb USA, Inc. +remarks: ISP located in US, but some RIR data for announced prefixes contain garbage +country: US + aut-num: AS8359 descr: MTS PJSC remarks: ISP located in RU, but some RIR data for announced prefixes contain garbage @@ -58,16 +68,36 @@ descr: Yes Networks Unlimited Ltd remarks: traces to UA, but some RIR entries seem to contain garbage (VG) country: UA +aut-num: AS24961 +descr: myLoc managed IT AG +remarks: ISP located in DE, but some RIR data for announced prefixes contain garbage +country: DE + aut-num: AS25098 descr: Netcalibre Ltd. -remarks: ISP located in GB, but some RIR data for announced prefixes contain garbage/ +remarks: ISP located in GB, but some RIR data for announced prefixes contain garbage country: GB +aut-num: AS27411 +descr: Leaseweb USA, Inc. +remarks: ISP located in US, but some RIR data for announced prefixes contain garbage +country: US + aut-num: AS28098 descr: ABGON Comunicaciones remarks: ISP located in CL, but some RIR data for announced prefixes contain garbage (BZ) country: CL +aut-num: AS28753 +descr: Leaseweb Deutschland GmbH +remarks: ISP located in Frankfurt/Main, DE, but many RIR data for announced prefixes contain garbage +country: DE + +aut-num: AS30633 +descr: Leaseweb USA, Inc. +remarks: ISP located in US, but some RIR data for announced prefixes contain garbage (BZ) +country: US + aut-num: AS35042 descr: IP Interactive UG (haftungsbeschraenkt) remarks: ISP located in BG, but RIR data for announced prefixes contain garbage @@ -163,6 +193,11 @@ descr: Global Colocation Limited remarks: part of a dirty ISP conglomerate most likely operating out of SE country: SE +aut-num: AS48721 +descr: ADM Service Ltd. +remarks: traces back to Vilnius, LT +country: LT + aut-num: AS49466 descr: KLAYER LLC remarks: part of the "Asline" IP hijacking gang, traces back to AP region @@ -233,6 +268,11 @@ descr: INNETRA PC remarks: another shady customer of "DDoS Guard Ltd.", jurisdiction is probably RU, but traceroutes dead-end somewhere else in EU country: EU +aut-num: AS59253 +descr: Leaseweb Asia Pacific pte. ltd. +remarks: ISP located in SG, but some RIR data for announced prefixes contain garbage +country: SG + aut-num: AS59580 descr: Batterflyai Media Ltd. remarks: ISP located in RU, but some RIR data for announced prefixes contain garbage @@ -248,6 +288,11 @@ descr: Inter Connects Inc. / Jing Yun remarks: part of a dirty ISP conglomerate operating most likely out of SE, hijacking AfriNIC networks country: SE +aut-num: AS60781 +descr: LeaseWeb Netherlands B.V. +remarks: ISP located in Amsterdam, NL, but many RIR data for announced prefixes contain garbage +country: NL + aut-num: AS61977 descr: Vivo Trade L.P. remarks: another shady customer of "DDoS Guard Ltd." @@ -283,11 +328,31 @@ descr: NForce Entertainment BV remarks: currently hijacks a single stolen /20 AfriNIC IPv4 net, hosted in NL country: NL +aut-num: AS132369 +descr: XIANGAO INTERNATIONAL TELECOMMUNICATION LIMITED +remarks: ISP located in HK, tampers with RIR data +country: HK + +aut-num: AS133752 +descr: Leaseweb Asia Pacific pte. ltd. +remarks: ISP located in HK, some RIR data for announced prefixes contain garbage +country: HK + +aut-num: AS134351 +descr: Leaseweb Japan K.K. +remarks: ISP located in JP, some RIR data for announced prefixes contain garbage +country: JP + aut-num: AS134548 descr: DXTL Tseung Kwan O Service remarks: tampers with RIR data, traces back to AP region country: AP +aut-num: AS136988 +descr: Leaseweb Australia Pty. Ltd. +remarks: ISP located in AU, some RIR data for announced prefixes contain garbage +country: AU + aut-num: AS137443 descr: Anchnet Asia Limited remarks: IP hijacker located in HK, tampers with RIR data @@ -298,11 +363,26 @@ descr: Clayer Limited remarks: part of the "Asline" IP hijacking gang, tampers with RIR data, traces back to AP region country: AP +aut-num: AS138571 +descr: SUPERCLOUDS LIMITED +remarks: ISP located in HK, tampers with RIR data +country: HK + +aut-num: AS139242 +descr: Cloudflare Sydney, LLC +remarks: ... but CF failed to set the country for announced prefixes to AU as well :-/ +country: AU + aut-num: AS139330 descr: SANREN DATA LIMITED remarks: IP hijacker located somewhere in AP region, tampers with RIR data country: AP +aut-num: AS139811 +descr: ANLIAN NETWORK TECHNOLOGY CO., LIMITED +remarks: ISP located in HK, tampers with RIR data +country: HK + aut-num: AS140733 descr: Wujidun Network Limited remarks: part of the "Asline" IP hijacking gang, tampers with RIR data, traces back to AP region @@ -323,6 +403,11 @@ descr: Datashield, Inc. remarks: fake offshore location (SC), traces back to NL country: NL +aut-num: AS200740 +descr: Network Management Ltd. +remarks: traceroutes dead-end somewhere in or near RU +country: RU + aut-num: AS200845 descr: AVATEL TELECOM remarks: ISP located in ES, but some RIR data for announced prefixes contain garbage @@ -388,6 +473,11 @@ descr: Hauer Hosting Services Limited remarks: ISP located in ES, but some RIR data for announced prefixes contain garbage country: ES +aut-num: AS205544 +descr: LEASEWEB UK LIMITED +remarks: ISP located in London, GB, but many RIR data for announced prefixes contain garbage +country: GB + aut-num: AS206397 descr: Genius Guard / Genius Security Ltd. remarks: another shady customer of "DDoS Guard Ltd.", probably located in RU @@ -403,6 +493,11 @@ descr: Xtudio Networks S.L.U. remarks: ISP located in ES, but some RIR data for announced prefixes contain garbage country: ES +aut-num: AS207569 +descr: Network Management Ltd. +remarks: traceroutes dead-end somewhere in or near RU +country: RU + aut-num: AS207616 descr: Altrosky Technology Ltd. remarks: fake offshore location (SC), traces back to CZ and NL @@ -458,6 +553,11 @@ descr: Harry Dowd remarks: ISP located in GB, but RIR data for announced prefixes contain garbage country: GB +aut-num: AS212962 +descr: Quality Area Ltd. +remarks: traceroutes dead-end somewhere near Crimera, UA +country: UA + aut-num: AS213035 descr: Serverion BV remarks: ISP located in NL, but RIR data for most announced prefixes contain garbage @@ -488,6 +588,16 @@ descr: Datapacket Maroc SARL remarks: bulletproof ISP (strongly linked to AS202425) located in NL country: NL +aut-num: AS394380 +descr: Leaseweb USA, Inc. +remarks: ISP located in Dallas, TX, US, but some RIR data for announced prefixes contain garbage +country: US + +aut-num: AS395954 +descr: Leaseweb USA, Inc. +remarks: ISP located in US, but some RIR data for announced prefixes contain garbage +country: US + aut-num: AS398478 descr: PEG TECH INC remarks: ISP located in HK, tampers with RIR data -- 2.26.2

1 0

[PATCH v2] location-importer.in: track original countries as well
by Peter Müller 02 Jun '21

02 Jun '21

This helps us to determine how many network objects have more than one country set, and what their original country code set looked like. The second version of this patch uses ALTER TABLE to add the column for original countries, preventing existing SQL setups from breaking. Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- src/python/location-importer.in | 52 ++++++++++++++++++++------------- 1 file changed, 32 insertions(+), 20 deletions(-) diff --git a/src/python/location-importer.in b/src/python/location-importer.in index e5f55af..c3f908a 100644 --- a/src/python/location-importer.in +++ b/src/python/location-importer.in @@ -165,6 +165,7 @@ class CLI(object): -- networks CREATE TABLE IF NOT EXISTS networks(network inet, country text); + ALTER TABLE networks ADD COLUMN IF NOT EXISTS original_countries text[]; CREATE UNIQUE INDEX IF NOT EXISTS networks_network ON networks(network); CREATE INDEX IF NOT EXISTS networks_family ON networks USING BTREE(family(network)); CREATE INDEX IF NOT EXISTS networks_search ON networks USING GIST(network inet_ops); @@ -377,7 +378,7 @@ class CLI(object): ON COMMIT DROP; CREATE UNIQUE INDEX _organizations_handle ON _organizations(handle); - CREATE TEMPORARY TABLE _rirdata(network inet NOT NULL, country text NOT NULL) + CREATE TEMPORARY TABLE _rirdata(network inet NOT NULL, country text NOT NULL, original_countries text[] NOT NULL) ON COMMIT DROP; CREATE INDEX _rirdata_search ON _rirdata USING BTREE(family(network), masklen(network)); CREATE UNIQUE INDEX _rirdata_network ON _rirdata(network); @@ -407,8 +408,8 @@ class CLI(object): for family in (row.family for row in families): smallest = self.db.get("SELECT MIN(masklen(network)) AS prefix FROM _rirdata WHERE family(network) = %s", family) - self.db.execute("INSERT INTO networks(network, country) \ - SELECT network, country FROM _rirdata WHERE masklen(network) = %s AND family(network) = %s", smallest.prefix, family) + self.db.execute("INSERT INTO networks(network, country, original_countries) \ + SELECT network, country, original_countries FROM _rirdata WHERE masklen(network) = %s AND family(network) = %s", smallest.prefix, family) # ... determine any other prefixes for this network family, ... prefixes = self.db.query("SELECT DISTINCT masklen(network) AS prefix FROM _rirdata \ @@ -421,7 +422,8 @@ class CLI(object): WITH candidates AS ( SELECT _rirdata.network, - _rirdata.country + _rirdata.country, + _rirdata.original_countries FROM _rirdata WHERE @@ -434,6 +436,7 @@ class CLI(object): DISTINCT ON (c.network) c.network, c.country, + c.original_countries, masklen(networks.network), networks.country AS parent_country FROM @@ -447,10 +450,11 @@ class CLI(object): masklen(networks.network) DESC NULLS LAST ) INSERT INTO - networks(network, country) + networks(network, country, original_countries) SELECT network, - country + country, + original_countries FROM filtered WHERE @@ -617,28 +621,36 @@ class CLI(object): inetnum[key] = [ipaddress.ip_network(val, strict=False)] elif key == "country": - inetnum[key] = val.upper() + # Catch RIR data objects with more than one country code... + if not key in inetnum.keys(): + inetnum[key] = [] + else: + if val.upper() in inetnum.get("country"): + # ... but keep this list distinct... + continue + + inetnum[key].append(val.upper()) # Skip empty objects if not inetnum or not "country" in inetnum: return + # Prepare skipping objects with unknown country codes... + invalidcountries = [singlecountry for singlecountry in inetnum.get("country") if singlecountry not in validcountries] + # Iterate through all networks enumerated from above, check them for plausibility and insert # them into the database, if _check_parsed_network() succeeded for single_network in inetnum.get("inet6num") or inetnum.get("inetnum"): if self._check_parsed_network(single_network): - - # Skip objects with unknown country codes - to avoid log spam for invalid or too small - # networks, this check is - kinda ugly - done at this point - if validcountries and inetnum.get("country") not in validcountries: - log.warning("Skipping network with bogus country '%s': %s" % \ - (inetnum.get("country"), inetnum.get("inet6num") or inetnum.get("inetnum"))) + # Skip objects with unknown country codes if they are valid to avoid log spam... + if validcountries and invalidcountries: + log.warning("Skipping network with bogus countr(y|ies) %s (original countries: %s): %s" % \ + (invalidcountries, inetnum.get("country"), inetnum.get("inet6num") or inetnum.get("inetnum"))) break - # Everything is fine here, run INSERT statement... - self.db.execute("INSERT INTO _rirdata(network, country) \ - VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country = excluded.country", - "%s" % single_network, inetnum.get("country"), + self.db.execute("INSERT INTO _rirdata(network, country, original_countries) \ + VALUES(%s, %s, %s) ON CONFLICT (network) DO UPDATE SET country = excluded.country", + "%s" % single_network, inetnum.get("country")[0], inetnum.get("country"), ) def _parse_org_block(self, block): @@ -729,10 +741,10 @@ class CLI(object): if not self._check_parsed_network(network): return - self.db.execute("INSERT INTO networks(network, country) \ - VALUES(%s, %s) ON CONFLICT (network) DO \ + self.db.execute("INSERT INTO networks(network, country, original_countries) \ + VALUES(%s, %s, %s) ON CONFLICT (network) DO \ UPDATE SET country = excluded.country", - "%s" % network, country, + "%s" % network, country, [country], ) def handle_update_announcements(self, ns): -- 2.20.1

2 6

[PATCH v2 1/2] location-importer.in: keep track of sources for networks, ASNs, and organisations
by Peter Müller 31 May '21

31 May '21

This allows us to trace back concrete changes or anomalies to their RIR source, without having to parse everything again. Further, it enables adding 3rd party sources such as IP feeds from Amazon, without loosing track of the changes introduced by them. The second version of this patchset uses ALTER TABLE to add the source columns, avoiding breaking existing SQL setups. Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- src/python/location-importer.in | 91 ++++++++++++++++++--------------- 1 file changed, 49 insertions(+), 42 deletions(-) diff --git a/src/python/location-importer.in b/src/python/location-importer.in index e5f55af..f796652 100644 --- a/src/python/location-importer.in +++ b/src/python/location-importer.in @@ -156,6 +156,7 @@ class CLI(object): -- autnums CREATE TABLE IF NOT EXISTS autnums(number bigint, name text NOT NULL); + ALTER TABLE autnums ADD COLUMN IF NOT EXISTS source text NOT NULL; CREATE UNIQUE INDEX IF NOT EXISTS autnums_number ON autnums(number); -- countries @@ -165,6 +166,7 @@ class CLI(object): -- networks CREATE TABLE IF NOT EXISTS networks(network inet, country text); + ALTER TABLE networks ADD COLUMN IF NOT EXISTS source text NOT NULL; CREATE UNIQUE INDEX IF NOT EXISTS networks_network ON networks(network); CREATE INDEX IF NOT EXISTS networks_family ON networks USING BTREE(family(network)); CREATE INDEX IF NOT EXISTS networks_search ON networks USING GIST(network inet_ops); @@ -369,15 +371,15 @@ class CLI(object): with self.db.transaction(): # Create some temporary tables to store parsed data self.db.execute(""" - CREATE TEMPORARY TABLE _autnums(number integer, organization text) + CREATE TEMPORARY TABLE _autnums(number integer, organization text, source text NOT NULL) ON COMMIT DROP; CREATE UNIQUE INDEX _autnums_number ON _autnums(number); - CREATE TEMPORARY TABLE _organizations(handle text, name text NOT NULL) + CREATE TEMPORARY TABLE _organizations(handle text, name text NOT NULL, source text NOT NULL) ON COMMIT DROP; CREATE UNIQUE INDEX _organizations_handle ON _organizations(handle); - CREATE TEMPORARY TABLE _rirdata(network inet NOT NULL, country text NOT NULL) + CREATE TEMPORARY TABLE _rirdata(network inet NOT NULL, country text NOT NULL, source text NOT NULL) ON COMMIT DROP; CREATE INDEX _rirdata_search ON _rirdata USING BTREE(family(network), masklen(network)); CREATE UNIQUE INDEX _rirdata_network ON _rirdata(network); @@ -395,10 +397,11 @@ class CLI(object): for row in rows: validcountries.append(row.country_code) - for source in location.importer.WHOIS_SOURCES: - with downloader.request(source, return_blocks=True) as f: - for block in f: - self._parse_block(block, validcountries) + for source_key in location.importer.WHOIS_SOURCES: + for single_url in location.importer.WHOIS_SOURCES[source_key]: + with downloader.request(single_url, return_blocks=True) as f: + for block in f: + self._parse_block(block, source_key, validcountries) # Process all parsed networks from every RIR we happen to have access to, # insert the largest network chunks into the networks table immediately... @@ -407,8 +410,8 @@ class CLI(object): for family in (row.family for row in families): smallest = self.db.get("SELECT MIN(masklen(network)) AS prefix FROM _rirdata WHERE family(network) = %s", family) - self.db.execute("INSERT INTO networks(network, country) \ - SELECT network, country FROM _rirdata WHERE masklen(network) = %s AND family(network) = %s", smallest.prefix, family) + self.db.execute("INSERT INTO networks(network, country, source) \ + SELECT network, country, source FROM _rirdata WHERE masklen(network) = %s AND family(network) = %s", smallest.prefix, family) # ... determine any other prefixes for this network family, ... prefixes = self.db.query("SELECT DISTINCT masklen(network) AS prefix FROM _rirdata \ @@ -421,7 +424,8 @@ class CLI(object): WITH candidates AS ( SELECT _rirdata.network, - _rirdata.country + _rirdata.country, + _rirdata.source FROM _rirdata WHERE @@ -434,6 +438,7 @@ class CLI(object): DISTINCT ON (c.network) c.network, c.country, + c.source, masklen(networks.network), networks.country AS parent_country FROM @@ -447,10 +452,11 @@ class CLI(object): masklen(networks.network) DESC NULLS LAST ) INSERT INTO - networks(network, country) + networks(network, country, source) SELECT network, - country + country, + source FROM filtered WHERE @@ -462,19 +468,20 @@ class CLI(object): ) self.db.execute(""" - INSERT INTO autnums(number, name) - SELECT _autnums.number, _organizations.name FROM _autnums + INSERT INTO autnums(number, name, source) + SELECT _autnums.number, _organizations.name, _organizations.source FROM _autnums JOIN _organizations ON _autnums.organization = _organizations.handle ON CONFLICT (number) DO UPDATE SET name = excluded.name; """) # Download all extended sources - for source in location.importer.EXTENDED_SOURCES: - with self.db.transaction(): - # Download data - with downloader.request(source) as f: - for line in f: - self._parse_line(line, validcountries) + for source_key in location.importer.EXTENDED_SOURCES: + for single_url in location.importer.EXTENDED_SOURCES[source_key]: + with self.db.transaction(): + # Download data + with downloader.request(single_url) as f: + for line in f: + self._parse_line(line, source_key, validcountries) def _check_parsed_network(self, network): """ @@ -539,23 +546,23 @@ class CLI(object): # be suitable for libloc consumption... return True - def _parse_block(self, block, validcountries = None): + def _parse_block(self, block, source_key, validcountries = None): # Get first line to find out what type of block this is line = block[0] # aut-num if line.startswith("aut-num:"): - return self._parse_autnum_block(block) + return self._parse_autnum_block(block, source_key) # inetnum if line.startswith("inet6num:") or line.startswith("inetnum:"): - return self._parse_inetnum_block(block, validcountries) + return self._parse_inetnum_block(block, source_key, validcountries) # organisation elif line.startswith("organisation:"): - return self._parse_org_block(block) + return self._parse_org_block(block, source_key) - def _parse_autnum_block(self, block): + def _parse_autnum_block(self, block, source_key): autnum = {} for line in block: # Split line @@ -574,13 +581,13 @@ class CLI(object): return # Insert into database - self.db.execute("INSERT INTO _autnums(number, organization) \ - VALUES(%s, %s) ON CONFLICT (number) DO UPDATE SET \ + self.db.execute("INSERT INTO _autnums(number, organization, source) \ + VALUES(%s, %s, %s) ON CONFLICT (number) DO UPDATE SET \ organization = excluded.organization", - autnum.get("asn"), autnum.get("org"), + autnum.get("asn"), autnum.get("org"), source_key, ) - def _parse_inetnum_block(self, block, validcountries = None): + def _parse_inetnum_block(self, block, source_key, validcountries = None): log.debug("Parsing inetnum block:") inetnum = {} @@ -636,12 +643,12 @@ class CLI(object): break # Everything is fine here, run INSERT statement... - self.db.execute("INSERT INTO _rirdata(network, country) \ - VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country = excluded.country", - "%s" % single_network, inetnum.get("country"), + self.db.execute("INSERT INTO _rirdata(network, country, source) \ + VALUES(%s, %s, %s) ON CONFLICT (network) DO UPDATE SET country = excluded.country", + "%s" % single_network, inetnum.get("country"), source_key, ) - def _parse_org_block(self, block): + def _parse_org_block(self, block, source_key): org = {} for line in block: # Split line @@ -656,13 +663,13 @@ class CLI(object): if not org: return - self.db.execute("INSERT INTO _organizations(handle, name) \ - VALUES(%s, %s) ON CONFLICT (handle) DO \ + self.db.execute("INSERT INTO _organizations(handle, name, source) \ + VALUES(%s, %s, %s) ON CONFLICT (handle) DO \ UPDATE SET name = excluded.name", - org.get("organisation"), org.get("org-name"), + org.get("organisation"), org.get("org-name"), source_key, ) - def _parse_line(self, line, validcountries = None): + def _parse_line(self, line, source_key, validcountries = None): # Skip version line if line.startswith("2"): return @@ -689,9 +696,9 @@ class CLI(object): return if type in ("ipv6", "ipv4"): - return self._parse_ip_line(country_code, type, line) + return self._parse_ip_line(country_code, type, line, source_key) - def _parse_ip_line(self, country, type, line): + def _parse_ip_line(self, country, type, line, source_key): try: address, prefix, date, status, organization = line.split("|") except ValueError: @@ -729,10 +736,10 @@ class CLI(object): if not self._check_parsed_network(network): return - self.db.execute("INSERT INTO networks(network, country) \ - VALUES(%s, %s) ON CONFLICT (network) DO \ + self.db.execute("INSERT INTO networks(network, country, source) \ + VALUES(%s, %s, %s) ON CONFLICT (network) DO \ UPDATE SET country = excluded.country", - "%s" % network, country, + "%s" % network, country, source_key, ) def handle_update_announcements(self, ns): -- 2.20.1

2 7

Re: Update AS, Country ...
by Peter Müller 22 May '21

22 May '21

Hello Sebastian, first, please keep the list CC'ed for transparency reasons (https://wiki.ipfire.org/devel/contact). Second: To the best of my knowledge, the Tor people currently fetch a version of location database a few days before they publish a new release, and convert it into the MaxMind database format they have used before. That implies (a) huge delays for getting updated databases to Tor users, especially if they use LTS versions packaged by Debian or similar distributions, which usually backport security issues only. Given the fact that the MaxMind database was even more inaccurate, I guess the status quo is a significant achievement already. Perhaps the Tor folks might slowly like to reduce delays of location database updates, but I don't know. (b) the statement "Tor is using the IPFire location database" might be kind of misleading. Technically, they do, but currently, they do not use _libloc_ as such. Not sure if that might change in the future; if so, shipping daily updates of the location databases to Tor users would be possible in theory, but comes with other caveats. :-) Sorry for not being able to help you on that issue. Please reach out to an appropriate Tor mailing list, if necessary. In case you do, I would be happy to be CC'ed on that conversation. :-) Thanks, and best regards, Peter Müller > Hi, > > Thanks for the fast reply. Im running Debian Buster with precompiled Tor and it seems there is no update script :-( Contabo Singapore is indeed correct but i guess i have to wait until they use a more recent db. > -- > > Sincerely yours / M.f.G. / Sincères salutations > > Sebastian Urbach > > > > Am 22. Mai 2021 12:35:43 schrieb Peter Müller <peter.mueller(a)ipfire.org>: > >> Hello Sebastian, >> >> thanks for reaching out. >> >> According to APNIC, that IP address nowadays belongs to "Contabo Asia Private Limited", having it's >> country code set to "SG" (Singapore) in the APNIC database. >> >> As far as I am concerned, the current version of IPFire's location database has this covered: >> >>> user@host:~$ location --version >>> location 0.9.6 >>> user@host:~$ location version >>> Sat, 22 May 2021 05:22:23 GMT >>> user@host:~$ location lookup 194.233.71.100 >>> 194.233.71.100: >>> Network : 194.233.68.0/22 >>> Country : Singapore >>> Autonomous System : AS141995 - Contabo Asia Private Limited >> >> May I ask you to run "location update" on your system (there should be a cron job or systemd timer >> for it, as we publish a new database every morning) and see if this solves your problem? >> >> Thanks, and best regards, >> Peter Müller > >

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Location