April 2021 - Location - lists.ipfire.org

[PATCH v3] po: Update translations
by Valters Jansons 15 Apr '21

15 Apr '21

POTFILES.in should not contain src/python/__init__.py file as it is not present in the committed tree. It has its respective .in file which is present instead. This commit further ensures po/POTFILES.in generator avoids such files that Git ignores (using git-check-ignore during find). Signed-off-by: Valters Jansons <valter.jansons(a)gmail.com> --- Makefile.am | 5 +++-- debian/control | 1 + po/POTFILES.in | 3 +-- po/de.po | 5 ++--- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/Makefile.am b/Makefile.am index e8ee791..dbdfd8e 100644 --- a/Makefile.am +++ b/Makefile.am @@ -79,8 +79,9 @@ update-po: po/POTFILES.in $(MAKE) -C po update-po po/POTFILES.in: Makefile - find $(abs_srcdir)/src | \ - grep -E "\.(in|py)$$" | sed -e "s@$(abs_srcdir)/@@g" | sort > $@ + find $(abs_srcdir)/src -type f $ -name '*.in' -o -name '*.py' $ \ + \! -exec git check-ignore -q {} \; -print | \ + sed -e "s@$(abs_srcdir)/@@g" | LC_ALL=C sort > $@ EXTRA_DIST += \ examples/private-key.pem \ diff --git a/debian/control b/debian/control index dc40927..118a111 100644 --- a/debian/control +++ b/debian/control @@ -16,6 +16,7 @@ Build-Depends: systemd, xsltproc <!nodoc>, docbook-xsl <!nodoc>, + git, Rules-Requires-Root: no Homepage: https://location.ipfire.org/ Vcs-Git: https://git.ipfire.org/pub/git/location/libloc.git diff --git a/po/POTFILES.in b/po/POTFILES.in index 9088986..5d2cc46 100644 --- a/po/POTFILES.in +++ b/po/POTFILES.in @@ -1,11 +1,10 @@ src/libloc.pc.in +src/python/__init__.py.in src/python/database.py src/python/downloader.py src/python/export.py src/python/i18n.py src/python/importer.py -src/python/__init__.py -src/python/__init__.py.in src/python/location-importer.in src/python/location.in src/python/logger.py diff --git a/po/de.po b/po/de.po index c35f348..8656453 100644 --- a/po/de.po +++ b/po/de.po @@ -7,7 +7,7 @@ msgid "" msgstr "" "Project-Id-Version: libloc 0\n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2020-08-04 15:44+0000\n" +"POT-Creation-Date: 2021-04-15 11:29+0000\n" "PO-Revision-Date: 2018-02-01 14:05+0000\n" "Last-Translator: Michael Tremer <michael.tremer(a)ipfire.org>\n" "Language-Team: German\n" @@ -217,8 +217,7 @@ msgstr "" msgid "AS%(asn)s belongs to %(name)s" msgstr "" -#, python-format -msgid "The datase has recently be updated recently (%s)" +msgid "The database has been updated recently" msgstr "" msgid "You must at least pass one flag" -- 2.31.1

1 0

[PATCH] po: Update translations
by Valters Jansons 14 Apr '21

14 Apr '21

In a fresh checkout of the repository, the src/python/__init__.py file is no longer present. As a result, removing from POTFILES.in. Signed-off-by: Valters Jansons <valter.jansons(a)gmail.com> --- po/POTFILES.in | 1 - po/de.po | 78 ++++++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 75 insertions(+), 4 deletions(-) diff --git a/po/POTFILES.in b/po/POTFILES.in index 9088986..41cbd63 100644 --- a/po/POTFILES.in +++ b/po/POTFILES.in @@ -4,7 +4,6 @@ src/python/downloader.py src/python/export.py src/python/i18n.py src/python/importer.py -src/python/__init__.py src/python/__init__.py.in src/python/location-importer.in src/python/location.in diff --git a/po/de.po b/po/de.po index c35f348..f01042e 100644 --- a/po/de.po +++ b/po/de.po @@ -7,7 +7,7 @@ msgid "" msgstr "" "Project-Id-Version: libloc 0\n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2020-08-04 15:44+0000\n" +"POT-Creation-Date: 2021-04-14 11:49+0000\n" "PO-Revision-Date: 2018-02-01 14:05+0000\n" "Last-Translator: Michael Tremer <michael.tremer(a)ipfire.org>\n" "Language-Team: German\n" @@ -17,240 +17,312 @@ msgstr "" "Content-Transfer-Encoding: 8bit\n" "Plural-Forms: nplurals=2; plural=(n != 1);\n" +#: ../src/python/location-importer.in:42 msgid "Location Importer Command Line Interface" msgstr "" +#: ../src/python/location-importer.in:48 ../src/python/location.in:52 msgid "Enable debug output" msgstr "" +#: ../src/python/location-importer.in:50 ../src/python/location.in:54 msgid "Enable quiet mode" msgstr "" +#: ../src/python/location-importer.in:58 msgid "Database Hostname" msgstr "" +#: ../src/python/location-importer.in:58 msgid "HOST" msgstr "" +#: ../src/python/location-importer.in:60 msgid "Database Name" msgstr "" +#: ../src/python/location-importer.in:60 msgid "NAME" msgstr "" +#: ../src/python/location-importer.in:62 msgid "Database Username" msgstr "" +#: ../src/python/location-importer.in:62 msgid "USERNAME" msgstr "" +#: ../src/python/location-importer.in:64 msgid "Database Password" msgstr "" +#: ../src/python/location-importer.in:64 msgid "PASSWORD" msgstr "" #. Write Database +#: ../src/python/location-importer.in:67 msgid "Write database to file" msgstr "" +#: ../src/python/location-importer.in:69 msgid "Database File" msgstr "" +#: ../src/python/location-importer.in:70 msgid "Signing Key" msgstr "" +#: ../src/python/location-importer.in:71 msgid "Backup Signing Key" msgstr "" +#: ../src/python/location-importer.in:72 msgid "Sets the vendor" msgstr "" +#: ../src/python/location-importer.in:73 msgid "Sets a description" msgstr "" +#: ../src/python/location-importer.in:74 msgid "Sets the license" msgstr "" +#: ../src/python/location-importer.in:75 msgid "Database Format Version" msgstr "" #. Update WHOIS +#: ../src/python/location-importer.in:78 msgid "Update WHOIS Information" msgstr "" +#: ../src/python/location-importer.in:83 msgid "Update BGP Annoucements" msgstr "" +#: ../src/python/location-importer.in:86 msgid "Route Server to connect to" msgstr "" +#: ../src/python/location-importer.in:86 msgid "SERVER" msgstr "" +#: ../src/python/location-importer.in:90 msgid "Update overrides" msgstr "" +#: ../src/python/location-importer.in:93 msgid "Files to import" msgstr "" +#: ../src/python/location-importer.in:99 msgid "Import countries" msgstr "" +#: ../src/python/location-importer.in:102 msgid "File to import" msgstr "" +#: ../src/python/location.in:46 msgid "Location Database Command Line Interface" msgstr "" +#: ../src/python/location.in:62 msgid "Path to database" msgstr "" +#: ../src/python/location.in:67 msgid "Public Signing Key" msgstr "" +#: ../src/python/location.in:72 msgid "Show database version" msgstr "" +#: ../src/python/location.in:77 msgid "Lookup one or multiple IP addresses" msgstr "" +#: ../src/python/location.in:84 msgid "Dump the entire database" msgstr "" #. Update +#: ../src/python/location.in:90 msgid "Update database" msgstr "" +#: ../src/python/location.in:92 msgid "Update the library only once per interval" msgstr "" +#: ../src/python/location.in:99 msgid "Verify the downloaded database" msgstr "" +#: ../src/python/location.in:104 msgid "Get information about one or multiple Autonomous Systems" msgstr "" +#: ../src/python/location.in:111 msgid "Search for Autonomous Systems that match the string" msgstr "" +#: ../src/python/location.in:118 msgid "Lists all networks in an AS" msgstr "" +#: ../src/python/location.in:128 msgid "Lists all networks in a country" msgstr "" +#: ../src/python/location.in:138 msgid "Lists all networks with flags" msgstr "" +#: ../src/python/location.in:141 msgid "Anonymous Proxies" msgstr "" +#: ../src/python/location.in:144 msgid "Satellite Providers" msgstr "" +#: ../src/python/location.in:147 msgid "Anycasts" msgstr "" +#: ../src/python/location.in:156 msgid "Lists all countries" msgstr "" +#: ../src/python/location.in:159 msgid "Show the name of the country" msgstr "" +#: ../src/python/location.in:162 msgid "Show the continent" msgstr "" +#: ../src/python/location.in:168 msgid "Exports data in many formats to load it into packet filters" msgstr "" +#: ../src/python/location.in:170 msgid "Output format" msgstr "" +#: ../src/python/location.in:172 msgid "Output directory" msgstr "" +#: ../src/python/location.in:174 msgid "Specify address family" msgstr "" +#: ../src/python/location.in:176 msgid "List country codes or ASNs to export" msgstr "" +#: ../src/python/location.in:255 #, python-format msgid "Invalid IP address: %s" msgstr "" +#: ../src/python/location.in:265 #, python-format msgid "Nothing found for %(address)s" msgstr "" +#: ../src/python/location.in:270 msgid "Network" msgstr "" +#: ../src/python/location.in:277 msgid "Country" msgstr "" +#: ../src/python/location.in:286 msgid "Autonomous System" msgstr "" +#: ../src/python/location.in:293 msgid "Anonymous Proxy" msgstr "" +#: ../src/python/location.in:293 ../src/python/location.in:299 +#: ../src/python/location.in:305 msgid "yes" msgstr "" +#: ../src/python/location.in:299 msgid "Satellite Provider" msgstr "" +#: ../src/python/location.in:305 msgid "Anycast" msgstr "" +#: ../src/python/location.in:377 #, python-format msgid "Invalid ASN: %s" msgstr "" +#: ../src/python/location.in:386 #, python-format msgid "Could not find AS%s" msgstr "" +#: ../src/python/location.in:390 #, python-format msgid "AS%(asn)s belongs to %(name)s" msgstr "" -#, python-format -msgid "The datase has recently be updated recently (%s)" +#: ../src/python/location.in:416 +msgid "The database has been updated recently" msgstr "" +#: ../src/python/location.in:527 msgid "You must at least pass one flag" msgstr "" +#: ../src/python/location.in:577 #, python-format msgid "One Day" msgid_plural "%(days)s Days" msgstr[0] "" msgstr[1] "" +#: ../src/python/location.in:583 #, python-format msgid "One Hour" msgid_plural "%(hours)s Hours" msgstr[0] "" msgstr[1] "" +#: ../src/python/location.in:589 #, python-format msgid "One Minute" msgid_plural "%(minutes)s Minutes" msgstr[0] "" msgstr[1] "" +#: ../src/python/location.in:595 #, python-format msgid "One Second" msgid_plural "%(seconds)s Seconds" msgstr[0] "" msgstr[1] "" +#: ../src/python/location.in:599 msgid "Now" msgstr "" +#: ../src/python/location.in:601 #, python-format msgid "%s ago" msgstr "" -- 2.31.1

1 1

[PATCH] debian: Rework historical changelog
by Valters Jansons 14 Apr '21

14 Apr '21

Rewriting history is generally considered a "not-so-good" thing, however here the historical data does not align with best practises and therefore it is beneficial to provide a better example going forward. There is only one initial release. Everything following that should list some kind of release notes or changelog, or at the very least just say something along the lines of "New version" rather than "Initial release". In this commit, the Git history is used for this task, filtering out "Makefile" changes as to retain only changes that are visible to users, excluding building tooling. For Debian packages, upon release, the target distribution should be updated to "unstable" (or "experimental" if preferred for any reason) when a release is finalized. During development, an invalid distribution name is expected to be there for tracking unreleased changes. That is why "UNRELEASED" is the standard way of specifying ongoing development, being an invalid distribution name itself. The "(Closes: #XXXXXX)" tag is intended for linking to Debian bug tracker, such as linking to the initial Intent to Package ticket, or later update/bugfix tickets. There does not appear to be a bug tracker in use for this task here, and the XXXXXX bug ticket number does not take you anywhere. It's therefore better to just remove it. --- debian/changelog | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/debian/changelog b/debian/changelog index e0be397..e58c0ca 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,11 +1,18 @@ -libloc (0.9.6-1) UNRELEASED; urgency=medium +libloc (0.9.6-1) unstable; urgency=medium - * Initial release. (Closes: #XXXXXX) + * location-importer.in: skip networks with unknown country codes + * location-importer.in: process unaligned IP ranges in RIR data files + correctly + * database: Free mmapped countries section + * location-importer.in: reduce log noise for unusable networks + * location-importer.in: delete 6to4 IPv6 space as well + * location-importer.in: fix typo + * location: Fix list-networks-by-as -- Michael Tremer <michael.tremer(a)ipfire.org> Wed, 31 Mar 2021 14:06:00 +0100 -libloc (0.9.5-1) UNRELEASED; urgency=medium +libloc (0.9.5-1) unstable; urgency=medium - * Initial release. (Closes: #XXXXXX) + * Initial release. -- Stefan Schantl <stefan.schantl(a)ipfire.org> Sun, 27 Oct 2019 18:55:44 +0100 -- 2.31.1

3 7

[PATCH] debian: Add missing '<' in copyright
by Valters Jansons 12 Apr '21

12 Apr '21

The email address information should be inside brackets. This commit ensures the missing bracket character issue is remedied. --- debian/copyright | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/debian/copyright b/debian/copyright index 151d53a..636af48 100644 --- a/debian/copyright +++ b/debian/copyright @@ -8,7 +8,7 @@ Copyright: 2017-2019 IPFire Development team <info(a)ipfire.org> License: LGPL-2.1 Files: debian/* -Copyright: 2019 Stefan Schantl stefan.schantl(a)ipfire.org> +Copyright: 2019 Stefan Schantl <stefan.schantl(a)ipfire.org> License: LGPL-2.1 License: LGPL-2.1 -- 2.31.1

2 2

[PATCH] override-{a{1,3},other}: regular batch of various overrides
by Peter Müller 11 Apr '21

11 Apr '21

These came to my attention last night. Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- overrides/override-a1.txt | 17 ++++++++++-- overrides/override-a3.txt | 5 ++++ overrides/override-other.txt | 50 ++++++++++++++++++++++++++++++++++++ 3 files changed, 70 insertions(+), 2 deletions(-) diff --git a/overrides/override-a1.txt b/overrides/override-a1.txt index ef3ec7d..0b50b76 100644 --- a/overrides/override-a1.txt +++ b/overrides/override-a1.txt @@ -99,9 +99,10 @@ remarks: VPN provider [high confidence, but not proofed] is-anonymous-proxy: yes aut-num: AS55303 -descr: Eagle Sky Co., Lt -remarks: Autonomous System registered to offshore company, abuse contact is a freemail address, address says "0 Market Square, P.O. Box 364, Belize" +descr: Eagle Sky Co., Lt[d ?] +remarks: Autonomous System registered to offshore company, abuse contact is a freemail address, address says "0 Market Square, P.O. Box 364, Belize", seems to trace to some location in AP vicinity is-anonymous-proxy: yes +country: AP aut-num: AS58546 descr: Astrill VPN @@ -128,6 +129,12 @@ descr: Anonymizer, Inc. remarks: VPN provider is-anonymous-proxy: yes +aut-num: AS206819 +descr: ANSON NETWORK LIMITED +remarks: Autonomous System registered to UK letterbox company, traces back through shady ISPs to TW +is-anonymous-proxy: yes +country: TW + aut-num: AS207688 descr: DataHome S.A. remarks: VPN provider located in BR [high confidence, but not proofed] @@ -634,6 +641,12 @@ descr: Perfect Privacy LTD remarks: VPN provider is-anonymous-proxy: yes +net: 85.92.100.0/22 +descr: LoadProxy, LLC +remarks: VPN provider +is-anonymous-proxy: yes +country: US + net: 85.203.23.0/24 descr: VPN Consumer Network / falco-networks.com remarks: VPN provider diff --git a/overrides/override-a3.txt b/overrides/override-a3.txt index dbf5dd7..36e03a3 100644 --- a/overrides/override-a3.txt +++ b/overrides/override-a3.txt @@ -160,6 +160,11 @@ descr: ipcom GmbH remarks: Generic anycast network is-anycast: yes +aut-num: AS209813 +descr: Fast Content Delivery Ltd. +remarks: Generic anycast network +is-anycast: yes + aut-num: AS210004 descr: Stichting Internet Domeinregistratie Nederland (SIDN) remarks: TLD operator's anycast network diff --git a/overrides/override-other.txt b/overrides/override-other.txt index a41d4da..d2c2423 100644 --- a/overrides/override-other.txt +++ b/overrides/override-other.txt @@ -128,6 +128,11 @@ descr: Cloud Management LLC remarks: tampers with RIR data, traces back to HK country: HK +aut-num: AS44015 +descr: Landgard Management Inc. +remarks: bulletproof ISP with strong links to RU +country: RU + aut-num: AS44477 descr: IP Oleinichenko Denis remarks: ISP located in RU, but some RIR data for announced prefixes contain garbage @@ -173,11 +178,21 @@ descr: REBA Communications BV remarks: bulletproof ISP (related to AS202425) located in NL country: NL +aut-num: AS56851 +descr: PE Skurykhin Mukola Volodumurovuch +remarks: tampers with RIR data, traces back to UA +country: UA + aut-num: AS57717 descr: FiberXpress BV remarks: bulletproof ISP (related to AS202425) located in NL country: NL +aut-num: AS57724 +descr: DDOS-GUARD LTD +remarks: shady ISP, customers massively tamper with RIR data, we cannot trust this network +country: RU + aut-num: AS57858 descr: Inter Connects Inc. remarks: part of a dirty ISP conglomerate operating most likely out of SE, hijacking stolen AfriNIC networks, massively tampers with RIR data @@ -193,6 +208,11 @@ descr: FOP Gubina Lubov Petrivna remarks: bulletproof ISP operating from a war zone in eastern UA country: UA +aut-num: AS58349 +descr: INNETRA PC +remarks: another shady customer of "DDoS Guard Ltd.", jurisdiction is probably RU, but traceroutes dead-end somewhere else in EU +country: EU + aut-num: AS59580 descr: Batterflyai Media Ltd. remarks: ISP located in RU, but some RIR data for announced prefixes contain garbage @@ -208,6 +228,11 @@ descr: Network Dedicated SAS remarks: bulletproof ISP and IP hijacker, claims to be located in CH, but traces to NL country: NL +aut-num: AS61977 +descr: Vivo Trade L.P. +remarks: another shady customer of "DDoS Guard Ltd." +country: RU + aut-num: AS62468 descr: VpsQuan L.L.C. remarks: claims to be located in US, but traces to HK @@ -248,6 +273,11 @@ descr: Wujidun Network Limited remarks: part of the "Asline" IP hijacking gang, tampers with RIR data, traces back to AP region country: AP +aut-num: AS200019 +descr: ALEXHOST SRL +remarks: ISP located in MD, majority of RIR data for announced prefixes contain garbage, we cannot trust this network +country: MD + aut-num: AS200699 descr: Datashield, Inc. remarks: fake offshore location (SC), traces back to NL @@ -273,6 +303,11 @@ descr: FutureNow Incorporated remarks: ISP located in BG, but RIR data for announced prefixes contain garbage country: BG +aut-num: AS202920 +descr: DataClub S.A. +remarks: another shady customer of "DDoS Guard Ltd." +country: RU + aut-num: AS202425 descr: IP Volume Inc. remarks: bulletproof ISP (aka: AS29073 / Ecatel Ltd. / Quasi Networks Ltd.) located in NL @@ -308,6 +343,11 @@ descr: Hauer Hosting Services Limited remarks: ISP located in ES, but some RIR data for announced prefixes contain garbage country: ES +aut-num: AS206397 +descr: Genius Guard / Genius Security Ltd. +remarks: another shady customer of "DDoS Guard Ltd.", probably located in RU +country: RU + aut-num: AS207046 descr: Xtudio Networks S.L.U. remarks: ISP located in ES, but some RIR data for announced prefixes contain garbage @@ -338,6 +378,11 @@ descr: Gudaev Maxim Amrakhovich remarks: announcements scatter across various places in EU (DE/CZ/??), but RIR data contain garbage country: EU +aut-num: AS211849 +descr: Kakharov Orinbassar Maratuly +remarks: ISP located in RU, but RIR data for announced prefixes contain garbage +country: RU + aut-num: AS213035 descr: Serverion BV remarks: ISP located in NL, but RIR data for most announced prefixes contain garbage @@ -458,6 +503,11 @@ descr: Golden Internet LLC remarks: fake location (KP), WHOIS contact points to RU country: RU +net: 91.243.32.0/19 +descr: Petersburg Internet Network Ltd. +remarks: RIR data for suballocations contain garbage, they are all located in RU +country: RU + net: 95.181.152.0/21 descr: QWARTA LLC remarks: fake location (US), WHOIS contact and traceroutes point to RU -- 2.26.2

1 0

[PATCH] overrides-a{1,2}: weekly batch of overrides
by Peter Müller 10 Apr '21

10 Apr '21

Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- overrides/override-a1.txt | 11 +++++++++++ overrides/override-a2.txt | 5 +++++ 2 files changed, 16 insertions(+) diff --git a/overrides/override-a1.txt b/overrides/override-a1.txt index 76a5a52..ef3ec7d 100644 --- a/overrides/override-a1.txt +++ b/overrides/override-a1.txt @@ -128,6 +128,12 @@ descr: Anonymizer, Inc. remarks: VPN provider is-anonymous-proxy: yes +aut-num: AS207688 +descr: DataHome S.A. +remarks: VPN provider located in BR [high confidence, but not proofed] +is-anonymous-proxy: yes +country: BR + aut-num: AS208294 descr: Markus Koch remarks: Tor relay provider @@ -748,6 +754,11 @@ descr: VPN Consumer Network remarks: VPN provider is-anonymous-proxy: yes +net: 104.165.145.0/24 +descr: Paceweb Proxy +remarks: VPN provider +is-anonymous-proxy: yes + net: 104.238.32.0/24 descr: VPN Consumer Network remarks: VPN provider diff --git a/overrides/override-a2.txt b/overrides/override-a2.txt index 72d1a1d..f53053b 100644 --- a/overrides/override-a2.txt +++ b/overrides/override-a2.txt @@ -157,6 +157,11 @@ descr: Emerging Markets Communications / SantanderTeleport / ... remarks: Satellite Internet provider is-satellite-provider: yes +aut-num: AS28429 +descr: Enlace de Datos y Redes SA de CV +remarks: Satellite Internet provider [high confidence, but not proofed] +is-satellite-provider: yes + aut-num: AS29137 descr: Eutelsat S.A. remarks: Satellite Internet provider -- 2.26.2

1 0

[PATCH] override-other: fix overrides for orphaned Yugoslavian networks
by Peter Müller 05 Apr '21

05 Apr '21

https://lists.ipfire.org/pipermail/location/2021-April/000267.html Cc: Michael Tremer <michael.tremer(a)ipfire.org> Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- overrides/override-other.txt | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/overrides/override-other.txt b/overrides/override-other.txt index b428d9f..a41d4da 100644 --- a/overrides/override-other.txt +++ b/overrides/override-other.txt @@ -543,6 +543,21 @@ descr: NetConn Services Ltd remarks: APNIC chunk owned by a HK-based company, routed to AP region, but assigned to SC country: AP +net: 195.66.165.0/24 +descr: Posta Crne Gore +remarks: Orphaned RIR data, see: https://lists.ipfire.org/pipermail/location/2021-April/000267.html +country: ME + +net: 195.252.115.0/24 +descr: Drenik ISP +remarks: Orphaned RIR data, see: https://lists.ipfire.org/pipermail/location/2021-April/000267.html +country: ME + +net: 195.252.120.0/24 +descr: AB SOFT +remarks: Orphaned RIR data, see: https://lists.ipfire.org/pipermail/location/2021-April/000267.html +country: ME + net: 198.144.120.0/23 descr: Amarutu Technology Ltd. / KoDDoS / ESecurity remarks: fake offshore location (BZ), traces back to NL -- 2.26.2

1 0

[PATCH v2] location-importer.in: skip networks with unknown country codes
by Peter Müller 04 Apr '21

04 Apr '21

There is no sense in parsing and storting networks whose country codes cannot be found in the ISO-3166-x country code table. This avoids side effects in applications using the location database, and introduces another sanity check to compensate bogus RIR data. On location02, this affects some networks from APNIC (country code: ZZ) as well as a bunch of smaller allocations within the RIPE region still tagged to CS or YU (Yugoslavia). To my surprise, no network tagged as SU (Soviet Union) was found - while the NIC for .su TLD is still operational. :-) Applying this patch causes the countries to be processed before update_whois() is called. In case no countries are present in the SQL table, this check is silently omitted. Fixes: #12510 Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- src/python/location-importer.in | 38 ++++++++++++++++++++++----------- 1 file changed, 26 insertions(+), 12 deletions(-) diff --git a/src/python/location-importer.in b/src/python/location-importer.in index e2f201b..1e08458 100644 --- a/src/python/location-importer.in +++ b/src/python/location-importer.in @@ -388,10 +388,17 @@ class CLI(object): TRUNCATE TABLE networks; """) + # Fetch all valid country codes to check parsed networks aganist... + rows = self.db.query("SELECT * FROM countries ORDER BY country_code") + validcountries = [] + + for row in rows: + validcountries.append(row.country_code) + for source in location.importer.WHOIS_SOURCES: with downloader.request(source, return_blocks=True) as f: for block in f: - self._parse_block(block) + self._parse_block(block, validcountries) # Process all parsed networks from every RIR we happen to have access to, # insert the largest network chunks into the networks table immediately... @@ -467,7 +474,7 @@ class CLI(object): # Download data with downloader.request(source) as f: for line in f: - self._parse_line(line) + self._parse_line(line, validcountries) def _check_parsed_network(self, network): """ @@ -532,7 +539,7 @@ class CLI(object): # be suitable for libloc consumption... return True - def _parse_block(self, block): + def _parse_block(self, block, validcountries = None): # Get first line to find out what type of block this is line = block[0] @@ -542,7 +549,7 @@ class CLI(object): # inetnum if line.startswith("inet6num:") or line.startswith("inetnum:"): - return self._parse_inetnum_block(block) + return self._parse_inetnum_block(block, validcountries) # organisation elif line.startswith("organisation:"): @@ -573,7 +580,7 @@ class CLI(object): autnum.get("asn"), autnum.get("org"), ) - def _parse_inetnum_block(self, block): + def _parse_inetnum_block(self, block, validcountries = None): log.debug("Parsing inetnum block:") inetnum = {} @@ -616,10 +623,10 @@ class CLI(object): if not inetnum or not "country" in inetnum: return - # Skip objects with bogus country code 'ZZ' - if inetnum.get("country") == "ZZ": - log.warning("Skipping network with bogus country 'ZZ': %s" % \ - (inetnum.get("inet6num") or inetnum.get("inetnum"))) + # Skip objects with unknown country codes + if validcountries and inetnum.get("country") not in validcountries: + log.warning("Skipping network with bogus country '%s': %s" % \ + (inetnum.get("country"), inetnum.get("inet6num") or inetnum.get("inetnum"))) return # Iterate through all networks enumerated from above, check them for plausibility and insert @@ -652,7 +659,7 @@ class CLI(object): org.get("organisation"), org.get("org-name"), ) - def _parse_line(self, line): + def _parse_line(self, line, validcountries = None): # Skip version line if line.startswith("2"): return @@ -667,8 +674,15 @@ class CLI(object): log.warning("Could not parse line: %s" % line) return - # Skip any lines that are for stats only - if country_code == "*": + # Skip any lines that are for stats only or do not have a country + # code at all (avoids log spam below) + if not country_code or country_code == '*': + return + + # Skip objects with unknown country codes + if validcountries and country_code not in validcountries: + log.warning("Skipping line with bogus country '%s': %s" % \ + (country_code, line)) return if type in ("ipv6", "ipv4"): -- 2.26.2

2 3

[PATCH] location-importer.in: process unaligned IP ranges in RIR data files correctly
by Peter Müller 01 Apr '21

01 Apr '21

The IP range given in an inetnum object apparently not necessarily matches distinct subnet boundaries. As a result, the current attempt to calculate its CIDR mask resulted in faulty subnets not covering the entire IP range. This patch leaves the task of enumerating subnets to the ipaddress module itself, which handles things much more robust. Since the output may contain of several subnets, a list for the inetnum key is necessary as well as a loop over them when conducting the SQL statements. Fixes: #12595 Cc: Michael Tremer <michael.tremer(a)ipfire.org> Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- src/python/location-importer.in | 31 +++++++++++-------------------- 1 file changed, 11 insertions(+), 20 deletions(-) diff --git a/src/python/location-importer.in b/src/python/location-importer.in index 2506925..e2f201b 100644 --- a/src/python/location-importer.in +++ b/src/python/location-importer.in @@ -3,7 +3,7 @@ # # # libloc - A library to determine the location of someone on the Internet # # # -# Copyright (C) 2020 IPFire Development Team <info(a)ipfire.org> # +# Copyright (C) 2020-2021 IPFire Development Team <info(a)ipfire.org> # # # # This library is free software; you can redistribute it and/or # # modify it under the terms of the GNU Lesser General Public # @@ -604,18 +604,10 @@ class CLI(object): log.warning("Could not parse line: %s" % line) return - # Set prefix to default - prefix = 32 - - # Count number of addresses in this subnet - num_addresses = int(end_address) - int(start_address) - if num_addresses: - prefix -= math.log(num_addresses, 2) - - inetnum["inetnum"] = "%s/%.0f" % (start_address, prefix) + inetnum["inetnum"] = list(ipaddress.summarize_address_range(start_address, end_address)) elif key == "inet6num": - inetnum[key] = val + inetnum[key] = [ipaddress.ip_network(val, strict=False)] elif key == "country": inetnum[key] = val.upper() @@ -630,15 +622,14 @@ class CLI(object): (inetnum.get("inet6num") or inetnum.get("inetnum"))) return - network = ipaddress.ip_network(inetnum.get("inet6num") or inetnum.get("inetnum"), strict=False) - - if not self._check_parsed_network(network): - return - - self.db.execute("INSERT INTO _rirdata(network, country) \ - VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country = excluded.country", - "%s" % network, inetnum.get("country"), - ) + # Iterate through all networks enumerated from above, check them for plausibility and insert + # them into the database, if _check_parsed_network() succeeded + for single_network in inetnum.get("inet6num") or inetnum.get("inetnum"): + if self._check_parsed_network(single_network): + self.db.execute("INSERT INTO _rirdata(network, country) \ + VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country = excluded.country", + "%s" % single_network, inetnum.get("country"), + ) def _parse_org_block(self, block): org = {} -- 2.26.2

2 7

Error in ' list-networks-by-cc'
by Gisle Vanem 01 Apr '21

01 Apr '21

When issuing: location.py list-networks-by-cc XX I get this error: for n in db.search_networks(country_code=country_code, family=ns.family): TypeError: 'country_code' is an invalid keyword argument for this function But this works better: @@ -506,7 +506,7 @@ f = writer(sys.stdout, prefix=country_code) # Print all matching networks - for n in db.search_networks(country_code=country_code, family=ns.family): + for n in db.search_networks(country_codes=[country_code.upper()], family=ns.family): f.write(n) --- with the added bonus that I can do 'no' or 'NO'. -- --gv

2 1

2024

2023

2022

2021

2020

2019

2018

2017

Location April 2021