- Location - lists.ipfire.org

Please update individual networks
by Martin Gebhardt 01 May '23

01 May '23

Hello Peter, I noticed while looking through the Tor metrics of my relays that I run at my employer that a few networks can be updated. https://metrics.torproject.org/rs.html#search/family:824862C40A81327DE807F4… Can you help again? :-) Reference in order of diff below: 37.252.254.0/24 https://apps.db.ripe.net/db-web-ui/query?searchtext=37.252.254.0%2F24 37.252.255.0/24 https://apps.db.ripe.net/db-web-ui/query?searchtext=37.252.255.0%2F24 217.146.2.0/24 https://apps.db.ripe.net/db-web-ui/query?searchtext=217.146.2.0%2F24 2a00:11c0:1b::/48 https://apps.db.ripe.net/db-web-ui/query?searchtext=2a00:11c0:1b:aa2::5 2a00:11c0:1c::/48 https://apps.db.ripe.net/db-web-ui/query?searchtext=2a00:11c0:1c:311::5 2a00:11c0:1f::/48 https://apps.db.ripe.net/db-web-ui/query?searchtext=2a00:11c0:1f:1::55 When checking the addresses, I also checked rDNS. More than a year ago, you pointed out to me that this is very suboptimal. At that time, I had opened a ticket with our NOC as promised. But unfortunately it's still bad... Partly it works with the addresses mentioned here, and partly it is.. meaningless. I didn't know if this was done in the override config or directly in the database.txt. But maybe it helps. diff --git a/database.txt b/database.txt index 34ed5d5..85f8e99 100644 --- a/database.txt +++ b/database.txt @@ -585009,11 +585009,11 @@ country: AT aut-num: 42473 net: 37.252.254.0/24 -country: AT +country: RU aut-num: 42473 net: 37.252.255.0/24 -country: AT +country: RU aut-num: 42473 net: 37.253.0.0/16 @@ -4506501,7 +4506501,7 @@ country: AT aut-num: 42473 net: 217.146.2.0/24 -country: AT +country: UA aut-num: 42473 net: 217.146.4.0/24 @@ -5241785,11 +5241785,11 @@ country: AT aut-num: 42473 net: 2a00:11c0:1b::/48 -country: AT +country: RU aut-num: 42473 net: 2a00:11c0:1c::/48 -country: AT +country: RU aut-num: 42473 net: 2a00:11c0:1d::/48 @@ -5241801,7 +5241801,7 @@ country: AT aut-num: 42473 net: 2a00:11c0:1f::/48 -country: AT +country: UA aut-num: 42473

1 0

Warning in '__loc_network_tree_count()'
by Gisle Vanem 04 Mar '23

04 Mar '23

Hello list. Compiling with 'clang-cl' on Windows-10, I saw this important warning: network.c(796,10): warning: variable 'counter' set but not used [-Wunused-but-set-variable] size_t* counter = (size_t*)data; ^ So shouldn't the function be: static int __loc_network_tree_count(struct loc_network* network, void* data) { size_t* counter = (size_t*)data; // Increase the counter for each network (*counter)++; return 0; } But I fail to see this ret-val is used for anything. BTW, disasm now is correct: inc qword ptr [rdx] xor eax,eax ret -- --gv

2 3

Re: Upload libloc to Debian
by Stefan Schantl 02 Mar '23

02 Mar '23

Hello Hans-Christoph, a big thanks for writing and sharing your script. I've added it to the official libloc source code by the following commit: ttps://git.ipfire.org/?p=location/libloc.git;a=commit;h=02a7d6ec0bb79f9 62ffe0746d311b3454b11a3db Best regards, -Stefan > > Hey all, > > I just uploaded a new version of the libloc Debian package that > includes bash > completion for the 'location' command. I would like to see this file > included > upstream in your git, so it is also attached to the email. > > If you want to try it, either install location 0.9.16-2 from Debian, > or stick > the attached file in /etc/bash_completion.d/location and open a new > bash shell. > > .hc > > Jochen Sprickerhof: > > Hi Michael, > > > > * Michael Tremer <michael.tremer(a)ipfire.org> [2022-08-16 10:00]: > > > > https://buildd.debian.org/status/package.php?p=libloc > > > > > > I installed a virtual machine with mips64el and the testsuite > > > weirdly runs > > > through. > > > > I was able to reproduce it using ppc64el: > > > > # echo "deb-src http://deb.debian.org/debian unstable main" >> > > /etc/apt/sources.list > > # apt update > > # apt build-dep libloc > > # apt source --compile libloc > > > > Interestingly the new version now also fails on mipsel, so maybe it > > is a flaky > > test? > > > > https://buildd.debian.org/status/logs.php?pkg=libloc&arch=mipsel > > > > Given that it compiled before this means we should try to fix the > > bug as it is > > blocks testing migration, otherwise: > > > > https://tracker.debian.org/pkg/libloc > > > > (The other option would be to request removal of the old mipsel > > version.) > > > > > Additionally, the packages don’t build for Debian any more using > > > my script. > > > > > > I opened a bug ticket with the error here: > > > https://bugzilla.ipfire.org/show_bug.cgi?id=12912 > > > > Looks like you try to cross build (installing crossbuild-essential- > > arm64:amd64), > > maybe that's currently broken. You can try installing qemu-user- > > static and > > replace --host with --arch in debian/build.sh. > > Btw. sbuild updates the chroot before building so there should be > > no need to > > throw it away (for a stable release). > > > > Cheers Jochen > > -- > Signal: +13478504872 > PGP fingerprint: EE66 20C7 136B 0D2C 456C 0A4D E9E2 8DEA 00AA 5556 > https://keys.openpgp.org/search?q=EE6620C7136B0D2C456C0A4DE9E28DEA00AA5556

2 1

[PATCH 1/2] Add bash-completion file for the location command.
by Stefan Schantl 24 Feb '23

24 Feb '23

From: Hans-Christoph Steiner <hans(a)guardianproject.info> Signed-off-by: Stefan Schantl <stefan.schantl(a)ipfire.org> --- bash-completion/location | 151 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 151 insertions(+) create mode 100644 bash-completion/location diff --git a/bash-completion/location b/bash-completion/location new file mode 100644 index 0000000..66fc6c1 --- /dev/null +++ b/bash-completion/location @@ -0,0 +1,151 @@ +# location(1) completion -*- shell-script -*- +# +# bash-completion - part of libloc +# +# Copyright (C) 2020,2023 Hans-Christoph Steiner <hans(a)eds.org> +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Lesser General Public License as published by +# the Free Software Foundation, either version 2.1 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. + +__location_init() { + if type -t _init_completion >/dev/null; then + _init_completion -n : || return + else + # manual initialization for older bash completion versions + COMPREPLY=() + cur="${COMP_WORDS[COMP_CWORD]}" + prev="${COMP_WORDS[COMP_CWORD-1]}" + fi + + (( $# >= 1 )) && __complete_${1} + __ltrim_colon_completions "$cur" +} + +__complete_options() { + case "${prev}" in + --directory) + _filedir -d + return 0;; + --cron) + COMPREPLY=( $( compgen -W "daily weekly monthly" -- $cur ) ) + return 0;; + --family) + COMPREPLY=( $( compgen -W "ipv6 ipv4" -- $cur ) ) + return 0;; + --format) + COMPREPLY=( $( compgen -W "ipset list nftables xt_geoip" -- $cur ) ) + return 0;; + esac + + case "$cur" in + -*) + COMPREPLY=( $( compgen -W "--help ${lopts}" -- $cur ) ) + return 0;; + esac +} + +__complete_dump() { + __complete_options +} + +__complete_get_as() { + __complete_options +} + +__complete_export() { + lopts="--directory --family --format" + __complete_options +} + +__complete_list_networks_by_as() { + lopts="--family --format" + __complete_options +} + +__complete_list_networks_by_cc() { + lopts="--family --format" + __complete_options +} + +__complete_list_networks_by_flags() { + lopts="--anonymous-proxy --satellite-provider --anycast --drop --family --format" + __complete_options +} + +__complete_list_bogons() { + lopts="--family --format" + __complete_options +} + +__complete_list_countries() { + lopts="--show-name --show-continent" + __complete_options +} + +__complete_lookup() { + __complete_options +} + +__complete_search_as() { + __complete_options +} + +__complete_update() { + lopts="--cron" + __complete_options +} + +__complete_version() { + __complete_options +} + +__complete_verify() { + __complete_options +} + +# for f in `location|grep -Eo '[a-z,-]+,[a-z,-]+'| sed 's/,/ /g'`; do printf '%s \\\n' $f; done|sort -u +__cmds=" \ +dump \ +export \ +get-as \ +list-bogons \ +list-countries \ +list-networks-by-as \ +list-networks-by-cc \ +list-networks-by-flags \ +lookup \ +search-as \ +update \ +verify \ +version \ +" + +for c in $__cmds; do + eval "_location_${c} () { + local cur prev lopts + __location_init ${c//-/_} + }" +done + +_location() { + local cmd + cmd=${COMP_WORDS[1]} + + [[ $__cmds == *\ $cmd\ * ]] && _location_${cmd} || { + (($COMP_CWORD == 1)) && COMPREPLY=( $( compgen -W "${__cmds}" -- $cmd ) ) + } +} + +complete -F _location location + +return 0 -- 2.30.2

1 1

Re: Location database updating every hour.
by Peter Müller 02 Dec '22

02 Dec '22

Hello Rob, thanks for reporting this. This problem was caused by a glitch in the location-importer, a component that is part of libloc and generates a new location database daily (in the early morning, UTC-speaking). The root cause of this glitch is that location-importer would not properly detect gzipped data if no matching MIME header was sent by the database server. RIPE, apparently, changed their database server infrastructure recently, and now only returns application/octet-stream for gzipped data. While generating the location database, this lead to RIPE data being omitted completely - which then caused sanity checks to fail, since we expect some data points served by RIPE to be present in a defined way. Michael fixed this in [1], but thanks to the load situation, some chaos the ongoing IPFire 3 "hackathon" caused, and a change we had to undergo in our infrastructure backend, it took us until today to be able to generate proper location databases again. The latest one was just generated and published shortly before you wrote your e-mail, on Fri, 02 Dec 2022 15:17:42 GMT [2]. You should now observe any and all IPFire installations to fetch this database once, and then behave like they usual do - one update attempt per day. Apologies for the hassle, and thanks for reaching out, Peter Müller [1] https://git.ipfire.org/?p=location/libloc.git;a=commit;h=c39b3b92b0c557fba4… [2] https://git.ipfire.org/?p=location/location-database.git;a=commit;h=d2d268a… > Since the beginning of this month the update-location-database script runs > hourly and isn't waiting for the expected next 7 day interval. > > I am not getting the expected "The database has been updated recently" > message in /var/log/messages since Dec 1 05:06:32 > > > On my development box IPFire 2.27 (x86_64) - Core-Update 170 > . > . > Nov 30 22:38:14 ipfire-dev2 The database has been updated recently > Nov 30 23:33:19 ipfire-dev2 The database has been updated recently > Dec 1 00:24:50 ipfire-dev2 The database has been updated recently > Dec 1 01:30:08 ipfire-dev2 The database has been updated recently > Dec 1 02:14:32 ipfire-dev2 The database has been updated recently > Dec 1 03:52:51 ipfire-dev2 The database has been updated recently > Dec 1 04:09:29 ipfire-dev2 The database has been updated recently > Dec 1 05:06:32 ipfire-dev2 The database has been updated recently > > and also on my production box IPFire 2.27 (x86_64) - Core Update 168 > . > . > Nov 30 22:45:58 griffith6 The database has been updated recently > Nov 30 23:22:58 griffith6 The database has been updated recently > Dec 1 00:53:45 griffith6 The database has been updated recently > Dec 1 01:53:41 griffith6 The database has been updated recently > Dec 1 02:45:16 griffith6 The database has been updated recently > Dec 1 03:20:54 griffith6 The database has been updated recently > Dec 1 04:34:41 griffith6 The database has been updated recently > Dec 1 05:48:56 griffith6 The database has been updated recently > Dec 1 06:28:10 griffith6 The database has been updated recently > > There doesn't seem to affect performance other than the location ipsets > are being updated every hour. > > location version is being set to the last update time: > # location version > Fri, 02 Dec 2022 15:17:42 GMT > > Could this be caused by the leading blank space in the day column after > the month change from November to December ? > > > > Rob

1 0

libloc 0.9.16 rootfile changes
by Peter Müller 21 Nov '22

21 Nov '22

Hello *, while trying to update libloc in IPFire 2.x, I noticed a possible quirk in the way the Perl module and man page files are currently installed. The rootfile diff looks as follows: diff --git a/config/rootfiles/common/libloc b/config/rootfiles/common/libloc index 7eb94217e..628ac343f 100644 --- a/config/rootfiles/common/libloc +++ b/config/rootfiles/common/libloc @@ -1,3 +1,8 @@ +#Location.3 +Location.pm +#auto +#auto/Location +auto/Location/Location.so #etc/cron.daily #etc/cron.daily/location-update usr/bin/location @@ -22,10 +27,6 @@ usr/bin/location #usr/lib/libloc.so usr/lib/libloc.so.1 usr/lib/libloc.so.1.0.2 -usr/lib/perl5/site_perl/5.36.0/xxxMACHINExxx-linux-thread-multi/Location.pm -#usr/lib/perl5/site_perl/5.36.0/xxxMACHINExxx-linux-thread-multi/auto/Location -#usr/lib/perl5/site_perl/5.36.0/xxxMACHINExxx-linux-thread-multi/auto/Location/.packlist -usr/lib/perl5/site_perl/5.36.0/xxxMACHINExxx-linux-thread-multi/auto/Location/Location.so #usr/lib/pkgconfig/libloc.pc #usr/lib/python3.10/site-packages/_location.la usr/lib/python3.10/site-packages/_location.so @@ -38,7 +39,6 @@ usr/lib/python3.10/site-packages/location/i18n.py #usr/lib/python3.10/site-packages/location/importer.py usr/lib/python3.10/site-packages/location/logger.py #usr/share/locale/de/LC_MESSAGES/libloc.mo -#usr/share/man/man3/Location.3 #var/lib/location var/lib/location/database.db var/lib/location/ipset It appears that, despite ./configure called with parameters such as "--disable-man-pages", libloc always installs a man page, and now places Perl files in the current working directory rather than something like /usr/lib/perl5/site_perl/5.36.0/... Does the ./configure call need additional parameters? Is this behaviour intended? Thanks, and best regards, Peter Müller

2 1

libloc 0.9.16 released
by Michael Tremer 29 Oct '22

29 Oct '22

Hello, This is the release announcement for libloc 0.9.16. It comes with the following changes: * The tarball now includes a recent default database which is going to be installed into the system. Having some data is better than having no data to start with. This will make packaging libloc easier for third-party vendors. * An issue has been resolved which lead to only one signature being written, even when multiple keys were passed to the database writer. There have also been further improvements on handling any errors during signature validation. * export: Exporting into the ipset format has been fixed * An issue with random strings being shown for vendor and description has been fixed when those attributes were not set at all * An incorrect default database path has been fixed * Some sanity checks for importing data from Spamhaus have been implemented to prevent importing any corrupted data * perl: Parallel builds were broken and the module is now installed into the correct path. Thanks to Petr Písař. * The man page for the “location” command is now being installed in section 1 instead of 8. Thanks to Petr Písař. Thank you to everyone who has contributed to this release. You can find the source here: https://git.ipfire.org/?p=location/libloc.git;a=shortlog;h=refs/tags/0.9.16 https://source.ipfire.org/releases/libloc/libloc-0.9.16.tar.gz Best, -Michael

1 0

[PATCH 01/10] importer: Store geofeed URLs from RIR data
by Michael Tremer 29 Oct '22

29 Oct '22

Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org> --- src/scripts/location-importer.in | 40 +++++++++++++++++++++++++++++++- 1 file changed, 39 insertions(+), 1 deletion(-) diff --git a/src/scripts/location-importer.in b/src/scripts/location-importer.in index 9faf23b..5bd5da3 100644 --- a/src/scripts/location-importer.in +++ b/src/scripts/location-importer.in @@ -182,6 +182,11 @@ class CLI(object): CREATE INDEX IF NOT EXISTS networks_family ON networks USING BTREE(family(network)); CREATE INDEX IF NOT EXISTS networks_search ON networks USING GIST(network inet_ops); + -- geofeeds + CREATE TABLE IF NOT EXISTS network_geofeeds(network inet, url text); + CREATE UNIQUE INDEX IF NOT EXISTS network_geofeeds_unique + ON network_geofeeds(network); + -- overrides CREATE TABLE IF NOT EXISTS autnum_overrides( number bigint NOT NULL, @@ -799,6 +804,16 @@ class CLI(object): inetnum[key].append(val) + # Parse the geofeed attribute + elif key == "geofeed": + inetnum["geofeed"] = val + + # Parse geofeed when used as a remark + elif key == "remark": + m = re.match(r"^(?:geofeed|Geofeed)\s+(https://.*)", val) + if m: + inetnum["geofeed"] = m.group(1) + # Skip empty objects if not inetnum or not "country" in inetnum: return @@ -810,7 +825,6 @@ class CLI(object): # them into the database, if _check_parsed_network() succeeded for single_network in inetnum.get("inet6num") or inetnum.get("inetnum"): if self._check_parsed_network(single_network): - # Skip objects with unknown country codes if they are valid to avoid log spam... if validcountries and invalidcountries: log.warning("Skipping network with bogus countr(y|ies) %s (original countries: %s): %s" % \ @@ -823,6 +837,30 @@ class CLI(object): "%s" % single_network, inetnum.get("country")[0], inetnum.get("country"), source_key, ) + # Update any geofeed information + geofeed = inetnum.get("geofeed", None) + + # Store/update any geofeeds + if geofeed: + self.db.execute(""" + INSERT INTO + network_geofeeds( + network, + url + ) + VALUES( + %s, %s + ) + ON CONFLICT (network) DO + UPDATE SET url = excluded.url""", + "%s" % single_network, geofeed, + ) + + # Delete any previous geofeeds + else: + self.db.execute("DELETE FROM network_geofeeds WHERE network = %s", + "%s" % single_network) + def _parse_org_block(self, block, source_key): org = {} for line in block: -- 2.30.2

2 11

[PATCH] location-importer.in: Conduct sanity checks per DROP list
by Peter Müller 04 Oct '22

04 Oct '22

Previously, the lack of distinction between different DROP lists caused only the last one to be persisted. Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- src/scripts/location-importer.in | 58 +++++++++++++++++++++----------- 1 file changed, 39 insertions(+), 19 deletions(-) diff --git a/src/scripts/location-importer.in b/src/scripts/location-importer.in index 8d47497..e4a9ab0 100644 --- a/src/scripts/location-importer.in +++ b/src/scripts/location-importer.in @@ -1427,18 +1427,21 @@ class CLI(object): def _update_overrides_for_spamhaus_drop(self): downloader = location.importer.Downloader() - ip_urls = [ - "https://www.spamhaus.org/drop/drop.txt", - "https://www.spamhaus.org/drop/edrop.txt", - "https://www.spamhaus.org/drop/dropv6.txt" + ip_lists = [ + ("Spamhaus DROP list", "https://www.spamhaus.org/drop/drop.txt"), + ("Spamhaus EDROP list", "https://www.spamhaus.org/drop/edrop.txt"), + ("Spamhaus DROPv6 list", "https://www.spamhaus.org/drop/dropv6.txt") ] - asn_urls = [ - "https://www.spamhaus.org/drop/asndrop.txt" + asn_lists = [ + ("Spamhaus ASN-DROP list", "https://www.spamhaus.org/drop/asndrop.txt") ] - for url in ip_urls: - # Fetch IP list + for list in ip_lists: + name = list[0] + url = list[1] + + # Fetch IP list from given URL f = downloader.retrieve(url) # Split into lines @@ -1448,11 +1451,11 @@ class CLI(object): # downloads. if len(fcontent) > 10: self.db.execute(""" - DELETE FROM autnum_overrides WHERE source = 'Spamhaus ASN-DROP list'; - DELETE FROM network_overrides WHERE source = 'Spamhaus DROP lists'; - """) + DELETE FROM network_overrides WHERE source = '%s'; + """ % name, + ) else: - log.error("Spamhaus DROP URL %s returned likely bogus file, ignored" % url) + log.error("%s (%s) returned likely bogus file, ignored" % (name, url)) continue # Iterate through every line, filter comments and add remaining networks to @@ -1475,8 +1478,8 @@ class CLI(object): # Sanitize parsed networks... if not self._check_parsed_network(network): - log.warning("Skipping bogus network found in Spamhaus DROP URL %s: %s" % \ - (url, network)) + log.warning("Skipping bogus network found in %s (%s): %s" % \ + (name, url, network)) continue # Conduct SQL statement... @@ -1488,14 +1491,31 @@ class CLI(object): ) VALUES (%s, %s, %s) ON CONFLICT (network) DO UPDATE SET is_drop = True""", "%s" % network, - "Spamhaus DROP lists", + name, True ) - for url in asn_urls: + for list in asn_lists: + name = list[0] + url = list[1] + # Fetch URL f = downloader.retrieve(url) + # Split into lines + fcontent = f.readlines() + + # Conduct a very basic sanity check to rule out CDN issues causing bogus DROP + # downloads. + if len(fcontent) > 10: + self.db.execute(""" + DELETE FROM autnum_overrides WHERE source = '%s'; + """ % name, + ) + else: + log.error("%s (%s) returned likely bogus file, ignored" % (name, url)) + continue + # Iterate through every line, filter comments and add remaining ASNs to # the override table in case they are valid... with self.db.transaction(): @@ -1518,8 +1538,8 @@ class CLI(object): # Filter invalid ASNs... if not self._check_parsed_asn(asn): - log.warning("Skipping bogus ASN found in Spamhaus DROP URL %s: %s" % \ - (url, asn)) + log.warning("Skipping bogus ASN found in %s (%s): %s" % \ + (name, url, asn)) continue # Conduct SQL statement... @@ -1531,7 +1551,7 @@ class CLI(object): ) VALUES (%s, %s, %s) ON CONFLICT (number) DO UPDATE SET is_drop = True""", "%s" % asn, - "Spamhaus ASN-DROP list", + name, True ) -- 2.35.3

3 9

Python location files moved from site-packages to dist-packages
by Peter Müller 02 Oct '22

02 Oct '22

Hello Michael, trying to update to libloc 0.9.15 in IPFire 2.x, I just noticed that all the Python files are now in /usr/lib/python3/dist-packages/location/, not /usr/lib/python3/site-packages/location/, where they used to be before. Thanks to that, "location" crashes with a ModuleNotFoundError. Adding > sys.path.append("/usr/lib/python3/dist-packages/") before the location import statements solves this, but is probably not a fix we strive for. :-) Am I overlooking something (such as changes to the LFS file)? Thanks, and best regards, Peter Müller

2 2

2024

2023

2022

2021

2020

2019

2018

2017

Location