[ANNOUNCE] libloc 0.9.20

[Michael Tremer <michael.tremer@ipfire.org>]

Hello everyone,

I have just tagged libloc 0.9.20. It is the first release since 0.9.19
in June and contains about 250 commits from five authors. I will not
list everything here, since most commit messages explain the problem and
the reasoning in some detail. They are the best place to read up on a
particular change. All changes since the last release:

  https://git.ipfire.org/?p=location/libloc.git;a=shortlog;h=refs/tags/0.9.20

The source tarball is at https://source.ipfire.org/releases/libloc/ and
the packages are on https://repo.ipfire.org/. Commits and bugs are
linked below the text.

Faster lookups

Looking up an address is a lot faster, mostly because of how the tree is
walked [1] and, for new databases, how it is written [2]. I measured
0.9.19 and 0.9.20 on the same machine, with the same database and the
same addresses (1,000,000 random IPv4 addresses and 1,000,000 IPv6
addresses from allocated networks), in one thread. Every lookup reads
the country and the ASN. These are the medians of seven runs, with the
database that is published today:

              0.9.19        0.9.20
  IPv4     1.8-2.1 us    0.31-0.32 us     about 6 times faster
  IPv6     1.2-1.3 us    0.60 us          about 2 times faster

With an older database from 2024, which has the old layout, IPv6
lookups take 1.7 us in 0.9.19 and 0.9-1.0 us in 0.9.20, and IPv4
lookups are as fast as above. The machine is a virtual machine with
Xeon E5-2690 v4 CPUs, which also ran other work, so please read these as
an indication and not as exact numbers. The benchmarks that compare
libloc with libmaxminddb are in the repository [3].

Other changes in short

* Databases can be reloaded. A database that has been opened with the
  new loc_database_open() finds out by itself that its file has been
  replaced, and loc_database_reload() does it on request. Python has
  Database.reload() now. A database can also be used from more than one
  thread. [4]

* The bindings have been cleaned up. AS numbers are unsigned in Python
  and Lua, Lua returns whole numbers as integers, Python releases the
  GIL while it writes and verifies databases, and the Perl bindings are
  packaged for Debian. The Python and Lua modules now have manual pages,
  and so does the whole C API. [5]

* "location update" is better at telling you what is wrong. It exits
  with zero if the database is already up to date, it says which file is
  missing if it cannot read the public key, and it no longer leaves a
  half-downloaded database behind. Proxies are taken from the
  environment, which the man page explains, including how to set them
  for the systemd unit. [6]

* The importer knows more. It imports the names of all ASes of the RIPE
  NCC, finds Geofeeds in remarks however they are spelled and in the
  route objects of ARIN and the other RIRs, and imports the IP ranges of
  Oracle Cloud and Cloudflare. It also stops marking documentation and
  other special-purpose ranges, like those from RFC 5737 and RFC 3849,
  as belonging to a country, and the SP-GiST indexes, which did not find
  all networks, have been dropped. [7]

* More platforms. There are packages for Arch Linux and RPM-based
  distributions now (the README has the repositories), the Debian
  packages cover more releases and architectures, and libloc builds and
  passes its tests on macOS again. [8]

When you upgrade

* The C functions of the library return zero or a negative error number
  everywhere now, and do not set errno any more. loc_set_log_fn() is
  deprecated. The new functions have been added to the library without
  changing its soname (libloc.so.1). [9]

* On Debian, the "location" package from 0.9.19-1 shipped an old copy of
  the database in /var/lib/location, which replaced the current one when
  the package was upgraded. The package does not do that any more. The
  first upgrade to this version removes that file again, so please run
  "location update" afterwards unless you use libloc-database. [10]

* "location update" exits with zero if there is nothing to do, so that
  the systemd unit and the cron job no longer have to translate the
  return code. [11]

Thanks

Many thanks to everyone who contributed to this release:

* Valters Jansons, for fifteen commits, which brought the Debian
  packaging up to date (Lintian, d/watch, copyright and the manual
  pages), the manual page for location-importer, and a number of fixes
  to the build.

* Spencer Norton, for getting libloc to build and pass its tests on
  macOS, for the importer filters that keep documentation ranges out of
  the database, and for adding the IP ranges of Oracle Cloud. Spencer
  also went through a number of old bug reports and told us which ones
  had been solved.

* Adolf Belka, for the patch for #14047.

* Bone NI, for the translation to Lao.

* Everyone who reported bugs, and everyone who translates libloc on
  Weblate.

As always, please report any problems at https://bugzilla.ipfire.org/ or
on this list.

-Michael


[1] https://git.ipfire.org/?p=location/libloc.git;a=commit;h=c85425827c23cde3327ae1f6df1897c8b6f1ed35
    https://git.ipfire.org/?p=location/libloc.git;a=commit;h=2db19f0a7db1384df2b86b29b45c69e90599474a
    https://git.ipfire.org/?p=location/libloc.git;a=commit;h=724347b86bad8f2200af95143774cad133e7c223
[2] https://git.ipfire.org/?p=location/libloc.git;a=commit;h=dd82fdb08a7dd65aba6b12f27137e3e11f61a645
[3] https://git.ipfire.org/?p=location/libloc.git;a=blob;f=benchmarks/README.md;hb=refs/tags/0.9.20
[4] https://git.ipfire.org/?p=location/libloc.git;a=commit;h=7bbc2007a6686120a40199fb8294a59187be85e3
    https://git.ipfire.org/?p=location/libloc.git;a=commit;h=4771f3351d3f5d4fa21ff3df535b33487d78770c
    https://git.ipfire.org/?p=location/libloc.git;a=commit;h=e0dcd36c5297ffd2d21c185f6df3cd96ccaaba0f
    https://git.ipfire.org/?p=location/libloc.git;a=commit;h=9f9518b683048da181ac06f651bb1c6ce60afb5e
[5] https://git.ipfire.org/?p=location/libloc.git;a=commit;h=9625935549a2be43ea6d6e6b29df2e9631ded4be
    https://git.ipfire.org/?p=location/libloc.git;a=commit;h=bf425b3473dd09e6cd073d353748c7cd52a3de3d
    https://git.ipfire.org/?p=location/libloc.git;a=commit;h=29b2560af5fbc2615fd78d242dc1bda2dacc2405
    https://git.ipfire.org/?p=location/libloc.git;a=commit;h=4141aa0450983763c002bd0607e5fa0d9d7f8d40
[6] https://git.ipfire.org/?p=location/libloc.git;a=commit;h=851a61a780e6a58dcb1977210317128edfe23bc3
    https://git.ipfire.org/?p=location/libloc.git;a=commit;h=c1e88295db3d52a2bd3de19848cfab7f551ceff7
    https://git.ipfire.org/?p=location/libloc.git;a=commit;h=979a3e7227680912d48dcb3a6bfb09ea565abcff
    https://bugzilla.ipfire.org/show_bug.cgi?id=13655
    https://bugzilla.ipfire.org/show_bug.cgi?id=14574
[7] https://git.ipfire.org/?p=location/libloc.git;a=commit;h=d3434740ac93e38bbb5a3f9e76382362a5aec929
    https://git.ipfire.org/?p=location/libloc.git;a=commit;h=84defc8be815353690f67a1e1503a8c3c7905441
    https://git.ipfire.org/?p=location/libloc.git;a=commit;h=01e15653595c9cbe1921c1f1bfc674119db1f92a
    https://git.ipfire.org/?p=location/libloc.git;a=commit;h=8a28c5d9cb957476369df89db3245cba034c3533
    https://git.ipfire.org/?p=location/libloc.git;a=commit;h=03267dba21c9cd1d310ce40c875a53fa21538329
    https://git.ipfire.org/?p=location/libloc.git;a=commit;h=6274fb37b8d63c9820720bb613418d8bc8a265ee
    https://git.ipfire.org/?p=location/libloc.git;a=commit;h=d2c25bc70dfc0b29adac4ddab160e82ecd4c3196
    https://bugzilla.ipfire.org/show_bug.cgi?id=13934
    https://bugzilla.ipfire.org/show_bug.cgi?id=14500
    https://bugzilla.ipfire.org/show_bug.cgi?id=14501
    https://bugzilla.ipfire.org/show_bug.cgi?id=14502
    https://bugzilla.ipfire.org/show_bug.cgi?id=14504
    https://bugzilla.ipfire.org/show_bug.cgi?id=14505
    https://bugzilla.ipfire.org/show_bug.cgi?id=14506
[8] https://git.ipfire.org/?p=location/libloc.git;a=commit;h=3b92cb0f03c396aa6b3ccdd6d66767e4d9929111
    https://git.ipfire.org/?p=location/libloc.git;a=commit;h=1f8925be1b1b637635e69eab5acfaf8567b6c876
    https://git.ipfire.org/?p=location/libloc.git;a=commit;h=6e992d0e10fecbc6288b721832cbed6f81305498
    https://bugzilla.ipfire.org/show_bug.cgi?id=12400
[9] https://git.ipfire.org/?p=location/libloc.git;a=commit;h=dc4dc084d23893cb6a54c73de11d7d609340fcea
    https://git.ipfire.org/?p=location/libloc.git;a=commit;h=1b050ab8b983f0448d9cf8b888bb164c56fa69fb
[10] https://git.ipfire.org/?p=location/libloc.git;a=commit;h=d97e70ed344bfb17dcc3e5f88fea941c820791bd
     https://bugzilla.ipfire.org/show_bug.cgi?id=14573
[11] https://git.ipfire.org/?p=location/libloc.git;a=commit;h=89f84289ba22f4ba8102bf6c0f60886092bdd6e1
     https://bugzilla.ipfire.org/show_bug.cgi?id=14047