Re: Location database updating every hour.

Re: Location database updating every hour.

[Peter Müller]

[-- Attachment #1: Type: text/plain, Size: 3571 bytes --]

Hello Rob,

thanks for reporting this.

This problem was caused by a glitch in the location-importer, a component
that is part of libloc and generates a new location database daily (in
the early morning, UTC-speaking). The root cause of this glitch is that
location-importer would not properly detect gzipped data if no matching
MIME header was sent by the database server.

RIPE, apparently, changed their database server infrastructure recently,
and now only returns application/octet-stream for gzipped data. While
generating the location database, this lead to RIPE data being omitted
completely - which then caused sanity checks to fail, since we expect some
data points served by RIPE to be present in a defined way.

Michael fixed this in [1], but thanks to the load situation, some chaos
the ongoing IPFire 3 "hackathon" caused, and a change we had to undergo
in our infrastructure backend, it took us until today to be able to generate
proper location databases again.

The latest one was just generated and published shortly before you wrote
your e-mail, on Fri, 02 Dec 2022 15:17:42 GMT [2]. You should now observe
any and all IPFire installations to fetch this database once, and then
behave like they usual do - one update attempt per day.

Apologies for the hassle, and thanks for reaching out,
Peter Müller

[1] https://git.ipfire.org/?p=location/libloc.git;a=commit;h=c39b3b92b0c557fba49a01ec63879189f3db2da1
[2] https://git.ipfire.org/?p=location/location-database.git;a=commit;h=d2d268a1fb754973a77a9f0c3be03054151edaaf


> Since the beginning of this month the update-location-database script runs 
> hourly and isn't waiting for the expected next 7 day interval.
> 
> I am not getting the expected "The database has been updated recently"  
> message in /var/log/messages since Dec  1 05:06:32
> 
> 
> On my development box IPFire 2.27 (x86_64) - Core-Update 170
> .
> .
> Nov 30 22:38:14 ipfire-dev2 The database has been updated recently
> Nov 30 23:33:19 ipfire-dev2 The database has been updated recently
> Dec  1 00:24:50 ipfire-dev2 The database has been updated recently
> Dec  1 01:30:08 ipfire-dev2 The database has been updated recently
> Dec  1 02:14:32 ipfire-dev2 The database has been updated recently
> Dec  1 03:52:51 ipfire-dev2 The database has been updated recently
> Dec  1 04:09:29 ipfire-dev2 The database has been updated recently
> Dec  1 05:06:32 ipfire-dev2 The database has been updated recently
> 
> and also on my production box IPFire 2.27 (x86_64) - Core Update 168
> .
> .
> Nov 30 22:45:58 griffith6 The database has been updated recently
> Nov 30 23:22:58 griffith6 The database has been updated recently
> Dec  1 00:53:45 griffith6 The database has been updated recently
> Dec  1 01:53:41 griffith6 The database has been updated recently
> Dec  1 02:45:16 griffith6 The database has been updated recently
> Dec  1 03:20:54 griffith6 The database has been updated recently
> Dec  1 04:34:41 griffith6 The database has been updated recently
> Dec  1 05:48:56 griffith6 The database has been updated recently
> Dec  1 06:28:10 griffith6 The database has been updated recently
> 
> There doesn't seem to affect performance other than the location ipsets 
> are being updated every hour.
> 
> location version is being set to the last update time:
> # location version
> Fri, 02 Dec 2022 15:17:42 GMT
> 
> Could this be caused by the leading blank space in the day column after 
> the month change from November to December ?
> 
> 
> 
> Rob