Since the beginning of this month the update-location-database script runs hourly and isn't waiting for the expected next 7 day interval.
I am not getting the expected "The database has been updated recently" message in /var/log/messages since Dec 1 05:06:32
On my development box IPFire 2.27 (x86_64) - Core-Update 170 . . Nov 30 22:38:14 ipfire-dev2 The database has been updated recently Nov 30 23:33:19 ipfire-dev2 The database has been updated recently Dec 1 00:24:50 ipfire-dev2 The database has been updated recently Dec 1 01:30:08 ipfire-dev2 The database has been updated recently Dec 1 02:14:32 ipfire-dev2 The database has been updated recently Dec 1 03:52:51 ipfire-dev2 The database has been updated recently Dec 1 04:09:29 ipfire-dev2 The database has been updated recently Dec 1 05:06:32 ipfire-dev2 The database has been updated recently
and also on my production box IPFire 2.27 (x86_64) - Core Update 168 . . Nov 30 22:45:58 griffith6 The database has been updated recently Nov 30 23:22:58 griffith6 The database has been updated recently Dec 1 00:53:45 griffith6 The database has been updated recently Dec 1 01:53:41 griffith6 The database has been updated recently Dec 1 02:45:16 griffith6 The database has been updated recently Dec 1 03:20:54 griffith6 The database has been updated recently Dec 1 04:34:41 griffith6 The database has been updated recently Dec 1 05:48:56 griffith6 The database has been updated recently Dec 1 06:28:10 griffith6 The database has been updated recently
There doesn't seem to affect performance other than the location ipsets are being updated every hour.
location version is being set to the last update time: # location version Fri, 02 Dec 2022 15:17:42 GMT
Could this be caused by the leading blank space in the day column after the month change from November to December ?
Rob
Hello Rob,
thanks for reporting this.
This problem was caused by a glitch in the location-importer, a component that is part of libloc and generates a new location database daily (in the early morning, UTC-speaking). The root cause of this glitch is that location-importer would not properly detect gzipped data if no matching MIME header was sent by the database server.
RIPE, apparently, changed their database server infrastructure recently, and now only returns application/octet-stream for gzipped data. While generating the location database, this lead to RIPE data being omitted completely - which then caused sanity checks to fail, since we expect some data points served by RIPE to be present in a defined way.
Michael fixed this in [1], but thanks to the load situation, some chaos the ongoing IPFire 3 "hackathon" caused, and a change we had to undergo in our infrastructure backend, it took us until today to be able to generate proper location databases again.
The latest one was just generated and published shortly before you wrote your e-mail, on Fri, 02 Dec 2022 15:17:42 GMT [2]. You should now observe any and all IPFire installations to fetch this database once, and then behave like they usual do - one update attempt per day.
Apologies for the hassle, and thanks for reaching out, Peter Müller
[1] https://git.ipfire.org/?p=location/libloc.git;a=commit;h=c39b3b92b0c557fba49... [2] https://git.ipfire.org/?p=location/location-database.git;a=commit;h=d2d268a1...
Since the beginning of this month the update-location-database script runs hourly and isn't waiting for the expected next 7 day interval.
I am not getting the expected "The database has been updated recently" message in /var/log/messages since Dec 1 05:06:32
On my development box IPFire 2.27 (x86_64) - Core-Update 170 . . Nov 30 22:38:14 ipfire-dev2 The database has been updated recently Nov 30 23:33:19 ipfire-dev2 The database has been updated recently Dec 1 00:24:50 ipfire-dev2 The database has been updated recently Dec 1 01:30:08 ipfire-dev2 The database has been updated recently Dec 1 02:14:32 ipfire-dev2 The database has been updated recently Dec 1 03:52:51 ipfire-dev2 The database has been updated recently Dec 1 04:09:29 ipfire-dev2 The database has been updated recently Dec 1 05:06:32 ipfire-dev2 The database has been updated recently
and also on my production box IPFire 2.27 (x86_64) - Core Update 168 . . Nov 30 22:45:58 griffith6 The database has been updated recently Nov 30 23:22:58 griffith6 The database has been updated recently Dec 1 00:53:45 griffith6 The database has been updated recently Dec 1 01:53:41 griffith6 The database has been updated recently Dec 1 02:45:16 griffith6 The database has been updated recently Dec 1 03:20:54 griffith6 The database has been updated recently Dec 1 04:34:41 griffith6 The database has been updated recently Dec 1 05:48:56 griffith6 The database has been updated recently Dec 1 06:28:10 griffith6 The database has been updated recently
There doesn't seem to affect performance other than the location ipsets are being updated every hour.
location version is being set to the last update time: # location version Fri, 02 Dec 2022 15:17:42 GMT
Could this be caused by the leading blank space in the day column after the month change from November to December ?
Rob
Hi Peter,
Pleased you were able to find the problem and thank you for your explanation.
I can confirm that the database is now updating as expected.
#location version Fri, 02 Dec 2022 15:17:42 GMT
Rob
On Fri, 2 Dec 2022 18:06:36 +0100, Peter Müller wrote:
Hello Rob,
thanks for reporting this.
This problem was caused by a glitch in the location-importer, a component that is part of libloc and generates a new location database daily (in the early morning, UTC-speaking). The root cause of this glitch is that location-importer would not properly detect gzipped data if no matching MIME header was sent by the database server.
RIPE, apparently, changed their database server infrastructure recently, and now only returns application/octet-stream for gzipped data. While generating the location database, this lead to RIPE data being omitted completely - which then caused sanity checks to fail, since we expect some data points served by RIPE to be present in a defined way.
Michael fixed this in [1], but thanks to the load situation, some chaos the ongoing IPFire 3 "hackathon" caused, and a change we had to undergo in our infrastructure backend, it took us until today to be able to generate proper location databases again.
The latest one was just generated and published shortly before you wrote your e-mail, on Fri, 02 Dec 2022 15:17:42 GMT [2]. You should now observe any and all IPFire installations to fetch this database once, and then behave like they usual do - one update attempt per day.
Apologies for the hassle, and thanks for reaching out, Peter Müller
libloc.git;a=commit;h=c39b3b92b0c557fba49a01ec63879189f3db2da1
database.git;a=commit;h=d2d268a1fb754973a77a9f0c3be03054151edaaf
Since the beginning of this month the update-location-database script runs hourly and isn't waiting for the expected next 7 day interval.
I am not getting the expected "The database has been updated recently" message in /var/log/messages since Dec 1 05:06:32
On my development box IPFire 2.27 (x86_64) - Core-Update 170 . . Nov 30 22:38:14 ipfire-dev2 The database has been updated recently Nov 30 23:33:19 ipfire-dev2 The database has been updated recently Dec 1 00:24:50 ipfire-dev2 The database has been updated recently Dec 1 01:30:08 ipfire-dev2 The database has been updated recently Dec 1 02:14:32 ipfire-dev2 The database has been updated recently Dec 1 03:52:51 ipfire-dev2 The database has been updated recently Dec 1 04:09:29 ipfire-dev2 The database has been updated recently Dec 1 05:06:32 ipfire-dev2 The database has been updated recently
and also on my production box IPFire 2.27 (x86_64) - Core Update 168 . . Nov 30 22:45:58 griffith6 The database has been updated recently Nov 30 23:22:58 griffith6 The database has been updated recently Dec 1 00:53:45 griffith6 The database has been updated recently Dec 1 01:53:41 griffith6 The database has been updated recently Dec 1 02:45:16 griffith6 The database has been updated recently Dec 1 03:20:54 griffith6 The database has been updated recently Dec 1 04:34:41 griffith6 The database has been updated recently Dec 1 05:48:56 griffith6 The database has been updated recently Dec 1 06:28:10 griffith6 The database has been updated recently
There doesn't seem to affect performance other than the location ipsets are being updated every hour.
location version is being set to the last update time: # location version Fri, 02 Dec 2022 15:17:42 GMT
Could this be caused by the leading blank space in the day column after the month change from November to December ?
Rob