Re: Location database updating every hour.

[Rob Brewer <ipfire-devel@grantura.co.uk>]

[-- Attachment #1: Type: text/plain, Size: 3862 bytes --]

Hi Peter,

Pleased you were able to find the problem and thank you for your 
explanation.

I can confirm that the database is now updating as expected.

#location version
Fri, 02 Dec 2022 15:17:42 GMT


Rob


On Fri, 2 Dec 2022 18:06:36 +0100, Peter Müller wrote:

> Hello Rob,
> 
> thanks for reporting this.
> 
> This problem was caused by a glitch in the location-importer, a
> component that is part of libloc and generates a new location database
> daily (in the early morning, UTC-speaking). The root cause of this
> glitch is that location-importer would not properly detect gzipped data
> if no matching MIME header was sent by the database server.
> 
> RIPE, apparently, changed their database server infrastructure recently,
> and now only returns application/octet-stream for gzipped data. While
> generating the location database, this lead to RIPE data being omitted
> completely - which then caused sanity checks to fail, since we expect
> some data points served by RIPE to be present in a defined way.
> 
> Michael fixed this in [1], but thanks to the load situation, some chaos
> the ongoing IPFire 3 "hackathon" caused, and a change we had to undergo
> in our infrastructure backend, it took us until today to be able to
> generate proper location databases again.
> 
> The latest one was just generated and published shortly before you wrote
> your e-mail, on Fri, 02 Dec 2022 15:17:42 GMT [2]. You should now
> observe any and all IPFire installations to fetch this database once,
> and then behave like they usual do - one update attempt per day.
> 
> Apologies for the hassle, and thanks for reaching out,
> Peter Müller
> 
> [1]
> https://git.ipfire.org/?p=location/
libloc.git;a=commit;h=c39b3b92b0c557fba49a01ec63879189f3db2da1
> [2]
> https://git.ipfire.org/?p=location/location-
database.git;a=commit;h=d2d268a1fb754973a77a9f0c3be03054151edaaf
> 
> 
>> Since the beginning of this month the update-location-database script
>> runs hourly and isn't waiting for the expected next 7 day interval.
>> 
>> I am not getting the expected "The database has been updated recently"
>> message in /var/log/messages since Dec  1 05:06:32
>> 
>> 
>> On my development box IPFire 2.27 (x86_64) - Core-Update 170 .
>> .
>> Nov 30 22:38:14 ipfire-dev2 The database has been updated recently Nov
>> 30 23:33:19 ipfire-dev2 The database has been updated recently Dec  1
>> 00:24:50 ipfire-dev2 The database has been updated recently Dec  1
>> 01:30:08 ipfire-dev2 The database has been updated recently Dec  1
>> 02:14:32 ipfire-dev2 The database has been updated recently Dec  1
>> 03:52:51 ipfire-dev2 The database has been updated recently Dec  1
>> 04:09:29 ipfire-dev2 The database has been updated recently Dec  1
>> 05:06:32 ipfire-dev2 The database has been updated recently
>> 
>> and also on my production box IPFire 2.27 (x86_64) - Core Update 168 .
>> .
>> Nov 30 22:45:58 griffith6 The database has been updated recently Nov 30
>> 23:22:58 griffith6 The database has been updated recently Dec  1
>> 00:53:45 griffith6 The database has been updated recently Dec  1
>> 01:53:41 griffith6 The database has been updated recently Dec  1
>> 02:45:16 griffith6 The database has been updated recently Dec  1
>> 03:20:54 griffith6 The database has been updated recently Dec  1
>> 04:34:41 griffith6 The database has been updated recently Dec  1
>> 05:48:56 griffith6 The database has been updated recently Dec  1
>> 06:28:10 griffith6 The database has been updated recently
>> 
>> There doesn't seem to affect performance other than the location ipsets
>> are being updated every hour.
>> 
>> location version is being set to the last update time:
>> # location version Fri, 02 Dec 2022 15:17:42 GMT
>> 
>> Could this be caused by the leading blank space in the day column after
>> the month change from November to December ?
>> 
>> 
>> 
>> Rob