Re: What generates database.db from database.txt?

[Michael Tremer <michael.tremer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 3698 bytes --]

Hey Gisle,

> On 20 Oct 2020, at 20:14, Gisle Vanem <gisle.vanem(a)gmail.com> wrote:
> 
> Michael Tremer wrote:
> 
>> Would you be willing to tidy up your code (if required) and submit them on here so that we can officially support Windows?
> 
> I've put a preliminary version of my patched 'libloc'
> in my 'Wsock-trace' project:
>  https://github.com/gvanem/wsock-trace/tree/master/src/Geo-IP/IPFire/libloc/src

Thanks. I will have a look at it.

What is your application you are planning to use this in?

And probably more importantly, why does the database that we provide not work for you?

>>> Running all the 'test-*' programs, it worked seemingly
>>> well. Until I built the Python3 module and played with
>>> e.g. 'location dump' and 'verify'. Nothing worked :-(
>> What didn’t work specifically? Reading and writing the database?
> 
> 'test-database' says:
>  Vendor doesn't match: LOCDBXX☺ != Test Vendor

Oh that sounds interesting.

We have a string pool which starts somewhere towards the end of the file and it looks like you got the first couple of bytes there with the magic value in it.

So reading from the string pool does not work.

> and 'test-network':
>  Could not look up 2001:db8::

And I suppose reading other blocks of data doesn’t work either then.

> I'm not sure it due to my 'mmap()' function.

We are using simple functions to read and write to memory. No idea why those should behave differently on Windows, but I am sure we will get to the bottom of it.

> But I noted a case where 'loc_stringpool_get()' could
> return NULL, this is used in 'strdup()'. Not sure that's
> legal everywhere.

loc_stringpool_get() can indeed return NULL, but we should not use that result in strdup() anywhere.

> So I did this patch:
> 
> --- a/libloc/src/as.c 2020-10-19 17:35:01
> +++ b/libloc/src/as.c 2020-10-20 19:37:19
> @@ -90,7 +90,7 @@
> }
> 
> LOC_EXPORT int loc_as_set_name(struct loc_as* as, const char* name) {
> -       as->name = strdup(name);
> +       as->name = name ? strdup(name) : strdup("");
> 
>        return 0;
> }

Good catch, but there is more to it. When resetting the name, the original value wasn’t freed. I added that and pushed it to the main repository:

https://git.ipfire.org/?p=location/libloc.git;a=commitdiff;h=a7d3a7a0565a0e09d3442e5829a0f30f016993b9

> 
>> What you will need to import the raw data is location-importer which comes with libloc and a PostgreSQL server.
> 
> No! I hoped that 'libloc' would be similar to
> IP2Location and/or GeoipCSV that I already support
> in my project. Running PostgreSQL is out of the question
> for me or any users of 'Wsock-trace'.

You won’t need it to make the database. You should simply download the binary version of the library and that will be it.

You will need the development tools old if you want to modify the database and publish your own. But I do not see any reason why anyone should be doing that.

> 
>> Generating the whole database might take a couple of hours
>> depending on how beefy your machine is.
> 
> Lacking in Python skills, I cooked up this 'compile-database'
> script that takes 2 minutes:
>  https://github.com/gvanem/wsock-trace/blob/master/src/Geo-IP/IPFire/database/compile-database.py
> 
> But it remains to see if it works. Or if I'll rewrite it
> into a C-program.

For performance-reasons Python should be fine.

I am still not sure what you are trying to achieve here. If you are looking for the data, simply run “location update” and the script should download the database, extract it, verify it and you can use it.

Best,
-Michael

> -- 
> --gv