What generates database.db from database.txt?

What generates database.db from database.txt?

[Gisle Vanem]

[-- Attachment #1: Type: text/plain, Size: 777 bytes --]

Hello list, my ports post.

I'm on Windows and have little interest in installing
IPFire or do any Linux related. But the 'libloc'
library caught my interest. So I did a port to
MSVC, MinGW and clang-cl (x86 only) in a hurry.

Running all the 'test-*' programs, it worked seemingly
well. Until I built the Python3 module and played with
e.g. 'location dump' and 'verify'. Nothing worked :-(
So I reckon the .db file is hosed somehow. AFAICS, it's
not an issue with structure packing or 'fopen(path, "r")'.

Anyway, I want to regenerate 'database.db' from 'database.txt'
'countries.txt' and the 'override' files myself. But how?

According to the git log at:
   git://git.ipfire.org/location/location-database.git

there once was a 'compile-database' script. Not now.

-- 
--gv

Re: What generates database.db from database.txt?

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 2653 bytes --]

Hello Gisle,

Thank you for your email.

> On 19 Oct 2020, at 16:53, Gisle Vanem <gisle.vanem(a)gmail.com> wrote:
> 
> Hello list, my ports post.
> 
> I'm on Windows and have little interest in installing
> IPFire or do any Linux related. But the 'libloc'
> library caught my interest. So I did a port to
> MSVC, MinGW and clang-cl (x86 only) in a hurry.

That sounds great!

We are looking for someone with some experience with Windows to port libloc. It should not be super complicated, but since I am not the expert I am not too sure.

Would you be willing to tidy up your code (if required) and submit them on here so that we can officially support Windows?

> Running all the 'test-*' programs, it worked seemingly
> well. Until I built the Python3 module and played with
> e.g. 'location dump' and 'verify'. Nothing worked :-(

What didn’t work specifically? Reading and writing the database?

> So I reckon the .db file is hosed somehow. AFAICS, it's
> not an issue with structure packing or 'fopen(path, "r")'.

Hmm… it would be interesting to debug that.

> Anyway, I want to regenerate 'database.db' from 'database.txt'
> 'countries.txt' and the 'override' files myself. But how?

We do not generate the database from database.txt. database.txt is being generated from the database. The text dump is just there so that we have a (sort of) human-readable version which we can diff against the previous one to manually verify any changes.

What you will need to import the raw data is location-importer which comes with libloc and a PostgreSQL server.

location-importer —help will show you how you pass the database credentials.

On our servers we regularly run “location-import update-overrides file-a.txt file-b.txt”, then "location-importer update-whois”, then we run “location-importer update-announcements” which connects to a locally running instance of Bird with a full BGP feed to import those.

Finally we run “location-importer write … database.db” (fill in the gaps with your vendor name, description, key, etc.) and finally you can run “location -d database.db dump” to export it into the text format.

Generating the whole database might take a couple of hours depending on how beefy your machine is.

Let me know if this works for you.

> According to the git log at:
>  git://git.ipfire.org/location/location-database.git
> 
> there once was a 'compile-database' script. Not now.

Yes, this has all been moved into “location-importer” so that we separate code and data in two different Git repositories.

Best,
-Michael

> 
> -- 
> --gv

Re: What generates database.db from database.txt?

[Gisle Vanem]

[-- Attachment #1: Type: text/plain, Size: 1921 bytes --]

Michael Tremer wrote:

> Would you be willing to tidy up your code (if required) and submit them on here so that we can officially support Windows?

I've put a preliminary version of my patched 'libloc'
in my 'Wsock-trace' project:
   https://github.com/gvanem/wsock-trace/tree/master/src/Geo-IP/IPFire/libloc/src

>> Running all the 'test-*' programs, it worked seemingly
>> well. Until I built the Python3 module and played with
>> e.g. 'location dump' and 'verify'. Nothing worked :-(
> 
> What didn’t work specifically? Reading and writing the database?

'test-database' says:
   Vendor doesn't match: LOCDBXX☺ != Test Vendor
and 'test-network':
   Could not look up 2001:db8::

I'm not sure it due to my 'mmap()' function.
But I noted a case where 'loc_stringpool_get()' could
return NULL, this is used in 'strdup()'. Not sure that's
legal everywhere. So I did this patch:

--- a/libloc/src/as.c 2020-10-19 17:35:01
+++ b/libloc/src/as.c 2020-10-20 19:37:19
@@ -90,7 +90,7 @@
  }

  LOC_EXPORT int loc_as_set_name(struct loc_as* as, const char* name) {
-       as->name = strdup(name);
+       as->name = name ? strdup(name) : strdup("");

         return 0;
  }

> What you will need to import the raw data is location-importer which comes 
> with libloc and a PostgreSQL server.

No! I hoped that 'libloc' would be similar to
IP2Location and/or GeoipCSV that I already support
in my project. Running PostgreSQL is out of the question
for me or any users of 'Wsock-trace'.

> Generating the whole database might take a couple of hours
> depending on how beefy your machine is.

Lacking in Python skills, I cooked up this 'compile-database'
script that takes 2 minutes:
   https://github.com/gvanem/wsock-trace/blob/master/src/Geo-IP/IPFire/database/compile-database.py

But it remains to see if it works. Or if I'll rewrite it
into a C-program.

-- 
--gv

Re: What generates database.db from database.txt?

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 3698 bytes --]

Hey Gisle,

> On 20 Oct 2020, at 20:14, Gisle Vanem <gisle.vanem(a)gmail.com> wrote:
> 
> Michael Tremer wrote:
> 
>> Would you be willing to tidy up your code (if required) and submit them on here so that we can officially support Windows?
> 
> I've put a preliminary version of my patched 'libloc'
> in my 'Wsock-trace' project:
>  https://github.com/gvanem/wsock-trace/tree/master/src/Geo-IP/IPFire/libloc/src

Thanks. I will have a look at it.

What is your application you are planning to use this in?

And probably more importantly, why does the database that we provide not work for you?

>>> Running all the 'test-*' programs, it worked seemingly
>>> well. Until I built the Python3 module and played with
>>> e.g. 'location dump' and 'verify'. Nothing worked :-(
>> What didn’t work specifically? Reading and writing the database?
> 
> 'test-database' says:
>  Vendor doesn't match: LOCDBXX☺ != Test Vendor

Oh that sounds interesting.

We have a string pool which starts somewhere towards the end of the file and it looks like you got the first couple of bytes there with the magic value in it.

So reading from the string pool does not work.

> and 'test-network':
>  Could not look up 2001:db8::

And I suppose reading other blocks of data doesn’t work either then.

> I'm not sure it due to my 'mmap()' function.

We are using simple functions to read and write to memory. No idea why those should behave differently on Windows, but I am sure we will get to the bottom of it.

> But I noted a case where 'loc_stringpool_get()' could
> return NULL, this is used in 'strdup()'. Not sure that's
> legal everywhere.

loc_stringpool_get() can indeed return NULL, but we should not use that result in strdup() anywhere.

> So I did this patch:
> 
> --- a/libloc/src/as.c 2020-10-19 17:35:01
> +++ b/libloc/src/as.c 2020-10-20 19:37:19
> @@ -90,7 +90,7 @@
> }
> 
> LOC_EXPORT int loc_as_set_name(struct loc_as* as, const char* name) {
> -       as->name = strdup(name);
> +       as->name = name ? strdup(name) : strdup("");
> 
>        return 0;
> }

Good catch, but there is more to it. When resetting the name, the original value wasn’t freed. I added that and pushed it to the main repository:

https://git.ipfire.org/?p=location/libloc.git;a=commitdiff;h=a7d3a7a0565a0e09d3442e5829a0f30f016993b9

> 
>> What you will need to import the raw data is location-importer which comes with libloc and a PostgreSQL server.
> 
> No! I hoped that 'libloc' would be similar to
> IP2Location and/or GeoipCSV that I already support
> in my project. Running PostgreSQL is out of the question
> for me or any users of 'Wsock-trace'.

You won’t need it to make the database. You should simply download the binary version of the library and that will be it.

You will need the development tools old if you want to modify the database and publish your own. But I do not see any reason why anyone should be doing that.

> 
>> Generating the whole database might take a couple of hours
>> depending on how beefy your machine is.
> 
> Lacking in Python skills, I cooked up this 'compile-database'
> script that takes 2 minutes:
>  https://github.com/gvanem/wsock-trace/blob/master/src/Geo-IP/IPFire/database/compile-database.py
> 
> But it remains to see if it works. Or if I'll rewrite it
> into a C-program.

For performance-reasons Python should be fine.

I am still not sure what you are trying to achieve here. If you are looking for the data, simply run “location update” and the script should download the database, extract it, verify it and you can use it.

Best,
-Michael

> -- 
> --gv

Re: What generates database.db from database.txt?

[Gisle Vanem]

[-- Attachment #1: Type: text/plain, Size: 2750 bytes --]

> What is your application you are planning to use this in?

Wsock-trace; a Winsock-trace library like I wrote.

> And probably more importantly, why does the database that we provide not work for you?

That's one of the things I'm trying to figure out.

>> LOC_EXPORT int loc_as_set_name(struct loc_as* as, const char* name) {
>> -       as->name = strdup(name);
>> +       as->name = name ? strdup(name) : strdup("");
>>
>>         return 0;
>> }
> 
> Good catch, but there is more to it. When resetting the name, the original
  value wasn’t freed. I added that and pushed it to the main repository:

With that patch, test-as crashes. I did this:
--- a/src/as.c 2020-10-20 23:23:34
+++ b/src/as.c 2020-10-20 23:55:51
@@ -142,7 +142,7 @@

  int loc_as_match_string(struct loc_as* as, const char* string) {
         // Match all AS when no search string is set
-       if (!string)
+       if (!string || !as->name)
                 return 1;

which no longer crashes. But when building in CRT debug-mode,
I note many mem-leaks in e.g. 'test-as.c' and 'test-country.c'.

Some of it is due to the error I get:
   Could not find country: YY

and you do 'exit(EXIT_FAILURE);' w/o cleaning up.
Such a behaviour could probably accumulate leaks
over time. But I do not know IPFire.

But I plugged other leaks using:
--- a/src/writer.c 2020-10-19 17:35:01
+++ b/src/writer.c 2020-10-20 23:15:20
@@ -155,6 +155,12 @@
         if (writer->networks)
                 loc_network_tree_unref(writer->networks);

+       if (writer->as)
+               free(writer->as);
+
+       if (writer->countries)
+               free(writer->countries);
+
         // Unref the string pool
         loc_stringpool_unref(writer->pool);

Never done AFAICS. The leak for 'as.c', I'm not sure
how to plug. You ought to check with valgrind etc.

> You won’t need it to make the database.
> You should simply download the binary version of the library and that will be it.

Where? Nothing under https://location.ipfire.org/databases/1/
But a 'adig -t txt _v1._db.location.ipfire.org' says:
   TXT Fri, 16 Oct 2020 07:54:08 GMT

I fail to find it. But that's another issue.

> I am still not sure what you are trying to achieve here. If you are looking for the data, simply run “location update” and the script should download the database, extract it, verify it and you can use it.

Here:
   py -3 location.py update
   https://location.ipfire.org/databases/ reported: HTTP Error 404: Not Found
   Could not download a new database

And have you ever used OpenSSL on Windows? Here:
   py -3 location.py verify
   OPENSSL_Uplink(640BC6B0,08): no OPENSSL_Applink

It's pure hell.

-- 
--gv

Re: What generates database.db from database.txt?

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 4468 bytes --]

Hi,

> On 20 Oct 2020, at 23:46, Gisle Vanem <gisle.vanem(a)gmail.com> wrote:
> 
>> What is your application you are planning to use this in?
> 
> Wsock-trace; a Winsock-trace library like I wrote.
> 
>> And probably more importantly, why does the database that we provide not work for you?
> 
> That's one of the things I'm trying to figure out.
> 
>>> LOC_EXPORT int loc_as_set_name(struct loc_as* as, const char* name) {
>>> -       as->name = strdup(name);
>>> +       as->name = name ? strdup(name) : strdup("");
>>> 
>>>        return 0;
>>> }
>> Good catch, but there is more to it. When resetting the name, the original
> value wasn’t freed. I added that and pushed it to the main repository:
> 
> With that patch, test-as crashes. I did this:
> --- a/src/as.c 2020-10-20 23:23:34
> +++ b/src/as.c 2020-10-20 23:55:51
> @@ -142,7 +142,7 @@
> 
> int loc_as_match_string(struct loc_as* as, const char* string) {
>        // Match all AS when no search string is set
> -       if (!string)
> +       if (!string || !as->name)
>                return 1;
> 
> which no longer crashes. But when building in CRT debug-mode,
> I note many mem-leaks in e.g. 'test-as.c' and 'test-country.c'.

The testsuite executed on my Debian box I used for testing.

But logically your patch is correct. I will import it. Thank you.

> Some of it is due to the error I get:
>  Could not find country: YY

This probably requires some more debugging and log output.

You can set LOC_LOG=7 as an environment variable and then run test/test-country.

You should see some logging output which will be helpful to get to the bottom of this.

> and you do 'exit(EXIT_FAILURE);' w/o cleaning up.
> Such a behaviour could probably accumulate leaks
> over time. But I do not know IPFire.

Yes it does. But it is the testsuite. We do clean up to test the cleanup routines, but if something goes wrong we terminate the whole process which will free all memory. This code isn’t shipped to the end user systems.

> But I plugged other leaks using:
> --- a/src/writer.c 2020-10-19 17:35:01
> +++ b/src/writer.c 2020-10-20 23:15:20
> @@ -155,6 +155,12 @@
>        if (writer->networks)
>                loc_network_tree_unref(writer->networks);
> 
> +       if (writer->as)
> +               free(writer->as);
> +
> +       if (writer->countries)
> +               free(writer->countries);
> +
>        // Unref the string pool
>        loc_stringpool_unref(writer->pool);
> 
> Never done AFAICS. The leak for 'as.c', I'm not sure
> how to plug. You ought to check with valgrind etc.

We employed someone to do this, but he said he didn’t find anything like this lol.

Just freeing the arrays is not enough. The allocated objects need to be freed, too.

I will have a look at it.

>> You won’t need it to make the database.
>> You should simply download the binary version of the library and that will be it.
> 
> Where? Nothing under https://location.ipfire.org/databases/1/
> But a 'adig -t txt _v1._db.location.ipfire.org' says:
>  TXT Fri, 16 Oct 2020 07:54:08 GMT
> 
> I fail to find it. But that's another issue.

I pulled the database last week because we had some issue with some data in it that triggered some bugs.

You can download it from the archive still until the updater works again:

  https://location.ipfire.org/databases/1/archive/location-2020-10-16.db.xz

> 
>> I am still not sure what you are trying to achieve here. If you are looking for the data, simply run “location update” and the script should download the database, extract it, verify it and you can use it.
> 
> Here:
>  py -3 location.py update
>  https://location.ipfire.org/databases/ reported: HTTP Error 404: Not Found
>  Could not download a new database
> 
> And have you ever used OpenSSL on Windows? Here:
>  py -3 location.py verify
>  OPENSSL_Uplink(640BC6B0,08): no OPENSSL_Applink
> 
> It's pure hell.

Yes, I assumed so. There are some tools that use but they might potentially just bundle it with their own software which I want to avoid.

It probably is possible to #ifdef the Windows crypto library here and check the signature that way. But since I have never worked with it, I do not know how to do that. Mac OS X has the same issue.

Have you ever worked with it?

However, OpenSSL is a very good choice in the *nix/BSD world.

Best,
-Michael

> 
> -- 
> --gv

Re: What generates database.db from database.txt?

[Gisle Vanem]

[-- Attachment #1: Type: text/plain, Size: 3189 bytes --]

Michael Tremer wrote:

> This probably requires some more debugging and log output.
> 
> You can set LOC_LOG=7 as an environment variable and then run test/test-country.

With this, test-country only reports an extra:
   libloc: loc_new: ctx 05021298 created

But with '-DENABLE_DEBUG' there's way too much trace
to be useful.

> I pulled the database last week because we had some issue with some data in it that triggered some bugs.
> 
> You can download it from the archive still until the updater works again:
> 
>    https://location.ipfire.org/databases/1/archive/location-2020-10-16.db.xz


> Yes, I assumed so. There are some tools that use but they might potentially just bundle it with their own software which I want to avoid.
> 
> It probably is possible to #ifdef the Windows crypto library here and check the signature that way. But since I have never worked with it, I do not know how to do that. Mac OS X has the same issue.

An issue with OpenSSL and 'loc_database_verify()'?

> Have you ever worked with it?

Yes, but 'test-signature' works fine. It's just when calling
'EVP_VerifyXX()' functions from a .dll (like a Python module), it
fails with 'no OPENSSL_AppLink".
Ref:
 
https://stackoverflow.com/questions/38933943/openssl-code-works-natively-but-as-dll-causes-openssl-uplink-no-openssl-applink

With your URL above,
  set loc_log=7 & py -3 location.py --database location-2020-10-16.db verify:

libloc: loc_new: ctx 0FDACEE8 created
libloc: loc_database_read_as_section_v1: Read 48148 ASes from the database
libloc: loc_database_read_network_nodes_section_v1: Read 2628939 network nodes from the database
libloc: loc_database_read_networks_section_v1: Read 1318972 networks from the database
libloc: loc_database_read_countries_section_v1: Read 253 countries from the database
libloc: loc_database_read: Opened database in 16.0000ms
libloc: loc_database_read_as_section_v1: Read 48148 ASes from the database
libloc: loc_database_read_network_nodes_section_v1: Read 2628939 network nodes from the database
libloc: loc_database_read_networks_section_v1: Read 1318972 networks from the database
libloc: loc_database_read_countries_section_v1: Read 253 countries from the database
libloc: loc_database_read: Opened database in 27.0000ms
OPENSSL_Uplink(640BC6B0,08): no OPENSSL_Applink

BUT, with 'location-2020-05-15.db', there is this error:

libloc: loc_new: ctx 0FA35190 created
libloc: loc_database_read_header: Incompatible database version: 0
Traceback (most recent call last):
   File "F:\gv\VC_project\ws_trace\src\Geo-IP\IPFire\libloc\src\py3_install\location.py", line 618, in <module>
     main()
   File "F:\gv\VC_project\ws_trace\src\Geo-IP\IPFire\libloc\src\py3_install\location.py", line 616, in main
     c.run()
   File "F:\gv\VC_project\ws_trace\src\Geo-IP\IPFire\libloc\src\py3_install\location.py", line 200, in run
     db = location.Database(args.database)
SystemError: <class 'location.Database'> returned NULL without setting an error
libloc: loc_unref: context 0FA35190 released

----

Check yourself if the 'location-2020-05-15.db.xz' is still in the archive.


-- 
--gv

Re: What generates database.db from database.txt?

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 3907 bytes --]

Hello,

> On 21 Oct 2020, at 12:07, Gisle Vanem <gisle.vanem(a)gmail.com> wrote:
> 
> Michael Tremer wrote:
> 
>> This probably requires some more debugging and log output.
>> You can set LOC_LOG=7 as an environment variable and then run test/test-country.
> 
> With this, test-country only reports an extra:
>  libloc: loc_new: ctx 05021298 created
> 
> But with '-DENABLE_DEBUG' there's way too much trace
> to be useful.

That is the kind of debug information I would need.

> 
>> I pulled the database last week because we had some issue with some data in it that triggered some bugs.
>> You can download it from the archive still until the updater works again:
>>   https://location.ipfire.org/databases/1/archive/location-2020-10-16.db.xz
> 
> 
>> Yes, I assumed so. There are some tools that use but they might potentially just bundle it with their own software which I want to avoid.
>> It probably is possible to #ifdef the Windows crypto library here and check the signature that way. But since I have never worked with it, I do not know how to do that. Mac OS X has the same issue.
> 
> An issue with OpenSSL and 'loc_database_verify()'?

No, that OpenSSL isn’t easily available on that platform without shipping it manually.

>> Have you ever worked with it?
> 
> Yes, but 'test-signature' works fine. It's just when calling
> 'EVP_VerifyXX()' functions from a .dll (like a Python module), it
> fails with 'no OPENSSL_AppLink".
> Ref:
> https://stackoverflow.com/questions/38933943/openssl-code-works-natively-but-as-dll-causes-openssl-uplink-no-openssl-applink
> 
> With your URL above,
> set loc_log=7 & py -3 location.py --database location-2020-10-16.db verify:
> 
> libloc: loc_new: ctx 0FDACEE8 created
> libloc: loc_database_read_as_section_v1: Read 48148 ASes from the database
> libloc: loc_database_read_network_nodes_section_v1: Read 2628939 network nodes from the database
> libloc: loc_database_read_networks_section_v1: Read 1318972 networks from the database
> libloc: loc_database_read_countries_section_v1: Read 253 countries from the database
> libloc: loc_database_read: Opened database in 16.0000ms
> libloc: loc_database_read_as_section_v1: Read 48148 ASes from the database
> libloc: loc_database_read_network_nodes_section_v1: Read 2628939 network nodes from the database
> libloc: loc_database_read_networks_section_v1: Read 1318972 networks from the database
> libloc: loc_database_read_countries_section_v1: Read 253 countries from the database
> libloc: loc_database_read: Opened database in 27.0000ms
> OPENSSL_Uplink(640BC6B0,08): no OPENSSL_Applink

I have absolutely no idea how to fix this unfortunately.

I do not even have access to a Windows system to reproduce this.

> BUT, with 'location-2020-05-15.db', there is this error:
> 
> libloc: loc_new: ctx 0FA35190 created
> libloc: loc_database_read_header: Incompatible database version: 0
> Traceback (most recent call last):
>  File "F:\gv\VC_project\ws_trace\src\Geo-IP\IPFire\libloc\src\py3_install\location.py", line 618, in <module>
>    main()
>  File "F:\gv\VC_project\ws_trace\src\Geo-IP\IPFire\libloc\src\py3_install\location.py", line 616, in main
>    c.run()
>  File "F:\gv\VC_project\ws_trace\src\Geo-IP\IPFire\libloc\src\py3_install\location.py", line 200, in run
>    db = location.Database(args.database)
> SystemError: <class 'location.Database'> returned NULL without setting an error
> libloc: loc_unref: context 0FA35190 released

This is a small bug where we do not raise the error message with Python.

However, the debug output should tell you why this didn’t work.

The database version that should be shown there is 1. You have a zero.

> ----
> 
> Check yourself if the 'location-2020-05-15.db.xz' is still in the archive.

Did you extract the file?

Best,
-Michael

> 
> 
> -- 
> --gv

Re: What generates database.db from database.txt?

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 4995 bytes --]

Hi,

> On 21 Oct 2020, at 14:58, Gisle Vanem <gisle.vanem(a)gmail.com> wrote:
> 
>>> OPENSSL_Uplink(640BC6B0,08): no OPENSSL_Applink
>> I have absolutely no idea how to fix this unfortunately.
> 
> It cannot easily be solve unless libloc.dll, _location.pyd
> and the Python .dll use the same CRT.

And a CRT is what?

>> This is a small bug where we do not raise the error message with Python.
> 
> Why not? Not very user friendly to have such obscure abort.

This is not intended. I need to know what error is happening at what point.

We catch the usual ones like “file does not exist”, or the file simply has some content that we did not expect. Potentially, windows is retuning a different error code somewhere which we didn’t catch.

>>> Check yourself if the 'location-2020-05-15.db.xz' is still in the archive.
>> Did you extract the file?
> 
> Duh! Yes.

Just double-checking.

> BTW, what's up with test-network.c and IPv4 addresses?
> I did this patch:
> 
> --- a/test-network.c 2020-10-19 17:35:01
> +++ b/test-network.c 2020-10-21 14:02:41
> @@ -26,6 +26,8 @@
> #include <loc/network.h>
> #include <loc/writer.h>
> 
> +#define IPV4_TEST
> +
> int main(int argc, char** argv) {
>        int err;
> 
> @@ -116,7 +118,7 @@
>        }
> 
>        size_t nodes = loc_network_tree_count_nodes(tree);
> -       printf("The tree has %zu nodes\n", nodes);
> +       printf("The tree has %zu IPv6-only nodes\n", nodes);
> 
>        // Check subnet function
>        err = loc_network_is_subnet_of(network1, network2);
> @@ -160,6 +162,23 @@
>        // Set ASN
>        loc_network_set_asn(network4, 1024);
> 
> +#ifdef IPV4_TEST
> +       struct loc_network* network5;
> +       err = loc_writer_add_network(writer, &network5, "1.2.3.4/16");
> +       if (err) {
> +               fprintf(stderr, "Could not add IPv4 network\n");
> +               exit(EXIT_FAILURE);
> +       }
> +
> +       // Set country code
> +       loc_network_set_country_code(network4, "ZZ");

I assume you meant to set the country code for network5?

> +
> +       // Set ASN
> +       loc_network_set_asn(network4, 1234);

Same as above.

> +       loc_network_tree_dump(tree);
> +       printf("The tree has %zu IPv4/6-mixed nodes\n", loc_network_tree_count_nodes(tree));
> +#endif
> +
>        FILE* f = tmpfile();
>        if (!f) {
>                fprintf(stderr, "Could not open file for writing: %s\n", strerror(errno));
> @@ -203,6 +222,15 @@
>        }
>        loc_network_unref(network1);
> 
> +#ifdef IPV4_TEST
> +       err = loc_database_lookup_from_string(db, "1.2.3.4", &network5);
> +       if (err) {
> +               fprintf(stderr, "Could not look up 1.2.3.4\n");
> +               exit(EXIT_FAILURE);
> +       }
> +       loc_network_unref(network5);
> +#endif
> +
>        loc_unref(ctx);
>        fclose(f);
> 
> ---------
> 
> But I get some errors from 'loc_parse_address()'.
> So maybe it's better to check for an IPv4 first?
> With this:
>  https://github.com/gvanem/wsock-trace/commit/4dc6dc6e09aa024fdcedf1a87dd3f578fdbcfcc9

I do not really understand, again, what you are trying to achieve here.

IPv6 and IPv4 addresses can easily be mixed in the same database.

You are adding "1.2.3.4/16” which actually should have been written as "1.2.0.0/16”. The database will automatically convert this into “1.2.0.0/16” for you.

The result of loc_parse_address() should be the same no matter which one you are trying first (IPv6 or IPv4). In my code, I always put IPv6 first.

Clearly “struct in6_addr” is defined differently in Windows and therefore the latter part of those changes might be required. There will be other places in the code that read from that structure bit by bit to put the IP address into the tree. It is quite likely, that this will go wrong too and therefore loc_database_lookup won’t find the correct entry.

> it works much better (no errors from inet_pton()).
> But the node-count is stuck at 49:
> 
> libloc: loc_new: ctx 049D4CF8 created
> libloc: __loc_network_tree_dump: 2001:db8::/32
> libloc: __loc_network_tree_dump: 2001:db8:ffff::/48
> The tree has 49 IPv6-only nodes
> libloc: __loc_network_tree_dump: 2001:db8::/32
> libloc: __loc_network_tree_dump: 2001:db8:ffff::/48
> The tree has 49 IPv/6-mixed nodes
> libloc: loc_database_read_as_section_v1: Read 0 ASes from the database
> libloc: loc_database_read_network_nodes_section_v1: Read 207 network nodes from the database
> libloc: loc_database_read_networks_section_v1: Read 3 networks from the database
> libloc: loc_database_read_countries_section_v1: Read 0 countries from the database
> libloc: loc_database_read: Opened database in 14.0000ms
> 
> Are we not supposed to add both AF_INET+AF_INET6 addresses to
> the same tree?

Yes.

Best,
-Michael

P.S. Please do not forget to keep the list copied in your replies.

> -- 
> --gv