Re: What generates database.db from database.txt?

Re: What generates database.db from database.txt?

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 4995 bytes --]

Hi,

> On 21 Oct 2020, at 14:58, Gisle Vanem <gisle.vanem(a)gmail.com> wrote:
> 
>>> OPENSSL_Uplink(640BC6B0,08): no OPENSSL_Applink
>> I have absolutely no idea how to fix this unfortunately.
> 
> It cannot easily be solve unless libloc.dll, _location.pyd
> and the Python .dll use the same CRT.

And a CRT is what?

>> This is a small bug where we do not raise the error message with Python.
> 
> Why not? Not very user friendly to have such obscure abort.

This is not intended. I need to know what error is happening at what point.

We catch the usual ones like “file does not exist”, or the file simply has some content that we did not expect. Potentially, windows is retuning a different error code somewhere which we didn’t catch.

>>> Check yourself if the 'location-2020-05-15.db.xz' is still in the archive.
>> Did you extract the file?
> 
> Duh! Yes.

Just double-checking.

> BTW, what's up with test-network.c and IPv4 addresses?
> I did this patch:
> 
> --- a/test-network.c 2020-10-19 17:35:01
> +++ b/test-network.c 2020-10-21 14:02:41
> @@ -26,6 +26,8 @@
> #include <loc/network.h>
> #include <loc/writer.h>
> 
> +#define IPV4_TEST
> +
> int main(int argc, char** argv) {
>        int err;
> 
> @@ -116,7 +118,7 @@
>        }
> 
>        size_t nodes = loc_network_tree_count_nodes(tree);
> -       printf("The tree has %zu nodes\n", nodes);
> +       printf("The tree has %zu IPv6-only nodes\n", nodes);
> 
>        // Check subnet function
>        err = loc_network_is_subnet_of(network1, network2);
> @@ -160,6 +162,23 @@
>        // Set ASN
>        loc_network_set_asn(network4, 1024);
> 
> +#ifdef IPV4_TEST
> +       struct loc_network* network5;
> +       err = loc_writer_add_network(writer, &network5, "1.2.3.4/16");
> +       if (err) {
> +               fprintf(stderr, "Could not add IPv4 network\n");
> +               exit(EXIT_FAILURE);
> +       }
> +
> +       // Set country code
> +       loc_network_set_country_code(network4, "ZZ");

I assume you meant to set the country code for network5?

> +
> +       // Set ASN
> +       loc_network_set_asn(network4, 1234);

Same as above.

> +       loc_network_tree_dump(tree);
> +       printf("The tree has %zu IPv4/6-mixed nodes\n", loc_network_tree_count_nodes(tree));
> +#endif
> +
>        FILE* f = tmpfile();
>        if (!f) {
>                fprintf(stderr, "Could not open file for writing: %s\n", strerror(errno));
> @@ -203,6 +222,15 @@
>        }
>        loc_network_unref(network1);
> 
> +#ifdef IPV4_TEST
> +       err = loc_database_lookup_from_string(db, "1.2.3.4", &network5);
> +       if (err) {
> +               fprintf(stderr, "Could not look up 1.2.3.4\n");
> +               exit(EXIT_FAILURE);
> +       }
> +       loc_network_unref(network5);
> +#endif
> +
>        loc_unref(ctx);
>        fclose(f);
> 
> ---------
> 
> But I get some errors from 'loc_parse_address()'.
> So maybe it's better to check for an IPv4 first?
> With this:
>  https://github.com/gvanem/wsock-trace/commit/4dc6dc6e09aa024fdcedf1a87dd3f578fdbcfcc9

I do not really understand, again, what you are trying to achieve here.

IPv6 and IPv4 addresses can easily be mixed in the same database.

You are adding "1.2.3.4/16” which actually should have been written as "1.2.0.0/16”. The database will automatically convert this into “1.2.0.0/16” for you.

The result of loc_parse_address() should be the same no matter which one you are trying first (IPv6 or IPv4). In my code, I always put IPv6 first.

Clearly “struct in6_addr” is defined differently in Windows and therefore the latter part of those changes might be required. There will be other places in the code that read from that structure bit by bit to put the IP address into the tree. It is quite likely, that this will go wrong too and therefore loc_database_lookup won’t find the correct entry.

> it works much better (no errors from inet_pton()).
> But the node-count is stuck at 49:
> 
> libloc: loc_new: ctx 049D4CF8 created
> libloc: __loc_network_tree_dump: 2001:db8::/32
> libloc: __loc_network_tree_dump: 2001:db8:ffff::/48
> The tree has 49 IPv6-only nodes
> libloc: __loc_network_tree_dump: 2001:db8::/32
> libloc: __loc_network_tree_dump: 2001:db8:ffff::/48
> The tree has 49 IPv/6-mixed nodes
> libloc: loc_database_read_as_section_v1: Read 0 ASes from the database
> libloc: loc_database_read_network_nodes_section_v1: Read 207 network nodes from the database
> libloc: loc_database_read_networks_section_v1: Read 3 networks from the database
> libloc: loc_database_read_countries_section_v1: Read 0 countries from the database
> libloc: loc_database_read: Opened database in 14.0000ms
> 
> Are we not supposed to add both AF_INET+AF_INET6 addresses to
> the same tree?

Yes.

Best,
-Michael

P.S. Please do not forget to keep the list copied in your replies.

> -- 
> --gv

Problem with 'loc_as_get_name()'

[Gisle Vanem]

[-- Attachment #1: Type: text/plain, Size: 1698 bytes --]

I've changed the subject since it's now off-topic
(solved that part).

> And a CRT is what?

A "C Run-time".


> +       // Set country code
>> +       loc_network_set_country_code(network4, "ZZ");

> I assume you meant to set the country code for network5?

Oh, yes. Silly me.

But other item. I've come a long way integrating 'libloc'
support in my program. Basically I do:

   struct in6_addr addr;
   struct loc_network *net;
   struct loc_as      *as = NULL;
   ...
   loc_database_lookup (libloc_data.db, &addr, &net);
   int         family  = loc_network_address_family (net);
   const char *country = loc_network_get_country_code (net);
   const char *AS_name = NULL;

   uint32 AS_num = loc_network_get_asn (net);
   is_anycast    = loc_network_has_flag (net, LOC_NETWORK_FLAG_ANYCAST);
   is_anon_proxy = loc_network_has_flag (net, LOC_NETWORK_FLAG_ANONYMOUS_PROXY);
   sat_provider  = loc_network_has_flag (net, LOC_NETWORK_FLAG_SATELLITE_PROVIDER);

   if (AS_num) {
     int r = loc_as_new (libloc_data.ctx, &as, AS_num);
     if (r == 0)
        AS_name = loc_as_get_name (as);
   }
   printf ("ASN: %lu, AF: %d, country: %s, name: %s (%d,%d,%d)",
           AS_num, family, country, AS_name, is_anycast,
           is_anon_proxy, sat_provider);
   if (as)
      loc_as_unref (as);
   loc_network_unref (net);

-------------

Everything except 'AS_name' is okay and prints sensible values
(I've checked all ret-vals). Since there isn't any 'loc_get_asn_by_number()'
function, what should I do to get a non-NULL 'AS_name' from 'AS_num'?

PS. do I really need this 'loc_network_unref()'?
   Or does 'loc_database_lookup()' handle it?

Re: Problem with 'loc_as_get_name()'

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 3307 bytes --]

Good morning,

> On 22 Oct 2020, at 14:39, Gisle Vanem <gisle.vanem(a)gmail.com> wrote:
> 
> I've changed the subject since it's now off-topic
> (solved that part).
> 
>> And a CRT is what?
> 
> A "C Run-time".
> 
> 
>> +       // Set country code
>>> +       loc_network_set_country_code(network4, "ZZ");
> 
>> I assume you meant to set the country code for network5?
> 
> Oh, yes. Silly me.
> 
> But other item. I've come a long way integrating 'libloc'
> support in my program. Basically I do:
> 
>  struct in6_addr addr;
>  struct loc_network *net;
>  struct loc_as      *as = NULL;
>  ...
>  loc_database_lookup (libloc_data.db, &addr, &net);
>  int         family  = loc_network_address_family (net);
>  const char *country = loc_network_get_country_code (net);
>  const char *AS_name = NULL;
> 
>  uint32 AS_num = loc_network_get_asn (net);
>  is_anycast    = loc_network_has_flag (net, LOC_NETWORK_FLAG_ANYCAST);
>  is_anon_proxy = loc_network_has_flag (net, LOC_NETWORK_FLAG_ANONYMOUS_PROXY);
>  sat_provider  = loc_network_has_flag (net, LOC_NETWORK_FLAG_SATELLITE_PROVIDER);
> 
>  if (AS_num) {
>    int r = loc_as_new (libloc_data.ctx, &as, AS_num);
>    if (r == 0)
>       AS_name = loc_as_get_name (as);
>  }
>  printf ("ASN: %lu, AF: %d, country: %s, name: %s (%d,%d,%d)",
>          AS_num, family, country, AS_name, is_anycast,
>          is_anon_proxy, sat_provider);
>  if (as)
>     loc_as_unref (as);
>  loc_network_unref (net);
> 
> -------------

Cool, this doesn’t look too bad.

First thing to know is that we use reference counting for all objects.

So every pointer that refers to an object has to decrement the counter when it no longer needs it. The object will be automatically cleaned up after nothing needs it any more. Therefore the loc_*_unref() calls are always needed. If you don’t call them, you will leak memory.

The next thing is that networks and AS are independent of each other. You can have a network giving you an ASN, but there might be no AS object in the database for it. That is because we might not have a name for that AS (ARIN doesn’t publicly share them) and therefore we do not need to waste memory on an empty object.

In your code, you are creating a new AS which is not what you want.

You want to call loc_database_get_as() with the AS number that you got from loc_network_get_asn(). That function with then either return an AS or NULL.

loc_as_get_name will give you the name. So your code might look like this:

struct loc_network* network = loc_database_lookup_from_string(db, "81.3.27.38");
if (network) {
	uint32_t asn = loc_network_get_asn(network);

	if (asn) {
		struct loc_as* a = loc_database_get_as(db, asn);
		if (a) {
			name = loc_as_get_name(a);
		}
	}

	loc_network_unref(network);
}

// Copy the name to somewhere
...

// Cleanup
if (a)
	loc_as_unref(a);


> Everything except 'AS_name' is okay and prints sensible values
> (I've checked all ret-vals). Since there isn't any 'loc_get_asn_by_number()'
> function, what should I do to get a non-NULL 'AS_name' from 'AS_num'?
> 
> PS. do I really need this 'loc_network_unref()'?
>  Or does 'loc_database_lookup()' handle it?

I hope this answers your questions :)

Best,
-Michael

Re: Problem with 'loc_as_get_name()'

[Gisle Vanem]

[-- Attachment #1: Type: text/plain, Size: 1847 bytes --]

Michael Tremer wrote:

> In your code, you are creating a new AS which is not what you want.

I figured that after I sent my previous message.

> struct loc_network* network = loc_database_lookup_from_string(db, "81.3.27.38");

I use 'loc_database_lookup()' directly since I do not want to
waste CPU-cycles on 'inet_pton()'. Works fine for me, except
I must always map an IPv4 to a IPv6-mapped address first.

In my code (and 'location lookup'), e.g. a '37.142.14.15' never works.
But this works '::ffff:37.142.14.15' always:
  Network                 : 37.142.0.0/20
  Country                 : Israel
  Autonomous System       : AS12849 - Hot-Net internet services Ltd.

> // Copy the name to somewhere

I found the AS-name can be quite long:
  AS49450 - Federal State Budget Institution NATIONAL MEDICAL RESEARCH CENTER FOR
            OBSTETRICS, GYNECOLOGY AND PERINATOLOGY named after academician
            V. I. Kulakov of the Ministry of Healthcare of the Russian Federation,

Longest I found, a whopping 214 characters.
I suppose 'libloc' handle any length?

Besides, how can I (if possible with libloc/location.py)
figure out all the Peers for an AS? Like what:
   https://dnslytics.com/bgp/as39029   (Redpill Linpro AS)

reports for "IP Prefixes and Peers". For IPv6:
   8473 Bahnhof AB
   2119 Telenor AS
  56655 TerraHost AS
  ...

If I do:
  location list-networks-by-as --family ipv6 39029
  2001:67c:21e0::/48  << !
  ...
and:
  location list-networks-by-as --family ipv6 8473
  2001:67c:107c::/48
  2001:67c:2fb8::/48  << ! this looks close to 2001:67c:21e0::
  ...

Is the relation with an AS and a peer only a routing (BGP)
thing? Where each of the AS'es have a common routing path?

I'm just beginning to understand all these concepts.
So sorry for nagging..

-- 
--gv

Re: Problem with 'loc_as_get_name()'

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 3193 bytes --]

Hi,

> On 23 Oct 2020, at 17:29, Gisle Vanem <gisle.vanem(a)gmail.com> wrote:
> 
> Michael Tremer wrote:
> 
>> In your code, you are creating a new AS which is not what you want.
> 
> I figured that after I sent my previous message.
> 
>> struct loc_network* network = loc_database_lookup_from_string(db, "81.3.27.38");
> 
> I use 'loc_database_lookup()' directly since I do not want to
> waste CPU-cycles on 'inet_pton()'. Works fine for me, except
> I must always map an IPv4 to a IPv6-mapped address first.
> 
> In my code (and 'location lookup'), e.g. a '37.142.14.15' never works.
> But this works '::ffff:37.142.14.15' always:
> Network                 : 37.142.0.0/20
> Country                 : Israel
> Autonomous System       : AS12849 - Hot-Net internet services Ltd.

We could add a call where you can pass struct in_addr and it automatically converts it.

We usually use the IP address as a string input in our own applications so that this was never necessary.

>> // Copy the name to somewhere
> 
> I found the AS-name can be quite long:
> AS49450 - Federal State Budget Institution NATIONAL MEDICAL RESEARCH CENTER FOR
>           OBSTETRICS, GYNECOLOGY AND PERINATOLOGY named after academician
>           V. I. Kulakov of the Ministry of Healthcare of the Russian Federation,
> 
> Longest I found, a whopping 214 characters.
> I suppose 'libloc' handle any length?

Yes, we do not have any limits on this. It might be good idea to introduce that though. That name is beyond reasonable.

> Besides, how can I (if possible with libloc/location.py)
> figure out all the Peers for an AS? Like what:
>  https://dnslytics.com/bgp/as39029   (Redpill Linpro AS)
> 
> reports for "IP Prefixes and Peers". For IPv6:
>  8473 Bahnhof AB
>  2119 Telenor AS
> 56655 TerraHost AS
> ...

You can use location list-networks-by-as. The code for that should tell you how to part that to your own application. Look for the database iterator.

> If I do:
> location list-networks-by-as --family ipv6 39029
> 2001:67c:21e0::/48  << !
> ...
> and:
> location list-networks-by-as --family ipv6 8473
> 2001:67c:107c::/48
> 2001:67c:2fb8::/48  << ! this looks close to 2001:67c:21e0::
> ...

Yes that might be. Subnets are being allocated like that without any gaps.

> Is the relation with an AS and a peer only a routing (BGP)
> thing? Where each of the AS'es have a common routing path?

I am not sure what you are asking here.

A peer can be in a subnet. That subnet might have an ASN (probably will if it is currently being routed or was recently routed).

If one network is only single-homed to another one is outside the scope of this library.

> I'm just beginning to understand all these concepts.
> So sorry for nagging..

Happy to help. Just try to help me helping you by explaining what your goal is and I can help you finding the best way to that.

It would be great if we could work on upstreaming your changes for Windows so that they become available for everyone and you no longer need to bundle your own version of libloc without receiving bugfixes from upstream.

Best.
-Michael

> 
> -- 
> --gv

Re: Problem with 'loc_as_get_name()'

[Gisle Vanem]

[-- Attachment #1: Type: text/plain, Size: 981 bytes --]

Michael Tremer wrote:

> It would be great if we could work on upstreaming your changes for
> Windows so that they become available for everyone and you no longer need to
> bundle your own version of libloc without receiving bugfixes from upstream.

Hi again. Sorry for the delay.

That would be tricky for me; I'm a n00b when it comes
to Git and pull-requests. But my my Win-sources are under
here:
   https://github.com/gvanem/wsock-trace/tree/master/src/Geo-IP/IPFire/libloc/src

You could perhaps pull them, create a diff and then
email me privately to discuss things if you really
want Windows support. It would be nice to see Location
get a much wider audience (competing with MaxMind and
IP2Location ...)

PS. One thing I see now, trying to support Cygwin too;
Your calls to 'mmap()' fails on Cygwin32 for a reason
I fail to understand; errno = EINVAL. So I use my
Win-mmap emulation for Cygwin too. Works fine.
Will see about Cygwin64.

-- 
--gv

Re: Problem with 'loc_as_get_name()'

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 1858 bytes --]

Hi,

> On 28 Oct 2020, at 11:50, Gisle Vanem <gisle.vanem(a)gmail.com> wrote:
> 
> Michael Tremer wrote:
> 
>> It would be great if we could work on upstreaming your changes for
>> Windows so that they become available for everyone and you no longer need to
>> bundle your own version of libloc without receiving bugfixes from upstream.
> 
> Hi again. Sorry for the delay.
> 
> That would be tricky for me; I'm a n00b when it comes
> to Git and pull-requests. But my my Win-sources are under
> here:
>  https://github.com/gvanem/wsock-trace/tree/master/src/Geo-IP/IPFire/libloc/src

I would recommend to learn Git then. It is everywhere and that is a skill paying off:

  https://wiki.ipfire.org/devel/submit-patches
  https://wiki.ipfire.org/devel/git

> You could perhaps pull them, create a diff and then
> email me privately to discuss things if you really
> want Windows support. It would be nice to see Location
> get a much wider audience (competing with MaxMind and
> IP2Location ...)

I would like to keep this on the list so that others can join in and help to catch any newly introduced bugs early.

And yes, I would like to see support for Windows, too.

> PS. One thing I see now, trying to support Cygwin too;
> Your calls to 'mmap()' fails on Cygwin32 for a reason
> I fail to understand; errno = EINVAL. So I use my
> Win-mmap emulation for Cygwin too. Works fine.
> Will see about Cygwin64.

It might simply be that mmap() is not supported on Win32. I am not sure if that architecture even needs to be supported by us at all. Is there even one officially supported version of Windows that is still 32 bit?

I would prefer to avoid pulling in a third-party wrapper for mmap() if we can find a different alternative. Those things tend to break really quickly.

Best,
-Michael

> 
> -- 
> --gv

What generates database.db from database.txt?

[Gisle Vanem]

[-- Attachment #1: Type: text/plain, Size: 777 bytes --]

Hello list, my ports post.

I'm on Windows and have little interest in installing
IPFire or do any Linux related. But the 'libloc'
library caught my interest. So I did a port to
MSVC, MinGW and clang-cl (x86 only) in a hurry.

Running all the 'test-*' programs, it worked seemingly
well. Until I built the Python3 module and played with
e.g. 'location dump' and 'verify'. Nothing worked :-(
So I reckon the .db file is hosed somehow. AFAICS, it's
not an issue with structure packing or 'fopen(path, "r")'.

Anyway, I want to regenerate 'database.db' from 'database.txt'
'countries.txt' and the 'override' files myself. But how?

According to the git log at:
   git://git.ipfire.org/location/location-database.git

there once was a 'compile-database' script. Not now.

-- 
--gv

Re: What generates database.db from database.txt?

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 2653 bytes --]

Hello Gisle,

Thank you for your email.

> On 19 Oct 2020, at 16:53, Gisle Vanem <gisle.vanem(a)gmail.com> wrote:
> 
> Hello list, my ports post.
> 
> I'm on Windows and have little interest in installing
> IPFire or do any Linux related. But the 'libloc'
> library caught my interest. So I did a port to
> MSVC, MinGW and clang-cl (x86 only) in a hurry.

That sounds great!

We are looking for someone with some experience with Windows to port libloc. It should not be super complicated, but since I am not the expert I am not too sure.

Would you be willing to tidy up your code (if required) and submit them on here so that we can officially support Windows?

> Running all the 'test-*' programs, it worked seemingly
> well. Until I built the Python3 module and played with
> e.g. 'location dump' and 'verify'. Nothing worked :-(

What didn’t work specifically? Reading and writing the database?

> So I reckon the .db file is hosed somehow. AFAICS, it's
> not an issue with structure packing or 'fopen(path, "r")'.

Hmm… it would be interesting to debug that.

> Anyway, I want to regenerate 'database.db' from 'database.txt'
> 'countries.txt' and the 'override' files myself. But how?

We do not generate the database from database.txt. database.txt is being generated from the database. The text dump is just there so that we have a (sort of) human-readable version which we can diff against the previous one to manually verify any changes.

What you will need to import the raw data is location-importer which comes with libloc and a PostgreSQL server.

location-importer —help will show you how you pass the database credentials.

On our servers we regularly run “location-import update-overrides file-a.txt file-b.txt”, then "location-importer update-whois”, then we run “location-importer update-announcements” which connects to a locally running instance of Bird with a full BGP feed to import those.

Finally we run “location-importer write … database.db” (fill in the gaps with your vendor name, description, key, etc.) and finally you can run “location -d database.db dump” to export it into the text format.

Generating the whole database might take a couple of hours depending on how beefy your machine is.

Let me know if this works for you.

> According to the git log at:
>  git://git.ipfire.org/location/location-database.git
> 
> there once was a 'compile-database' script. Not now.

Yes, this has all been moved into “location-importer” so that we separate code and data in two different Git repositories.

Best,
-Michael

> 
> -- 
> --gv

Re: What generates database.db from database.txt?

[Gisle Vanem]

[-- Attachment #1: Type: text/plain, Size: 1921 bytes --]

Michael Tremer wrote:

> Would you be willing to tidy up your code (if required) and submit them on here so that we can officially support Windows?

I've put a preliminary version of my patched 'libloc'
in my 'Wsock-trace' project:
   https://github.com/gvanem/wsock-trace/tree/master/src/Geo-IP/IPFire/libloc/src

>> Running all the 'test-*' programs, it worked seemingly
>> well. Until I built the Python3 module and played with
>> e.g. 'location dump' and 'verify'. Nothing worked :-(
> 
> What didn’t work specifically? Reading and writing the database?

'test-database' says:
   Vendor doesn't match: LOCDBXX☺ != Test Vendor
and 'test-network':
   Could not look up 2001:db8::

I'm not sure it due to my 'mmap()' function.
But I noted a case where 'loc_stringpool_get()' could
return NULL, this is used in 'strdup()'. Not sure that's
legal everywhere. So I did this patch:

--- a/libloc/src/as.c 2020-10-19 17:35:01
+++ b/libloc/src/as.c 2020-10-20 19:37:19
@@ -90,7 +90,7 @@
  }

  LOC_EXPORT int loc_as_set_name(struct loc_as* as, const char* name) {
-       as->name = strdup(name);
+       as->name = name ? strdup(name) : strdup("");

         return 0;
  }

> What you will need to import the raw data is location-importer which comes 
> with libloc and a PostgreSQL server.

No! I hoped that 'libloc' would be similar to
IP2Location and/or GeoipCSV that I already support
in my project. Running PostgreSQL is out of the question
for me or any users of 'Wsock-trace'.

> Generating the whole database might take a couple of hours
> depending on how beefy your machine is.

Lacking in Python skills, I cooked up this 'compile-database'
script that takes 2 minutes:
   https://github.com/gvanem/wsock-trace/blob/master/src/Geo-IP/IPFire/database/compile-database.py

But it remains to see if it works. Or if I'll rewrite it
into a C-program.

-- 
--gv

Re: What generates database.db from database.txt?

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 3698 bytes --]

Hey Gisle,

> On 20 Oct 2020, at 20:14, Gisle Vanem <gisle.vanem(a)gmail.com> wrote:
> 
> Michael Tremer wrote:
> 
>> Would you be willing to tidy up your code (if required) and submit them on here so that we can officially support Windows?
> 
> I've put a preliminary version of my patched 'libloc'
> in my 'Wsock-trace' project:
>  https://github.com/gvanem/wsock-trace/tree/master/src/Geo-IP/IPFire/libloc/src

Thanks. I will have a look at it.

What is your application you are planning to use this in?

And probably more importantly, why does the database that we provide not work for you?

>>> Running all the 'test-*' programs, it worked seemingly
>>> well. Until I built the Python3 module and played with
>>> e.g. 'location dump' and 'verify'. Nothing worked :-(
>> What didn’t work specifically? Reading and writing the database?
> 
> 'test-database' says:
>  Vendor doesn't match: LOCDBXX☺ != Test Vendor

Oh that sounds interesting.

We have a string pool which starts somewhere towards the end of the file and it looks like you got the first couple of bytes there with the magic value in it.

So reading from the string pool does not work.

> and 'test-network':
>  Could not look up 2001:db8::

And I suppose reading other blocks of data doesn’t work either then.

> I'm not sure it due to my 'mmap()' function.

We are using simple functions to read and write to memory. No idea why those should behave differently on Windows, but I am sure we will get to the bottom of it.

> But I noted a case where 'loc_stringpool_get()' could
> return NULL, this is used in 'strdup()'. Not sure that's
> legal everywhere.

loc_stringpool_get() can indeed return NULL, but we should not use that result in strdup() anywhere.

> So I did this patch:
> 
> --- a/libloc/src/as.c 2020-10-19 17:35:01
> +++ b/libloc/src/as.c 2020-10-20 19:37:19
> @@ -90,7 +90,7 @@
> }
> 
> LOC_EXPORT int loc_as_set_name(struct loc_as* as, const char* name) {
> -       as->name = strdup(name);
> +       as->name = name ? strdup(name) : strdup("");
> 
>        return 0;
> }

Good catch, but there is more to it. When resetting the name, the original value wasn’t freed. I added that and pushed it to the main repository:

https://git.ipfire.org/?p=location/libloc.git;a=commitdiff;h=a7d3a7a0565a0e09d3442e5829a0f30f016993b9

> 
>> What you will need to import the raw data is location-importer which comes with libloc and a PostgreSQL server.
> 
> No! I hoped that 'libloc' would be similar to
> IP2Location and/or GeoipCSV that I already support
> in my project. Running PostgreSQL is out of the question
> for me or any users of 'Wsock-trace'.

You won’t need it to make the database. You should simply download the binary version of the library and that will be it.

You will need the development tools old if you want to modify the database and publish your own. But I do not see any reason why anyone should be doing that.

> 
>> Generating the whole database might take a couple of hours
>> depending on how beefy your machine is.
> 
> Lacking in Python skills, I cooked up this 'compile-database'
> script that takes 2 minutes:
>  https://github.com/gvanem/wsock-trace/blob/master/src/Geo-IP/IPFire/database/compile-database.py
> 
> But it remains to see if it works. Or if I'll rewrite it
> into a C-program.

For performance-reasons Python should be fine.

I am still not sure what you are trying to achieve here. If you are looking for the data, simply run “location update” and the script should download the database, extract it, verify it and you can use it.

Best,
-Michael

> -- 
> --gv

Re: What generates database.db from database.txt?

[Gisle Vanem]

[-- Attachment #1: Type: text/plain, Size: 2750 bytes --]

> What is your application you are planning to use this in?

Wsock-trace; a Winsock-trace library like I wrote.

> And probably more importantly, why does the database that we provide not work for you?

That's one of the things I'm trying to figure out.

>> LOC_EXPORT int loc_as_set_name(struct loc_as* as, const char* name) {
>> -       as->name = strdup(name);
>> +       as->name = name ? strdup(name) : strdup("");
>>
>>         return 0;
>> }
> 
> Good catch, but there is more to it. When resetting the name, the original
  value wasn’t freed. I added that and pushed it to the main repository:

With that patch, test-as crashes. I did this:
--- a/src/as.c 2020-10-20 23:23:34
+++ b/src/as.c 2020-10-20 23:55:51
@@ -142,7 +142,7 @@

  int loc_as_match_string(struct loc_as* as, const char* string) {
         // Match all AS when no search string is set
-       if (!string)
+       if (!string || !as->name)
                 return 1;

which no longer crashes. But when building in CRT debug-mode,
I note many mem-leaks in e.g. 'test-as.c' and 'test-country.c'.

Some of it is due to the error I get:
   Could not find country: YY

and you do 'exit(EXIT_FAILURE);' w/o cleaning up.
Such a behaviour could probably accumulate leaks
over time. But I do not know IPFire.

But I plugged other leaks using:
--- a/src/writer.c 2020-10-19 17:35:01
+++ b/src/writer.c 2020-10-20 23:15:20
@@ -155,6 +155,12 @@
         if (writer->networks)
                 loc_network_tree_unref(writer->networks);

+       if (writer->as)
+               free(writer->as);
+
+       if (writer->countries)
+               free(writer->countries);
+
         // Unref the string pool
         loc_stringpool_unref(writer->pool);

Never done AFAICS. The leak for 'as.c', I'm not sure
how to plug. You ought to check with valgrind etc.

> You won’t need it to make the database.
> You should simply download the binary version of the library and that will be it.

Where? Nothing under https://location.ipfire.org/databases/1/
But a 'adig -t txt _v1._db.location.ipfire.org' says:
   TXT Fri, 16 Oct 2020 07:54:08 GMT

I fail to find it. But that's another issue.

> I am still not sure what you are trying to achieve here. If you are looking for the data, simply run “location update” and the script should download the database, extract it, verify it and you can use it.

Here:
   py -3 location.py update
   https://location.ipfire.org/databases/ reported: HTTP Error 404: Not Found
   Could not download a new database

And have you ever used OpenSSL on Windows? Here:
   py -3 location.py verify
   OPENSSL_Uplink(640BC6B0,08): no OPENSSL_Applink

It's pure hell.

-- 
--gv

Re: What generates database.db from database.txt?

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 4468 bytes --]

Hi,

> On 20 Oct 2020, at 23:46, Gisle Vanem <gisle.vanem(a)gmail.com> wrote:
> 
>> What is your application you are planning to use this in?
> 
> Wsock-trace; a Winsock-trace library like I wrote.
> 
>> And probably more importantly, why does the database that we provide not work for you?
> 
> That's one of the things I'm trying to figure out.
> 
>>> LOC_EXPORT int loc_as_set_name(struct loc_as* as, const char* name) {
>>> -       as->name = strdup(name);
>>> +       as->name = name ? strdup(name) : strdup("");
>>> 
>>>        return 0;
>>> }
>> Good catch, but there is more to it. When resetting the name, the original
> value wasn’t freed. I added that and pushed it to the main repository:
> 
> With that patch, test-as crashes. I did this:
> --- a/src/as.c 2020-10-20 23:23:34
> +++ b/src/as.c 2020-10-20 23:55:51
> @@ -142,7 +142,7 @@
> 
> int loc_as_match_string(struct loc_as* as, const char* string) {
>        // Match all AS when no search string is set
> -       if (!string)
> +       if (!string || !as->name)
>                return 1;
> 
> which no longer crashes. But when building in CRT debug-mode,
> I note many mem-leaks in e.g. 'test-as.c' and 'test-country.c'.

The testsuite executed on my Debian box I used for testing.

But logically your patch is correct. I will import it. Thank you.

> Some of it is due to the error I get:
>  Could not find country: YY

This probably requires some more debugging and log output.

You can set LOC_LOG=7 as an environment variable and then run test/test-country.

You should see some logging output which will be helpful to get to the bottom of this.

> and you do 'exit(EXIT_FAILURE);' w/o cleaning up.
> Such a behaviour could probably accumulate leaks
> over time. But I do not know IPFire.

Yes it does. But it is the testsuite. We do clean up to test the cleanup routines, but if something goes wrong we terminate the whole process which will free all memory. This code isn’t shipped to the end user systems.

> But I plugged other leaks using:
> --- a/src/writer.c 2020-10-19 17:35:01
> +++ b/src/writer.c 2020-10-20 23:15:20
> @@ -155,6 +155,12 @@
>        if (writer->networks)
>                loc_network_tree_unref(writer->networks);
> 
> +       if (writer->as)
> +               free(writer->as);
> +
> +       if (writer->countries)
> +               free(writer->countries);
> +
>        // Unref the string pool
>        loc_stringpool_unref(writer->pool);
> 
> Never done AFAICS. The leak for 'as.c', I'm not sure
> how to plug. You ought to check with valgrind etc.

We employed someone to do this, but he said he didn’t find anything like this lol.

Just freeing the arrays is not enough. The allocated objects need to be freed, too.

I will have a look at it.

>> You won’t need it to make the database.
>> You should simply download the binary version of the library and that will be it.
> 
> Where? Nothing under https://location.ipfire.org/databases/1/
> But a 'adig -t txt _v1._db.location.ipfire.org' says:
>  TXT Fri, 16 Oct 2020 07:54:08 GMT
> 
> I fail to find it. But that's another issue.

I pulled the database last week because we had some issue with some data in it that triggered some bugs.

You can download it from the archive still until the updater works again:

  https://location.ipfire.org/databases/1/archive/location-2020-10-16.db.xz

> 
>> I am still not sure what you are trying to achieve here. If you are looking for the data, simply run “location update” and the script should download the database, extract it, verify it and you can use it.
> 
> Here:
>  py -3 location.py update
>  https://location.ipfire.org/databases/ reported: HTTP Error 404: Not Found
>  Could not download a new database
> 
> And have you ever used OpenSSL on Windows? Here:
>  py -3 location.py verify
>  OPENSSL_Uplink(640BC6B0,08): no OPENSSL_Applink
> 
> It's pure hell.

Yes, I assumed so. There are some tools that use but they might potentially just bundle it with their own software which I want to avoid.

It probably is possible to #ifdef the Windows crypto library here and check the signature that way. But since I have never worked with it, I do not know how to do that. Mac OS X has the same issue.

Have you ever worked with it?

However, OpenSSL is a very good choice in the *nix/BSD world.

Best,
-Michael

> 
> -- 
> --gv

Re: What generates database.db from database.txt?

[Gisle Vanem]

[-- Attachment #1: Type: text/plain, Size: 3189 bytes --]

Michael Tremer wrote:

> This probably requires some more debugging and log output.
> 
> You can set LOC_LOG=7 as an environment variable and then run test/test-country.

With this, test-country only reports an extra:
   libloc: loc_new: ctx 05021298 created

But with '-DENABLE_DEBUG' there's way too much trace
to be useful.

> I pulled the database last week because we had some issue with some data in it that triggered some bugs.
> 
> You can download it from the archive still until the updater works again:
> 
>    https://location.ipfire.org/databases/1/archive/location-2020-10-16.db.xz


> Yes, I assumed so. There are some tools that use but they might potentially just bundle it with their own software which I want to avoid.
> 
> It probably is possible to #ifdef the Windows crypto library here and check the signature that way. But since I have never worked with it, I do not know how to do that. Mac OS X has the same issue.

An issue with OpenSSL and 'loc_database_verify()'?

> Have you ever worked with it?

Yes, but 'test-signature' works fine. It's just when calling
'EVP_VerifyXX()' functions from a .dll (like a Python module), it
fails with 'no OPENSSL_AppLink".
Ref:
 
https://stackoverflow.com/questions/38933943/openssl-code-works-natively-but-as-dll-causes-openssl-uplink-no-openssl-applink

With your URL above,
  set loc_log=7 & py -3 location.py --database location-2020-10-16.db verify:

libloc: loc_new: ctx 0FDACEE8 created
libloc: loc_database_read_as_section_v1: Read 48148 ASes from the database
libloc: loc_database_read_network_nodes_section_v1: Read 2628939 network nodes from the database
libloc: loc_database_read_networks_section_v1: Read 1318972 networks from the database
libloc: loc_database_read_countries_section_v1: Read 253 countries from the database
libloc: loc_database_read: Opened database in 16.0000ms
libloc: loc_database_read_as_section_v1: Read 48148 ASes from the database
libloc: loc_database_read_network_nodes_section_v1: Read 2628939 network nodes from the database
libloc: loc_database_read_networks_section_v1: Read 1318972 networks from the database
libloc: loc_database_read_countries_section_v1: Read 253 countries from the database
libloc: loc_database_read: Opened database in 27.0000ms
OPENSSL_Uplink(640BC6B0,08): no OPENSSL_Applink

BUT, with 'location-2020-05-15.db', there is this error:

libloc: loc_new: ctx 0FA35190 created
libloc: loc_database_read_header: Incompatible database version: 0
Traceback (most recent call last):
   File "F:\gv\VC_project\ws_trace\src\Geo-IP\IPFire\libloc\src\py3_install\location.py", line 618, in <module>
     main()
   File "F:\gv\VC_project\ws_trace\src\Geo-IP\IPFire\libloc\src\py3_install\location.py", line 616, in main
     c.run()
   File "F:\gv\VC_project\ws_trace\src\Geo-IP\IPFire\libloc\src\py3_install\location.py", line 200, in run
     db = location.Database(args.database)
SystemError: <class 'location.Database'> returned NULL without setting an error
libloc: loc_unref: context 0FA35190 released

----

Check yourself if the 'location-2020-05-15.db.xz' is still in the archive.


-- 
--gv

Re: What generates database.db from database.txt?

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 3907 bytes --]

Hello,

> On 21 Oct 2020, at 12:07, Gisle Vanem <gisle.vanem(a)gmail.com> wrote:
> 
> Michael Tremer wrote:
> 
>> This probably requires some more debugging and log output.
>> You can set LOC_LOG=7 as an environment variable and then run test/test-country.
> 
> With this, test-country only reports an extra:
>  libloc: loc_new: ctx 05021298 created
> 
> But with '-DENABLE_DEBUG' there's way too much trace
> to be useful.

That is the kind of debug information I would need.

> 
>> I pulled the database last week because we had some issue with some data in it that triggered some bugs.
>> You can download it from the archive still until the updater works again:
>>   https://location.ipfire.org/databases/1/archive/location-2020-10-16.db.xz
> 
> 
>> Yes, I assumed so. There are some tools that use but they might potentially just bundle it with their own software which I want to avoid.
>> It probably is possible to #ifdef the Windows crypto library here and check the signature that way. But since I have never worked with it, I do not know how to do that. Mac OS X has the same issue.
> 
> An issue with OpenSSL and 'loc_database_verify()'?

No, that OpenSSL isn’t easily available on that platform without shipping it manually.

>> Have you ever worked with it?
> 
> Yes, but 'test-signature' works fine. It's just when calling
> 'EVP_VerifyXX()' functions from a .dll (like a Python module), it
> fails with 'no OPENSSL_AppLink".
> Ref:
> https://stackoverflow.com/questions/38933943/openssl-code-works-natively-but-as-dll-causes-openssl-uplink-no-openssl-applink
> 
> With your URL above,
> set loc_log=7 & py -3 location.py --database location-2020-10-16.db verify:
> 
> libloc: loc_new: ctx 0FDACEE8 created
> libloc: loc_database_read_as_section_v1: Read 48148 ASes from the database
> libloc: loc_database_read_network_nodes_section_v1: Read 2628939 network nodes from the database
> libloc: loc_database_read_networks_section_v1: Read 1318972 networks from the database
> libloc: loc_database_read_countries_section_v1: Read 253 countries from the database
> libloc: loc_database_read: Opened database in 16.0000ms
> libloc: loc_database_read_as_section_v1: Read 48148 ASes from the database
> libloc: loc_database_read_network_nodes_section_v1: Read 2628939 network nodes from the database
> libloc: loc_database_read_networks_section_v1: Read 1318972 networks from the database
> libloc: loc_database_read_countries_section_v1: Read 253 countries from the database
> libloc: loc_database_read: Opened database in 27.0000ms
> OPENSSL_Uplink(640BC6B0,08): no OPENSSL_Applink

I have absolutely no idea how to fix this unfortunately.

I do not even have access to a Windows system to reproduce this.

> BUT, with 'location-2020-05-15.db', there is this error:
> 
> libloc: loc_new: ctx 0FA35190 created
> libloc: loc_database_read_header: Incompatible database version: 0
> Traceback (most recent call last):
>  File "F:\gv\VC_project\ws_trace\src\Geo-IP\IPFire\libloc\src\py3_install\location.py", line 618, in <module>
>    main()
>  File "F:\gv\VC_project\ws_trace\src\Geo-IP\IPFire\libloc\src\py3_install\location.py", line 616, in main
>    c.run()
>  File "F:\gv\VC_project\ws_trace\src\Geo-IP\IPFire\libloc\src\py3_install\location.py", line 200, in run
>    db = location.Database(args.database)
> SystemError: <class 'location.Database'> returned NULL without setting an error
> libloc: loc_unref: context 0FA35190 released

This is a small bug where we do not raise the error message with Python.

However, the debug output should tell you why this didn’t work.

The database version that should be shown there is 1. You have a zero.

> ----
> 
> Check yourself if the 'location-2020-05-15.db.xz' is still in the archive.

Did you extract the file?

Best,
-Michael

> 
> 
> -- 
> --gv