Re: What generates database.db from database.txt?

Re: What generates database.db from database.txt?

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 4995 bytes --]

Hi,

> On 21 Oct 2020, at 14:58, Gisle Vanem <gisle.vanem(a)gmail.com> wrote:
> 
>>> OPENSSL_Uplink(640BC6B0,08): no OPENSSL_Applink
>> I have absolutely no idea how to fix this unfortunately.
> 
> It cannot easily be solve unless libloc.dll, _location.pyd
> and the Python .dll use the same CRT.

And a CRT is what?

>> This is a small bug where we do not raise the error message with Python.
> 
> Why not? Not very user friendly to have such obscure abort.

This is not intended. I need to know what error is happening at what point.

We catch the usual ones like “file does not exist”, or the file simply has some content that we did not expect. Potentially, windows is retuning a different error code somewhere which we didn’t catch.

>>> Check yourself if the 'location-2020-05-15.db.xz' is still in the archive.
>> Did you extract the file?
> 
> Duh! Yes.

Just double-checking.

> BTW, what's up with test-network.c and IPv4 addresses?
> I did this patch:
> 
> --- a/test-network.c 2020-10-19 17:35:01
> +++ b/test-network.c 2020-10-21 14:02:41
> @@ -26,6 +26,8 @@
> #include <loc/network.h>
> #include <loc/writer.h>
> 
> +#define IPV4_TEST
> +
> int main(int argc, char** argv) {
>        int err;
> 
> @@ -116,7 +118,7 @@
>        }
> 
>        size_t nodes = loc_network_tree_count_nodes(tree);
> -       printf("The tree has %zu nodes\n", nodes);
> +       printf("The tree has %zu IPv6-only nodes\n", nodes);
> 
>        // Check subnet function
>        err = loc_network_is_subnet_of(network1, network2);
> @@ -160,6 +162,23 @@
>        // Set ASN
>        loc_network_set_asn(network4, 1024);
> 
> +#ifdef IPV4_TEST
> +       struct loc_network* network5;
> +       err = loc_writer_add_network(writer, &network5, "1.2.3.4/16");
> +       if (err) {
> +               fprintf(stderr, "Could not add IPv4 network\n");
> +               exit(EXIT_FAILURE);
> +       }
> +
> +       // Set country code
> +       loc_network_set_country_code(network4, "ZZ");

I assume you meant to set the country code for network5?

> +
> +       // Set ASN
> +       loc_network_set_asn(network4, 1234);

Same as above.

> +       loc_network_tree_dump(tree);
> +       printf("The tree has %zu IPv4/6-mixed nodes\n", loc_network_tree_count_nodes(tree));
> +#endif
> +
>        FILE* f = tmpfile();
>        if (!f) {
>                fprintf(stderr, "Could not open file for writing: %s\n", strerror(errno));
> @@ -203,6 +222,15 @@
>        }
>        loc_network_unref(network1);
> 
> +#ifdef IPV4_TEST
> +       err = loc_database_lookup_from_string(db, "1.2.3.4", &network5);
> +       if (err) {
> +               fprintf(stderr, "Could not look up 1.2.3.4\n");
> +               exit(EXIT_FAILURE);
> +       }
> +       loc_network_unref(network5);
> +#endif
> +
>        loc_unref(ctx);
>        fclose(f);
> 
> ---------
> 
> But I get some errors from 'loc_parse_address()'.
> So maybe it's better to check for an IPv4 first?
> With this:
>  https://github.com/gvanem/wsock-trace/commit/4dc6dc6e09aa024fdcedf1a87dd3f578fdbcfcc9

I do not really understand, again, what you are trying to achieve here.

IPv6 and IPv4 addresses can easily be mixed in the same database.

You are adding "1.2.3.4/16” which actually should have been written as "1.2.0.0/16”. The database will automatically convert this into “1.2.0.0/16” for you.

The result of loc_parse_address() should be the same no matter which one you are trying first (IPv6 or IPv4). In my code, I always put IPv6 first.

Clearly “struct in6_addr” is defined differently in Windows and therefore the latter part of those changes might be required. There will be other places in the code that read from that structure bit by bit to put the IP address into the tree. It is quite likely, that this will go wrong too and therefore loc_database_lookup won’t find the correct entry.

> it works much better (no errors from inet_pton()).
> But the node-count is stuck at 49:
> 
> libloc: loc_new: ctx 049D4CF8 created
> libloc: __loc_network_tree_dump: 2001:db8::/32
> libloc: __loc_network_tree_dump: 2001:db8:ffff::/48
> The tree has 49 IPv6-only nodes
> libloc: __loc_network_tree_dump: 2001:db8::/32
> libloc: __loc_network_tree_dump: 2001:db8:ffff::/48
> The tree has 49 IPv/6-mixed nodes
> libloc: loc_database_read_as_section_v1: Read 0 ASes from the database
> libloc: loc_database_read_network_nodes_section_v1: Read 207 network nodes from the database
> libloc: loc_database_read_networks_section_v1: Read 3 networks from the database
> libloc: loc_database_read_countries_section_v1: Read 0 countries from the database
> libloc: loc_database_read: Opened database in 14.0000ms
> 
> Are we not supposed to add both AF_INET+AF_INET6 addresses to
> the same tree?

Yes.

Best,
-Michael

P.S. Please do not forget to keep the list copied in your replies.

> -- 
> --gv

Problem with 'loc_as_get_name()'

[Gisle Vanem]

[-- Attachment #1: Type: text/plain, Size: 1698 bytes --]

I've changed the subject since it's now off-topic
(solved that part).

> And a CRT is what?

A "C Run-time".


> +       // Set country code
>> +       loc_network_set_country_code(network4, "ZZ");

> I assume you meant to set the country code for network5?

Oh, yes. Silly me.

But other item. I've come a long way integrating 'libloc'
support in my program. Basically I do:

   struct in6_addr addr;
   struct loc_network *net;
   struct loc_as      *as = NULL;
   ...
   loc_database_lookup (libloc_data.db, &addr, &net);
   int         family  = loc_network_address_family (net);
   const char *country = loc_network_get_country_code (net);
   const char *AS_name = NULL;

   uint32 AS_num = loc_network_get_asn (net);
   is_anycast    = loc_network_has_flag (net, LOC_NETWORK_FLAG_ANYCAST);
   is_anon_proxy = loc_network_has_flag (net, LOC_NETWORK_FLAG_ANONYMOUS_PROXY);
   sat_provider  = loc_network_has_flag (net, LOC_NETWORK_FLAG_SATELLITE_PROVIDER);

   if (AS_num) {
     int r = loc_as_new (libloc_data.ctx, &as, AS_num);
     if (r == 0)
        AS_name = loc_as_get_name (as);
   }
   printf ("ASN: %lu, AF: %d, country: %s, name: %s (%d,%d,%d)",
           AS_num, family, country, AS_name, is_anycast,
           is_anon_proxy, sat_provider);
   if (as)
      loc_as_unref (as);
   loc_network_unref (net);

-------------

Everything except 'AS_name' is okay and prints sensible values
(I've checked all ret-vals). Since there isn't any 'loc_get_asn_by_number()'
function, what should I do to get a non-NULL 'AS_name' from 'AS_num'?

PS. do I really need this 'loc_network_unref()'?
   Or does 'loc_database_lookup()' handle it?

Re: Problem with 'loc_as_get_name()'

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 3307 bytes --]

Good morning,

> On 22 Oct 2020, at 14:39, Gisle Vanem <gisle.vanem(a)gmail.com> wrote:
> 
> I've changed the subject since it's now off-topic
> (solved that part).
> 
>> And a CRT is what?
> 
> A "C Run-time".
> 
> 
>> +       // Set country code
>>> +       loc_network_set_country_code(network4, "ZZ");
> 
>> I assume you meant to set the country code for network5?
> 
> Oh, yes. Silly me.
> 
> But other item. I've come a long way integrating 'libloc'
> support in my program. Basically I do:
> 
>  struct in6_addr addr;
>  struct loc_network *net;
>  struct loc_as      *as = NULL;
>  ...
>  loc_database_lookup (libloc_data.db, &addr, &net);
>  int         family  = loc_network_address_family (net);
>  const char *country = loc_network_get_country_code (net);
>  const char *AS_name = NULL;
> 
>  uint32 AS_num = loc_network_get_asn (net);
>  is_anycast    = loc_network_has_flag (net, LOC_NETWORK_FLAG_ANYCAST);
>  is_anon_proxy = loc_network_has_flag (net, LOC_NETWORK_FLAG_ANONYMOUS_PROXY);
>  sat_provider  = loc_network_has_flag (net, LOC_NETWORK_FLAG_SATELLITE_PROVIDER);
> 
>  if (AS_num) {
>    int r = loc_as_new (libloc_data.ctx, &as, AS_num);
>    if (r == 0)
>       AS_name = loc_as_get_name (as);
>  }
>  printf ("ASN: %lu, AF: %d, country: %s, name: %s (%d,%d,%d)",
>          AS_num, family, country, AS_name, is_anycast,
>          is_anon_proxy, sat_provider);
>  if (as)
>     loc_as_unref (as);
>  loc_network_unref (net);
> 
> -------------

Cool, this doesn’t look too bad.

First thing to know is that we use reference counting for all objects.

So every pointer that refers to an object has to decrement the counter when it no longer needs it. The object will be automatically cleaned up after nothing needs it any more. Therefore the loc_*_unref() calls are always needed. If you don’t call them, you will leak memory.

The next thing is that networks and AS are independent of each other. You can have a network giving you an ASN, but there might be no AS object in the database for it. That is because we might not have a name for that AS (ARIN doesn’t publicly share them) and therefore we do not need to waste memory on an empty object.

In your code, you are creating a new AS which is not what you want.

You want to call loc_database_get_as() with the AS number that you got from loc_network_get_asn(). That function with then either return an AS or NULL.

loc_as_get_name will give you the name. So your code might look like this:

struct loc_network* network = loc_database_lookup_from_string(db, "81.3.27.38");
if (network) {
	uint32_t asn = loc_network_get_asn(network);

	if (asn) {
		struct loc_as* a = loc_database_get_as(db, asn);
		if (a) {
			name = loc_as_get_name(a);
		}
	}

	loc_network_unref(network);
}

// Copy the name to somewhere
...

// Cleanup
if (a)
	loc_as_unref(a);


> Everything except 'AS_name' is okay and prints sensible values
> (I've checked all ret-vals). Since there isn't any 'loc_get_asn_by_number()'
> function, what should I do to get a non-NULL 'AS_name' from 'AS_num'?
> 
> PS. do I really need this 'loc_network_unref()'?
>  Or does 'loc_database_lookup()' handle it?

I hope this answers your questions :)

Best,
-Michael

Re: Problem with 'loc_as_get_name()'

[Gisle Vanem]

[-- Attachment #1: Type: text/plain, Size: 1847 bytes --]

Michael Tremer wrote:

> In your code, you are creating a new AS which is not what you want.

I figured that after I sent my previous message.

> struct loc_network* network = loc_database_lookup_from_string(db, "81.3.27.38");

I use 'loc_database_lookup()' directly since I do not want to
waste CPU-cycles on 'inet_pton()'. Works fine for me, except
I must always map an IPv4 to a IPv6-mapped address first.

In my code (and 'location lookup'), e.g. a '37.142.14.15' never works.
But this works '::ffff:37.142.14.15' always:
  Network                 : 37.142.0.0/20
  Country                 : Israel
  Autonomous System       : AS12849 - Hot-Net internet services Ltd.

> // Copy the name to somewhere

I found the AS-name can be quite long:
  AS49450 - Federal State Budget Institution NATIONAL MEDICAL RESEARCH CENTER FOR
            OBSTETRICS, GYNECOLOGY AND PERINATOLOGY named after academician
            V. I. Kulakov of the Ministry of Healthcare of the Russian Federation,

Longest I found, a whopping 214 characters.
I suppose 'libloc' handle any length?

Besides, how can I (if possible with libloc/location.py)
figure out all the Peers for an AS? Like what:
   https://dnslytics.com/bgp/as39029   (Redpill Linpro AS)

reports for "IP Prefixes and Peers". For IPv6:
   8473 Bahnhof AB
   2119 Telenor AS
  56655 TerraHost AS
  ...

If I do:
  location list-networks-by-as --family ipv6 39029
  2001:67c:21e0::/48  << !
  ...
and:
  location list-networks-by-as --family ipv6 8473
  2001:67c:107c::/48
  2001:67c:2fb8::/48  << ! this looks close to 2001:67c:21e0::
  ...

Is the relation with an AS and a peer only a routing (BGP)
thing? Where each of the AS'es have a common routing path?

I'm just beginning to understand all these concepts.
So sorry for nagging..

-- 
--gv

Re: Problem with 'loc_as_get_name()'

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 3193 bytes --]

Hi,

> On 23 Oct 2020, at 17:29, Gisle Vanem <gisle.vanem(a)gmail.com> wrote:
> 
> Michael Tremer wrote:
> 
>> In your code, you are creating a new AS which is not what you want.
> 
> I figured that after I sent my previous message.
> 
>> struct loc_network* network = loc_database_lookup_from_string(db, "81.3.27.38");
> 
> I use 'loc_database_lookup()' directly since I do not want to
> waste CPU-cycles on 'inet_pton()'. Works fine for me, except
> I must always map an IPv4 to a IPv6-mapped address first.
> 
> In my code (and 'location lookup'), e.g. a '37.142.14.15' never works.
> But this works '::ffff:37.142.14.15' always:
> Network                 : 37.142.0.0/20
> Country                 : Israel
> Autonomous System       : AS12849 - Hot-Net internet services Ltd.

We could add a call where you can pass struct in_addr and it automatically converts it.

We usually use the IP address as a string input in our own applications so that this was never necessary.

>> // Copy the name to somewhere
> 
> I found the AS-name can be quite long:
> AS49450 - Federal State Budget Institution NATIONAL MEDICAL RESEARCH CENTER FOR
>           OBSTETRICS, GYNECOLOGY AND PERINATOLOGY named after academician
>           V. I. Kulakov of the Ministry of Healthcare of the Russian Federation,
> 
> Longest I found, a whopping 214 characters.
> I suppose 'libloc' handle any length?

Yes, we do not have any limits on this. It might be good idea to introduce that though. That name is beyond reasonable.

> Besides, how can I (if possible with libloc/location.py)
> figure out all the Peers for an AS? Like what:
>  https://dnslytics.com/bgp/as39029   (Redpill Linpro AS)
> 
> reports for "IP Prefixes and Peers". For IPv6:
>  8473 Bahnhof AB
>  2119 Telenor AS
> 56655 TerraHost AS
> ...

You can use location list-networks-by-as. The code for that should tell you how to part that to your own application. Look for the database iterator.

> If I do:
> location list-networks-by-as --family ipv6 39029
> 2001:67c:21e0::/48  << !
> ...
> and:
> location list-networks-by-as --family ipv6 8473
> 2001:67c:107c::/48
> 2001:67c:2fb8::/48  << ! this looks close to 2001:67c:21e0::
> ...

Yes that might be. Subnets are being allocated like that without any gaps.

> Is the relation with an AS and a peer only a routing (BGP)
> thing? Where each of the AS'es have a common routing path?

I am not sure what you are asking here.

A peer can be in a subnet. That subnet might have an ASN (probably will if it is currently being routed or was recently routed).

If one network is only single-homed to another one is outside the scope of this library.

> I'm just beginning to understand all these concepts.
> So sorry for nagging..

Happy to help. Just try to help me helping you by explaining what your goal is and I can help you finding the best way to that.

It would be great if we could work on upstreaming your changes for Windows so that they become available for everyone and you no longer need to bundle your own version of libloc without receiving bugfixes from upstream.

Best.
-Michael

> 
> -- 
> --gv

Re: Problem with 'loc_as_get_name()'

[Gisle Vanem]

[-- Attachment #1: Type: text/plain, Size: 981 bytes --]

Michael Tremer wrote:

> It would be great if we could work on upstreaming your changes for
> Windows so that they become available for everyone and you no longer need to
> bundle your own version of libloc without receiving bugfixes from upstream.

Hi again. Sorry for the delay.

That would be tricky for me; I'm a n00b when it comes
to Git and pull-requests. But my my Win-sources are under
here:
   https://github.com/gvanem/wsock-trace/tree/master/src/Geo-IP/IPFire/libloc/src

You could perhaps pull them, create a diff and then
email me privately to discuss things if you really
want Windows support. It would be nice to see Location
get a much wider audience (competing with MaxMind and
IP2Location ...)

PS. One thing I see now, trying to support Cygwin too;
Your calls to 'mmap()' fails on Cygwin32 for a reason
I fail to understand; errno = EINVAL. So I use my
Win-mmap emulation for Cygwin too. Works fine.
Will see about Cygwin64.

-- 
--gv

Re: Problem with 'loc_as_get_name()'

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 1858 bytes --]

Hi,

> On 28 Oct 2020, at 11:50, Gisle Vanem <gisle.vanem(a)gmail.com> wrote:
> 
> Michael Tremer wrote:
> 
>> It would be great if we could work on upstreaming your changes for
>> Windows so that they become available for everyone and you no longer need to
>> bundle your own version of libloc without receiving bugfixes from upstream.
> 
> Hi again. Sorry for the delay.
> 
> That would be tricky for me; I'm a n00b when it comes
> to Git and pull-requests. But my my Win-sources are under
> here:
>  https://github.com/gvanem/wsock-trace/tree/master/src/Geo-IP/IPFire/libloc/src

I would recommend to learn Git then. It is everywhere and that is a skill paying off:

  https://wiki.ipfire.org/devel/submit-patches
  https://wiki.ipfire.org/devel/git

> You could perhaps pull them, create a diff and then
> email me privately to discuss things if you really
> want Windows support. It would be nice to see Location
> get a much wider audience (competing with MaxMind and
> IP2Location ...)

I would like to keep this on the list so that others can join in and help to catch any newly introduced bugs early.

And yes, I would like to see support for Windows, too.

> PS. One thing I see now, trying to support Cygwin too;
> Your calls to 'mmap()' fails on Cygwin32 for a reason
> I fail to understand; errno = EINVAL. So I use my
> Win-mmap emulation for Cygwin too. Works fine.
> Will see about Cygwin64.

It might simply be that mmap() is not supported on Win32. I am not sure if that architecture even needs to be supported by us at all. Is there even one officially supported version of Windows that is still 32 bit?

I would prefer to avoid pulling in a third-party wrapper for mmap() if we can find a different alternative. Those things tend to break really quickly.

Best,
-Michael

> 
> -- 
> --gv