October 2019 - Location - lists.ipfire.org

libloc: Approaching feature-completeness
by Michael Tremer 04 Oct '19

04 Oct '19

Good morning ladies and gentlemen, I would like to bring everyone up to speed since we have made huge progress on libloc and I would like to keep this momentum! What has happened in the last couple of weeks? Stefan has been working on Perl bindings which are essential to launch this on IPFire 2. The web UI makes a lot of use of looking up IP addresses and displaying the correct flag. That needs to be fast and therefore we could not rely on a CLI wrapper for location-query(8). The Perl module is super-lightweight and does not implement all features of libloc. Mainly it is ready-only since we do not need another tool to compose a database. For our purposes this is sufficient and easier to maintain. If anyone else needs more features, they can always be added later. The other thing I have been working on (with a lot of sweat, blood and tears) is walking through the tree to search it. This has been an outstanding issue for a long time and was fairly difficult to implement since a standard DFS was not possible to use. We have encoded information into the edges to store less information and this made it extremely difficult to extract again - on top of that comes that the implementation could not be recursive to be used in the Python wrapper. We now have a re-entrant search which performs super fast and uses very little memory. The depth of the tree is limited to 128, although we would never get too close to that. That results in a maximum stack size of 256 + 1 nodes and we walk through every edge and node only once which results in a performance of O(1) for computing and memory. It could not get better than that. Finally, I have extended the Python bindings to make use of all of this so that all networks belonging to an AS can now be listed with “list-networks-by-as” and all networks belonging to a country can be listed with “list-networks-by-cc”. I have added a man-page for location-query(8) which can be found here: https://man-pages.ipfire.org/libloc/location-query.html As far as I can see this now, we are approaching to be feature-complete for a 1.0 release. Of course there is still a long way to test this and make sure that we do not have any bugs in the database format or the code. Another major road block is that we do not have a database that is ready for prime time, yet, but the good news is that the toolkit is almost there! Best, -Michael

1 0

[PATCHv2] perl: Testsuite: Add tests for lookup_asn()
by Stefan Schantl 02 Oct '19

02 Oct '19

Signed-off-by: Stefan Schantl <stefan.schantl(a)ipfire.org> --- src/perl/t/Location.t | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/src/perl/t/Location.t b/src/perl/t/Location.t index 55a18f3..fd43946 100644 --- a/src/perl/t/Location.t +++ b/src/perl/t/Location.t @@ -11,7 +11,7 @@ use warnings; # Where to find the test database. my $testdb = $ENV{'database'}; -use Test::More tests => 5; +use Test::More tests => 6; BEGIN { use_ok('Location') }; ######################### @@ -42,3 +42,12 @@ if(defined($country_code)) { fail("Test 5 - Lookup country code for address not $country_code = &Location::lookup_country_code($db, "a.b.c.d"); if(defined($country_code)) { fail("Test 6 - Lookup country code for invalid address.") } + +my $as_number = &Location::lookup_asn($db, $address); +ok($as_number eq "204867", "Test 7 - Lookup Autonomous System Number for $address."); + +$as_number = &Location::lookup_asn($db, "1.1.1.1"); +if(defined($as_number)) { fail("Test 8 - Lookup Autonomous System Number for address not in Database.") } + +$as_number = &Location::lookup_asn($db, "a.b.c.d"); +if(defined($as_number)) { fail("Test 9 - Lookup Autonomous System Number for invalid address.") } -- 2.20.1

1 0

[PATCHv2] perl: Add function to perform AS number lookups.
by Stefan Schantl 02 Oct '19

02 Oct '19

Signed-off-by: Stefan Schantl <stefan.schantl(a)ipfire.org> --- src/perl/Location.xs | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/src/perl/Location.xs b/src/perl/Location.xs index cdbc7e7..b6c4dc5 100644 --- a/src/perl/Location.xs +++ b/src/perl/Location.xs @@ -113,6 +113,29 @@ lookup_country_code(db, address) OUTPUT: RETVAL +SV* +lookup_asn(db, address) + struct loc_database* db; + char* address; + + CODE: + RETVAL = &PL_sv_undef; + + // Lookup network + struct loc_network *network; + int err = loc_database_lookup_from_string(db, address, &network); + if (!err) { + // Extract the ASN + unsigned int as_number = loc_network_get_asn(network); + if (as_number > 0) { + RETVAL = newSViv(as_number); + } + + loc_network_unref(network); + } + OUTPUT: + RETVAL + void DESTROY(db) struct loc_database* db; -- 2.20.1

1 0

[PATCH 1/2] perl: Fix lookup if given address is invalid or not in DB.
by Stefan Schantl 02 Oct '19

02 Oct '19

In this case now undef will be returned. Signed-off-by: Stefan Schantl <stefan.schantl(a)ipfire.org> --- src/perl/Location.xs | 20 ++++++++------------ 1 file changed, 8 insertions(+), 12 deletions(-) diff --git a/src/perl/Location.xs b/src/perl/Location.xs index ab27f0b..cdbc7e7 100644 --- a/src/perl/Location.xs +++ b/src/perl/Location.xs @@ -92,27 +92,23 @@ get_license(db) # # Lookup functions # -char* +SV* lookup_country_code(db, address) struct loc_database* db; char* address; CODE: + RETVAL = &PL_sv_undef; + // Lookup network struct loc_network *network; int err = loc_database_lookup_from_string(db, address, &network); - if (err) { - croak("Could not look up for %s\n", address); - } - - // Extract the country code - const char* country_code = loc_network_get_country_code(network); - loc_network_unref(network); + if (!err) { + // Extract the country code + const char* country_code = loc_network_get_country_code(network); + RETVAL = newSVpv(country_code, strlen(country_code)); - if (country_code) { - RETVAL = strdup(country_code); - } else { - RETVAL = NULL; + loc_network_unref(network); } OUTPUT: RETVAL -- 2.20.1

1 1

How to update the database daily?
by Michael Tremer 02 Oct '19

02 Oct '19

Hello, I just wanted to summarise and archive a conversation that I had with Peter yesterday on the phone: We were talking about what mechanisms would be best to distribute the database on a daily basis. There are two options of which the first one is a downloader script which is called by systemd or cron on a regular basis, downloads the database, verifies it and done. The second option is to use the package updating mechanism of distributions, i.e. sending a daily update of a package with the latest version of the database. That way, no extra configuration for systems behind a proxy and without any access to the internet would be necessary. I personally do not think that any distribution would be able or willing to push a daily package update regardless of its size. There is no precedent for it in any distribution I know, but Peter was going to check with Debian if this is a possible path and report back. Best, -Michael

1 0

[PATCH] perl: Add function to perform AS number lookups.
by Stefan Schantl 01 Oct '19

01 Oct '19

Signed-off-by: Stefan Schantl <stefan.schantl(a)ipfire.org> --- src/perl/Location.xs | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/src/perl/Location.xs b/src/perl/Location.xs index ab27f0b..63410c4 100644 --- a/src/perl/Location.xs +++ b/src/perl/Location.xs @@ -117,6 +117,32 @@ lookup_country_code(db, address) OUTPUT: RETVAL +unsigned int +lookup_asn(db, address) + struct loc_database* db; + char* address; + + CODE: + // Lookup network + struct loc_network *network; + int err = loc_database_lookup_from_string(db, address, &network); + if (err) { + croak("Could not look up for %s\n", address); + } + + // Extract the ASN + unsigned int as_number = loc_network_get_asn(network); + loc_network_unref(network); + + if (as_number) { + RETVAL = as_number; + } else { + //RETVAL = NULL; + RETVAL = 0; + } + OUTPUT: + RETVAL + void DESTROY(db) struct loc_database* db; -- 2.20.1

2 1

[PATCH] perl: Testsuite: Add test for lookup_asn()
by Stefan Schantl 01 Oct '19

01 Oct '19

Signed-off-by: Stefan Schantl <stefan.schantl(a)ipfire.org> --- src/perl/t/Location.t | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/perl/t/Location.t b/src/perl/t/Location.t index aef15c3..af5e258 100644 --- a/src/perl/t/Location.t +++ b/src/perl/t/Location.t @@ -11,7 +11,7 @@ use warnings; # Where to find the test database. my $testdb = "../../testdata/test.db"; -use Test::More tests => 5; +use Test::More tests => 6; BEGIN { use_ok('Location') }; ######################### @@ -36,3 +36,6 @@ ok($description eq "This is a geo location database", "Test 3 - Get Database Des my $country_code = &Location::lookup_country_code($db, $address); ok($country_code eq "DE", "Test 4 - Lookup country code for $address"); + +my $as_number = &Location::lookup_asn($db, $address); +ok($as_number eq "204867", "Test 5 - Lookup Autonomous System for $address"); -- 2.20.1

2 1

[PATCH] perl: Add tests to testsuite.
by Stefan Schantl 01 Oct '19

01 Oct '19

Signed-off-by: Stefan Schantl <stefan.schantl(a)ipfire.org> --- src/perl/t/Location.t | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/src/perl/t/Location.t b/src/perl/t/Location.t index a68c094..aef15c3 100644 --- a/src/perl/t/Location.t +++ b/src/perl/t/Location.t @@ -8,10 +8,31 @@ use strict; use warnings; -use Test::More tests => 1; +# Where to find the test database. +my $testdb = "../../testdata/test.db"; + +use Test::More tests => 5; BEGIN { use_ok('Location') }; ######################### # Insert your test code below, the Test::More module is use()ed here so read # its man page ( perldoc Test::More ) for help writing this test script. + +# Address which should be used for database lookup. +my $address = "2a07:1c44:5800::1"; + +# Connect to the database. +my $db = &Location::init("$testdb"); + +my $vendor = &Location::get_vendor($db); +ok($vendor eq "IPFire Project", "Test 1 - Get Database Vendor"); + +my $license = &Location::get_license($db); +ok($license eq "CC", "Test 2 - Get Database license"); + +my $description = &Location::get_description($db); +ok($description eq "This is a geo location database", "Test 3 - Get Database Description"); + +my $country_code = &Location::lookup_country_code($db, $address); +ok($country_code eq "DE", "Test 4 - Lookup country code for $address"); -- 2.20.1

1 0

[PATCH] Add example database for testing purposes.
by Stefan Schantl 01 Oct '19

01 Oct '19

This example database and it's content can be used for various testing purposes. The testsuite of the perl bindings depends on it. Signed-off-by: Stefan Schantl <stefan.schantl(a)ipfire.org> --- testdata/Readme.txt | 12 ++++++++++++ testdata/test.db | Bin 0 -> 16459 bytes 2 files changed, 12 insertions(+) create mode 100644 testdata/Readme.txt create mode 100644 testdata/test.db diff --git a/testdata/Readme.txt b/testdata/Readme.txt new file mode 100644 index 0000000..25e7f59 --- /dev/null +++ b/testdata/Readme.txt @@ -0,0 +1,12 @@ +The database in this folder can be used to run various tests against it and +has been created by using the "create-database.py" script in the examples directory. + +It contains the following content: + +Vendor: "IPFire Project" +Description: "This is a geo location database" +License: "CC" + +AS: "AS204867 (Lightning Wire Labs GmbH)" +Subnet: "2a07:1c44:5800::/40" +Country Code: "DE" diff --git a/testdata/test.db b/testdata/test.db new file mode 100644 index 0000000000000000000000000000000000000000..5840aca7fa48bed4459803c6005a3e7ac2eea146 GIT binary patch literal 16459 zcmeI&NlpSm6oBDEoF{OerJUJm+`2I~PLV_wCbDtSU<cYLoj~usgJ<&?-a+eW#k{a} zL*%cJ)ZdTlPV=oP-v*_#YSqYkcsqIj4r6j|KQtyUm!rlU`Q5rb^I^=XT;CL5Wn&?L z00IagfB*srAb<b@2q1t!&kJO2;NJ1L=iieTL;wK<5I_I{1Q0*~0R#|00D-><WaQ>g zx7;?MS~BbBgI)6>KOgRzkNCO!&%p09$5fB2o=`ohdP=qAw11{_PIcb*jC|pHR`r}} z$$53}g6c)zOX~cx>J`<JtN#3&ny;(gP%XLX&ll9(s@_s9x$V#Ibj^4DT*iC8_tlvL z)kW2kECdih009ILKmY**5I_KdJ{KsRJAj}60Q9*8KOX@E5I_I{1Q0*~0R#|00D=E2 za9z2G+mWrblc%Vjn!9G)vF;PvMwHm+q#mYm(z1_X8rH&2WP-qy<3=-W#jS?DmpjX0 Ltz$1=YFFk9;D*Fd literal 0 HcmV?d00001 -- 2.20.1

1 0

[PATCH] examples: python/create-database.py: Add license to database
by Stefan Schantl 01 Oct '19

01 Oct '19

Signed-off-by: Stefan Schantl <stefan.schantl(a)ipfire.org> --- examples/python/create-database.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/examples/python/create-database.py b/examples/python/create-database.py index 284c85f..651709c 100644 --- a/examples/python/create-database.py +++ b/examples/python/create-database.py @@ -10,6 +10,9 @@ w.vendor = "IPFire Project" # Set a description w.description = "This is a geo location database" +# Set a license +w.license = "CC" + # Add an AS a = w.add_as(204867) a.name = "Lightning Wire Labs GmbH" -- 2.20.1

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Location October 2019