Hello list, my ports post.
I'm on Windows and have little interest in installing IPFire or do any Linux related. But the 'libloc' library caught my interest. So I did a port to MSVC, MinGW and clang-cl (x86 only) in a hurry.
Running all the 'test-*' programs, it worked seemingly well. Until I built the Python3 module and played with e.g. 'location dump' and 'verify'. Nothing worked :-( So I reckon the .db file is hosed somehow. AFAICS, it's not an issue with structure packing or 'fopen(path, "r")'.
Anyway, I want to regenerate 'database.db' from 'database.txt' 'countries.txt' and the 'override' files myself. But how?
According to the git log at: git://git.ipfire.org/location/location-database.git
there once was a 'compile-database' script. Not now.
Hello Gisle,
Thank you for your email.
On 19 Oct 2020, at 16:53, Gisle Vanem gisle.vanem@gmail.com wrote:
Hello list, my ports post.
I'm on Windows and have little interest in installing IPFire or do any Linux related. But the 'libloc' library caught my interest. So I did a port to MSVC, MinGW and clang-cl (x86 only) in a hurry.
That sounds great!
We are looking for someone with some experience with Windows to port libloc. It should not be super complicated, but since I am not the expert I am not too sure.
Would you be willing to tidy up your code (if required) and submit them on here so that we can officially support Windows?
Running all the 'test-*' programs, it worked seemingly well. Until I built the Python3 module and played with e.g. 'location dump' and 'verify'. Nothing worked :-(
What didn’t work specifically? Reading and writing the database?
So I reckon the .db file is hosed somehow. AFAICS, it's not an issue with structure packing or 'fopen(path, "r")'.
Hmm… it would be interesting to debug that.
Anyway, I want to regenerate 'database.db' from 'database.txt' 'countries.txt' and the 'override' files myself. But how?
We do not generate the database from database.txt. database.txt is being generated from the database. The text dump is just there so that we have a (sort of) human-readable version which we can diff against the previous one to manually verify any changes.
What you will need to import the raw data is location-importer which comes with libloc and a PostgreSQL server.
location-importer —help will show you how you pass the database credentials.
On our servers we regularly run “location-import update-overrides file-a.txt file-b.txt”, then "location-importer update-whois”, then we run “location-importer update-announcements” which connects to a locally running instance of Bird with a full BGP feed to import those.
Finally we run “location-importer write … database.db” (fill in the gaps with your vendor name, description, key, etc.) and finally you can run “location -d database.db dump” to export it into the text format.
Generating the whole database might take a couple of hours depending on how beefy your machine is.
Let me know if this works for you.
According to the git log at: git://git.ipfire.org/location/location-database.git
there once was a 'compile-database' script. Not now.
Yes, this has all been moved into “location-importer” so that we separate code and data in two different Git repositories.
Best, -Michael
-- --gv
Michael Tremer wrote:
Would you be willing to tidy up your code (if required) and submit them on here so that we can officially support Windows?
I've put a preliminary version of my patched 'libloc' in my 'Wsock-trace' project: https://github.com/gvanem/wsock-trace/tree/master/src/Geo-IP/IPFire/libloc/s...
Running all the 'test-*' programs, it worked seemingly well. Until I built the Python3 module and played with e.g. 'location dump' and 'verify'. Nothing worked :-(
What didn’t work specifically? Reading and writing the database?
'test-database' says: Vendor doesn't match: LOCDBXX☺ != Test Vendor and 'test-network': Could not look up 2001:db8::
I'm not sure it due to my 'mmap()' function. But I noted a case where 'loc_stringpool_get()' could return NULL, this is used in 'strdup()'. Not sure that's legal everywhere. So I did this patch:
--- a/libloc/src/as.c 2020-10-19 17:35:01 +++ b/libloc/src/as.c 2020-10-20 19:37:19 @@ -90,7 +90,7 @@ }
LOC_EXPORT int loc_as_set_name(struct loc_as* as, const char* name) { - as->name = strdup(name); + as->name = name ? strdup(name) : strdup("");
return 0; }
What you will need to import the raw data is location-importer which comes with libloc and a PostgreSQL server.
No! I hoped that 'libloc' would be similar to IP2Location and/or GeoipCSV that I already support in my project. Running PostgreSQL is out of the question for me or any users of 'Wsock-trace'.
Generating the whole database might take a couple of hours depending on how beefy your machine is.
Lacking in Python skills, I cooked up this 'compile-database' script that takes 2 minutes: https://github.com/gvanem/wsock-trace/blob/master/src/Geo-IP/IPFire/database...
But it remains to see if it works. Or if I'll rewrite it into a C-program.
Hey Gisle,
On 20 Oct 2020, at 20:14, Gisle Vanem gisle.vanem@gmail.com wrote:
Michael Tremer wrote:
Would you be willing to tidy up your code (if required) and submit them on here so that we can officially support Windows?
I've put a preliminary version of my patched 'libloc' in my 'Wsock-trace' project: https://github.com/gvanem/wsock-trace/tree/master/src/Geo-IP/IPFire/libloc/s...
Thanks. I will have a look at it.
What is your application you are planning to use this in?
And probably more importantly, why does the database that we provide not work for you?
Running all the 'test-*' programs, it worked seemingly well. Until I built the Python3 module and played with e.g. 'location dump' and 'verify'. Nothing worked :-(
What didn’t work specifically? Reading and writing the database?
'test-database' says: Vendor doesn't match: LOCDBXX☺ != Test Vendor
Oh that sounds interesting.
We have a string pool which starts somewhere towards the end of the file and it looks like you got the first couple of bytes there with the magic value in it.
So reading from the string pool does not work.
and 'test-network': Could not look up 2001:db8::
And I suppose reading other blocks of data doesn’t work either then.
I'm not sure it due to my 'mmap()' function.
We are using simple functions to read and write to memory. No idea why those should behave differently on Windows, but I am sure we will get to the bottom of it.
But I noted a case where 'loc_stringpool_get()' could return NULL, this is used in 'strdup()'. Not sure that's legal everywhere.
loc_stringpool_get() can indeed return NULL, but we should not use that result in strdup() anywhere.
So I did this patch:
--- a/libloc/src/as.c 2020-10-19 17:35:01 +++ b/libloc/src/as.c 2020-10-20 19:37:19 @@ -90,7 +90,7 @@ }
LOC_EXPORT int loc_as_set_name(struct loc_as* as, const char* name) {
as->name = strdup(name);
as->name = name ? strdup(name) : strdup(""); return 0;}
Good catch, but there is more to it. When resetting the name, the original value wasn’t freed. I added that and pushed it to the main repository:
https://git.ipfire.org/?p=location/libloc.git;a=commitdiff;h=a7d3a7a0565a0e0...
What you will need to import the raw data is location-importer which comes with libloc and a PostgreSQL server.
No! I hoped that 'libloc' would be similar to IP2Location and/or GeoipCSV that I already support in my project. Running PostgreSQL is out of the question for me or any users of 'Wsock-trace'.
You won’t need it to make the database. You should simply download the binary version of the library and that will be it.
You will need the development tools old if you want to modify the database and publish your own. But I do not see any reason why anyone should be doing that.
Generating the whole database might take a couple of hours depending on how beefy your machine is.
Lacking in Python skills, I cooked up this 'compile-database' script that takes 2 minutes: https://github.com/gvanem/wsock-trace/blob/master/src/Geo-IP/IPFire/database...
But it remains to see if it works. Or if I'll rewrite it into a C-program.
For performance-reasons Python should be fine.
I am still not sure what you are trying to achieve here. If you are looking for the data, simply run “location update” and the script should download the database, extract it, verify it and you can use it.
Best, -Michael
-- --gv
What is your application you are planning to use this in?
Wsock-trace; a Winsock-trace library like I wrote.
And probably more importantly, why does the database that we provide not work for you?
That's one of the things I'm trying to figure out.
LOC_EXPORT int loc_as_set_name(struct loc_as* as, const char* name) {
as->name = strdup(name);
as->name = name ? strdup(name) : strdup(""); return 0;}
Good catch, but there is more to it. When resetting the name, the original
value wasn’t freed. I added that and pushed it to the main repository:
With that patch, test-as crashes. I did this: --- a/src/as.c 2020-10-20 23:23:34 +++ b/src/as.c 2020-10-20 23:55:51 @@ -142,7 +142,7 @@
int loc_as_match_string(struct loc_as* as, const char* string) { // Match all AS when no search string is set - if (!string) + if (!string || !as->name) return 1;
which no longer crashes. But when building in CRT debug-mode, I note many mem-leaks in e.g. 'test-as.c' and 'test-country.c'.
Some of it is due to the error I get: Could not find country: YY
and you do 'exit(EXIT_FAILURE);' w/o cleaning up. Such a behaviour could probably accumulate leaks over time. But I do not know IPFire.
But I plugged other leaks using: --- a/src/writer.c 2020-10-19 17:35:01 +++ b/src/writer.c 2020-10-20 23:15:20 @@ -155,6 +155,12 @@ if (writer->networks) loc_network_tree_unref(writer->networks);
+ if (writer->as) + free(writer->as); + + if (writer->countries) + free(writer->countries); + // Unref the string pool loc_stringpool_unref(writer->pool);
Never done AFAICS. The leak for 'as.c', I'm not sure how to plug. You ought to check with valgrind etc.
You won’t need it to make the database. You should simply download the binary version of the library and that will be it.
Where? Nothing under https://location.ipfire.org/databases/1/ But a 'adig -t txt _v1._db.location.ipfire.org' says: TXT Fri, 16 Oct 2020 07:54:08 GMT
I fail to find it. But that's another issue.
I am still not sure what you are trying to achieve here. If you are looking for the data, simply run “location update” and the script should download the database, extract it, verify it and you can use it.
Here: py -3 location.py update https://location.ipfire.org/databases/ reported: HTTP Error 404: Not Found Could not download a new database
And have you ever used OpenSSL on Windows? Here: py -3 location.py verify OPENSSL_Uplink(640BC6B0,08): no OPENSSL_Applink
It's pure hell.
Hi,
On 20 Oct 2020, at 23:46, Gisle Vanem gisle.vanem@gmail.com wrote:
What is your application you are planning to use this in?
Wsock-trace; a Winsock-trace library like I wrote.
And probably more importantly, why does the database that we provide not work for you?
That's one of the things I'm trying to figure out.
LOC_EXPORT int loc_as_set_name(struct loc_as* as, const char* name) {
as->name = strdup(name);
as->name = name ? strdup(name) : strdup(""); return 0;}
Good catch, but there is more to it. When resetting the name, the original
value wasn’t freed. I added that and pushed it to the main repository:
With that patch, test-as crashes. I did this: --- a/src/as.c 2020-10-20 23:23:34 +++ b/src/as.c 2020-10-20 23:55:51 @@ -142,7 +142,7 @@
int loc_as_match_string(struct loc_as* as, const char* string) { // Match all AS when no search string is set
if (!string)
if (!string || !as->name) return 1;which no longer crashes. But when building in CRT debug-mode, I note many mem-leaks in e.g. 'test-as.c' and 'test-country.c'.
The testsuite executed on my Debian box I used for testing.
But logically your patch is correct. I will import it. Thank you.
Some of it is due to the error I get: Could not find country: YY
This probably requires some more debugging and log output.
You can set LOC_LOG=7 as an environment variable and then run test/test-country.
You should see some logging output which will be helpful to get to the bottom of this.
and you do 'exit(EXIT_FAILURE);' w/o cleaning up. Such a behaviour could probably accumulate leaks over time. But I do not know IPFire.
Yes it does. But it is the testsuite. We do clean up to test the cleanup routines, but if something goes wrong we terminate the whole process which will free all memory. This code isn’t shipped to the end user systems.
But I plugged other leaks using: --- a/src/writer.c 2020-10-19 17:35:01 +++ b/src/writer.c 2020-10-20 23:15:20 @@ -155,6 +155,12 @@ if (writer->networks) loc_network_tree_unref(writer->networks);
if (writer->as)free(writer->as);if (writer->countries)free(writer->countries);// Unref the string pool loc_stringpool_unref(writer->pool);Never done AFAICS. The leak for 'as.c', I'm not sure how to plug. You ought to check with valgrind etc.
We employed someone to do this, but he said he didn’t find anything like this lol.
Just freeing the arrays is not enough. The allocated objects need to be freed, too.
I will have a look at it.
You won’t need it to make the database. You should simply download the binary version of the library and that will be it.
Where? Nothing under https://location.ipfire.org/databases/1/ But a 'adig -t txt _v1._db.location.ipfire.org' says: TXT Fri, 16 Oct 2020 07:54:08 GMT
I fail to find it. But that's another issue.
I pulled the database last week because we had some issue with some data in it that triggered some bugs.
You can download it from the archive still until the updater works again:
https://location.ipfire.org/databases/1/archive/location-2020-10-16.db.xz
I am still not sure what you are trying to achieve here. If you are looking for the data, simply run “location update” and the script should download the database, extract it, verify it and you can use it.
Here: py -3 location.py update https://location.ipfire.org/databases/ reported: HTTP Error 404: Not Found Could not download a new database
And have you ever used OpenSSL on Windows? Here: py -3 location.py verify OPENSSL_Uplink(640BC6B0,08): no OPENSSL_Applink
It's pure hell.
Yes, I assumed so. There are some tools that use but they might potentially just bundle it with their own software which I want to avoid.
It probably is possible to #ifdef the Windows crypto library here and check the signature that way. But since I have never worked with it, I do not know how to do that. Mac OS X has the same issue.
Have you ever worked with it?
However, OpenSSL is a very good choice in the *nix/BSD world.
Best, -Michael
-- --gv
Michael Tremer wrote:
This probably requires some more debugging and log output.
You can set LOC_LOG=7 as an environment variable and then run test/test-country.
With this, test-country only reports an extra: libloc: loc_new: ctx 05021298 created
But with '-DENABLE_DEBUG' there's way too much trace to be useful.
I pulled the database last week because we had some issue with some data in it that triggered some bugs.
You can download it from the archive still until the updater works again:
https://location.ipfire.org/databases/1/archive/location-2020-10-16.db.xz
Yes, I assumed so. There are some tools that use but they might potentially just bundle it with their own software which I want to avoid.
It probably is possible to #ifdef the Windows crypto library here and check the signature that way. But since I have never worked with it, I do not know how to do that. Mac OS X has the same issue.
An issue with OpenSSL and 'loc_database_verify()'?
Have you ever worked with it?
Yes, but 'test-signature' works fine. It's just when calling 'EVP_VerifyXX()' functions from a .dll (like a Python module), it fails with 'no OPENSSL_AppLink". Ref:
https://stackoverflow.com/questions/38933943/openssl-code-works-natively-but...
With your URL above, set loc_log=7 & py -3 location.py --database location-2020-10-16.db verify:
libloc: loc_new: ctx 0FDACEE8 created libloc: loc_database_read_as_section_v1: Read 48148 ASes from the database libloc: loc_database_read_network_nodes_section_v1: Read 2628939 network nodes from the database libloc: loc_database_read_networks_section_v1: Read 1318972 networks from the database libloc: loc_database_read_countries_section_v1: Read 253 countries from the database libloc: loc_database_read: Opened database in 16.0000ms libloc: loc_database_read_as_section_v1: Read 48148 ASes from the database libloc: loc_database_read_network_nodes_section_v1: Read 2628939 network nodes from the database libloc: loc_database_read_networks_section_v1: Read 1318972 networks from the database libloc: loc_database_read_countries_section_v1: Read 253 countries from the database libloc: loc_database_read: Opened database in 27.0000ms OPENSSL_Uplink(640BC6B0,08): no OPENSSL_Applink
BUT, with 'location-2020-05-15.db', there is this error:
libloc: loc_new: ctx 0FA35190 created libloc: loc_database_read_header: Incompatible database version: 0 Traceback (most recent call last): File "F:\gv\VC_project\ws_trace\src\Geo-IP\IPFire\libloc\src\py3_install\location.py", line 618, in <module> main() File "F:\gv\VC_project\ws_trace\src\Geo-IP\IPFire\libloc\src\py3_install\location.py", line 616, in main c.run() File "F:\gv\VC_project\ws_trace\src\Geo-IP\IPFire\libloc\src\py3_install\location.py", line 200, in run db = location.Database(args.database) SystemError: <class 'location.Database'> returned NULL without setting an error libloc: loc_unref: context 0FA35190 released
----
Check yourself if the 'location-2020-05-15.db.xz' is still in the archive.
Hello,
On 21 Oct 2020, at 12:07, Gisle Vanem gisle.vanem@gmail.com wrote:
Michael Tremer wrote:
This probably requires some more debugging and log output. You can set LOC_LOG=7 as an environment variable and then run test/test-country.
With this, test-country only reports an extra: libloc: loc_new: ctx 05021298 created
But with '-DENABLE_DEBUG' there's way too much trace to be useful.
That is the kind of debug information I would need.
I pulled the database last week because we had some issue with some data in it that triggered some bugs. You can download it from the archive still until the updater works again: https://location.ipfire.org/databases/1/archive/location-2020-10-16.db.xz
Yes, I assumed so. There are some tools that use but they might potentially just bundle it with their own software which I want to avoid. It probably is possible to #ifdef the Windows crypto library here and check the signature that way. But since I have never worked with it, I do not know how to do that. Mac OS X has the same issue.
An issue with OpenSSL and 'loc_database_verify()'?
No, that OpenSSL isn’t easily available on that platform without shipping it manually.
Have you ever worked with it?
Yes, but 'test-signature' works fine. It's just when calling 'EVP_VerifyXX()' functions from a .dll (like a Python module), it fails with 'no OPENSSL_AppLink". Ref: https://stackoverflow.com/questions/38933943/openssl-code-works-natively-but...
With your URL above, set loc_log=7 & py -3 location.py --database location-2020-10-16.db verify:
libloc: loc_new: ctx 0FDACEE8 created libloc: loc_database_read_as_section_v1: Read 48148 ASes from the database libloc: loc_database_read_network_nodes_section_v1: Read 2628939 network nodes from the database libloc: loc_database_read_networks_section_v1: Read 1318972 networks from the database libloc: loc_database_read_countries_section_v1: Read 253 countries from the database libloc: loc_database_read: Opened database in 16.0000ms libloc: loc_database_read_as_section_v1: Read 48148 ASes from the database libloc: loc_database_read_network_nodes_section_v1: Read 2628939 network nodes from the database libloc: loc_database_read_networks_section_v1: Read 1318972 networks from the database libloc: loc_database_read_countries_section_v1: Read 253 countries from the database libloc: loc_database_read: Opened database in 27.0000ms OPENSSL_Uplink(640BC6B0,08): no OPENSSL_Applink
I have absolutely no idea how to fix this unfortunately.
I do not even have access to a Windows system to reproduce this.
BUT, with 'location-2020-05-15.db', there is this error:
libloc: loc_new: ctx 0FA35190 created libloc: loc_database_read_header: Incompatible database version: 0 Traceback (most recent call last): File "F:\gv\VC_project\ws_trace\src\Geo-IP\IPFire\libloc\src\py3_install\location.py", line 618, in <module> main() File "F:\gv\VC_project\ws_trace\src\Geo-IP\IPFire\libloc\src\py3_install\location.py", line 616, in main c.run() File "F:\gv\VC_project\ws_trace\src\Geo-IP\IPFire\libloc\src\py3_install\location.py", line 200, in run db = location.Database(args.database) SystemError: <class 'location.Database'> returned NULL without setting an error libloc: loc_unref: context 0FA35190 released
This is a small bug where we do not raise the error message with Python.
However, the debug output should tell you why this didn’t work.
The database version that should be shown there is 1. You have a zero.
Check yourself if the 'location-2020-05-15.db.xz' is still in the archive.
Did you extract the file?
Best, -Michael
-- --gv
-
Gisle Vanem -
Michael Tremer