From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: location@lists.ipfire.org Subject: Re: What generates database.db from database.txt? Date: Tue, 20 Oct 2020 21:49:28 +0100 Message-ID: In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============5552399386226271214==" List-Id: --===============5552399386226271214== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hey Gisle, > On 20 Oct 2020, at 20:14, Gisle Vanem wrote: >=20 > Michael Tremer wrote: >=20 >> Would you be willing to tidy up your code (if required) and submit them on= here so that we can officially support Windows? >=20 > I've put a preliminary version of my patched 'libloc' > in my 'Wsock-trace' project: > https://github.com/gvanem/wsock-trace/tree/master/src/Geo-IP/IPFire/libloc= /src Thanks. I will have a look at it. What is your application you are planning to use this in? And probably more importantly, why does the database that we provide not work= for you? >>> Running all the 'test-*' programs, it worked seemingly >>> well. Until I built the Python3 module and played with >>> e.g. 'location dump' and 'verify'. Nothing worked :-( >> What didn=E2=80=99t work specifically? Reading and writing the database? >=20 > 'test-database' says: > Vendor doesn't match: LOCDBXX=E2=98=BA !=3D Test Vendor Oh that sounds interesting. We have a string pool which starts somewhere towards the end of the file and = it looks like you got the first couple of bytes there with the magic value in= it. So reading from the string pool does not work. > and 'test-network': > Could not look up 2001:db8:: And I suppose reading other blocks of data doesn=E2=80=99t work either then. > I'm not sure it due to my 'mmap()' function. We are using simple functions to read and write to memory. No idea why those = should behave differently on Windows, but I am sure we will get to the bottom= of it. > But I noted a case where 'loc_stringpool_get()' could > return NULL, this is used in 'strdup()'. Not sure that's > legal everywhere. loc_stringpool_get() can indeed return NULL, but we should not use that resul= t in strdup() anywhere. > So I did this patch: >=20 > --- a/libloc/src/as.c 2020-10-19 17:35:01 > +++ b/libloc/src/as.c 2020-10-20 19:37:19 > @@ -90,7 +90,7 @@ > } >=20 > LOC_EXPORT int loc_as_set_name(struct loc_as* as, const char* name) { > - as->name =3D strdup(name); > + as->name =3D name ? strdup(name) : strdup(""); >=20 > return 0; > } Good catch, but there is more to it. When resetting the name, the original va= lue wasn=E2=80=99t freed. I added that and pushed it to the main repository: https://git.ipfire.org/?p=3Dlocation/libloc.git;a=3Dcommitdiff;h=3Da7d3a7a056= 5a0e09d3442e5829a0f30f016993b9 >=20 >> What you will need to import the raw data is location-importer which comes= with libloc and a PostgreSQL server. >=20 > No! I hoped that 'libloc' would be similar to > IP2Location and/or GeoipCSV that I already support > in my project. Running PostgreSQL is out of the question > for me or any users of 'Wsock-trace'. You won=E2=80=99t need it to make the database. You should simply download th= e binary version of the library and that will be it. You will need the development tools old if you want to modify the database an= d publish your own. But I do not see any reason why anyone should be doing th= at. >=20 >> Generating the whole database might take a couple of hours >> depending on how beefy your machine is. >=20 > Lacking in Python skills, I cooked up this 'compile-database' > script that takes 2 minutes: > https://github.com/gvanem/wsock-trace/blob/master/src/Geo-IP/IPFire/databa= se/compile-database.py >=20 > But it remains to see if it works. Or if I'll rewrite it > into a C-program. For performance-reasons Python should be fine. I am still not sure what you are trying to achieve here. If you are looking f= or the data, simply run =E2=80=9Clocation update=E2=80=9D and the script shou= ld download the database, extract it, verify it and you can use it. Best, -Michael > --=20 > --gv --===============5552399386226271214==--