From mboxrd@z Thu Jan  1 00:00:00 1970
From: Michael Tremer <michael.tremer@ipfire.org>
To: location@lists.ipfire.org
Subject: Re: What generates database.db from database.txt?
Date: Wed, 21 Oct 2020 10:34:19 +0100
Message-ID: <339DD4F5-353D-4466-A59F-DCB43289F22B@ipfire.org>
In-Reply-To: <6ec26d86-805d-dde9-dfbf-db2fbd1c29f5@gmail.com>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============2003747100128718582=="
List-Id: <location.lists.ipfire.org>

--===============2003747100128718582==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

Hi,

> On 20 Oct 2020, at 23:46, Gisle Vanem <gisle.vanem(a)gmail.com> wrote:
>=20
>> What is your application you are planning to use this in?
>=20
> Wsock-trace; a Winsock-trace library like I wrote.
>=20
>> And probably more importantly, why does the database that we provide not w=
ork for you?
>=20
> That's one of the things I'm trying to figure out.
>=20
>>> LOC_EXPORT int loc_as_set_name(struct loc_as* as, const char* name) {
>>> -       as->name =3D strdup(name);
>>> +       as->name =3D name ? strdup(name) : strdup("");
>>>=20
>>>        return 0;
>>> }
>> Good catch, but there is more to it. When resetting the name, the original
> value wasn=E2=80=99t freed. I added that and pushed it to the main reposito=
ry:
>=20
> With that patch, test-as crashes. I did this:
> --- a/src/as.c 2020-10-20 23:23:34
> +++ b/src/as.c 2020-10-20 23:55:51
> @@ -142,7 +142,7 @@
>=20
> int loc_as_match_string(struct loc_as* as, const char* string) {
>        // Match all AS when no search string is set
> -       if (!string)
> +       if (!string || !as->name)
>                return 1;
>=20
> which no longer crashes. But when building in CRT debug-mode,
> I note many mem-leaks in e.g. 'test-as.c' and 'test-country.c'.

The testsuite executed on my Debian box I used for testing.

But logically your patch is correct. I will import it. Thank you.

> Some of it is due to the error I get:
>  Could not find country: YY

This probably requires some more debugging and log output.

You can set LOC_LOG=3D7 as an environment variable and then run test/test-cou=
ntry.

You should see some logging output which will be helpful to get to the bottom=
 of this.

> and you do 'exit(EXIT_FAILURE);' w/o cleaning up.
> Such a behaviour could probably accumulate leaks
> over time. But I do not know IPFire.

Yes it does. But it is the testsuite. We do clean up to test the cleanup rout=
ines, but if something goes wrong we terminate the whole process which will f=
ree all memory. This code isn=E2=80=99t shipped to the end user systems.

> But I plugged other leaks using:
> --- a/src/writer.c 2020-10-19 17:35:01
> +++ b/src/writer.c 2020-10-20 23:15:20
> @@ -155,6 +155,12 @@
>        if (writer->networks)
>                loc_network_tree_unref(writer->networks);
>=20
> +       if (writer->as)
> +               free(writer->as);
> +
> +       if (writer->countries)
> +               free(writer->countries);
> +
>        // Unref the string pool
>        loc_stringpool_unref(writer->pool);
>=20
> Never done AFAICS. The leak for 'as.c', I'm not sure
> how to plug. You ought to check with valgrind etc.

We employed someone to do this, but he said he didn=E2=80=99t find anything l=
ike this lol.

Just freeing the arrays is not enough. The allocated objects need to be freed=
, too.

I will have a look at it.

>> You won=E2=80=99t need it to make the database.
>> You should simply download the binary version of the library and that will=
 be it.
>=20
> Where? Nothing under https://location.ipfire.org/databases/1/
> But a 'adig -t txt _v1._db.location.ipfire.org' says:
>  TXT Fri, 16 Oct 2020 07:54:08 GMT
>=20
> I fail to find it. But that's another issue.

I pulled the database last week because we had some issue with some data in i=
t that triggered some bugs.

You can download it from the archive still until the updater works again:

  https://location.ipfire.org/databases/1/archive/location-2020-10-16.db.xz

>=20
>> I am still not sure what you are trying to achieve here. If you are lookin=
g for the data, simply run =E2=80=9Clocation update=E2=80=9D and the script s=
hould download the database, extract it, verify it and you can use it.
>=20
> Here:
>  py -3 location.py update
>  https://location.ipfire.org/databases/ reported: HTTP Error 404: Not Found
>  Could not download a new database
>=20
> And have you ever used OpenSSL on Windows? Here:
>  py -3 location.py verify
>  OPENSSL_Uplink(640BC6B0,08): no OPENSSL_Applink
>=20
> It's pure hell.

Yes, I assumed so. There are some tools that use but they might potentially j=
ust bundle it with their own software which I want to avoid.

It probably is possible to #ifdef the Windows crypto library here and check t=
he signature that way. But since I have never worked with it, I do not know h=
ow to do that. Mac OS X has the same issue.

Have you ever worked with it?

However, OpenSSL is a very good choice in the *nix/BSD world.

Best,
-Michael

>=20
> --=20
> --gv


--===============2003747100128718582==--