Problem in urllib?

Problem in urllib?

[Gisle Vanem]

[-- Attachment #1: Type: text/plain, Size: 1218 bytes --]

I have a problem with he Python-module
and downloading a new .db-file. Used as:

   py -3 location.py --debug update

always returns:
   HTTP GET Request to location.ipfire.org
           URL: https://location.ipfire.org/databases/1/location.db.xz
           Headers:
                 If-modified-since: Tue, 30 Mar 2021 04:04:14 GMT
                 User-agent: location/0.9.4
   HTTP Response: 410
         Headers:
                 date: Tue, 30 Mar 2021 08:32:16 GMT
                 content-length: 282
                 content-type: text/html; charset=iso-8859-1
                 strict-transport-security: max-age=31536000; includeSubDomains; preload
                 connection: close
   https://location.ipfire.org/databases/ reported: HTTP Error 410: Gone
   Could not download a new database

But when using wget or Chrome, the above URL downloads just fine.

Maybe there is a problem with urllib in Windows' version of
Python 3.6.5? Or it's a server and CA-path problem since my
curl says:
  subject alt name(s) or common name do not match "location.ipfire.org"

(but a 'curl -k ..' works).

How can I solve this? I've tried several ca-bundle files to no avail.

-- 
--gv

Re: Problem in urllib?

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 1438 bytes --]

Hello,

Please just update to the 0.9.5 release and this problem will be fixed.

Best,
-Michael

> On 30 Mar 2021, at 10:08, Gisle Vanem <gisle.vanem(a)gmail.com> wrote:
> 
> I have a problem with he Python-module
> and downloading a new .db-file. Used as:
> 
>  py -3 location.py --debug update
> 
> always returns:
>  HTTP GET Request to location.ipfire.org
>          URL: https://location.ipfire.org/databases/1/location.db.xz
>          Headers:
>                If-modified-since: Tue, 30 Mar 2021 04:04:14 GMT
>                User-agent: location/0.9.4
>  HTTP Response: 410
>        Headers:
>                date: Tue, 30 Mar 2021 08:32:16 GMT
>                content-length: 282
>                content-type: text/html; charset=iso-8859-1
>                strict-transport-security: max-age=31536000; includeSubDomains; preload
>                connection: close
>  https://location.ipfire.org/databases/ reported: HTTP Error 410: Gone
>  Could not download a new database
> 
> But when using wget or Chrome, the above URL downloads just fine.
> 
> Maybe there is a problem with urllib in Windows' version of
> Python 3.6.5? Or it's a server and CA-path problem since my
> curl says:
> subject alt name(s) or common name do not match "location.ipfire.org"
> 
> (but a 'curl -k ..' works).
> 
> How can I solve this? I've tried several ca-bundle files to no avail.
> 
> -- 
> --gv

Re: Problem in urllib?

[Gisle Vanem]

[-- Attachment #1: Type: text/plain, Size: 1725 bytes --]

Michael Tremer wrote:

> Please just update to the 0.9.5 release and this problem will be fixed.

I seems to have that; git describe --tags
0.9.5-9-g1814283

It was only that the '0.9.4' was a left-over in my
build. What specific change after 0.9.4 did fix this?

But rebuilding such that 'User-agent: location/0.9.5',
and a little path to 'python/database.c', it seems to
work better:
...
HTTP Response: 200
         Headers:
                 date: Tue, 30 Mar 2021 10:52:03 GMT
                 last-modified: Tue, 30 Mar 2021 05:06:07 GMT
                 etag: "425158-5beb9f4bc1867"
                 accept-ranges: bytes
                 content-length: 4346200
                 x-content-type-options: nosniff
                 x-frame-options: deny
                 referrer-policy: strict-origin
                 x-xss-protection: 1; mode=block
                 content-type: application/x-xz
                 strict-transport-security: max-age=31536000; includeSubDomains; preload
                 connection: close
Downloaded new database from Tue, 30 Mar 2021 05:04:14 GMT
Traceback (most recent call last):
   File "f:\ProgramFiler\Python36\lib\shutil.py", line 544, in move
     os.rename(src, real_dst)
FileExistsError: [WinError 183] Kan ikke opprette en fil når
filen allerede finnes: 'f:\\gv\\VC_project\\ws_trace\\tmp82
8kmqif' -> 'f:\\gv\\VC_project\\ws_trace\\IPFire-database.db'

-----------

Some issue with shutil when overwriting/renaming an existing
.db-file. But updating w/o a .db-file works fine.

PS. it would be nice that 'location.py --debug' would honour
the setting of 'LOC_LOG'. w/o this it's very hard to see
what goes on in the Python module.

Re: Problem in urllib?

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 2468 bytes --]

Hello,

> On 30 Mar 2021, at 12:14, Gisle Vanem <gisle.vanem(a)gmail.com> wrote:
> 
> Michael Tremer wrote:
> 
>> Please just update to the 0.9.5 release and this problem will be fixed.
> 
> I seems to have that; git describe --tags
> 0.9.5-9-g1814283
> 
> It was only that the '0.9.4' was a left-over in my
> build. What specific change after 0.9.4 did fix this?

This was a server-side change where we blocked clients with version 0.9.4 or older to download the databases. They have been changed which could have caused some damage on some systems, so we took this precautionary step.

> But rebuilding such that 'User-agent: location/0.9.5',
> and a little path to 'python/database.c', it seems to
> work better:
> ...
> HTTP Response: 200
>        Headers:
>                date: Tue, 30 Mar 2021 10:52:03 GMT
>                last-modified: Tue, 30 Mar 2021 05:06:07 GMT
>                etag: "425158-5beb9f4bc1867"
>                accept-ranges: bytes
>                content-length: 4346200
>                x-content-type-options: nosniff
>                x-frame-options: deny
>                referrer-policy: strict-origin
>                x-xss-protection: 1; mode=block
>                content-type: application/x-xz
>                strict-transport-security: max-age=31536000; includeSubDomains; preload
>                connection: close
> Downloaded new database from Tue, 30 Mar 2021 05:04:14 GMT
> Traceback (most recent call last):
>  File "f:\ProgramFiler\Python36\lib\shutil.py", line 544, in move
>    os.rename(src, real_dst)
> FileExistsError: [WinError 183] Kan ikke opprette en fil når
> filen allerede finnes: 'f:\\gv\\VC_project\\ws_trace\\tmp82
> 8kmqif' -> 'f:\\gv\\VC_project\\ws_trace\\IPFire-database.db'
> 
> -----------
> 
> Some issue with shutil when overwriting/renaming an existing
> .db-file. But updating w/o a .db-file works fine.

I cannot exactly decipher why Windows would complain here. We are replacing the file because we cannot change it when it is open using mmap(). Maybe Windows does not allow that?

> PS. it would be nice that 'location.py --debug' would honour
> the setting of 'LOC_LOG'. w/o this it's very hard to see
> what goes on in the Python module.

You can always set LOC_LOG=debug, but you will need to build the library with debugging to see more output.

The Python code should always print debugging information if enabled.

-Michael

Re: Problem in urllib?

[Gisle Vanem]

[-- Attachment #1: Type: text/plain, Size: 564 bytes --]

Michael Tremer wrote:

>> Some issue with shutil when overwriting/renaming an existing
>> .db-file. But updating w/o a .db-file works fine.
> 
> I cannot exactly decipher why Windows would complain here. We are replacing the file because we cannot change it when it is open using mmap(). Maybe Windows does not allow that?

It was Python's fault; 'os.chmod()' with those bits, made the
file read-only. When I did this patch:
  + if sys.platform != "win32":
      os.chmod(t.name, stat.S_IRUSR|stat.S_IRGRP|stat.S_IROTH)

it works fine.


-- 
--gv