- Location - lists.ipfire.org

A quick status update about the location database
by Peter Müller 24 Jan '18

24 Jan '18

Hello, just a quick status update about the location database ("libloc"). I did some research about how (and where) to gather the data for it the other day. Basically, we just need these information: - IPv4/IPv6 network range - assigned country code - Autonomous System Number (ASN) the network belongs to - description of the network [helpful, but optional in first place] The RIPE provides a documentation about the different files on their FTP server here: https://www.ripe.net/manage-ips-and-asns/db/support/documentation/update-re… The database (either split or en bloc) can be downloaded here: https://ftp.ripe.net/ripe/dbase/ According to the first link, the files ripe.db.inetnum and ripe.db.inet6num contain the allocations of the IP address space. This does not mean that they are actually announced via BGP at the moment, but we agreed not to care about that since a network might be down for maintenance or some other reasons. However, only the network ranges and country codes can be extracted from inet(6)num directly. It seems like we have to look at the route database for anything else (ASN, description), which is located at ripe.db.route. The network range needs to be converted to CIDR first, since the formats differ between inet(6)num and route. But since an AS might announce bigger ranges (say a /15) than the network's size itself (i.e. /29), it might make sense to extract the route announcements in fist place. We already know ASN and some descriptions then. After that, inet(6)num can be parsed to gather networks in that range, and to extract their actual network range and country code. I think it makes sense to keep the description of the AS set here. The last two steps consist in checking if either the complete ASN or a certain network is listed in one of the override files. Finally, the stuff can be dumped into the libloc database. Apart from LACNIC (Latin America and Caribbean), every RIR offers a complete database for download: - ARIN: https://ftp.arin.net/pub/rr/arin.db - AFRINIC: https://ftp.afrinic.net/pub/dbase/afrinic.db.gz - APNIC: https://ftp.apnic.net/apnic/whois/ (split DB only) - RIPE: https://ftp.ripe.net/ripe/dbase/ripe.db.gz LACNIC only provides a list of IP ranges (and decimal encoded subnets - *ugh*) and country codes over here: http://ftp.lacnic.net/pub/stats/lacnic/delegated-lacnic-extended-latest I have no idea at the moment how to enumerate ASN and ASdescr for them... Overall, the RIPE DB looks quite tidy, but we always have a huge number of side effects (one network range is announced by two ASNs, ...) making thinks ugly. I tried to write a Python script for the jobs listed above, but eventually failed since my programming skills are very limited and I needed some layer (perhaps a SQLite DB?) to store partial information. Unfortunately, it seems like that job is much more complicated than I initially thought. Does anybody have an idea about how to store incomplete information here? Let's say, we already know ASN, description and CIDR, but neither all possible networks in that range nor their country... Any help is highly appreciated. Thanks in advance. Best regards, Peter Müller

2 1

[PATCH] add override country code to A1 overrides
by Peter Müller 18 Jan '18

18 Jan '18

Adding a "newcountry" flag for each AS (no IP ranges currently listed here) in the A1 override file. This makes parsing of several override files much easier, since we do not need to care about file names, for example. Signed-off-by: Peter Müller <peter.mueller(a)link38.eu> --- country-overrides/override-a1.txt | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/country-overrides/override-a1.txt b/country-overrides/override-a1.txt index edb91e0..3de3001 100644 --- a/country-overrides/override-a1.txt +++ b/country-overrides/override-a1.txt @@ -19,15 +19,19 @@ net: AS10914 descr: Cypherpunk Privacy Network remarks: VPN provider +newcountry: A1 net: AS37560 descr: Cyberdyne S.A. remarks: VPN and Tor relay provider +newcountry: A1 net: AS396507 descr: Emerald Onion remarks: Tor relay provider +newcountry: A1 net: AS51852 descr: Private Layer INC remarks: VPN provider +newcountry: A1 -- 2.13.6

1 0

[PATCH] A1 overrides: Sort them by ASN
by Peter Müller 16 Jan '18

16 Jan '18

Signed-off-by: Peter Müller <peter.mueller(a)link38.eu> --- country-overrides/override-a1.txt | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/country-overrides/override-a1.txt b/country-overrides/override-a1.txt index a99acf8..edb91e0 100644 --- a/country-overrides/override-a1.txt +++ b/country-overrides/override-a1.txt @@ -16,18 +16,18 @@ # generated by another script. # -net: AS37560 -descr: Cyberdyne S.A. -remarks: VPN and Tor relay provider - net: AS10914 descr: Cypherpunk Privacy Network remarks: VPN provider -net: AS51852 -descr: Private Layer INC -remarks: VPN provider +net: AS37560 +descr: Cyberdyne S.A. +remarks: VPN and Tor relay provider net: AS396507 descr: Emerald Onion remarks: Tor relay provider + +net: AS51852 +descr: Private Layer INC +remarks: VPN provider -- 2.13.6

1 0

[PATCH] country override files: inital checkin
by Peter Müller 07 Jan '18

07 Jan '18

Initially create manual override files for country data. Signed-off-by: Peter Müller <peter.mueller(a)link38.eu> --- country-overrides/override-a1.txt | 18 ++++++++++++++++++ country-overrides/override-a2.txt | 11 +++++++++++ country-overrides/override-a3.txt | 10 ++++++++++ country-overrides/override-other.txt | 12 ++++++++++++ 4 files changed, 51 insertions(+) create mode 100644 country-overrides/override-a1.txt create mode 100644 country-overrides/override-a2.txt create mode 100644 country-overrides/override-a3.txt create mode 100644 country-overrides/override-other.txt diff --git a/country-overrides/override-a1.txt b/country-overrides/override-a1.txt new file mode 100644 index 0000000..08eabfd --- /dev/null +++ b/country-overrides/override-a1.txt @@ -0,0 +1,18 @@ +# +# override-a1 [.txt] +# +# This file contains Autonomous Systems (AS) or IP networks/adresses +# which are - in fact or with a high level of confidence - anonymous +# proxies (special country code: A1). +# +# Since it does not make sense to assign them to a county, they +# will be flagged as "A1" in the database. +# +# Although we do not consider them to be bad entirely, they might +# be unwanted in certain scenarios. +# +# Please note only long-living Tor relay providers with static IPs +# are listed here, as the list of all Tor relays will be dynamically +# generated by another script. +# + diff --git a/country-overrides/override-a2.txt b/country-overrides/override-a2.txt new file mode 100644 index 0000000..2ce00d0 --- /dev/null +++ b/country-overrides/override-a2.txt @@ -0,0 +1,11 @@ +# +# override-a2 [.txt] +# +# This file contains Autonomous Systems (AS) or IP networks/addresses +# which are - in fact or with a high level of confidence - belonging +# to satellite network providers (special country code: A2). +# +# Since a satellite uplink connection is possible from almost +# anywhere in the world, it does not make sense to assign them to a +# specific country. They will be flagged as "A2" in the database. +# diff --git a/country-overrides/override-a3.txt b/country-overrides/override-a3.txt new file mode 100644 index 0000000..4fbcdeb --- /dev/null +++ b/country-overrides/override-a3.txt @@ -0,0 +1,10 @@ +# +# override-a3 [.txt] +# +# This file contains Autonomous Systems (AS) or IP networks/addresses +# which are - in fact or with a high level of confidence - believed +# to be worldwide anycast instances (special country codes: A3). +# +# It does not make sense to assign them to a certain country, they +# will be flagged as "A3" in the database. +# diff --git a/country-overrides/override-other.txt b/country-overrides/override-other.txt new file mode 100644 index 0000000..d7ba5b9 --- /dev/null +++ b/country-overrides/override-other.txt @@ -0,0 +1,12 @@ +# +# override-other [.txt] +# +# This file contains Autonomous Systems (AS) or IP networks/addresses +# whose country information in the WHOIS is not identical to their +# physical location and which do not match to one of the special categories +# A[1-3]. +# +# Such networks might be legitimate (poorly maintained WHOIS data) +# or hostile (faked WHOIS data in order to bypass country filtering or +# to hide the owners' identity). +# -- 2.13.6

2 3

[PATCH] A1 override: add AS396507 (Emerald Onion)
by Peter Müller 07 Jan '18

07 Jan '18

Signed-off-by: Peter Müller <peter.mueller(a)link38.eu> --- country-overrides/override-a1.txt | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/country-overrides/override-a1.txt b/country-overrides/override-a1.txt index 0936994..a99acf8 100644 --- a/country-overrides/override-a1.txt +++ b/country-overrides/override-a1.txt @@ -27,3 +27,7 @@ remarks: VPN provider net: AS51852 descr: Private Layer INC remarks: VPN provider + +net: AS396507 +descr: Emerald Onion +remarks: Tor relay provider -- 2.13.6

2 2

[PATCH] A1 override: add AS51852 (Private Layer INC)
by Peter Müller 19 Dec '17

19 Dec '17

Signed-off-by: Peter Müller <peter.mueller(a)link38.eu> --- country-overrides/override-a1.txt | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/country-overrides/override-a1.txt b/country-overrides/override-a1.txt index 985b9a4..0936994 100644 --- a/country-overrides/override-a1.txt +++ b/country-overrides/override-a1.txt @@ -23,3 +23,7 @@ remarks: VPN and Tor relay provider net: AS10914 descr: Cypherpunk Privacy Network remarks: VPN provider + +net: AS51852 +descr: Private Layer INC +remarks: VPN provider -- 2.13.6

1 0

[PATCH] A1 override: add AS10914 (Cypherpunk Privacy Network)
by Peter Müller 19 Dec '17

19 Dec '17

Signed-off-by: Peter Müller <peter.mueller(a)link38.eu> --- country-overrides/override-a1.txt | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/country-overrides/override-a1.txt b/country-overrides/override-a1.txt index ac67184..985b9a4 100644 --- a/country-overrides/override-a1.txt +++ b/country-overrides/override-a1.txt @@ -19,3 +19,7 @@ net: AS37560 descr: Cyberdyne S.A. remarks: VPN and Tor relay provider + +net: AS10914 +descr: Cypherpunk Privacy Network +remarks: VPN provider -- 2.13.6

1 0

[PATCH] A1 override: add AS37560 (Cyberdyne S.A.)
by Peter Müller 19 Dec '17

19 Dec '17

Signed-off-by: Peter Müller <peter.mueller(a)link38.eu> --- country-overrides/override-a1.txt | 3 +++ 1 file changed, 3 insertions(+) diff --git a/country-overrides/override-a1.txt b/country-overrides/override-a1.txt index 08eabfd..ac67184 100644 --- a/country-overrides/override-a1.txt +++ b/country-overrides/override-a1.txt @@ -16,3 +16,6 @@ # generated by another script. # +net: AS37560 +descr: Cyberdyne S.A. +remarks: VPN and Tor relay provider -- 2.13.6

1 0

C library and database design
by Michael Tremer 11 Dec '17

11 Dec '17

Hello, I have had a little bit of a head start and implemented a few things that I wanted to present here: I built a C library that is supposed to implement the core functionality or reading and writing the database as well as performing the lookups. https://git.ipfire.org/?p=people/ms/libloc.git;a=summary So far there is an implementation of a string pool which will later hold all strings like the names of countries and ASes, etc. The pool will probably at the end of the database and keeps all strings separated by a NULL byte. Therefore it is easy to just jump to the right place and read the string from there until you find NULL. That makes a string lookup perform in O(1) no matter how large the database is. When you add a string to the pool it is checking for duplicates so that adding the same string twice will make the pool only store it once and return the same address both times. This operation is O(n) but that should be fine since the database is not going to be very very large (i.e. gigabytes) and we write it once and read if very often which makes this a good optimisation. We could potentially compress the string pool later if we needed to, but so far I didn't see the point there and it is kind of nice to being able to open the database in a hex editor and see what is going on inside. So what does the database look like to far? [root@rice-oxley libloc]# hexdump -C test.db 00000000 4c 4f 43 44 42 58 58 00 00 00 00 00 00 00 00 0c |LOCDBXX.........| 00000010 00 00 00 18 00 00 01 55 54 65 73 74 20 56 65 6e |.......UTest Ven| 00000020 64 6f 72 00 4c 6f 72 65 6d 20 69 70 73 75 6d 20 |dor.Lorem ipsum | 00000030 64 6f 6c 6f 72 20 73 69 74 20 61 6d 65 74 2c 20 |dolor sit amet, | 00000040 63 6f 6e 73 65 63 74 65 74 75 72 20 61 64 69 70 |consectetur adip| 00000050 69 73 63 69 6e 67 20 65 6c 69 74 2e 20 50 72 6f |iscing elit. Pro| 00000060 69 6e 20 75 6c 74 72 69 63 65 73 20 70 75 6c 76 |in ultrices pulv| 00000070 69 6e 61 72 20 64 6f 6c 6f 72 2c 20 65 74 20 73 |inar dolor, et s| 00000080 6f 6c 6c 69 63 69 74 75 64 69 6e 20 65 72 6f 73 |ollicitudin eros| 00000090 20 75 6c 74 72 69 63 69 65 73 20 76 69 74 61 65 | ultricies vitae| 000000a0 2e 20 4e 61 6d 20 69 6e 20 76 6f 6c 75 74 70 61 |. Nam in volutpa| 000000b0 74 20 6c 69 62 65 72 6f 2e 20 4e 75 6c 6c 61 20 |t libero. Nulla | 000000c0 66 61 63 69 6c 69 73 69 2e 20 50 65 6c 6c 65 6e |facilisi. Pellen| 000000d0 74 65 73 71 75 65 20 74 65 6d 70 6f 72 20 66 65 |tesque tempor fe| 000000e0 6c 69 73 20 65 6e 69 6d 2e 20 49 6e 74 65 67 65 |lis enim. Intege| 000000f0 72 20 63 6f 6e 67 75 65 20 6e 69 73 69 20 69 6e |r congue nisi in| 00000100 20 6d 61 78 69 6d 75 73 20 70 72 65 74 69 75 6d | maximus pretium| 00000110 2e 20 50 65 6c 6c 65 6e 74 65 73 71 75 65 20 65 |. Pellentesque e| 00000120 74 20 74 75 72 70 69 73 20 65 6c 65 6d 65 6e 74 |t turpis element| 00000130 75 6d 2c 20 6c 75 63 74 75 73 20 6d 69 20 61 74 |um, luctus mi at| 00000140 2c 20 69 6e 74 65 72 64 75 6d 20 65 72 61 74 2e |, interdum erat.| 00000150 20 4d 61 65 63 65 6e 61 73 20 75 74 20 76 65 6e | Maecenas ut ven| 00000160 65 6e 61 74 69 73 20 6e 75 6e 63 2e 00 |enatis nunc..| The database starts with a magic value which I have set to LOCDBXX at the moment and we need to find a better one probably. After that there is a version field that we change if the format changes. So we can have any changes of the database format later if we need to. Then, there is a pointer to a string of the "vendor" of the database. That allows us to set where the database is from and there is also a "description" pointer where we can just write some text and put useful information about the database. The pointers just point to a string in the string pool. Do we need more like those? I could imaging license and URL where this database came from? But we could put this into the description, too. Thoughts? Then there is an offset where the string pool starts in the file (i.e. at which byte) and the length of the string pool. There is no trailer yet to make sure that the database hasn't been altered. I guess we should do that with some sort of a signature inside the database. But I haven't really thought about that too much. I want to involve everyone into the process of designing this and also peer-review as much as possible so that we really have a smart and flexible design that won't create any future problems. The library will get some bindings for other programming languages later. That allows us to use it easily in Python, Perl and what else people use. I have good experience with writing the Python bindings. We will need perl for using this database in IPFire 2. Anything else should be contributed by third parties. The library itself is under LGPLv2.1 or later now which allows closed source applications to use it as well. It has some unit tests to load the database, etc. So far I am only depending on the C standard library and it should stay as that as long as we can because that makes porting the database to other OSes easier. If we want to use compression we might need other libraries and of course the language bindings need that their respective libraries. So this is where I am now. I haven't really made a plan for the next steps. So please send me comments and suggestions and I will try to draft out something. Best, -Michael

2 2

Welcome!
by Michael Tremer 08 Dec '17

08 Dec '17

I just wanted to briefly say hello and introduce this list to everyone. Since we have recently talked about attempting to build an "Internet location database", this is the place where we will have all the discussion about design, implementation and probably even bugs and support. So please feel free to post! Best, -Michael

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Location