- Location - lists.ipfire.org

Next steps of IPFire Location
by Michael Tremer 28 Nov '20

28 Nov '20

Hello, Since there are now so many people involved in this project, I would like to send a little email to report what has happened and what the next steps are going to be… I have spent (unfortunately) a lot of time to add the ability that we can “flatten the tree” when we export it. That means that no network will overlap with another one and we do not need to care about any orders of iptables rules being added. That was quite a slow process in Python and since IPFire has to run on small hardware it needed to be ported into C. The tree can now be exported in about 10 seconds on my development box which is running an Intel Xeon E5-2630 processor. I hope that we can reach acceptable times of only a few minutes on the slowest systems and way under a minute on an average system. To find out about that, I need you help: I just pushed all changes into the next tree and would everyone who can to install that and report how long this command runs: mkdir tmp time location export --directory=tmp --format=xt_geoip It would also be great if you could all check the output if this is still correct. I have done what I could, but I want to have an extra pair of eyes if possible. If this works well, and no more regressions are being found, I will tag this release and we will close Core Update 153. We will then roll out all recent changes that make the database better, but had to be held back because the tree needs to be flattened for most use-cases now. All older clients will then no longer receive an updated database any more. Happy testing. Best, -Michael

3 8

Re: Next steps of IPFire Location
by Michael Tremer 26 Nov '20

26 Nov '20

Hello Arne, Thank you for spotting this. I have fixed the issue with some of the networks not being sorted correctly. Since this has affected performance slightly, the code has now become slightly faster again. Regarding the 32 bit issue: This is a pain to debug now because I do not have a working 32 bit build environment with a debugger. I am compiling this now and I will work on it tomorrow. Your export of A1 was incorrect because it wasn’t exported at all. That must have been old data because I forgot to copy the flags, too. Best, -Michael > On 26 Nov 2020, at 07:51, Arne Fitzenreiter <arne_f(a)ipfire.org> wrote: > > Bad news with current nightly. The export segfault on 32bit > > Also the exported data on RPi3 was still not in > the correct order. In my export the fourth entry of A1.ipv4 is lower than the third. > > Arne > > Am 2020-11-25 21:24, schrieb Michael Tremer: >> Ladies and gentlemen, >> After days and days of looking into this, and neglecting all the more >> important things I should be doing, I have updated libloc in the >> testing tree. >> The good news is that I think that I have fixed the issue. I would >> like you all to check this for me if you could :) >> The bad news is that the reason the output was incorrect was that the >> more complicated code path was not being used when it was required. >> This has been fixed now and it is actually fast enough I think. That >> again has to be tested. >> If you all would repeat the last test with the updated version as soon >> as it finishes its build. >> It will run for many many minutes, if not an hour. I want to know that >> figure. We will bring it down. >> Best, >> -Michael >>> On 23 Nov 2020, at 11:16, Michael Tremer <michael.tremer(a)ipfire.org> wrote: >>> Hi, >>> Yes, I can confirm this. The subnets returned are not unique. >>> -Michael >>>> On 22 Nov 2020, at 11:59, Peter Müller <peter.mueller(a)ipfire.org> wrote: >>>> Hello *, >>>> I just stumbled across a strange issue on my testing machine strongly reminding me of a problem with the xt_geoip module if we printed the location database contents non-planar: >>>> For debugging and monitoring reasons, a NRPE job pinging ping.ipfire.org (which resolves to 81.3.27.38) is executed periodically. Since a few hours, this job fails due to ICMP packets to this IP address being dropped by the outgoing firewall engine. >>>> Its policy is set to "drop", with a rule allowing ICMP traffic to DE. While "location lookup 81.3.27.38" correctly returns DE for the prefix 31.8.0.0/18, it seems as the xt_geoip module did not get that and misclassifies this IP address. >>>> Is somebody able to verify or falsify this issue on Core Development Build 153? >>>> Thanks, and best regards, >>>> Peter Müller

1 0

Re: Next steps of IPFire Location
by Michael Tremer 23 Nov '20

23 Nov '20

Thank you for all your feedback. It helped a lot. Or rather… It would have if Arne didn’t find that this whole algorithm doesn’t work :) It simply does not do what we expect it to do and after my fix this takes a long time again. I hope that I will find some time this week to improve the implementation further and will get back to you all very soon :) Best, -Michael > On 22 Nov 2020, at 20:06, Adolf Belka <ahb.ipfire(a)gmail.com> wrote: > > Dear All, > > > I have the results now for the location testing on my Intel(R) Celeron(R) CPU J1900 @ 1.99GHz production box. > > Stable > real 0m37.240s > user 0m37.086s > sys 0m0.070s > > > Unstable > real 0m25.827s > user 0m25.568s > sys 0m0.123s > > > Regards, > > Adolf. > > On 19/11/2020 18:48, Adolf Belka wrote: >> I have tested it out on my IPFire virtual machine test bed. >> >> I got between 4.8 and 5 secs. >> >> The computer is an i5-8400 CPU @ 2.80GHz >> >> I will also be testing it out on my production box which is a Intel(R) Celeron(R) CPU J1900 @ 1.99GHz but I have to prepare for that as I will need to re-install Core152 after carrying out the test. Will email the results when I have them. >> >> Regards, >> Adolf. >> >> >> On 19/11/2020 14:10, Michael Tremer wrote: >>> I managed to export the database on my IPFire Mini Appliance in about one minute: >>> >>> [root@fw01 libloc]# time location export --directory=tmp --format=xt_geoip >>> >>> real 1m3.423s >>> user 1m1.470s >>> sys 0m1.009s >>> >>> The system was under a little bit of load due to some network activity, but I am generally quite happy with this. >>> >>> I added some more fine-tuning and could decrease the export time on my Debian test machine by another 40% to around six seconds. >>> >>> -Michael >>> >>>> On 18 Nov 2020, at 19:26, Michael Tremer <michael.tremer(a)ipfire.org> wrote: >>>> >>>> Hello, >>>> >>>> Since there are now so many people involved in this project, I would like to send a little email to report what has happened and what the next steps are going to be… >>>> >>>> I have spent (unfortunately) a lot of time to add the ability that we can “flatten the tree” when we export it. That means that no network will overlap with another one and we do not need to care about any orders of iptables rules being added. >>>> >>>> That was quite a slow process in Python and since IPFire has to run on small hardware it needed to be ported into C. The tree can now be exported in about 10 seconds on my development box which is running an Intel Xeon E5-2630 processor. >>>> >>>> I hope that we can reach acceptable times of only a few minutes on the slowest systems and way under a minute on an average system. To find out about that, I need you help: >>>> >>>> I just pushed all changes into the next tree and would everyone who can to install that and report how long this command runs: >>>> >>>> mkdir tmp >>>> time location export --directory=tmp --format=xt_geoip >>>> >>>> It would also be great if you could all check the output if this is still correct. I have done what I could, but I want to have an extra pair of eyes if possible. >>>> >>>> If this works well, and no more regressions are being found, I will tag this release and we will close Core Update 153. >>>> >>>> We will then roll out all recent changes that make the database better, but had to be held back because the tree needs to be flattened for most use-cases now. All older clients will then no longer receive an updated database any more. >>>> >>>> Happy testing. >>>> >>>> Best, >>>> -Michael >>>

1 0

[PATCH] countries: treat Cyprus as being part of the European area
by Peter Müller 16 Nov '20

16 Nov '20

While this country is politically split, it's IP space is maintained by RIPE, which is why we have to tag it as "EU" rather than "AS" to ensure ipinfo.cgi asking the right WHOIS servers. Fixes: #12524 Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- countries.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/countries.txt b/countries.txt index ced8aa9..faaa902 100644 --- a/countries.txt +++ b/countries.txt @@ -54,7 +54,7 @@ CU NA Cuba CV AF Cabo Verde CW SA Curaçao CX AS Christmas Island -CY AS Cyprus +CY EU Cyprus CZ EU Czechia DE EU Germany DJ AF Djibouti -- 2.26.2

1 0

[PATCH] override-{a[1-3],other}: regular batch of various overrides
by Peter Müller 16 Nov '20

16 Nov '20

Since the "Asline" IP hijacking gang tampers with RIR data, probably to evade location based firewall rules, their Autonomous Systems were pinned to the AP region (the given Hong Kong contact address seems to be bogus for at least one /16 stolen AFRINIC chunk) for safety reasons. Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- overrides/override-a1.txt | 25 +++++++++++++++---------- overrides/override-a2.txt | 12 ++++++++++++ overrides/override-a3.txt | 5 +++++ overrides/override-other.txt | 35 +++++++++++++++++++++++++++++++++++ 4 files changed, 67 insertions(+), 10 deletions(-) diff --git a/overrides/override-a1.txt b/overrides/override-a1.txt index e81d6c2..7aca339 100644 --- a/overrides/override-a1.txt +++ b/overrides/override-a1.txt @@ -596,16 +596,6 @@ descr: ThinkTech Technology Industrial CO. Limited remarks: VPN provider is-anonymous-proxy: yes -net: 94.199.160.0/23 -descr: MIK Telecom VPN pool -remarks: VPN provider -is-anonymous-proxy: yes - -net: 95.129.56.0/21 -descr: Azimut-R VPN Service -remarks: VPN provider -is-anonymous-proxy: yes - net: 91.193.75.0/24 descr: KGB Hosting d.o.o. / David Craig remarks: (Rogue) VPN provider @@ -616,6 +606,21 @@ descr: Privax LTD remarks: VPN provider is-anonymous-proxy: yes +net: 92.118.39.0/24 +descr: CloudMine NET +remarks: VPN provider [high confidence, but not proofed] +is-anonymous-proxy: yes + +net: 94.199.160.0/23 +descr: MIK Telecom VPN pool +remarks: VPN provider +is-anonymous-proxy: yes + +net: 95.129.56.0/21 +descr: Azimut-R VPN Service +remarks: VPN provider +is-anonymous-proxy: yes + net: 95.154.64.0/18 descr: Octopusnet VPN remarks: VPN provider diff --git a/overrides/override-a2.txt b/overrides/override-a2.txt index 8f03159..a55c940 100644 --- a/overrides/override-a2.txt +++ b/overrides/override-a2.txt @@ -152,6 +152,12 @@ descr: Arab Satellite Communications Organization remarks: Satellite Internet provider is-satellite-provider: yes +aut-num: AS42962 +descr: CoreLink Communications +remarks: Chinese satellite Internet provider [high confidence, but not proofed] +is-satellite-provider: yes +country: AP + aut-num: AS43905 descr: Telenor Satellite AS remarks: Satellite Internet provider @@ -1616,3 +1622,9 @@ net: 2a04:2880::/30 descr: Satellite Solutions Worldwide Ltd remarks: Satellite Internet provider is-satellite-provider: yes + +net: 2a0a:2840::/29 +descr: CoreLink Communications +remarks: Chinese satellite Internet provider [high confidence, but not proofed] +is-satellite-provider: yes +country: AP diff --git a/overrides/override-a3.txt b/overrides/override-a3.txt index 924c859..07b2621 100644 --- a/overrides/override-a3.txt +++ b/overrides/override-a3.txt @@ -1527,6 +1527,11 @@ descr: marbis GmbH remarks: Generic anycast network [high confidence, but not proofed] is-anycast: yes +net: 2a05:7f00::/29 +descr: nic.at GmbH and friends +remarks: TLD operator's anycast network +is-anycast: yes + net: 2a06:e881:4001::/48 descr: Thomas Harwood remarks: Public anycast DNS resolver diff --git a/overrides/override-other.txt b/overrides/override-other.txt index d4c3f5b..98ea79b 100644 --- a/overrides/override-other.txt +++ b/overrides/override-other.txt @@ -18,6 +18,16 @@ descr: Iron Mountain Data Center remarks: ISP located in US, but some RIR data for announced prefixes contain garbage country: US +aut-num: AS18013 +descr: ASLINE LIMITED +remarks: IP hijacker, traces back to AP region +country: AP + +aut-num: AS18254 +descr: KLAYER LLC +remarks: part of the "Asline" IP hijacking gang, traces back to AP region +country: AP + aut-num: AS24700 descr: Yes Networks Unlimited Ltd remarks: traces to UA, but some RIR entries seem to contain garbage (VG) @@ -33,6 +43,11 @@ descr: IP Interactive UG (haftungsbeschraenkt) remarks: ISP located in BG, but RIR data for announced prefixes contain garbage country: BG +aut-num: AS35478 +descr: Buena Telecom SRL +remarks: ISP located in RO, but RIR data for announced prefixes contain garbage +country: RO + aut-num: AS37518 descr: Fiber Grid Inc. remarks: tampers with RIR data, traces back to SE @@ -73,6 +88,11 @@ descr: PPTECHNOLOGY LIMITED remarks: bulletproof ISP (related to AS204655) located in NL country: NL +aut-num: AS49466 +descr: KLAYER LLC +remarks: part of the "Asline" IP hijacking gang, traces back to AP region +country: AP + aut-num: AS49505 descr: Selectel remarks: ISP located in RU, but some RIR data for announced prefixes contain garbage @@ -108,6 +128,11 @@ descr: DXTL Tseung Kwan O Service remarks: tampers with RIR data, traces back to AP region country: AP +aut-num: AS137951 +descr: Clayer Limited +remarks: part of the "Asline" IP hijacking gang, tampers with RIR data, traces back to AP region +country: AP + aut-num: AS201133 descr: Verdina Ltd. remarks: ISP located in BG, but RIR data for announced prefixes contain garbage @@ -138,6 +163,11 @@ descr: Altrosky Technology Ltd. remarks: fake offshore location (SC), traces back to CZ and NL country: EU +aut-num: AS208046 +descr: Maximilian Kutzner trading as HostSlick +remarks: traces back to NL, but some RIR data for announced prefixes contain garbage +country: NL + aut-num: AS209132 descr: Alviva Holding Limited remarks: ISP located in BG, but RIR data for announced prefixes contain garbage @@ -158,6 +188,11 @@ descr: IP Connect Inc. remarks: fake offshore location (SC), traces back to NL country: NL +aut-num: AS398478 +descr: PEG TECH INC +remarks: ISP located in HK, tampers with RIR data +country: HK + net: 5.252.32.0/22 descr: StormWall s.r.o. remarks: claims to be located in DE, but traces back to somewhere else in central Europe -- 2.26.2

1 0

[PATCH] location-importer.in: always convert organisation handles into upper cases
by Peter Müller 03 Nov '20

03 Nov '20

Fixes: #12523 Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- src/python/location-importer.in | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/python/location-importer.in b/src/python/location-importer.in index 864eab1..2dec89e 100644 --- a/src/python/location-importer.in +++ b/src/python/location-importer.in @@ -560,7 +560,7 @@ class CLI(object): autnum["asn"] = m.group(2) elif key == "org": - autnum[key] = val + autnum[key] = val.upper() # Skip empty objects if not autnum: @@ -646,7 +646,9 @@ class CLI(object): # Split line key, val = split_line(line) - if key in ("organisation", "org-name"): + if key == "organisation": + org[key] = val.upper() + elif key == "org-name": org[key] = val # Skip empty objects -- 2.20.1

1 0

Thoughts about (automated) countermeasures against bogus countries
by Peter Müller 31 Oct '20

31 Oct '20

Hello location folks, since I happen to be the "master of disaster" when it comes to the actual contents of the location database, I observe networks whose country code is inaccurate on an almost daily basis. Once a week (or in a similar interval), I batch them into a single commit and send them to this list, as you probably all have noticed by now. Besides things like being an anonymous proxy, a satellite dial-up network, or an anycasted network service (which all cannot be very reliably determined automatically), there is a significant amount of inaccurate countries - on purpose or by chance, such as bypassing location-based firewall rules. Since it has never been clarified whether that country refers to the actual physical location of the machines behind that network or its jurisdiction, we just take the country information we are able to obtain, and do not make any legal/political/... decisions there on our own. We simply have no alternative. However, there are some examples of forging country information (let's put it that way) I would like to make stop: (a) Country codes of unpopulated islands The Bouvet Island in the Atlantic Ocean is a notable example of a ISO-3166-x country code (BV) which should not show up in RIR data: Literally nobody is using IP networks on that island, since it has no inhabitants who might use it. Same goes for AQ (Antarctica): While there might be some scientists there, they would probably not delegate a /24 or a /48 to this country - in fact, I would be surprised to hear that anything else than a satellite uplink is possible there. (b) Networks registered to offshore letterbox companies Especially folks with a higher need of privacy (not necessarily malicious, although there are some bulletproof ISPs or professional IP hijackers in this group) register IP networks to letterbox companies in offshore locations, such as the Seychelles (SC). What annoys me here is some of those operators put int obviously unhelpful country information into their networks, so we have things like a network used in Romania (RO), but, according to it's RIR data, is "located" in an offshore jurisdiction. We have agreed to correct such data manually, as they do not provide helpful information to our users. This usually involves running "mtr" against that network and try to find out which country we end up in - thanks to all backbone operators who care about informative PTRs for their core infrastructure. :-) (c) Highly obvious example of forged country information A popular example of this category is a Russian ISP tagging it's networks as being located in the United States (US), since there is no reasonable way of blocking traffic from or to this country entirely. I am currently not aware of an opposite example. While this is not that easy to spot if the original and forged country are geographically located close to each other (e.g.: Russia and Finland), we might be able to automate this by running traceroutes into those networks, and try to find out which countries we traverse - if they are located in a completely different continent than we expect the network to be located in, things start to look suspicious. At the time of writing, my ideas about detecting those cases automatically are as follows: (a) Reject or flag networks located in certain countries automatically In my humble opinion, it is safe to consider any network located in BV as being suspicious. We could flag those to indicate they probably need further investigation (preferred), or reject them entirely while building our database. (b) Determine contact details for networks in certain countries and flag known letterbox companies At least for some offshore locations (such as Seychelles [SC] or Panama [PA]), some networks located there are indeed used by local telcos, so we cannot apply (a) as such to those countries. However, a bunch of letterbox companies tends to trace back to the same postal address, so we might be able to detect those networks with a reasonable sensitivity by check their contact details against lists of known letterboxes (the ICIJ offshore leaks come to mind). This, however, is time-consuming and does not work for regions covered by LACNIC and ARIN. (c) Try to detect traceroute anomalies I've discussed this idea with Michael several times on the phone, and while it is certainly not an elegant solution, I have not came up with a better solution at the time of writing. The technical setup would be as follows: - An unmetered server runs a traceroute against a given networks. - The hops are parsed into a list of IP addresses. - The location results for the last two or three hops are resolved. In case they are not part of the network in question (possibly not even the same AS) and differ completely from what we are expecting (i.e. a different country), the network is being flagged. For the records, I wanted to document those ideas here. In case anybody is willingly to discuss them, has a better idea or some comments on his/her own, I would be delighted to hear of them. Thanks, and best regards, Peter Müller P.S.: At the time of writing, the "AQ" and "BV" networks in question are as follows: > location=# SELECT * FROM networks WHERE country = 'AQ' ORDER BY family(network), masklen(network); > network | country > ---------------------+--------- > 185.192.56.0/22 | AQ > 202.144.198.0/23 | AQ > 139.28.204.0/23 | AQ > 156.0.201.0/24 | AQ > 185.121.177.0/24 | AQ > 2a0a:2840::/30 | AQ > 2a0a:2844::/30 | AQ > 2a0e:46c6::/40 | AQ > 2a07:1c44:1000::/40 | AQ > 2a07:1c44:6700::/40 | AQ > 2a0a:2846:230::/44 | AQ > 2a0e:b107:d10::/44 | AQ > 2a0c:3b80:4151::/48 | AQ > 2a0c:3b80:6171::/48 | AQ > 2a0c:b641:21e::/48 | AQ > 2a0d:1a40:cafe::/48 | AQ > 2a0d:1a45:128::/48 | AQ > 2a0d:1a45:3213::/48 | AQ > 2a0e:b107:b7f::/48 | AQ > 2a05:dfc7:5::/48 | AQ > 2a05:dfc7:5353::/48 | AQ > 2a05:dfc7:beef::/48 | AQ > 2a05:dfc7:dfc7::/48 | AQ > 2a05:dfc7:dfc8::/48 | AQ > 2a06:1e86:5555::/48 | AQ > 2a07:a905:ffe8::/48 | AQ > 2a0e:b107:198::/48 | AQ > (27 rows) > location=# SELECT * FROM networks WHERE country = 'BV' ORDER BY family(network), masklen(network); > network | country > ---------------------+--------- > 185.193.126.0/23 | BV > 80.78.16.0/24 | BV > 185.193.124.0/24 | BV > 185.193.125.0/24 | BV > 2a0c:3b80:4256::/48 | BV > 2a0c:3b80:6276::/48 | BV > (6 rows)

1 0

[PATCH 1/2] overrides/overrides-{a{1,3},other}: weekly batch of override updates
by Peter Müller 30 Oct '20

30 Oct '20

Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- overrides/override-a1.txt | 10 ++++++++++ overrides/override-a3.txt | 5 +++++ overrides/override-other.txt | 15 +++++++++++++++ 3 files changed, 30 insertions(+) diff --git a/overrides/override-a1.txt b/overrides/override-a1.txt index a9fdf96..5fb75cc 100644 --- a/overrides/override-a1.txt +++ b/overrides/override-a1.txt @@ -384,6 +384,11 @@ descr: VPN Consumer Network remarks: VPN provider is-anonymous-proxy: yes +net: 45.74.0.0/18 +descr: Secure Internet LLC +remarks: VPN provider +is-anonymous-proxy: yes + net: 45.151.115.0/24 descr: ikoProxies [high confidence, but not proofed] remarks: VPN provider located in NL @@ -545,6 +550,11 @@ descr: SecuredConnectivity remarks: VPN provider is-anonymous-proxy: yes +net: 69.171.214.0/24 +descr: icedoutproxies.com +remarks: VPN provider [high confidence, but not proofed] +is-anonymous-proxy: yes + net: 80.254.74.0/20 descr: Monzoon / SwissVPN remarks: VPN provider [high confidence, but not proofed] diff --git a/overrides/override-a3.txt b/overrides/override-a3.txt index eb17927..d98544f 100644 --- a/overrides/override-a3.txt +++ b/overrides/override-a3.txt @@ -50,6 +50,11 @@ descr: Canadian Internet Registration Authority (CIRA) remarks: Generic anycast network is-anycast: yes +aut-num: AS42388 +descr: ANEXIA Internetdienstleistungs GmbH +remarks: Public anycast DNS nameserver network [high confidence, but not proofed] +is-anycast: yes + aut-num: AS48550 descr: Pascal Mathis trading as SnapServ Mathis remarks: Generic anycast network diff --git a/overrides/override-other.txt b/overrides/override-other.txt index ce936aa..93e1780 100644 --- a/overrides/override-other.txt +++ b/overrides/override-other.txt @@ -23,6 +23,11 @@ descr: Yes Networks Unlimited Ltd remarks: traces to UA, but some RIR entries seem to contain garbage (VG) country: UA +aut-num: AS28098 +descr: ABGON Comunicaciones +remarks: ISP located in CL, but some RIR data for announced prefixes contain garbage (BZ) +country: CL + aut-num: AS35042 descr: IP Interactive UG (haftungsbeschraenkt) remarks: ISP located in BG, but RIR data for announced prefixes contain garbage @@ -53,11 +58,21 @@ descr: UKSERVERS remarks: ISP located in GB, but some RIR data for announced prefixes contain garbage country: GB +aut-num: AS44446 +descr: OOO SibirInvest +remarks: bulletproof ISP (related to AS204655), traces back to NL and BG +country: EU + aut-num: AS48090 descr: PPTECHNOLOGY LIMITED remarks: bulletproof ISP (related to AS204655) located in NL country: NL +aut-num: AS49505 +descr: Selectel +remarks: ISP located in RU, but some RIR data for announced prefixes contain garbage +country: RU + aut-num: AS51558 descr: Smart Telecom S.A.R.L remarks: tampers with RIR data, traces back to RU -- 2.26.2

1 1

Problem with 'loc_as_get_name()'
by Gisle Vanem 28 Oct '20

28 Oct '20

I've changed the subject since it's now off-topic (solved that part). > And a CRT is what? A "C Run-time". > + // Set country code >> + loc_network_set_country_code(network4, "ZZ"); > I assume you meant to set the country code for network5? Oh, yes. Silly me. But other item. I've come a long way integrating 'libloc' support in my program. Basically I do: struct in6_addr addr; struct loc_network *net; struct loc_as *as = NULL; ... loc_database_lookup (libloc_data.db, &addr, &net); int family = loc_network_address_family (net); const char *country = loc_network_get_country_code (net); const char *AS_name = NULL; uint32 AS_num = loc_network_get_asn (net); is_anycast = loc_network_has_flag (net, LOC_NETWORK_FLAG_ANYCAST); is_anon_proxy = loc_network_has_flag (net, LOC_NETWORK_FLAG_ANONYMOUS_PROXY); sat_provider = loc_network_has_flag (net, LOC_NETWORK_FLAG_SATELLITE_PROVIDER); if (AS_num) { int r = loc_as_new (libloc_data.ctx, &as, AS_num); if (r == 0) AS_name = loc_as_get_name (as); } printf ("ASN: %lu, AF: %d, country: %s, name: %s (%d,%d,%d)", AS_num, family, country, AS_name, is_anycast, is_anon_proxy, sat_provider); if (as) loc_as_unref (as); loc_network_unref (net); ------------- Everything except 'AS_name' is okay and prints sensible values (I've checked all ret-vals). Since there isn't any 'loc_get_asn_by_number()' function, what should I do to get a non-NULL 'AS_name' from 'AS_num'? PS. do I really need this 'loc_network_unref()'? Or does 'loc_database_lookup()' handle it?

2 5

[PATCH] export: Flatten the tree before exporting it
by Peter Müller 27 Oct '20

27 Oct '20

From: Michael Tremer <michael.tremer(a)ipfire.org> This patch removes the possibility that any IP address ranges might show up in multiple exported files. If this was content from the database: * 10.0.0.0/16 - DE * 10.0.1.0/24 - FR Then the IP address 10.0.1.1 would match for both countries. The algorithm will now break the larger /16 subnet apart into smaller subnets so that 10.0.1.0/24 is no longer overlapped. There was some time spent on this to make this as efficient as possible. Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org> --- src/python/export.py | 154 ++++++++++++++++++++++++++++++------------- 1 file changed, 110 insertions(+), 44 deletions(-) diff --git a/src/python/export.py b/src/python/export.py index 5eaf43f..dd44332 100644 --- a/src/python/export.py +++ b/src/python/export.py @@ -89,28 +89,55 @@ class OutputWriter(object): def write(self, network, subnets): if self.flatten and self._flatten(network): - log.debug("Skipping writing network %s" % network) + log.debug("Skipping writing network %s (last one was %s)" % (network, self._last_network)) return + # Convert network into a Python object + _network = ipaddress.ip_network("%s" % network) + # Write the network when it has no subnets if not subnets: - network = ipaddress.ip_network("%s" % network) - return self._write_network(network) + log.debug("Writing %s to %s" % (_network, self.f)) + return self._write_network(_network) + + # Convert subnets into Python objects + _subnets = [ipaddress.ip_network("%s" % subnet) for subnet in subnets] + + # Split the network into smaller bits so that + # we can accomodate for any gaps in it later + to_check = set() + for _subnet in _subnets: + to_check.update( + _network.address_exclude(_subnet) + ) + + # Clear the list of all subnets + subnets = [] + + # Check if all subnets to not overlap with anything else + while to_check: + subnet_to_check = to_check.pop() - # Collect all matching subnets - matching_subnets = [] + for _subnet in _subnets: + # Drop this subnet if it equals one of the subnets + # or if it is subnet of one of them + if subnet_to_check == _subnet or subnet_to_check.subnet_of(_subnet): + break - for subnet in sorted(subnets): - # Try to find the subnet in the database - n = self.db.lookup("%s" % subnet.network_address) + # Break it down if it overlaps + if subnet_to_check.overlaps(_subnet): + to_check.update( + subnet_to_check.address_exclude(_subnet) + ) + break - # No entry or matching country means those IP addresses belong here - if not n or n.country_code == network.country_code: - matching_subnets.append(subnet) + # Add the subnet again as it passed the check + else: + subnets.append(subnet_to_check) # Write all networks as compact as possible - for network in ipaddress.collapse_addresses(matching_subnets): - log.debug("Writing %s to database" % network) + for network in ipaddress.collapse_addresses(subnets): + log.debug("Writing %s to %s" % (network, self.f)) self._write_network(network) def finish(self): @@ -206,40 +233,40 @@ class Exporter(object): # Get all networks that match the family networks = self.db.search_networks(family=family) - # Materialise the generator into a list (uses quite some memory) - networks = list(networks) + # Create a stack with all networks in order where we can put items back + # again and retrieve them in the next iteration. + networks = BufferedStack(networks) # Walk through all networks - for i, network in enumerate(networks): - _network = ipaddress.ip_network("%s" % network) - - # Search for all subnets - subnets = set() - - while i < len(networks): - subnet = networks[i+1] - - if subnet.is_subnet_of(network): - _subnet = ipaddress.ip_network("%s" % subnet) - - subnets.add(_subnet) - subnets.update(_network.address_exclude(_subnet)) - - i += 1 - else: + for network in networks: + # Collect all networks which are a subnet of network + subnets = [] + for subnet in networks: + # If the next subnet was not a subnet, we have to push + # it back on the stack and break this loop + if not subnet.is_subnet_of(network): + networks.push(subnet) break + subnets.append(subnet) + # Write matching countries - try: - writers[network.country_code].write(network, subnets) - except KeyError: - pass + if network.country_code and network.country_code in writers: + # Mismatching subnets + gaps = [ + subnet for subnet in subnets if not network.country_code == subnet.country_code + ] + + writers[network.country_code].write(network, gaps) # Write matching ASNs - try: - writers[network.asn].write(network, subnets) - except KeyError: - pass + if network.asn and network.asn in writers: + # Mismatching subnets + gaps = [ + subnet for subnet in subnets if not network.asn == subnet.asn + ] + + writers[network.asn].write(network, gaps) # Handle flags for flag in flags: @@ -247,10 +274,19 @@ class Exporter(object): # Fetch the "fake" country code country = flags[flag] - try: - writers[country].write(network, subnets) - except KeyError: - pass + if not country in writers: + continue + + gaps = [ + subnet for subnet in subnets + if not subnet.has_flag(flag) + ] + + writers[country].write(network, gaps) + + # Push all subnets back onto the stack + for subnet in reversed(subnets): + networks.push(subnet) # Write everything to the filesystem for writer in writers.values(): @@ -262,3 +298,33 @@ class Exporter(object): ) return os.path.join(directory, filename) + + +class BufferedStack(object): + """ + This class takes an iterator and when being iterated + over it returns objects from that iterator for as long + as there are any. + + It additionally has a function to put an item back on + the back so that it will be returned again at the next + iteration. + """ + def __init__(self, iterator): + self.iterator = iterator + self.stack = [] + + def __iter__(self): + return self + + def __next__(self): + if self.stack: + return self.stack.pop(0) + + return next(self.iterator) + + def push(self, elem): + """ + Takes an element and puts it on the stack + """ + self.stack.insert(0, elem) -- 2.20.1

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Location