Thank you for all your feedback. It helped a lot.

Or rather… It would have if Arne didn’t find that this whole algorithm doesn’t work :)

It simply does not do what we expect it to do and after my fix this takes a long time again.

I hope that I will find some time this week to improve the implementation further and will get back to you all very soon :)

Best, -Michael

On 22 Nov 2020, at 20:06, Adolf Belka ahb.ipfire@gmail.com wrote:

Dear All,

I have the results now for the location testing on my Intel(R) Celeron(R) CPU J1900 @ 1.99GHz production box.

Stable real 0m37.240s user 0m37.086s sys 0m0.070s

Unstable real 0m25.827s user 0m25.568s sys 0m0.123s

Regards,

Adolf.

On 19/11/2020 18:48, Adolf Belka wrote:

...

I have tested it out on my IPFire virtual machine test bed.

I got between 4.8 and 5 secs.

The computer is an i5-8400 CPU @ 2.80GHz

I will also be testing it out on my production box which is a Intel(R) Celeron(R) CPU J1900 @ 1.99GHz but I have to prepare for that as I will need to re-install Core152 after carrying out the test. Will email the results when I have them.

Regards, Adolf.

On 19/11/2020 14:10, Michael Tremer wrote:

...

I managed to export the database on my IPFire Mini Appliance in about one minute:

[root@fw01 libloc]# time location export --directory=tmp --format=xt_geoip

real 1m3.423s user 1m1.470s sys 0m1.009s

The system was under a little bit of load due to some network activity, but I am generally quite happy with this.

I added some more fine-tuning and could decrease the export time on my Debian test machine by another 40% to around six seconds.

-Michael

...

On 18 Nov 2020, at 19:26, Michael Tremer michael.tremer@ipfire.org wrote:

Hello,

Since there are now so many people involved in this project, I would like to send a little email to report what has happened and what the next steps are going to be…

I have spent (unfortunately) a lot of time to add the ability that we can “flatten the tree” when we export it. That means that no network will overlap with another one and we do not need to care about any orders of iptables rules being added.

That was quite a slow process in Python and since IPFire has to run on small hardware it needed to be ported into C. The tree can now be exported in about 10 seconds on my development box which is running an Intel Xeon E5-2630 processor.

I hope that we can reach acceptable times of only a few minutes on the slowest systems and way under a minute on an average system. To find out about that, I need you help:

I just pushed all changes into the next tree and would everyone who can to install that and report how long this command runs:

mkdir tmp time location export --directory=tmp --format=xt_geoip

It would also be great if you could all check the output if this is still correct. I have done what I could, but I want to have an extra pair of eyes if possible.

If this works well, and no more regressions are being found, I will tag this release and we will close Core Update 153.

We will then roll out all recent changes that make the database better, but had to be held back because the tree needs to be flattened for most use-cases now. All older clients will then no longer receive an updated database any more.

Happy testing.

Best, -Michael