I managed to export the database on my IPFire Mini Appliance in about one minute:
[root@fw01 libloc]# time location export --directory=tmp --format=xt_geoip
real 1m3.423s user 1m1.470s sys 0m1.009s
The system was under a little bit of load due to some network activity, but I am generally quite happy with this.
I added some more fine-tuning and could decrease the export time on my Debian test machine by another 40% to around six seconds.
-Michael
On 18 Nov 2020, at 19:26, Michael Tremer michael.tremer@ipfire.org wrote:
Hello,
Since there are now so many people involved in this project, I would like to send a little email to report what has happened and what the next steps are going to be…
I have spent (unfortunately) a lot of time to add the ability that we can “flatten the tree” when we export it. That means that no network will overlap with another one and we do not need to care about any orders of iptables rules being added.
That was quite a slow process in Python and since IPFire has to run on small hardware it needed to be ported into C. The tree can now be exported in about 10 seconds on my development box which is running an Intel Xeon E5-2630 processor.
I hope that we can reach acceptable times of only a few minutes on the slowest systems and way under a minute on an average system. To find out about that, I need you help:
I just pushed all changes into the next tree and would everyone who can to install that and report how long this command runs:
mkdir tmp time location export --directory=tmp --format=xt_geoip
It would also be great if you could all check the output if this is still correct. I have done what I could, but I want to have an extra pair of eyes if possible.
If this works well, and no more regressions are being found, I will tag this release and we will close Core Update 153.
We will then roll out all recent changes that make the database better, but had to be held back because the tree needs to be flattened for most use-cases now. All older clients will then no longer receive an updated database any more.
Happy testing.
Best, -Michael