Thoughts on importing IP feeds from Amazon, second attempt (was: Re: [PATCH] location-importer.in: import additional IP information for Amazon AWS IP networks)

["Peter Müller" <peter.mueller@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 1385 bytes --]

Hello Michael,
hello *,

before I start coding, I just wanted to share my current idea of importing IP feeds from Amazon AWS
in a less insecure way. Comments, etc. are appreciated. :-)

(a) Run "location-importer update-whois" and "location-importer update-announcements", as we did before.
(b) Introduce something like "location-importer update-3rd-party-feeds", which is a blanket function for
    updating all the 3rd party feeds we will have at some day, as Amazon for sure won't be the only one.
(c) In case of Amazon, download their feed, parse it and put the results in a temporary table.
(d) Process a list of Autonomous Systems owned or controlled by Amazon.
(d) Delete every IP network from this temporary table which is not announced by one of the Autonomous
    Systems. That way, we limit potential damage by a broken or manipulated Amazon IP feed to their ASNs.
(e) Anything left in the temporary table is safe to go, and will be merged into the overrides table.

Sounds a bit complicated than my first patch looked like, but is more versatile and robust. :-)

Speaking of robustness, do we want a "source" column for the overrides table as well? Although it won't
appear in the generated database or it's .txt dump, it might be worth having, so we still have transparency
on 3rd party feeds at this point.

Thanks, and best regards,
Peter Müller