Hello Michael, hello *,
before I start coding, I just wanted to share my current idea of importing IP feeds from Amazon AWS in a less insecure way. Comments, etc. are appreciated. :-)
(a) Run "location-importer update-whois" and "location-importer update-announcements", as we did before. (b) Introduce something like "location-importer update-3rd-party-feeds", which is a blanket function for updating all the 3rd party feeds we will have at some day, as Amazon for sure won't be the only one. (c) In case of Amazon, download their feed, parse it and put the results in a temporary table. (d) Process a list of Autonomous Systems owned or controlled by Amazon. (d) Delete every IP network from this temporary table which is not announced by one of the Autonomous Systems. That way, we limit potential damage by a broken or manipulated Amazon IP feed to their ASNs. (e) Anything left in the temporary table is safe to go, and will be merged into the overrides table.
Sounds a bit complicated than my first patch looked like, but is more versatile and robust. :-)
Speaking of robustness, do we want a "source" column for the overrides table as well? Although it won't appear in the generated database or it's .txt dump, it might be worth having, so we still have transparency on 3rd party feeds at this point.
Thanks, and best regards, Peter Müller