Hello Michael,
thanks for your reply.
(b) Introduce something like "location-importer update-3rd-party-feeds", which is a blanket function for updating all the 3rd party feeds we will have at some day, as Amazon for sure won't be the only one.
Does this need a third command? Why can this not be part of “update-whois”?
Because we do not necessarily have the BGP data available at this step. If we want to build in AS-based safeguards, we will have to parse 3rd party feeds after running "location-importer update-announcements".
I would say that this is a difficult dependency. We should have the “owner” of that subnet somewhere in the WHOIS data which is deterministic while the BGP isn’t.
There are “route” objects which should allow you to do what you want to do.
Unfortunately, we do not have these data for ARIN and LACNIC space - at least not in a manner we can properly use in an automated way. For the rest of the RIRs, I agree.
(c) In case of Amazon, download their feed, parse it and put the results in a temporary table. (d) Process a list of Autonomous Systems owned or controlled by Amazon.
Where is this list coming from?
Something similar to "countries.txt", I guess. It would definitely be something we will have to maintain on our own. A simple .txt file per 3rd party source, containing one ASN per line, would do it in my point of view.
Hmm, okay.
(d) Delete every IP network from this temporary table which is not announced by one of the Autonomous Systems. That way, we limit potential damage by a broken or manipulated Amazon IP feed to their ASNs.
This is your second step (d).
?
You mis-numbered them. Never mind.
When you say you are comparing this, what is the authority for this? The BGP feed? Whois?
The BGP feed. We cannot rely on RIR data for this job, as they do not reflect reality and we don't have them for ARIN- and LACNIC-maintained space.
Well, interesting. I would have said the opposite.
Normative power of the factual (a bit phony, I guess): If something is announced via BGP, traffic to it will be routed towards the announced ASN - things like RPKI not taken into account here -, no matter what a RIR database contains for this network.
I don't like saying that, but I guess this is what we have.
(e) Anything left in the temporary table is safe to go, and will be merged into the overrides table.
Sounds a bit complicated than my first patch looked like, but is more versatile and robust. :-)
I kind of liked the first patch. It was simple and it worked.
Indeed. But it allowed Amazon to inject arbitrary data. This is bad enough for RIRs already, I do not want to extend the list of entities being able to do this to some profit-oriented companies...
I consider that a different problem. Not necessarily less important, but probably we should not try to fix all problems in only one patch.
All right, I agree. Given the state of discussion, I propose to work and submit a two-part patchset: The first one adds a source column to the network_overrides, so we can debug things better on our systems at least, while the second one imports Amazon AWS feed in the same way as I did initially.
We would then care about additional safeguards later... *fingers crossed* :-]
Would you agree on that?
Thanks, and best regards, Peter Müller