Re: Thoughts on importing IP feeds from Amazon, second attempt

["Peter Müller" <peter.mueller@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 3568 bytes --]

Hello Michael,

thanks for your reply.

>>>> (b) Introduce something like "location-importer update-3rd-party-feeds", which is a blanket function for
>>>>   updating all the 3rd party feeds we will have at some day, as Amazon for sure won't be the only one.
>>>
>>> Does this need a third command? Why can this not be part of “update-whois”?
>>
>> Because we do not necessarily have the BGP data available at this step. If we want to build in AS-based
>> safeguards, we will have to parse 3rd party feeds after running "location-importer update-announcements".
> 
> I would say that this is a difficult dependency. We should have the “owner” of that subnet somewhere in the WHOIS data which is deterministic while the BGP isn’t.
> 
> There are “route” objects which should allow you to do what you want to do.

Unfortunately, we do not have these data for ARIN and LACNIC space - at least not in a manner we can properly
use in an automated way. For the rest of the RIRs, I agree.

> 
>>>
>>>> (c) In case of Amazon, download their feed, parse it and put the results in a temporary table.
>>>> (d) Process a list of Autonomous Systems owned or controlled by Amazon.
>>>
>>> Where is this list coming from?
>>
>> Something similar to "countries.txt", I guess. It would definitely be something we will have to maintain
>> on our own. A simple .txt file per 3rd party source, containing one ASN per line, would do it in my point
>> of view.
> 
> Hmm, okay.
> 
>>>
>>>> (d) Delete every IP network from this temporary table which is not announced by one of the Autonomous
>>>>   Systems. That way, we limit potential damage by a broken or manipulated Amazon IP feed to their ASNs.
>>>
>>> This is your second step (d).
>>
>> ?
> 
> You mis-numbered them. Never mind.
> 
>>
>>>
>>> When you say you are comparing this, what is the authority for this? The BGP feed? Whois?
>>
>> The BGP feed. We cannot rely on RIR data for this job, as they do not reflect reality and we don't have them
>> for ARIN- and LACNIC-maintained space.
> 
> Well, interesting. I would have said the opposite.

Normative power of the factual (a bit phony, I guess): If something is announced via BGP, traffic to it will
be routed towards the announced ASN - things like RPKI not taken into account here -, no matter what a RIR database
contains for this network.

I don't like saying that, but I guess this is what we have.

> 
>>>> (e) Anything left in the temporary table is safe to go, and will be merged into the overrides table.
>>>>
>>>> Sounds a bit complicated than my first patch looked like, but is more versatile and robust. :-)
>>>
>>> I kind of liked the first patch. It was simple and it worked.
>>
>> Indeed. But it allowed Amazon to inject arbitrary data. This is bad enough for RIRs already, I do not want
>> to extend the list of entities being able to do this to some profit-oriented companies...
> 
> I consider that a different problem. Not necessarily less important, but probably we should not try to fix all problems in only one patch.

All right, I agree. Given the state of discussion, I propose to work and submit a two-part patchset: The first
one adds a source column to the network_overrides, so we can debug things better on our systems at least, while
the second one imports Amazon AWS feed in the same way as I did initially.

We would then care about additional safeguards later... *fingers crossed* :-]

Would you agree on that?

Thanks, and best regards,
Peter Müller