Re: Thoughts on importing IP feeds from Amazon, second attempt

["Peter Müller" <peter.mueller@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 3750 bytes --]

Hello Michael,

thanks for your reply.

> Hello,
> 
> First, is there a need to constantly rename subjects? I find this more confusing than helpful to keep track of a conversation on this list.

Personally, I like the idea of changing the subject as soon as the discussion leaves the proposed patch as such,
shifting towards a more general issue. That way, I thought it might be easier to differ between remarks targetting
the _actual_ patch, and general discussions.

If you'll object, I will stop doing that. You are the boss around here... :-)

> 
>> On 30 May 2021, at 10:15, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>
>> Hello Michael,
>> hello *,
>>
>> before I start coding, I just wanted to share my current idea of importing IP feeds from Amazon AWS
>> in a less insecure way. Comments, etc. are appreciated. :-)
> 
> You already submitted some code before. What happened to that?

It is still available, although I would not consider it being safe for production anymore.

> 
>> (a) Run "location-importer update-whois" and "location-importer update-announcements", as we did before.
>> (b) Introduce something like "location-importer update-3rd-party-feeds", which is a blanket function for
>>    updating all the 3rd party feeds we will have at some day, as Amazon for sure won't be the only one.
> 
> Does this need a third command? Why can this not be part of “update-whois”?

Because we do not necessarily have the BGP data available at this step. If we want to build in AS-based
safeguards, we will have to parse 3rd party feeds after running "location-importer update-announcements".

> 
>> (c) In case of Amazon, download their feed, parse it and put the results in a temporary table.
>> (d) Process a list of Autonomous Systems owned or controlled by Amazon.
> 
> Where is this list coming from?

Something similar to "countries.txt", I guess. It would definitely be something we will have to maintain
on our own. A simple .txt file per 3rd party source, containing one ASN per line, would do it in my point
of view.

> 
>> (d) Delete every IP network from this temporary table which is not announced by one of the Autonomous
>>    Systems. That way, we limit potential damage by a broken or manipulated Amazon IP feed to their ASNs.
> 
> This is your second step (d).

?

> 
> When you say you are comparing this, what is the authority for this? The BGP feed? Whois?

The BGP feed. We cannot rely on RIR data for this job, as they do not reflect reality and we don't have them
for ARIN- and LACNIC-maintained space.

> 
>> (e) Anything left in the temporary table is safe to go, and will be merged into the overrides table.
>>
>> Sounds a bit complicated than my first patch looked like, but is more versatile and robust. :-)
> 
> I kind of liked the first patch. It was simple and it worked.

Indeed. But it allowed Amazon to inject arbitrary data. This is bad enough for RIRs already, I do not want
to extend the list of entities being able to do this to some profit-oriented companies...

> 
>> Speaking of robustness, do we want a "source" column for the overrides table as well? Although it won't
>> appear in the generated database or it's .txt dump, it might be worth having, so we still have transparency
>> on 3rd party feeds at this point.
> 
> I do not think it is worth it, because it is easy to check. If you want it, I wouldn’t object either.

Hm, it might not be that easy in production, since we do not store the raw contents of our IP feeds. Especially
if there is a delta, finding out which entry in the overrides table came from with source could be tricky,
eventually.

Thanks, and best regards,
Peter Müller