Re: Thoughts on importing IP feeds from Amazon, second attempt

[Michael Tremer <michael.tremer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 3998 bytes --]

Hello,

> On 5 Jun 2021, at 13:40, Peter Müller <peter.mueller(a)ipfire.org> wrote:
> 
> Hello Michael,
> 
> thanks for your reply.
> 
>>>>> (b) Introduce something like "location-importer update-3rd-party-feeds", which is a blanket function for
>>>>>  updating all the 3rd party feeds we will have at some day, as Amazon for sure won't be the only one.
>>>> 
>>>> Does this need a third command? Why can this not be part of “update-whois”?
>>> 
>>> Because we do not necessarily have the BGP data available at this step. If we want to build in AS-based
>>> safeguards, we will have to parse 3rd party feeds after running "location-importer update-announcements".
>> 
>> I would say that this is a difficult dependency. We should have the “owner” of that subnet somewhere in the WHOIS data which is deterministic while the BGP isn’t.
>> 
>> There are “route” objects which should allow you to do what you want to do.
> 
> Unfortunately, we do not have these data for ARIN and LACNIC space - at least not in a manner we can properly
> use in an automated way. For the rest of the RIRs, I agree.

I hate that we have to have a lot of duplicate code to deal with two RIRs.

>> 
>>>> 
>>>>> (c) In case of Amazon, download their feed, parse it and put the results in a temporary table.
>>>>> (d) Process a list of Autonomous Systems owned or controlled by Amazon.
>>>> 
>>>> Where is this list coming from?
>>> 
>>> Something similar to "countries.txt", I guess. It would definitely be something we will have to maintain
>>> on our own. A simple .txt file per 3rd party source, containing one ASN per line, would do it in my point
>>> of view.
>> 
>> Hmm, okay.
>> 
>>>> 
>>>>> (d) Delete every IP network from this temporary table which is not announced by one of the Autonomous
>>>>>  Systems. That way, we limit potential damage by a broken or manipulated Amazon IP feed to their ASNs.
>>>> 
>>>> This is your second step (d).
>>> 
>>> ?
>> 
>> You mis-numbered them. Never mind.
>> 
>>> 
>>>> 
>>>> When you say you are comparing this, what is the authority for this? The BGP feed? Whois?
>>> 
>>> The BGP feed. We cannot rely on RIR data for this job, as they do not reflect reality and we don't have them
>>> for ARIN- and LACNIC-maintained space.
>> 
>> Well, interesting. I would have said the opposite.
> 
> Normative power of the factual (a bit phony, I guess): If something is announced via BGP, traffic to it will
> be routed towards the announced ASN - things like RPKI not taken into account here -, no matter what a RIR database
> contains for this network.
> 
> I don't like saying that, but I guess this is what we have.

RPKI uses the RIR database. So we would just enforce RPKI for the entire database. I guess it is pretty much the same.

> 
>> 
>>>>> (e) Anything left in the temporary table is safe to go, and will be merged into the overrides table.
>>>>> 
>>>>> Sounds a bit complicated than my first patch looked like, but is more versatile and robust. :-)
>>>> 
>>>> I kind of liked the first patch. It was simple and it worked.
>>> 
>>> Indeed. But it allowed Amazon to inject arbitrary data. This is bad enough for RIRs already, I do not want
>>> to extend the list of entities being able to do this to some profit-oriented companies...
>> 
>> I consider that a different problem. Not necessarily less important, but probably we should not try to fix all problems in only one patch.
> 
> All right, I agree. Given the state of discussion, I propose to work and submit a two-part patchset: The first
> one adds a source column to the network_overrides, so we can debug things better on our systems at least, while
> the second one imports Amazon AWS feed in the same way as I did initially.
> 
> We would then care about additional safeguards later... *fingers crossed* :-]
> 
> Would you agree on that?

Okay. Merged.

> 
> Thanks, and best regards,
> Peter Müller