From mboxrd@z Thu Jan 1 00:00:00 1970 From: Peter =?utf-8?q?M=C3=BCller?= To: location@lists.ipfire.org Subject: Re: Thoughts on importing IP feeds from Amazon, second attempt Date: Wed, 02 Jun 2021 23:12:25 +0200 Message-ID: <39ae49e1-db28-277b-35b5-c710612bd4b5@ipfire.org> In-Reply-To: <0105552B-5866-44B9-BFEF-4470E92C8BCD@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============8264796545921933180==" List-Id: --===============8264796545921933180== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello Michael, thanks for your reply. > Hello, >=20 > First, is there a need to constantly rename subjects? I find this more conf= using than helpful to keep track of a conversation on this list. Personally, I like the idea of changing the subject as soon as the discussion= leaves the proposed patch as such, shifting towards a more general issue. That way, I thought it might be easier= to differ between remarks targetting the _actual_ patch, and general discussions. If you'll object, I will stop doing that. You are the boss around here... :-) >=20 >> On 30 May 2021, at 10:15, Peter M=C3=BCller w= rote: >> >> Hello Michael, >> hello *, >> >> before I start coding, I just wanted to share my current idea of importing= IP feeds from Amazon AWS >> in a less insecure way. Comments, etc. are appreciated. :-) >=20 > You already submitted some code before. What happened to that? It is still available, although I would not consider it being safe for produc= tion anymore. >=20 >> (a) Run "location-importer update-whois" and "location-importer update-ann= ouncements", as we did before. >> (b) Introduce something like "location-importer update-3rd-party-feeds", w= hich is a blanket function for >> updating all the 3rd party feeds we will have at some day, as Amazon fo= r sure won't be the only one. >=20 > Does this need a third command? Why can this not be part of =E2=80=9Cupdate= -whois=E2=80=9D? Because we do not necessarily have the BGP data available at this step. If we= want to build in AS-based safeguards, we will have to parse 3rd party feeds after running "location-imp= orter update-announcements". >=20 >> (c) In case of Amazon, download their feed, parse it and put the results i= n a temporary table. >> (d) Process a list of Autonomous Systems owned or controlled by Amazon. >=20 > Where is this list coming from? Something similar to "countries.txt", I guess. It would definitely be somethi= ng we will have to maintain on our own. A simple .txt file per 3rd party source, containing one ASN per l= ine, would do it in my point of view. >=20 >> (d) Delete every IP network from this temporary table which is not announc= ed by one of the Autonomous >> Systems. That way, we limit potential damage by a broken or manipulated= Amazon IP feed to their ASNs. >=20 > This is your second step (d). ? >=20 > When you say you are comparing this, what is the authority for this? The BG= P feed? Whois? The BGP feed. We cannot rely on RIR data for this job, as they do not reflect= reality and we don't have them for ARIN- and LACNIC-maintained space. >=20 >> (e) Anything left in the temporary table is safe to go, and will be merged= into the overrides table. >> >> Sounds a bit complicated than my first patch looked like, but is more vers= atile and robust. :-) >=20 > I kind of liked the first patch. It was simple and it worked. Indeed. But it allowed Amazon to inject arbitrary data. This is bad enough fo= r RIRs already, I do not want to extend the list of entities being able to do this to some profit-oriented = companies... >=20 >> Speaking of robustness, do we want a "source" column for the overrides tab= le as well? Although it won't >> appear in the generated database or it's .txt dump, it might be worth havi= ng, so we still have transparency >> on 3rd party feeds at this point. >=20 > I do not think it is worth it, because it is easy to check. If you want it,= I wouldn=E2=80=99t object either. Hm, it might not be that easy in production, since we do not store the raw co= ntents of our IP feeds. Especially if there is a delta, finding out which entry in the overrides table came from= with source could be tricky, eventually. Thanks, and best regards, Peter M=C3=BCller --===============8264796545921933180==--