From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: location@lists.ipfire.org Subject: Re: Thoughts on importing IP feeds from Amazon, second attempt Date: Thu, 10 Jun 2021 10:25:43 +0100 Message-ID: <0E10270D-5A52-4F90-A24B-2C6F70EC6948@ipfire.org> In-Reply-To: <72d8bef9-32a6-ab82-4f8a-86aee5111a1c@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0180994059910083927==" List-Id: --===============0180994059910083927== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello, > On 5 Jun 2021, at 13:40, Peter M=C3=BCller wro= te: >=20 > Hello Michael, >=20 > thanks for your reply. >=20 >>>>> (b) Introduce something like "location-importer update-3rd-party-feeds"= , which is a blanket function for >>>>> updating all the 3rd party feeds we will have at some day, as Amazon f= or sure won't be the only one. >>>>=20 >>>> Does this need a third command? Why can this not be part of =E2=80=9Cupd= ate-whois=E2=80=9D? >>>=20 >>> Because we do not necessarily have the BGP data available at this step. I= f we want to build in AS-based >>> safeguards, we will have to parse 3rd party feeds after running "location= -importer update-announcements". >>=20 >> I would say that this is a difficult dependency. We should have the =E2=80= =9Cowner=E2=80=9D of that subnet somewhere in the WHOIS data which is determi= nistic while the BGP isn=E2=80=99t. >>=20 >> There are =E2=80=9Croute=E2=80=9D objects which should allow you to do wha= t you want to do. >=20 > Unfortunately, we do not have these data for ARIN and LACNIC space - at lea= st not in a manner we can properly > use in an automated way. For the rest of the RIRs, I agree. I hate that we have to have a lot of duplicate code to deal with two RIRs. >>=20 >>>>=20 >>>>> (c) In case of Amazon, download their feed, parse it and put the result= s in a temporary table. >>>>> (d) Process a list of Autonomous Systems owned or controlled by Amazon. >>>>=20 >>>> Where is this list coming from? >>>=20 >>> Something similar to "countries.txt", I guess. It would definitely be som= ething we will have to maintain >>> on our own. A simple .txt file per 3rd party source, containing one ASN p= er line, would do it in my point >>> of view. >>=20 >> Hmm, okay. >>=20 >>>>=20 >>>>> (d) Delete every IP network from this temporary table which is not anno= unced by one of the Autonomous >>>>> Systems. That way, we limit potential damage by a broken or manipulate= d Amazon IP feed to their ASNs. >>>>=20 >>>> This is your second step (d). >>>=20 >>> ? >>=20 >> You mis-numbered them. Never mind. >>=20 >>>=20 >>>>=20 >>>> When you say you are comparing this, what is the authority for this? The= BGP feed? Whois? >>>=20 >>> The BGP feed. We cannot rely on RIR data for this job, as they do not ref= lect reality and we don't have them >>> for ARIN- and LACNIC-maintained space. >>=20 >> Well, interesting. I would have said the opposite. >=20 > Normative power of the factual (a bit phony, I guess): If something is anno= unced via BGP, traffic to it will > be routed towards the announced ASN - things like RPKI not taken into accou= nt here -, no matter what a RIR database > contains for this network. >=20 > I don't like saying that, but I guess this is what we have. RPKI uses the RIR database. So we would just enforce RPKI for the entire data= base. I guess it is pretty much the same. >=20 >>=20 >>>>> (e) Anything left in the temporary table is safe to go, and will be mer= ged into the overrides table. >>>>>=20 >>>>> Sounds a bit complicated than my first patch looked like, but is more v= ersatile and robust. :-) >>>>=20 >>>> I kind of liked the first patch. It was simple and it worked. >>>=20 >>> Indeed. But it allowed Amazon to inject arbitrary data. This is bad enoug= h for RIRs already, I do not want >>> to extend the list of entities being able to do this to some profit-orien= ted companies... >>=20 >> I consider that a different problem. Not necessarily less important, but p= robably we should not try to fix all problems in only one patch. >=20 > All right, I agree. Given the state of discussion, I propose to work and su= bmit a two-part patchset: The first > one adds a source column to the network_overrides, so we can debug things b= etter on our systems at least, while > the second one imports Amazon AWS feed in the same way as I did initially. >=20 > We would then care about additional safeguards later... *fingers crossed* := -] >=20 > Would you agree on that? Okay. Merged. >=20 > Thanks, and best regards, > Peter M=C3=BCller --===============0180994059910083927==--