From mboxrd@z Thu Jan 1 00:00:00 1970 From: Peter =?utf-8?q?M=C3=BCller?= To: location@lists.ipfire.org Subject: Re: Thoughts on importing IP feeds from Amazon, second attempt Date: Sat, 05 Jun 2021 14:40:02 +0200 Message-ID: <72d8bef9-32a6-ab82-4f8a-86aee5111a1c@ipfire.org> In-Reply-To: <8A99C94A-15B3-4BCD-9EA7-BAA099C34C94@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============9126964844203425236==" List-Id: --===============9126964844203425236== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello Michael, thanks for your reply. >>>> (b) Introduce something like "location-importer update-3rd-party-feeds",= which is a blanket function for >>>> updating all the 3rd party feeds we will have at some day, as Amazon f= or sure won't be the only one. >>> >>> Does this need a third command? Why can this not be part of =E2=80=9Cupda= te-whois=E2=80=9D? >> >> Because we do not necessarily have the BGP data available at this step. If= we want to build in AS-based >> safeguards, we will have to parse 3rd party feeds after running "location-= importer update-announcements". >=20 > I would say that this is a difficult dependency. We should have the =E2=80= =9Cowner=E2=80=9D of that subnet somewhere in the WHOIS data which is determi= nistic while the BGP isn=E2=80=99t. >=20 > There are =E2=80=9Croute=E2=80=9D objects which should allow you to do what= you want to do. Unfortunately, we do not have these data for ARIN and LACNIC space - at least= not in a manner we can properly use in an automated way. For the rest of the RIRs, I agree. >=20 >>> >>>> (c) In case of Amazon, download their feed, parse it and put the results= in a temporary table. >>>> (d) Process a list of Autonomous Systems owned or controlled by Amazon. >>> >>> Where is this list coming from? >> >> Something similar to "countries.txt", I guess. It would definitely be some= thing we will have to maintain >> on our own. A simple .txt file per 3rd party source, containing one ASN pe= r line, would do it in my point >> of view. >=20 > Hmm, okay. >=20 >>> >>>> (d) Delete every IP network from this temporary table which is not annou= nced by one of the Autonomous >>>> Systems. That way, we limit potential damage by a broken or manipulate= d Amazon IP feed to their ASNs. >>> >>> This is your second step (d). >> >> ? >=20 > You mis-numbered them. Never mind. >=20 >> >>> >>> When you say you are comparing this, what is the authority for this? The = BGP feed? Whois? >> >> The BGP feed. We cannot rely on RIR data for this job, as they do not refl= ect reality and we don't have them >> for ARIN- and LACNIC-maintained space. >=20 > Well, interesting. I would have said the opposite. Normative power of the factual (a bit phony, I guess): If something is announ= ced via BGP, traffic to it will be routed towards the announced ASN - things like RPKI not taken into account= here -, no matter what a RIR database contains for this network. I don't like saying that, but I guess this is what we have. >=20 >>>> (e) Anything left in the temporary table is safe to go, and will be merg= ed into the overrides table. >>>> >>>> Sounds a bit complicated than my first patch looked like, but is more ve= rsatile and robust. :-) >>> >>> I kind of liked the first patch. It was simple and it worked. >> >> Indeed. But it allowed Amazon to inject arbitrary data. This is bad enough= for RIRs already, I do not want >> to extend the list of entities being able to do this to some profit-orient= ed companies... >=20 > I consider that a different problem. Not necessarily less important, but pr= obably we should not try to fix all problems in only one patch. All right, I agree. Given the state of discussion, I propose to work and subm= it a two-part patchset: The first one adds a source column to the network_overrides, so we can debug things bet= ter on our systems at least, while the second one imports Amazon AWS feed in the same way as I did initially. We would then care about additional safeguards later... *fingers crossed* :-] Would you agree on that? Thanks, and best regards, Peter M=C3=BCller --===============9126964844203425236==--