ASN support for iptables

ASN support for iptables

[Peter Müller]

[-- Attachment #1: Type: text/plain, Size: 2321 bytes --]

Hello development list,

today, I'd like to discuss whether a new feature in
the firewall engine of IPFire makes sense or not.

Since Core Update 90, IPFire supports GeoIP based firewall
rules, which goes beyond simple IP addresses or CIDR blocks
and makes firewalling easier.

The idea I had in mind is to add ASN (Autonomous System
Number) support for firewall rules, too.

An AS (Autonomous System) can be described as an administrative
instance on top of IP: For example, several IP blocks belong
to an AS, i.e. to the same company, university or whatever.
Although these blocks may be used for completely different purposes
in completely different countries, they share the same owner.

Every AS has a number (ASN) and a description (sometimes
abbreviated to ASDescr), while the number is unique.

There are some scenarios in which AS based firewall rules
make sense, since AS information change less seldom than
IP ranges:

(a) One wants to block malicious traffic, but blocking entire
countries is too much since there are some legitimate partners,
customers, ... out there. With AS support, it is possible to
grant them access by simply permitting their AS. The rest of
the country may now safely be blocked.

(b) In some cases, IP ranges change very often, making firewall
rules very complex and hard to maintain, or the exact IP address
of a machine cannot be determined (dial-up connections). In
both cases, the AS (mostly) stays the same and allows firewall
rules without permitting access to a whole country.

(c) Rogue ISPs (networks which are controlled/operated by professional
spammers or worse, such as the "Russian Business Network" (RBN),
which died in end-2007) sometimes run networks located in "good"
countries such as US or NL. Blocking them by GeoIP is not an
option because of many false-positives. AS based rules may help
here.

Since the data behind this can be extracted from BGP feeds,
no external databases (such as MaxMind) are required.

Unfortunately, my programming skills are too low for implementing
this feature. Thereof, if it is decided to do this, I will need
some help here. :-)

Technically, this is similar to the GeoIP firewall stuff (just
another database), so I assume most of the work done there can
just be copied.

Any thoughts on this idea?

Best regards,
Peter Müller

Re: ASN support for iptables

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 3872 bytes --]

Hi,

On Sun, 2017-11-19 at 15:52 +0100, Peter Müller wrote:
> Hello development list,
> 
> today, I'd like to discuss whether a new feature in
> the firewall engine of IPFire makes sense or not.
> 
> Since Core Update 90, IPFire supports GeoIP based firewall
> rules, which goes beyond simple IP addresses or CIDR blocks
> and makes firewalling easier.
> 
> The idea I had in mind is to add ASN (Autonomous System
> Number) support for firewall rules, too.
> 
> An AS (Autonomous System) can be described as an administrative
> instance on top of IP: For example, several IP blocks belong
> to an AS, i.e. to the same company, university or whatever.
> Although these blocks may be used for completely different purposes
> in completely different countries, they share the same owner.
> 
> Every AS has a number (ASN) and a description (sometimes
> abbreviated to ASDescr), while the number is unique.

I think this makes sense and I would welcome that as a new feature in IPFire. It
will in the end have some similar problems like the GeoIP blocker, but that is
not too bad.

> There are some scenarios in which AS based firewall rules
> make sense, since AS information change less seldom than
> IP ranges:
> 
> (a) One wants to block malicious traffic, but blocking entire
> countries is too much since there are some legitimate partners,
> customers, ... out there. With AS support, it is possible to
> grant them access by simply permitting their AS. The rest of
> the country may now safely be blocked.

True. This might work well in some situations, but is probably quite useless
when fighting against a botnet.

What would also be good is to open a port forwarding only from a certain AS.
Let's call that whitelisting.

> (b) In some cases, IP ranges change very often, making firewall
> rules very complex and hard to maintain, or the exact IP address
> of a machine cannot be determined (dial-up connections). In
> both cases, the AS (mostly) stays the same and allows firewall
> rules without permitting access to a whole country.

That will be the biggest challenge here. The database will need to be complete
and needs regular updates. We don't really care if someone is actually
announcing their prefix, but if they have one assigned, we should block/permit
access.

> (c) Rogue ISPs (networks which are controlled/operated by professional
> spammers or worse, such as the "Russian Business Network" (RBN),
> which died in end-2007) sometimes run networks located in "good"
> countries such as US or NL. Blocking them by GeoIP is not an
> option because of many false-positives. AS based rules may help
> here.

The US is practically unblockable on the GeoIP filter, because too much is
hosted in the US (at least according to the database by businesses that have
their HQ there).

So this would be a good extension to blocking more granular.

> Since the data behind this can be extracted from BGP feeds,
> no external databases (such as MaxMind) are required.

If we would use a BGP feed, we will only have the networks in the database that
are currently announced. Wouldn't scraping the WHOIS database be better?

Why not MaxMind? Not that I am in favour of that, but I am interested why it is
not an option.

> Unfortunately, my programming skills are too low for implementing
> this feature. Thereof, if it is decided to do this, I will need
> some help here. :-)

*raises hand*

> Technically, this is similar to the GeoIP firewall stuff (just
> another database), so I assume most of the work done there can
> just be copied.

The GeoIP block uses an iptables extension which parses the database. We
wouldn't use that here but would either build something with ipset or similar.

> Any thoughts on this idea?
> 
> Best regards,
> Peter Müller

-Michael

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: ASN support for iptables

[Peter Müller]

[-- Attachment #1: Type: text/plain, Size: 6314 bytes --]

Hello Michael,

> Hi,
> 
> On Sun, 2017-11-19 at 15:52 +0100, Peter Müller wrote:
> > Hello development list,
> > 
> > today, I'd like to discuss whether a new feature in
> > the firewall engine of IPFire makes sense or not.
> > 
> > Since Core Update 90, IPFire supports GeoIP based firewall
> > rules, which goes beyond simple IP addresses or CIDR blocks
> > and makes firewalling easier.
> > 
> > The idea I had in mind is to add ASN (Autonomous System
> > Number) support for firewall rules, too.
> > 
> > An AS (Autonomous System) can be described as an administrative
> > instance on top of IP: For example, several IP blocks belong
> > to an AS, i.e. to the same company, university or whatever.
> > Although these blocks may be used for completely different purposes
> > in completely different countries, they share the same owner.
> > 
> > Every AS has a number (ASN) and a description (sometimes
> > abbreviated to ASDescr), while the number is unique.  
> 
> I think this makes sense and I would welcome that as a new feature in IPFire. It
> will in the end have some similar problems like the GeoIP blocker, but that is
> not too bad.
Just for curiosity: Which ones are they?

I only see the (potential) problem of _another_ incomplete database here (see below).
Since all necessary data for this AS stuff are more or less public, inaccurate
data and extensive manual research are not necessary.

All we need is to write some scripts that convert the RIPE, ... databases
to a suitable format.

Second, in contrast to the GeoIP database, false-positives are extremely rare
when dealing with AS information since they have to be technically correct. As
always, there are some "good" and "bad" ASNs, and a lot "grey" ones, which stays
the same.
> 
> > There are some scenarios in which AS based firewall rules
> > make sense, since AS information change less seldom than
> > IP ranges:
> > 
> > (a) One wants to block malicious traffic, but blocking entire
> > countries is too much since there are some legitimate partners,
> > customers, ... out there. With AS support, it is possible to
> > grant them access by simply permitting their AS. The rest of
> > the country may now safely be blocked.  
> 
> True. This might work well in some situations, but is probably quite useless
> when fighting against a botnet.
I am not sure about this. In some cases, blocking access to some bullet proof/
rogue ISPs where many C&C servers are hosted improves the situation. Of course,
if there is a simple DDoS attack from many different IPs, this will not help
very much.
> 
> What would also be good is to open a port forwarding only from a certain AS.
> Let's call that whitelisting.
Yes, that is a very good use case. :-)

And the combination of GeoIP and AS filtering (inbound and outbound)
opens up fare more possibilities...
> 
> > (b) In some cases, IP ranges change very often, making firewall
> > rules very complex and hard to maintain, or the exact IP address
> > of a machine cannot be determined (dial-up connections). In
> > both cases, the AS (mostly) stays the same and allows firewall
> > rules without permitting access to a whole country.  
> 
> That will be the biggest challenge here. The database will need to be complete
> and needs regular updates. We don't really care if someone is actually
> announcing their prefix, but if they have one assigned, we should block/permit
> access.
I agree with you.

We could simply ship an updated version with every Core Update (I
had that in mind for the CA list, too, so we don't get too big deltas).
> 
> > (c) Rogue ISPs (networks which are controlled/operated by professional
> > spammers or worse, such as the "Russian Business Network" (RBN),
> > which died in end-2007) sometimes run networks located in "good"
> > countries such as US or NL. Blocking them by GeoIP is not an
> > option because of many false-positives. AS based rules may help
> > here.  
> 
> The US is practically unblockable on the GeoIP filter, because too much is
> hosted in the US (at least according to the database by businesses that have
> their HQ there).
Indeed, and there are several ASs over there the internet would be a better
place without. But I guess that applies to most other countries, too.
> 
> So this would be a good extension to blocking more granular.
> 
> > Since the data behind this can be extracted from BGP feeds,
> > no external databases (such as MaxMind) are required.  
> 
> If we would use a BGP feed, we will only have the networks in the database that
> are currently announced. Wouldn't scraping the WHOIS database be better?
Good idea. For example, RIPE has one here: http://ftp.ripe.net/ripe/dbase/split/inet(6)num.gz
> 
> Why not MaxMind? Not that I am in favour of that, but I am interested why it is
> not an option.
Well, technically, it is an option. But since there is some trouble with
their GeoIP database, as mentioned somewhere else some time before, I'd
like to move away from them.

And since we can build our own database here, I do not see why we should
rely on 3rd party data.
> 
> > Unfortunately, my programming skills are too low for implementing
> > this feature. Thereof, if it is decided to do this, I will need
> > some help here. :-)  
> 
> *raises hand*
Thank you very much. :-)
> 
> > Technically, this is similar to the GeoIP firewall stuff (just
> > another database), so I assume most of the work done there can
> > just be copied.  
> 
> The GeoIP block uses an iptables extension which parses the database. We
> wouldn't use that here but would either build something with ipset or similar.
Hmmm, not sure how well this performs: If there is only a small AS with
few IP ranges to parse, ipset would be sufficient.

However, if we have large networks (Level 3, Hurricane Electric, DTAG, ...),
ipset might become slow and cause many iptables rules. And if we update
the database (whomever created it), we need to reload the firewall rules.
Using a database lookup module similar to GeoIP avoids that issues here, as
far as I am concerned.

Best regards,
Peter Müller
> 
> > Any thoughts on this idea?
> > 
> > Best regards,
> > Peter Müller  
> 
> -Michael

Re: ASN support for iptables

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 8330 bytes --]

Hi,

On Mon, 2017-11-20 at 20:03 +0100, Peter Müller wrote:
> Hello Michael,
> 
> > Hi,
> > 
> > On Sun, 2017-11-19 at 15:52 +0100, Peter Müller wrote:
> > > Hello development list,
> > > 
> > > today, I'd like to discuss whether a new feature in
> > > the firewall engine of IPFire makes sense or not.
> > > 
> > > Since Core Update 90, IPFire supports GeoIP based firewall
> > > rules, which goes beyond simple IP addresses or CIDR blocks
> > > and makes firewalling easier.
> > > 
> > > The idea I had in mind is to add ASN (Autonomous System
> > > Number) support for firewall rules, too.
> > > 
> > > An AS (Autonomous System) can be described as an administrative
> > > instance on top of IP: For example, several IP blocks belong
> > > to an AS, i.e. to the same company, university or whatever.
> > > Although these blocks may be used for completely different purposes
> > > in completely different countries, they share the same owner.
> > > 
> > > Every AS has a number (ASN) and a description (sometimes
> > > abbreviated to ASDescr), while the number is unique.  
> > 
> > I think this makes sense and I would welcome that as a new feature in
> > IPFire. It
> > will in the end have some similar problems like the GeoIP blocker, but that
> > is
> > not too bad.
> 
> Just for curiosity: Which ones are they?

Apologies for nothing into depths of that. I just thought of the usual ones like
not actually blocking what the user intends to block. Rules changing because
prefixes might be moved around, etc.

It is just not 100% determinable. But indeed it is better than GeoIP.

> I only see the (potential) problem of _another_ incomplete database here (see
> below).
> Since all necessary data for this AS stuff are more or less public, inaccurate
> data and extensive manual research are not necessary.
> 
> All we need is to write some scripts that convert the RIPE, ... databases
> to a suitable format.

Does such a public database of the RIRs exist? I don't know of one.

For IPv4 handing out address space has slowed down a lot. And we should not use
the BGP feed and parse that because then we might miss prefixes that are
currently not announced - either deliberately or because of a downtime.

I know this:
  http://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest

But it doesn't connect the prefixes to an ASN AFAIK.

> Second, in contrast to the GeoIP database, false-positives are extremely rare
> when dealing with AS information since they have to be technically correct. As
> always, there are some "good" and "bad" ASNs, and a lot "grey" ones, which
> stays
> the same.

We wouldn't really care about that, would we? That is up to the user to decide
what they want to block/whitelist.

> > 
> > > There are some scenarios in which AS based firewall rules
> > > make sense, since AS information change less seldom than
> > > IP ranges:
> > > 
> > > (a) One wants to block malicious traffic, but blocking entire
> > > countries is too much since there are some legitimate partners,
> > > customers, ... out there. With AS support, it is possible to
> > > grant them access by simply permitting their AS. The rest of
> > > the country may now safely be blocked.  
> > 
> > True. This might work well in some situations, but is probably quite useless
> > when fighting against a botnet.
> 
> I am not sure about this. In some cases, blocking access to some bullet proof/
> rogue ISPs where many C&C servers are hosted improves the situation. Of
> course,
> if there is a simple DDoS attack from many different IPs, this will not help
> very much.
> > 
> > What would also be good is to open a port forwarding only from a certain AS.
> > Let's call that whitelisting.
> 
> Yes, that is a very good use case. :-)
> 
> And the combination of GeoIP and AS filtering (inbound and outbound)
> opens up fare more possibilities...

How would we combine this?

> > > (b) In some cases, IP ranges change very often, making firewall
> > > rules very complex and hard to maintain, or the exact IP address
> > > of a machine cannot be determined (dial-up connections). In
> > > both cases, the AS (mostly) stays the same and allows firewall
> > > rules without permitting access to a whole country.  
> > 
> > That will be the biggest challenge here. The database will need to be
> > complete
> > and needs regular updates. We don't really care if someone is actually
> > announcing their prefix, but if they have one assigned, we should
> > block/permit
> > access.
> 
> I agree with you.
> 
> We could simply ship an updated version with every Core Update (I
> had that in mind for the CA list, too, so we don't get too big deltas).

If we don't convert this into some nice binary format this will indeed become a
huge database of probably ~10MB?! That would be too much to be included in every
Core Update.

> > > (c) Rogue ISPs (networks which are controlled/operated by professional
> > > spammers or worse, such as the "Russian Business Network" (RBN),
> > > which died in end-2007) sometimes run networks located in "good"
> > > countries such as US or NL. Blocking them by GeoIP is not an
> > > option because of many false-positives. AS based rules may help
> > > here.  
> > 
> > The US is practically unblockable on the GeoIP filter, because too much is
> > hosted in the US (at least according to the database by businesses that have
> > their HQ there).
> 
> Indeed, and there are several ASs over there the internet would be a better
> place without. But I guess that applies to most other countries, too.
> > 
> > So this would be a good extension to blocking more granular.
> > 
> > > Since the data behind this can be extracted from BGP feeds,
> > > no external databases (such as MaxMind) are required.  
> > 
> > If we would use a BGP feed, we will only have the networks in the database
> > that
> > are currently announced. Wouldn't scraping the WHOIS database be better?
> 
> Good idea. For example, RIPE has one here:
> http://ftp.ripe.net/ripe/dbase/split/inet(6)num.gz

And where is the link to the ASN?

> > 
> > Why not MaxMind? Not that I am in favour of that, but I am interested why it
> > is
> > not an option.
> 
> Well, technically, it is an option. But since there is some trouble with
> their GeoIP database, as mentioned somewhere else some time before, I'd
> like to move away from them.

What trouble?

> And since we can build our own database here, I do not see why we should
> rely on 3rd party data.
> > 
> > > Unfortunately, my programming skills are too low for implementing
> > > this feature. Thereof, if it is decided to do this, I will need
> > > some help here. :-)  
> > 
> > *raises hand*
> 
> Thank you very much. :-)
> > 
> > > Technically, this is similar to the GeoIP firewall stuff (just
> > > another database), so I assume most of the work done there can
> > > just be copied.  
> > 
> > The GeoIP block uses an iptables extension which parses the database. We
> > wouldn't use that here but would either build something with ipset or
> > similar.
> 
> Hmmm, not sure how well this performs: If there is only a small AS with
> few IP ranges to parse, ipset would be sufficient.
> 
> However, if we have large networks (Level 3, Hurricane Electric, DTAG, ...),
> ipset might become slow and cause many iptables rules. And if we update
> the database (whomever created it), we need to reload the firewall rules.
> Using a database lookup module similar to GeoIP avoids that issues here, as
> far as I am concerned.

I mean it is an option to build an iptables extension, but I am not sure if that
is the best thing.

ipset is O(1) for each lookup. So it doesn't matter how large the AS is.
iptables rules would be O(n) which is not very nice for larger ASNs, but we
should have a look since I do not expect ASNs to announce hundreds of prefixes.
The average might be way less and I have no idea what to expect as the worst-
case performance. That needs investigation.

Best,
-Michael

> 
> Best regards,
> Peter Müller
> > 
> > > Any thoughts on this idea?
> > > 
> > > Best regards,
> > > Peter Müller  
> > 
> > -Michael
> 
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: ASN support for iptables

[Peter Müller]

[-- Attachment #1: Type: text/plain, Size: 13805 bytes --]

Hello,

> Hi,
> 
> On Mon, 2017-11-20 at 20:03 +0100, Peter Müller wrote:
> > Hello Michael,
> >   
> > > Hi,
> > > 
> > > On Sun, 2017-11-19 at 15:52 +0100, Peter Müller wrote:  
> > > > Hello development list,
> > > > 
> > > > today, I'd like to discuss whether a new feature in
> > > > the firewall engine of IPFire makes sense or not.
> > > > 
> > > > Since Core Update 90, IPFire supports GeoIP based firewall
> > > > rules, which goes beyond simple IP addresses or CIDR blocks
> > > > and makes firewalling easier.
> > > > 
> > > > The idea I had in mind is to add ASN (Autonomous System
> > > > Number) support for firewall rules, too.
> > > > 
> > > > An AS (Autonomous System) can be described as an administrative
> > > > instance on top of IP: For example, several IP blocks belong
> > > > to an AS, i.e. to the same company, university or whatever.
> > > > Although these blocks may be used for completely different purposes
> > > > in completely different countries, they share the same owner.
> > > > 
> > > > Every AS has a number (ASN) and a description (sometimes
> > > > abbreviated to ASDescr), while the number is unique.    
> > > 
> > > I think this makes sense and I would welcome that as a new feature in
> > > IPFire. It
> > > will in the end have some similar problems like the GeoIP blocker, but that
> > > is
> > > not too bad.  
> > 
> > Just for curiosity: Which ones are they?  
> 
> Apologies for nothing into depths of that. I just thought of the usual ones like
> not actually blocking what the user intends to block. Rules changing because
> prefixes might be moved around, etc.
Ah, yes, the usual suspects. :-)
> 
> It is just not 100% determinable. But indeed it is better than GeoIP.
True.
> 
> > I only see the (potential) problem of _another_ incomplete database here (see
> > below).
> > Since all necessary data for this AS stuff are more or less public, inaccurate
> > data and extensive manual research are not necessary.
> > 
> > All we need is to write some scripts that convert the RIPE, ... databases
> > to a suitable format.  
> 
> Does such a public database of the RIRs exist? I don't know of one.
Yes, they do exist, but mostly without the connection between an IP range and
the ASN (see below). At least for RIPE, I have not checked the others in detail.
> 
> For IPv4 handing out address space has slowed down a lot. And we should not use
> the BGP feed and parse that because then we might miss prefixes that are
> currently not announced - either deliberately or because of a downtime.
> 
> I know this:
>   http://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest
> 
> But it doesn't connect the prefixes to an ASN AFAIK.
> 
> > Second, in contrast to the GeoIP database, false-positives are extremely rare
> > when dealing with AS information since they have to be technically correct. As
> > always, there are some "good" and "bad" ASNs, and a lot "grey" ones, which
> > stays
> > the same.  
> 
> We wouldn't really care about that, would we? That is up to the user to decide
> what they want to block/whitelist.
> 
> > >   
> > > > There are some scenarios in which AS based firewall rules
> > > > make sense, since AS information change less seldom than
> > > > IP ranges:
> > > > 
> > > > (a) One wants to block malicious traffic, but blocking entire
> > > > countries is too much since there are some legitimate partners,
> > > > customers, ... out there. With AS support, it is possible to
> > > > grant them access by simply permitting their AS. The rest of
> > > > the country may now safely be blocked.    
> > > 
> > > True. This might work well in some situations, but is probably quite useless
> > > when fighting against a botnet.  
> > 
> > I am not sure about this. In some cases, blocking access to some bullet proof/
> > rogue ISPs where many C&C servers are hosted improves the situation. Of
> > course,
> > if there is a simple DDoS attack from many different IPs, this will not help
> > very much.  
> > > 
> > > What would also be good is to open a port forwarding only from a certain AS.
> > > Let's call that whitelisting.  
> > 
> > Yes, that is a very good use case. :-)
> > 
> > And the combination of GeoIP and AS filtering (inbound and outbound)
> > opens up fare more possibilities...  
> 
> How would we combine this?
I meant that it opens up new possibilities for creating firewall rules (allowing
one AS and block/throttle the rest of a country, ...).
> 
> > > > (b) In some cases, IP ranges change very often, making firewall
> > > > rules very complex and hard to maintain, or the exact IP address
> > > > of a machine cannot be determined (dial-up connections). In
> > > > both cases, the AS (mostly) stays the same and allows firewall
> > > > rules without permitting access to a whole country.    
> > > 
> > > That will be the biggest challenge here. The database will need to be
> > > complete
> > > and needs regular updates. We don't really care if someone is actually
> > > announcing their prefix, but if they have one assigned, we should
> > > block/permit
> > > access.  
> > 
> > I agree with you.
> > 
> > We could simply ship an updated version with every Core Update (I
> > had that in mind for the CA list, too, so we don't get too big deltas).  
> 
> If we don't convert this into some nice binary format this will indeed become a
> huge database of probably ~10MB?! That would be too much to be included in every
> Core Update.
Is shipping database deltas an option? An maybe every 20th Core Update the
complete one so we do not have too much trouble with incremental updates.

If there is a nice small binary format, we should use it indeed. I am not very
deep into this subject. :-|

Shipping the database with every update avoids fetching them by the systems
itself, in which I see a set of issues:

(a) A download server for the database must be reachable. In some networks
with restrictive policies, this might be a problem. On the other hand, if
people install a Core Update, everybody understands that IPFire needs to
connect to some mirror servers. So why not shipping the databases via this way.

(The idea behind this is that every network connection can go wrong. A bit
ancient, I know. :-) )

(b) Error handling for database servers is problematic. In the given scenario:
What if downloading the AS database fails? Simply using the old version without
notifying is not an option. On the other hand, a notification (i.e. a warning
in the WebUI) needs to be noticed by the administrator. If they install
an update, they will see errors instantly.

(c) A download server is an unnecessary information leak. Since all systems
fetch their databases from a - hopefully redundant - server, this server sees
all IP addresses (and wget user agents, ...) belonging to firewall systems.
Although it may not be obvious at the first glance that IP x.y.z is an IPFire machine,
it is a good point for hacking tools to start from.

This problem becomes more important when talking about third party sites
(MaxMind!) since nobody knows what they are actually doing with the logs.
Even worse, if "geolite.maxmind.com" is going offline permanently, there
is no fallback server where the data can be fetched from.

I do not consider the IPFire mirrors as problematic here, since there is
redundancy, and the traffic is split up to more than one system.

(d) Integrity checking for downloaded databases is problematic. In most
cases, they are just .tar.gz-files. No HTTPS, no DNSSEC/DANE, no checksum
file, no GPG signature. Great.

And a firewall system is _relying_ on these data???

Of course, this problem needs to be fixed by mirror operators in first place.
But it is also a question of design: If one assumes that the internet in
general and the mirror servers in particular cannot be trusted, he/she/it
will provide signed (and GPG encrypted) data, as IPFire does. Again, why
not using this way?
> 
> > > > (c) Rogue ISPs (networks which are controlled/operated by professional
> > > > spammers or worse, such as the "Russian Business Network" (RBN),
> > > > which died in end-2007) sometimes run networks located in "good"
> > > > countries such as US or NL. Blocking them by GeoIP is not an
> > > > option because of many false-positives. AS based rules may help
> > > > here.    
> > > 
> > > The US is practically unblockable on the GeoIP filter, because too much is
> > > hosted in the US (at least according to the database by businesses that have
> > > their HQ there).  
> > 
> > Indeed, and there are several ASs over there the internet would be a better
> > place without. But I guess that applies to most other countries, too.  
> > > 
> > > So this would be a good extension to blocking more granular.
> > >   
> > > > Since the data behind this can be extracted from BGP feeds,
> > > > no external databases (such as MaxMind) are required.    
> > > 
> > > If we would use a BGP feed, we will only have the networks in the database
> > > that
> > > are currently announced. Wouldn't scraping the WHOIS database be better?  
> > 
> > Good idea. For example, RIPE has one here:
> > http://ftp.ripe.net/ripe/dbase/split/inet(6)num.gz  
> 
> And where is the link to the ASN?
Sorry, I used the wrong link while researching for the GeoIP stuff. Here is
the correct one: https://ftp.ripe.net/ripe/dbase/ripe.db.gz

As far as I know, the link between "as-block" and an IP range can be done
via the "mnt-by" or the "origin" tags. 
> 
> > > 
> > > Why not MaxMind? Not that I am in favour of that, but I am interested why it
> > > is
> > > not an option.  
> > 
> > Well, technically, it is an option. But since there is some trouble with
> > their GeoIP database, as mentioned somewhere else some time before, I'd
> > like to move away from them.  
> 
> What trouble?
Well, some of them are mentioned above already. Here are some more:

(i) There is no changelog or release note. Once in a month, the GeoIP database
is updated - and there is no (simple) way of telling which changes were made.

For example, the IP ranges belonging to Cloudflare were recently deleted from
this database. As a result, some websites became unreachable from today to
tomorrow (see bug #11482 for details) since the iptables module permits access
in case an IP is not listed (which is the correct behaviour).

Not sure how much went wrong here in productive networks, but it certainly
is an issue only MaxMind can fix.

(ii) The database update is performed silently. In case of troubles, this
procedure might not be obvious. On the other hand, if people install a Core
Update, they are in maintenance mode, scheduled downtime, or whatever and
are somewhat prepared for trouble.

(iii) The quality of the GeoIP database by MaxMind we use is poor.
For example, allocations to countries are incorrect (especially in case of
"A1 - Anonymous Proxies") even if the correct data is known. Much worse,
sometimes the allocations are incorrect without being obvious (anycast networks
are assigned to "US", or simply dropped in case of Cloudflare). It seems
like they are only scraping WHOIS, too.

There should be an "A3 - Anycast Networks" section for companies like Google,
similar to "EU" or "AP" which also only indicate a continent without going down
to the country level. A3 would be a step further.

Note that I am not talking about their chargeable products or the city
databases here.


Because of these issues, I'd like to build our own GeoIP database, but that
is off-topic here, and will be mentioned in another mail. ;-)
> 
> > And since we can build our own database here, I do not see why we should
> > rely on 3rd party data.  
> > >   
> > > > Unfortunately, my programming skills are too low for implementing
> > > > this feature. Thereof, if it is decided to do this, I will need
> > > > some help here. :-)    
> > > 
> > > *raises hand*  
> > 
> > Thank you very much. :-)  
> > >   
> > > > Technically, this is similar to the GeoIP firewall stuff (just
> > > > another database), so I assume most of the work done there can
> > > > just be copied.    
> > > 
> > > The GeoIP block uses an iptables extension which parses the database. We
> > > wouldn't use that here but would either build something with ipset or
> > > similar.  
> > 
> > Hmmm, not sure how well this performs: If there is only a small AS with
> > few IP ranges to parse, ipset would be sufficient.
> > 
> > However, if we have large networks (Level 3, Hurricane Electric, DTAG, ...),
> > ipset might become slow and cause many iptables rules. And if we update
> > the database (whomever created it), we need to reload the firewall rules.
> > Using a database lookup module similar to GeoIP avoids that issues here, as
> > far as I am concerned.  
> 
> I mean it is an option to build an iptables extension, but I am not sure if that
> is the best thing.
> 
> ipset is O(1) for each lookup. So it doesn't matter how large the AS is.
> iptables rules would be O(n) which is not very nice for larger ASNs, but we
> should have a look since I do not expect ASNs to announce hundreds of prefixes.
> The average might be way less and I have no idea what to expect as the worst-
> case performance. That needs investigation.
All right.

@Everybody else: Comments?

Best regards,
Peter Müller
> 
> Best,
> -Michael
> 
> > 
> > Best regards,
> > Peter Müller  
> > >   
> > > > Any thoughts on this idea?
> > > > 
> > > > Best regards,
> > > > Peter Müller    
> > > 
> > > -Michael  
> > 
> >   

Re: ASN support for iptables

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 17311 bytes --]

On Fri, 2017-11-24 at 19:31 +0100, Peter Müller wrote:
> Hello,
> 
> > Hi,
> > 
> > On Mon, 2017-11-20 at 20:03 +0100, Peter Müller wrote:
> > > Hello Michael,
> > >   
> > > > Hi,
> > > > 
> > > > On Sun, 2017-11-19 at 15:52 +0100, Peter Müller wrote:  
> > > > > Hello development list,
> > > > > 
> > > > > today, I'd like to discuss whether a new feature in
> > > > > the firewall engine of IPFire makes sense or not.
> > > > > 
> > > > > Since Core Update 90, IPFire supports GeoIP based firewall
> > > > > rules, which goes beyond simple IP addresses or CIDR blocks
> > > > > and makes firewalling easier.
> > > > > 
> > > > > The idea I had in mind is to add ASN (Autonomous System
> > > > > Number) support for firewall rules, too.
> > > > > 
> > > > > An AS (Autonomous System) can be described as an administrative
> > > > > instance on top of IP: For example, several IP blocks belong
> > > > > to an AS, i.e. to the same company, university or whatever.
> > > > > Although these blocks may be used for completely different purposes
> > > > > in completely different countries, they share the same owner.
> > > > > 
> > > > > Every AS has a number (ASN) and a description (sometimes
> > > > > abbreviated to ASDescr), while the number is unique.    
> > > > 
> > > > I think this makes sense and I would welcome that as a new feature in
> > > > IPFire. It
> > > > will in the end have some similar problems like the GeoIP blocker, but
> > > > that
> > > > is
> > > > not too bad.  
> > > 
> > > Just for curiosity: Which ones are they?  
> > 
> > Apologies for nothing into depths of that. I just thought of the usual ones
> > like
> > not actually blocking what the user intends to block. Rules changing because
> > prefixes might be moved around, etc.
> 
> Ah, yes, the usual suspects. :-)
> > 
> > It is just not 100% determinable. But indeed it is better than GeoIP.
> 
> True.
> > 
> > > I only see the (potential) problem of _another_ incomplete database here
> > > (see
> > > below).
> > > Since all necessary data for this AS stuff are more or less public,
> > > inaccurate
> > > data and extensive manual research are not necessary.
> > > 
> > > All we need is to write some scripts that convert the RIPE, ... databases
> > > to a suitable format.  
> > 
> > Does such a public database of the RIRs exist? I don't know of one.
> 
> Yes, they do exist, but mostly without the connection between an IP range and
> the ASN (see below). At least for RIPE, I have not checked the others in
> detail.
> > 
> > For IPv4 handing out address space has slowed down a lot. And we should not
> > use
> > the BGP feed and parse that because then we might miss prefixes that are
> > currently not announced - either deliberately or because of a downtime.
> > 
> > I know this:
> >   http://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest
> > 
> > But it doesn't connect the prefixes to an ASN AFAIK.
> > 
> > > Second, in contrast to the GeoIP database, false-positives are extremely
> > > rare
> > > when dealing with AS information since they have to be technically
> > > correct. As
> > > always, there are some "good" and "bad" ASNs, and a lot "grey" ones, which
> > > stays
> > > the same.  
> > 
> > We wouldn't really care about that, would we? That is up to the user to
> > decide
> > what they want to block/whitelist.
> > 
> > > >   
> > > > > There are some scenarios in which AS based firewall rules
> > > > > make sense, since AS information change less seldom than
> > > > > IP ranges:
> > > > > 
> > > > > (a) One wants to block malicious traffic, but blocking entire
> > > > > countries is too much since there are some legitimate partners,
> > > > > customers, ... out there. With AS support, it is possible to
> > > > > grant them access by simply permitting their AS. The rest of
> > > > > the country may now safely be blocked.    
> > > > 
> > > > True. This might work well in some situations, but is probably quite
> > > > useless
> > > > when fighting against a botnet.  
> > > 
> > > I am not sure about this. In some cases, blocking access to some bullet
> > > proof/
> > > rogue ISPs where many C&C servers are hosted improves the situation. Of
> > > course,
> > > if there is a simple DDoS attack from many different IPs, this will not
> > > help
> > > very much.  
> > > > 
> > > > What would also be good is to open a port forwarding only from a certain
> > > > AS.
> > > > Let's call that whitelisting.  
> > > 
> > > Yes, that is a very good use case. :-)
> > > 
> > > And the combination of GeoIP and AS filtering (inbound and outbound)
> > > opens up fare more possibilities...  
> > 
> > How would we combine this?
> 
> I meant that it opens up new possibilities for creating firewall rules
> (allowing
> one AS and block/throttle the rest of a country, ...).

LOL

> > 
> > > > > (b) In some cases, IP ranges change very often, making firewall
> > > > > rules very complex and hard to maintain, or the exact IP address
> > > > > of a machine cannot be determined (dial-up connections). In
> > > > > both cases, the AS (mostly) stays the same and allows firewall
> > > > > rules without permitting access to a whole country.    
> > > > 
> > > > That will be the biggest challenge here. The database will need to be
> > > > complete
> > > > and needs regular updates. We don't really care if someone is actually
> > > > announcing their prefix, but if they have one assigned, we should
> > > > block/permit
> > > > access.  
> > > 
> > > I agree with you.
> > > 
> > > We could simply ship an updated version with every Core Update (I
> > > had that in mind for the CA list, too, so we don't get too big deltas).  
> > 
> > If we don't convert this into some nice binary format this will indeed
> > become a
> > huge database of probably ~10MB?! That would be too much to be included in
> > every
> > Core Update.
> 
> Is shipping database deltas an option? An maybe every 20th Core Update the
> complete one so we do not have too much trouble with incremental updates.

I don't think that this is worth it since we have to have a format that is
possible to being delta-ed or we have to create a delta format and a tool which
applies the delta. Way too much work for a different problem.

In that case, text probably won't be the worst format and just applying a diff.

> If there is a nice small binary format, we should use it indeed. I am not very
> deep into this subject. :-|

Not that I know of.

> Shipping the database with every update avoids fetching them by the systems
> itself, in which I see a set of issues:
> 
> (a) A download server for the database must be reachable. In some networks
> with restrictive policies, this might be a problem. On the other hand, if
> people install a Core Update, everybody understands that IPFire needs to
> connect to some mirror servers. So why not shipping the databases via this
> way.
> 
> (The idea behind this is that every network connection can go wrong. A bit
> ancient, I know. :-) )

We will have to find out if we are free to redistribute this data.

I wouldn't really like if we would distribute this through our mirrors. What
could be an option is only to put this into a package that is always installed
and just regularly updated - but that would be a manual process again and not
possible to update the package without installing all recent Core Updates.

> (b) Error handling for database servers is problematic. In the given scenario:
> What if downloading the AS database fails? Simply using the old version
> without
> notifying is not an option. On the other hand, a notification (i.e. a warning
> in the WebUI) needs to be noticed by the administrator. If they install
> an update, they will see errors instantly.

Yes, continue to use the old version.

> (c) A download server is an unnecessary information leak. Since all systems
> fetch their databases from a - hopefully redundant - server, this server sees
> all IP addresses (and wget user agents, ...) belonging to firewall systems.
> Although it may not be obvious at the first glance that IP x.y.z is an IPFire
> machine,
> it is a good point for hacking tools to start from.

We do have that already for the GeoIP database which I do not like either, but I
think the license doesn't allow us to do anything else.

Some systems are being blocked Internet access by a proxy and probably can't
download anything at all. We need a contingency plan for that.

> This problem becomes more important when talking about third party sites
> (MaxMind!) since nobody knows what they are actually doing with the logs.
> Even worse, if "geolite.maxmind.com" is going offline permanently, there
> is no fallback server where the data can be fetched from.

We would update the URL then...

> I do not consider the IPFire mirrors as problematic here, since there is
> redundancy, and the traffic is split up to more than one system.
> 
> (d) Integrity checking for downloaded databases is problematic. In most
> cases, they are just .tar.gz-files. No HTTPS, no DNSSEC/DANE, no checksum
> file, no GPG signature. Great.

Certainly is a must if you want to go that way... The timestamp also has to be
signed so that an older database cannot be used to replace a newer one.

But we don't do any of this for GeoIP because we can only download this from the
third-party servers.

> And a firewall system is _relying_ on these data???

Yes. If you want features like this you have no other option.

> Of course, this problem needs to be fixed by mirror operators in first place.
> But it is also a question of design: If one assumes that the internet in
> general and the mirror servers in particular cannot be trusted, he/she/it
> will provide signed (and GPG encrypted) data, as IPFire does. Again, why
> not using this way?

We don't trust the mirrors. They are operated by third parties and I have no
idea if and how well they are secured against hacks, etc.

They are also constantly out of sync and slow and just integrity of the files
because of disk corruption isn't guaranteed.

And most people who are operating the servers or have databases like MaxMind
don't seem to know or care about these problems.

> > > > > (c) Rogue ISPs (networks which are controlled/operated by professional
> > > > > spammers or worse, such as the "Russian Business Network" (RBN),
> > > > > which died in end-2007) sometimes run networks located in "good"
> > > > > countries such as US or NL. Blocking them by GeoIP is not an
> > > > > option because of many false-positives. AS based rules may help
> > > > > here.    
> > > > 
> > > > The US is practically unblockable on the GeoIP filter, because too much
> > > > is
> > > > hosted in the US (at least according to the database by businesses that
> > > > have
> > > > their HQ there).  
> > > 
> > > Indeed, and there are several ASs over there the internet would be a
> > > better
> > > place without. But I guess that applies to most other countries, too.  
> > > > 
> > > > So this would be a good extension to blocking more granular.
> > > >   
> > > > > Since the data behind this can be extracted from BGP feeds,
> > > > > no external databases (such as MaxMind) are required.    
> > > > 
> > > > If we would use a BGP feed, we will only have the networks in the
> > > > database
> > > > that
> > > > are currently announced. Wouldn't scraping the WHOIS database be
> > > > better?  
> > > 
> > > Good idea. For example, RIPE has one here:
> > > http://ftp.ripe.net/ripe/dbase/split/inet(6)num.gz  
> > 
> > And where is the link to the ASN?
> 
> Sorry, I used the wrong link while researching for the GeoIP stuff. Here is
> the correct one: https://ftp.ripe.net/ripe/dbase/ripe.db.gz
> 
> As far as I know, the link between "as-block" and an IP range can be done
> via the "mnt-by" or the "origin" tags. 

Okay.

> > 
> > > > 
> > > > Why not MaxMind? Not that I am in favour of that, but I am interested
> > > > why it
> > > > is
> > > > not an option.  
> > > 
> > > Well, technically, it is an option. But since there is some trouble with
> > > their GeoIP database, as mentioned somewhere else some time before, I'd
> > > like to move away from them.  
> > 
> > What trouble?
> 
> Well, some of them are mentioned above already. Here are some more:
> 
> (i) There is no changelog or release note. Once in a month, the GeoIP database
> is updated - and there is no (simple) way of telling which changes were made.
> 
> For example, the IP ranges belonging to Cloudflare were recently deleted from
> this database. As a result, some websites became unreachable from today to
> tomorrow (see bug #11482 for details) since the iptables module permits access
> in case an IP is not listed (which is the correct behaviour).
> 
> Not sure how much went wrong here in productive networks, but it certainly
> is an issue only MaxMind can fix.

That is a problem that every blacklist has. Same goes for the proxy URL
blacklists.

It is hard to track what is on there and probably most people don't really care
much. But in case of disputing if something should be on it or not it is never
good to have a commercial company to make that decision for you.

I suppose we can be much more "objective" about this here with the AS because we
just export information from the RIRs.

> (ii) The database update is performed silently. In case of troubles, this
> procedure might not be obvious. On the other hand, if people install a Core
> Update, they are in maintenance mode, scheduled downtime, or whatever and
> are somewhat prepared for trouble.

Yes. Saying what?

> (iii) The quality of the GeoIP database by MaxMind we use is poor.
> For example, allocations to countries are incorrect (especially in case of
> "A1 - Anonymous Proxies") even if the correct data is known. Much worse,
> sometimes the allocations are incorrect without being obvious (anycast
> networks
> are assigned to "US", or simply dropped in case of Cloudflare). It seems
> like they are only scraping WHOIS, too.

I don't have much resources to verify the quality of it too much. I was hoping
they would do a better job at this.

If they just get this from the RIRs databases, we can do this ourselves.

> There should be an "A3 - Anycast Networks" section for companies like Google,
> similar to "EU" or "AP" which also only indicate a continent without going
> down
> to the country level. A3 would be a step further.

I don't really appreciate the A* categories because they are just a random
collection of what didn't fit anywhere else...

> Note that I am not talking about their chargeable products or the city
> databases here.

> Because of these issues, I'd like to build our own GeoIP database, but that
> is off-topic here, and will be mentioned in another mail. ;-)

Oooookay...

> > 
> > > And since we can build our own database here, I do not see why we should
> > > rely on 3rd party data.  
> > > >   
> > > > > Unfortunately, my programming skills are too low for implementing
> > > > > this feature. Thereof, if it is decided to do this, I will need
> > > > > some help here. :-)    
> > > > 
> > > > *raises hand*  
> > > 
> > > Thank you very much. :-)  
> > > >   
> > > > > Technically, this is similar to the GeoIP firewall stuff (just
> > > > > another database), so I assume most of the work done there can
> > > > > just be copied.    
> > > > 
> > > > The GeoIP block uses an iptables extension which parses the database. We
> > > > wouldn't use that here but would either build something with ipset or
> > > > similar.  
> > > 
> > > Hmmm, not sure how well this performs: If there is only a small AS with
> > > few IP ranges to parse, ipset would be sufficient.
> > > 
> > > However, if we have large networks (Level 3, Hurricane Electric, DTAG,
> > > ...),
> > > ipset might become slow and cause many iptables rules. And if we update
> > > the database (whomever created it), we need to reload the firewall rules.
> > > Using a database lookup module similar to GeoIP avoids that issues here,
> > > as
> > > far as I am concerned.  
> > 
> > I mean it is an option to build an iptables extension, but I am not sure if
> > that
> > is the best thing.
> > 
> > ipset is O(1) for each lookup. So it doesn't matter how large the AS is.
> > iptables rules would be O(n) which is not very nice for larger ASNs, but we
> > should have a look since I do not expect ASNs to announce hundreds of
> > prefixes.
> > The average might be way less and I have no idea what to expect as the
> > worst-
> > case performance. That needs investigation.
> 
> All right.
> 
> @Everybody else: Comments?
> 
> Best regards,
> Peter Müller
> > 
> > Best,
> > -Michael
> > 
> > > 
> > > Best regards,
> > > Peter Müller  
> > > >   
> > > > > Any thoughts on this idea?
> > > > > 
> > > > > Best regards,
> > > > > Peter Müller    
> > > > 
> > > > -Michael  
> > > 
> > >   
> 
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: ASN support for iptables

[Peter Müller]

[-- Attachment #1: Type: text/plain, Size: 20350 bytes --]

Hello,

> On Fri, 2017-11-24 at 19:31 +0100, Peter Müller wrote:
> > Hello,
> >   
> > > Hi,
> > > 
> > > On Mon, 2017-11-20 at 20:03 +0100, Peter Müller wrote:  
> > > > Hello Michael,
> > > >     
> > > > > Hi,
> > > > > 
> > > > > On Sun, 2017-11-19 at 15:52 +0100, Peter Müller wrote:    
> > > > > > Hello development list,
> > > > > > 
> > > > > > today, I'd like to discuss whether a new feature in
> > > > > > the firewall engine of IPFire makes sense or not.
> > > > > > 
> > > > > > Since Core Update 90, IPFire supports GeoIP based firewall
> > > > > > rules, which goes beyond simple IP addresses or CIDR blocks
> > > > > > and makes firewalling easier.
> > > > > > 
> > > > > > The idea I had in mind is to add ASN (Autonomous System
> > > > > > Number) support for firewall rules, too.
> > > > > > 
> > > > > > An AS (Autonomous System) can be described as an administrative
> > > > > > instance on top of IP: For example, several IP blocks belong
> > > > > > to an AS, i.e. to the same company, university or whatever.
> > > > > > Although these blocks may be used for completely different purposes
> > > > > > in completely different countries, they share the same owner.
> > > > > > 
> > > > > > Every AS has a number (ASN) and a description (sometimes
> > > > > > abbreviated to ASDescr), while the number is unique.      
> > > > > 
> > > > > I think this makes sense and I would welcome that as a new feature in
> > > > > IPFire. It
> > > > > will in the end have some similar problems like the GeoIP blocker, but
> > > > > that
> > > > > is
> > > > > not too bad.    
> > > > 
> > > > Just for curiosity: Which ones are they?    
> > > 
> > > Apologies for nothing into depths of that. I just thought of the usual ones
> > > like
> > > not actually blocking what the user intends to block. Rules changing because
> > > prefixes might be moved around, etc.  
> > 
> > Ah, yes, the usual suspects. :-)  
> > > 
> > > It is just not 100% determinable. But indeed it is better than GeoIP.  
> > 
> > True.  
> > >   
> > > > I only see the (potential) problem of _another_ incomplete database here
> > > > (see
> > > > below).
> > > > Since all necessary data for this AS stuff are more or less public,
> > > > inaccurate
> > > > data and extensive manual research are not necessary.
> > > > 
> > > > All we need is to write some scripts that convert the RIPE, ... databases
> > > > to a suitable format.    
> > > 
> > > Does such a public database of the RIRs exist? I don't know of one.  
> > 
> > Yes, they do exist, but mostly without the connection between an IP range and
> > the ASN (see below). At least for RIPE, I have not checked the others in
> > detail.  
> > > 
> > > For IPv4 handing out address space has slowed down a lot. And we should not
> > > use
> > > the BGP feed and parse that because then we might miss prefixes that are
> > > currently not announced - either deliberately or because of a downtime.
> > > 
> > > I know this:
> > >   http://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest
> > > 
> > > But it doesn't connect the prefixes to an ASN AFAIK.
> > >   
> > > > Second, in contrast to the GeoIP database, false-positives are extremely
> > > > rare
> > > > when dealing with AS information since they have to be technically
> > > > correct. As
> > > > always, there are some "good" and "bad" ASNs, and a lot "grey" ones, which
> > > > stays
> > > > the same.    
> > > 
> > > We wouldn't really care about that, would we? That is up to the user to
> > > decide
> > > what they want to block/whitelist.
> > >   
> > > > >     
> > > > > > There are some scenarios in which AS based firewall rules
> > > > > > make sense, since AS information change less seldom than
> > > > > > IP ranges:
> > > > > > 
> > > > > > (a) One wants to block malicious traffic, but blocking entire
> > > > > > countries is too much since there are some legitimate partners,
> > > > > > customers, ... out there. With AS support, it is possible to
> > > > > > grant them access by simply permitting their AS. The rest of
> > > > > > the country may now safely be blocked.      
> > > > > 
> > > > > True. This might work well in some situations, but is probably quite
> > > > > useless
> > > > > when fighting against a botnet.    
> > > > 
> > > > I am not sure about this. In some cases, blocking access to some bullet
> > > > proof/
> > > > rogue ISPs where many C&C servers are hosted improves the situation. Of
> > > > course,
> > > > if there is a simple DDoS attack from many different IPs, this will not
> > > > help
> > > > very much.    
> > > > > 
> > > > > What would also be good is to open a port forwarding only from a certain
> > > > > AS.
> > > > > Let's call that whitelisting.    
> > > > 
> > > > Yes, that is a very good use case. :-)
> > > > 
> > > > And the combination of GeoIP and AS filtering (inbound and outbound)
> > > > opens up fare more possibilities...    
> > > 
> > > How would we combine this?  
> > 
> > I meant that it opens up new possibilities for creating firewall rules
> > (allowing
> > one AS and block/throttle the rest of a country, ...).  
> 
> LOL
> 
> > >   
> > > > > > (b) In some cases, IP ranges change very often, making firewall
> > > > > > rules very complex and hard to maintain, or the exact IP address
> > > > > > of a machine cannot be determined (dial-up connections). In
> > > > > > both cases, the AS (mostly) stays the same and allows firewall
> > > > > > rules without permitting access to a whole country.      
> > > > > 
> > > > > That will be the biggest challenge here. The database will need to be
> > > > > complete
> > > > > and needs regular updates. We don't really care if someone is actually
> > > > > announcing their prefix, but if they have one assigned, we should
> > > > > block/permit
> > > > > access.    
> > > > 
> > > > I agree with you.
> > > > 
> > > > We could simply ship an updated version with every Core Update (I
> > > > had that in mind for the CA list, too, so we don't get too big deltas).    
> > > 
> > > If we don't convert this into some nice binary format this will indeed
> > > become a
> > > huge database of probably ~10MB?! That would be too much to be included in
> > > every
> > > Core Update.  
> > 
> > Is shipping database deltas an option? An maybe every 20th Core Update the
> > complete one so we do not have too much trouble with incremental updates.  
> 
> I don't think that this is worth it since we have to have a format that is
> possible to being delta-ed or we have to create a delta format and a tool which
> applies the delta. Way too much work for a different problem.
Hm, I thought about that a little bit. It surprises me that shipping lager
files (even bzipped or something) is a problem.

In fact, MaxMinds GeoIP database is just a .tar.gz-file containing an aggregated
CSV. So if the machines are able to handle such a download once a month, I thought
they would also handle updates which are a few MBytes lager.
> 
> In that case, text probably won't be the worst format and just applying a diff.
Yes, that's what I had in mind.
> 
> > If there is a nice small binary format, we should use it indeed. I am not very
> > deep into this subject. :-|  
> 
> Not that I know of.
> 
> > Shipping the database with every update avoids fetching them by the systems
> > itself, in which I see a set of issues:
> > 
> > (a) A download server for the database must be reachable. In some networks
> > with restrictive policies, this might be a problem. On the other hand, if
> > people install a Core Update, everybody understands that IPFire needs to
> > connect to some mirror servers. So why not shipping the databases via this
> > way.
> > 
> > (The idea behind this is that every network connection can go wrong. A bit
> > ancient, I know. :-) )  
> 
> We will have to find out if we are free to redistribute this data.
> 
> I wouldn't really like if we would distribute this through our mirrors. What
> could be an option is only to put this into a package that is always installed
> and just regularly updated - but that would be a manual process again and not
> possible to update the package without installing all recent Core Updates.
Again, I thought that data amount would be fine. Glad you are mentioning it is not.
> 
> > (b) Error handling for database servers is problematic. In the given scenario:
> > What if downloading the AS database fails? Simply using the old version
> > without
> > notifying is not an option. On the other hand, a notification (i.e. a warning
> > in the WebUI) needs to be noticed by the administrator. If they install
> > an update, they will see errors instantly.  
> 
> Yes, continue to use the old version.
Not very nice, is it? I saw some of the download servers being out of sync for
~ two months, which is also very difficult to handle since there is no DNS entry
showing the current version, similar to what ClamAV does.

I would prefer a notification in the WebUI here, too, so the admins know that
their machine is using probably outdated information.
> 
> > (c) A download server is an unnecessary information leak. Since all systems
> > fetch their databases from a - hopefully redundant - server, this server sees
> > all IP addresses (and wget user agents, ...) belonging to firewall systems.
> > Although it may not be obvious at the first glance that IP x.y.z is an IPFire
> > machine,
> > it is a good point for hacking tools to start from.  
> 
> We do have that already for the GeoIP database which I do not like either, but I
> think the license doesn't allow us to do anything else.
> 
> Some systems are being blocked Internet access by a proxy and probably can't
> download anything at all. We need a contingency plan for that.
> 
> > This problem becomes more important when talking about third party sites
> > (MaxMind!) since nobody knows what they are actually doing with the logs.
> > Even worse, if "geolite.maxmind.com" is going offline permanently, there
> > is no fallback server where the data can be fetched from.  
> 
> We would update the URL then...
> 
> > I do not consider the IPFire mirrors as problematic here, since there is
> > redundancy, and the traffic is split up to more than one system.
> > 
> > (d) Integrity checking for downloaded databases is problematic. In most
> > cases, they are just .tar.gz-files. No HTTPS, no DNSSEC/DANE, no checksum
> > file, no GPG signature. Great.  
> 
> Certainly is a must if you want to go that way... The timestamp also has to be
> signed so that an older database cannot be used to replace a newer one.
Indeed.
> 
> But we don't do any of this for GeoIP because we can only download this from the
> third-party servers.
> 
> > And a firewall system is _relying_ on these data???  
> 
> Yes. If you want features like this you have no other option.
Except building and shipping the databases (ASN and GeoIP) by ourselves.
> 
> > Of course, this problem needs to be fixed by mirror operators in first place.
> > But it is also a question of design: If one assumes that the internet in
> > general and the mirror servers in particular cannot be trusted, he/she/it
> > will provide signed (and GPG encrypted) data, as IPFire does. Again, why
> > not using this way?  
> 
> We don't trust the mirrors. They are operated by third parties and I have no
> idea if and how well they are secured against hacks, etc.
> 
> They are also constantly out of sync and slow and just integrity of the files
> because of disk corruption isn't guaranteed.
> 
> And most people who are operating the servers or have databases like MaxMind
> don't seem to know or care about these problems.
> 
> > > > > > (c) Rogue ISPs (networks which are controlled/operated by professional
> > > > > > spammers or worse, such as the "Russian Business Network" (RBN),
> > > > > > which died in end-2007) sometimes run networks located in "good"
> > > > > > countries such as US or NL. Blocking them by GeoIP is not an
> > > > > > option because of many false-positives. AS based rules may help
> > > > > > here.      
> > > > > 
> > > > > The US is practically unblockable on the GeoIP filter, because too much
> > > > > is
> > > > > hosted in the US (at least according to the database by businesses that
> > > > > have
> > > > > their HQ there).    
> > > > 
> > > > Indeed, and there are several ASs over there the internet would be a
> > > > better
> > > > place without. But I guess that applies to most other countries, too.    
> > > > > 
> > > > > So this would be a good extension to blocking more granular.
> > > > >     
> > > > > > Since the data behind this can be extracted from BGP feeds,
> > > > > > no external databases (such as MaxMind) are required.      
> > > > > 
> > > > > If we would use a BGP feed, we will only have the networks in the
> > > > > database
> > > > > that
> > > > > are currently announced. Wouldn't scraping the WHOIS database be
> > > > > better?    
> > > > 
> > > > Good idea. For example, RIPE has one here:
> > > > http://ftp.ripe.net/ripe/dbase/split/inet(6)num.gz    
> > > 
> > > And where is the link to the ASN?  
> > 
> > Sorry, I used the wrong link while researching for the GeoIP stuff. Here is
> > the correct one: https://ftp.ripe.net/ripe/dbase/ripe.db.gz
> > 
> > As far as I know, the link between "as-block" and an IP range can be done
> > via the "mnt-by" or the "origin" tags.   
> 
> Okay.
> 
> > >   
> > > > > 
> > > > > Why not MaxMind? Not that I am in favour of that, but I am interested
> > > > > why it
> > > > > is
> > > > > not an option.    
> > > > 
> > > > Well, technically, it is an option. But since there is some trouble with
> > > > their GeoIP database, as mentioned somewhere else some time before, I'd
> > > > like to move away from them.    
> > > 
> > > What trouble?  
> > 
> > Well, some of them are mentioned above already. Here are some more:
> > 
> > (i) There is no changelog or release note. Once in a month, the GeoIP database
> > is updated - and there is no (simple) way of telling which changes were made.
> > 
> > For example, the IP ranges belonging to Cloudflare were recently deleted from
> > this database. As a result, some websites became unreachable from today to
> > tomorrow (see bug #11482 for details) since the iptables module permits access
> > in case an IP is not listed (which is the correct behaviour).
> > 
> > Not sure how much went wrong here in productive networks, but it certainly
> > is an issue only MaxMind can fix.  
> 
> That is a problem that every blacklist has. Same goes for the proxy URL
> blacklists.
> 
> It is hard to track what is on there and probably most people don't really care
> much. But in case of disputing if something should be on it or not it is never
> good to have a commercial company to make that decision for you.
> 
> I suppose we can be much more "objective" about this here with the AS because we
> just export information from the RIRs.
I agree.
> 
> > (ii) The database update is performed silently. In case of troubles, this
> > procedure might not be obvious. On the other hand, if people install a Core
> > Update, they are in maintenance mode, scheduled downtime, or whatever and
> > are somewhat prepared for trouble.  
> 
> Yes. Saying what?
Saying that people are aware that updating might cause problems, so they are
prepared to test and fix something afterwards. In case a new database version
causes network trouble because some firewall rules do not apply anymore, they
can react to this instantly without fixing a productive system.

If the database is updated silently at a random time, trouble occur out of
nowhere and might break things since the firewall is actively used.

Because of that, I like updating the AS database via a Core Update better.
> 
> > (iii) The quality of the GeoIP database by MaxMind we use is poor.
> > For example, allocations to countries are incorrect (especially in case of
> > "A1 - Anonymous Proxies") even if the correct data is known. Much worse,
> > sometimes the allocations are incorrect without being obvious (anycast
> > networks
> > are assigned to "US", or simply dropped in case of Cloudflare). It seems
> > like they are only scraping WHOIS, too.  
> 
> I don't have much resources to verify the quality of it too much. I was hoping
> they would do a better job at this.
Not sure what they are doing with their commercial database editions, but the
free one is not that good indeed. For example, all AOL IP ranges are listed in
US, even if they are used in Brazil or other countries.
> 
> If they just get this from the RIRs databases, we can do this ourselves.
I think so but currently doing some research how to build this thing.
> 
> > There should be an "A3 - Anycast Networks" section for companies like Google,
> > similar to "EU" or "AP" which also only indicate a continent without going
> > down
> > to the country level. A3 would be a step further.  
> 
> I don't really appreciate the A* categories because they are just a random
> collection of what didn't fit anywhere else...
But they are actually needed since proxies, satellite and anycast network _do_
not fit into anything else. We also have EU for networks somewhere in Europe,
and there is AP for locations in Asia/Pacific. I did not really get your point here. :-)
> 
> > Note that I am not talking about their chargeable products or the city
> > databases here.  
> 
> > Because of these issues, I'd like to build our own GeoIP database, but that
> > is off-topic here, and will be mentioned in another mail. ;-)  
> 
> Oooookay...
> 
> > >   
> > > > And since we can build our own database here, I do not see why we should
> > > > rely on 3rd party data.    
> > > > >     
> > > > > > Unfortunately, my programming skills are too low for implementing
> > > > > > this feature. Thereof, if it is decided to do this, I will need
> > > > > > some help here. :-)      
> > > > > 
> > > > > *raises hand*    
> > > > 
> > > > Thank you very much. :-)    
> > > > >     
> > > > > > Technically, this is similar to the GeoIP firewall stuff (just
> > > > > > another database), so I assume most of the work done there can
> > > > > > just be copied.      
> > > > > 
> > > > > The GeoIP block uses an iptables extension which parses the database. We
> > > > > wouldn't use that here but would either build something with ipset or
> > > > > similar.    
> > > > 
> > > > Hmmm, not sure how well this performs: If there is only a small AS with
> > > > few IP ranges to parse, ipset would be sufficient.
> > > > 
> > > > However, if we have large networks (Level 3, Hurricane Electric, DTAG,
> > > > ...),
> > > > ipset might become slow and cause many iptables rules. And if we update
> > > > the database (whomever created it), we need to reload the firewall rules.
> > > > Using a database lookup module similar to GeoIP avoids that issues here,
> > > > as
> > > > far as I am concerned.    
> > > 
> > > I mean it is an option to build an iptables extension, but I am not sure if
> > > that
> > > is the best thing.
> > > 
> > > ipset is O(1) for each lookup. So it doesn't matter how large the AS is.
> > > iptables rules would be O(n) which is not very nice for larger ASNs, but we
> > > should have a look since I do not expect ASNs to announce hundreds of
> > > prefixes.
> > > The average might be way less and I have no idea what to expect as the
> > > worst-
> > > case performance. That needs investigation.  
> > 
> > All right.
> > 
> > @Everybody else: Comments?
> > 
> > Best regards,
> > Peter Müller  
> > > 
> > > Best,
> > > -Michael
> > >   
> > > > 
> > > > Best regards,
> > > > Peter Müller    
> > > > >     
> > > > > > Any thoughts on this idea?
> > > > > > 
> > > > > > Best regards,
> > > > > > Peter Müller      
> > > > > 
> > > > > -Michael    
> > > > 
> > > >     
> > 
> >   
Best regards,
Peter Müller

Re: ASN support for iptables

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 23046 bytes --]

On Mon, 2017-12-04 at 17:47 +0100, Peter Müller wrote:
> Hello,
> 
> > On Fri, 2017-11-24 at 19:31 +0100, Peter Müller wrote:
> > > Hello,
> > >   
> > > > Hi,
> > > > 
> > > > On Mon, 2017-11-20 at 20:03 +0100, Peter Müller wrote:  
> > > > > Hello Michael,
> > > > >     
> > > > > > Hi,
> > > > > > 
> > > > > > On Sun, 2017-11-19 at 15:52 +0100, Peter Müller wrote:    
> > > > > > > Hello development list,
> > > > > > > 
> > > > > > > today, I'd like to discuss whether a new feature in
> > > > > > > the firewall engine of IPFire makes sense or not.
> > > > > > > 
> > > > > > > Since Core Update 90, IPFire supports GeoIP based firewall
> > > > > > > rules, which goes beyond simple IP addresses or CIDR blocks
> > > > > > > and makes firewalling easier.
> > > > > > > 
> > > > > > > The idea I had in mind is to add ASN (Autonomous System
> > > > > > > Number) support for firewall rules, too.
> > > > > > > 
> > > > > > > An AS (Autonomous System) can be described as an administrative
> > > > > > > instance on top of IP: For example, several IP blocks belong
> > > > > > > to an AS, i.e. to the same company, university or whatever.
> > > > > > > Although these blocks may be used for completely different
> > > > > > > purposes
> > > > > > > in completely different countries, they share the same owner.
> > > > > > > 
> > > > > > > Every AS has a number (ASN) and a description (sometimes
> > > > > > > abbreviated to ASDescr), while the number is unique.      
> > > > > > 
> > > > > > I think this makes sense and I would welcome that as a new feature
> > > > > > in
> > > > > > IPFire. It
> > > > > > will in the end have some similar problems like the GeoIP blocker,
> > > > > > but
> > > > > > that
> > > > > > is
> > > > > > not too bad.    
> > > > > 
> > > > > Just for curiosity: Which ones are they?    
> > > > 
> > > > Apologies for nothing into depths of that. I just thought of the usual
> > > > ones
> > > > like
> > > > not actually blocking what the user intends to block. Rules changing
> > > > because
> > > > prefixes might be moved around, etc.  
> > > 
> > > Ah, yes, the usual suspects. :-)  
> > > > 
> > > > It is just not 100% determinable. But indeed it is better than GeoIP.  
> > > 
> > > True.  
> > > >   
> > > > > I only see the (potential) problem of _another_ incomplete database
> > > > > here
> > > > > (see
> > > > > below).
> > > > > Since all necessary data for this AS stuff are more or less public,
> > > > > inaccurate
> > > > > data and extensive manual research are not necessary.
> > > > > 
> > > > > All we need is to write some scripts that convert the RIPE, ...
> > > > > databases
> > > > > to a suitable format.    
> > > > 
> > > > Does such a public database of the RIRs exist? I don't know of one.  
> > > 
> > > Yes, they do exist, but mostly without the connection between an IP range
> > > and
> > > the ASN (see below). At least for RIPE, I have not checked the others in
> > > detail.  
> > > > 
> > > > For IPv4 handing out address space has slowed down a lot. And we should
> > > > not
> > > > use
> > > > the BGP feed and parse that because then we might miss prefixes that are
> > > > currently not announced - either deliberately or because of a downtime.
> > > > 
> > > > I know this:
> > > >   http://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-lates
> > > > t
> > > > 
> > > > But it doesn't connect the prefixes to an ASN AFAIK.
> > > >   
> > > > > Second, in contrast to the GeoIP database, false-positives are
> > > > > extremely
> > > > > rare
> > > > > when dealing with AS information since they have to be technically
> > > > > correct. As
> > > > > always, there are some "good" and "bad" ASNs, and a lot "grey" ones,
> > > > > which
> > > > > stays
> > > > > the same.    
> > > > 
> > > > We wouldn't really care about that, would we? That is up to the user to
> > > > decide
> > > > what they want to block/whitelist.
> > > >   
> > > > > >     
> > > > > > > There are some scenarios in which AS based firewall rules
> > > > > > > make sense, since AS information change less seldom than
> > > > > > > IP ranges:
> > > > > > > 
> > > > > > > (a) One wants to block malicious traffic, but blocking entire
> > > > > > > countries is too much since there are some legitimate partners,
> > > > > > > customers, ... out there. With AS support, it is possible to
> > > > > > > grant them access by simply permitting their AS. The rest of
> > > > > > > the country may now safely be blocked.      
> > > > > > 
> > > > > > True. This might work well in some situations, but is probably quite
> > > > > > useless
> > > > > > when fighting against a botnet.    
> > > > > 
> > > > > I am not sure about this. In some cases, blocking access to some
> > > > > bullet
> > > > > proof/
> > > > > rogue ISPs where many C&C servers are hosted improves the situation.
> > > > > Of
> > > > > course,
> > > > > if there is a simple DDoS attack from many different IPs, this will
> > > > > not
> > > > > help
> > > > > very much.    
> > > > > > 
> > > > > > What would also be good is to open a port forwarding only from a
> > > > > > certain
> > > > > > AS.
> > > > > > Let's call that whitelisting.    
> > > > > 
> > > > > Yes, that is a very good use case. :-)
> > > > > 
> > > > > And the combination of GeoIP and AS filtering (inbound and outbound)
> > > > > opens up fare more possibilities...    
> > > > 
> > > > How would we combine this?  
> > > 
> > > I meant that it opens up new possibilities for creating firewall rules
> > > (allowing
> > > one AS and block/throttle the rest of a country, ...).  
> > 
> > LOL
> > 
> > > >   
> > > > > > > (b) In some cases, IP ranges change very often, making firewall
> > > > > > > rules very complex and hard to maintain, or the exact IP address
> > > > > > > of a machine cannot be determined (dial-up connections). In
> > > > > > > both cases, the AS (mostly) stays the same and allows firewall
> > > > > > > rules without permitting access to a whole country.      
> > > > > > 
> > > > > > That will be the biggest challenge here. The database will need to
> > > > > > be
> > > > > > complete
> > > > > > and needs regular updates. We don't really care if someone is
> > > > > > actually
> > > > > > announcing their prefix, but if they have one assigned, we should
> > > > > > block/permit
> > > > > > access.    
> > > > > 
> > > > > I agree with you.
> > > > > 
> > > > > We could simply ship an updated version with every Core Update (I
> > > > > had that in mind for the CA list, too, so we don't get too big
> > > > > deltas).    
> > > > 
> > > > If we don't convert this into some nice binary format this will indeed
> > > > become a
> > > > huge database of probably ~10MB?! That would be too much to be included
> > > > in
> > > > every
> > > > Core Update.  
> > > 
> > > Is shipping database deltas an option? An maybe every 20th Core Update the
> > > complete one so we do not have too much trouble with incremental
> > > updates.  
> > 
> > I don't think that this is worth it since we have to have a format that is
> > possible to being delta-ed or we have to create a delta format and a tool
> > which
> > applies the delta. Way too much work for a different problem.
> 
> Hm, I thought about that a little bit. It surprises me that shipping lager
> files (even bzipped or something) is a problem.
> 
> In fact, MaxMinds GeoIP database is just a .tar.gz-file containing an
> aggregated
> CSV. So if the machines are able to handle such a download once a month, I
> thought
> they would also handle updates which are a few MBytes lager.

The problem we have is a maximum size for the core update. With some other big
things like the kernel being shipped we get quite close to that limit.

Therefore it would be nice to avoid a certain amount of that size being used up
by those databases.

> > 
> > In that case, text probably won't be the worst format and just applying a
> > diff.
> 
> Yes, that's what I had in mind.
> > 
> > > If there is a nice small binary format, we should use it indeed. I am not
> > > very
> > > deep into this subject. :-|  
> > 
> > Not that I know of.
> > 
> > > Shipping the database with every update avoids fetching them by the
> > > systems
> > > itself, in which I see a set of issues:
> > > 
> > > (a) A download server for the database must be reachable. In some networks
> > > with restrictive policies, this might be a problem. On the other hand, if
> > > people install a Core Update, everybody understands that IPFire needs to
> > > connect to some mirror servers. So why not shipping the databases via this
> > > way.
> > > 
> > > (The idea behind this is that every network connection can go wrong. A bit
> > > ancient, I know. :-) )  
> > 
> > We will have to find out if we are free to redistribute this data.
> > 
> > I wouldn't really like if we would distribute this through our mirrors. What
> > could be an option is only to put this into a package that is always
> > installed
> > and just regularly updated - but that would be a manual process again and
> > not
> > possible to update the package without installing all recent Core Updates.
> 
> Again, I thought that data amount would be fine. Glad you are mentioning it is
> not.
> > 
> > > (b) Error handling for database servers is problematic. In the given
> > > scenario:
> > > What if downloading the AS database fails? Simply using the old version
> > > without
> > > notifying is not an option. On the other hand, a notification (i.e. a
> > > warning
> > > in the WebUI) needs to be noticed by the administrator. If they install
> > > an update, they will see errors instantly.  
> > 
> > Yes, continue to use the old version.
> 
> Not very nice, is it? I saw some of the download servers being out of sync for
> ~ two months, which is also very difficult to handle since there is no DNS
> entry
> showing the current version, similar to what ClamAV does.

I quite like the ClamAV approach. However, that information needs to be signed
(either by DNSSEC or something else).

And another downside is that it brings another protocol in which can cause more
problems when being blocked behind another firewall or when there are other
problems. Want to say: It makes the solution more complex.

> I would prefer a notification in the WebUI here, too, so the admins know that
> their machine is using probably outdated information.

I never considered the database to be important to be up to date. We don't know
how good and recent the information inside it is and assume it is rather old
sometime. Therefore it is very often the case that the information is not of
good quality. What ever the reason for that is.

> > 
> > > (c) A download server is an unnecessary information leak. Since all
> > > systems
> > > fetch their databases from a - hopefully redundant - server, this server
> > > sees
> > > all IP addresses (and wget user agents, ...) belonging to firewall
> > > systems.
> > > Although it may not be obvious at the first glance that IP x.y.z is an
> > > IPFire
> > > machine,
> > > it is a good point for hacking tools to start from.  
> > 
> > We do have that already for the GeoIP database which I do not like either,
> > but I
> > think the license doesn't allow us to do anything else.
> > 
> > Some systems are being blocked Internet access by a proxy and probably can't
> > download anything at all. We need a contingency plan for that.
> > 
> > > This problem becomes more important when talking about third party sites
> > > (MaxMind!) since nobody knows what they are actually doing with the logs.
> > > Even worse, if "geolite.maxmind.com" is going offline permanently, there
> > > is no fallback server where the data can be fetched from.  
> > 
> > We would update the URL then...
> > 
> > > I do not consider the IPFire mirrors as problematic here, since there is
> > > redundancy, and the traffic is split up to more than one system.
> > > 
> > > (d) Integrity checking for downloaded databases is problematic. In most
> > > cases, they are just .tar.gz-files. No HTTPS, no DNSSEC/DANE, no checksum
> > > file, no GPG signature. Great.  
> > 
> > Certainly is a must if you want to go that way... The timestamp also has to
> > be
> > signed so that an older database cannot be used to replace a newer one.
> 
> Indeed.
> > 
> > But we don't do any of this for GeoIP because we can only download this from
> > the
> > third-party servers.
> > 
> > > And a firewall system is _relying_ on these data???  
> > 
> > Yes. If you want features like this you have no other option.
> 
> Except building and shipping the databases (ASN and GeoIP) by ourselves.

Let's do that then!

> > 
> > > Of course, this problem needs to be fixed by mirror operators in first
> > > place.
> > > But it is also a question of design: If one assumes that the internet in
> > > general and the mirror servers in particular cannot be trusted, he/she/it
> > > will provide signed (and GPG encrypted) data, as IPFire does. Again, why
> > > not using this way?  
> > 
> > We don't trust the mirrors. They are operated by third parties and I have no
> > idea if and how well they are secured against hacks, etc.
> > 
> > They are also constantly out of sync and slow and just integrity of the
> > files
> > because of disk corruption isn't guaranteed.
> > 
> > And most people who are operating the servers or have databases like MaxMind
> > don't seem to know or care about these problems.
> > 
> > > > > > > (c) Rogue ISPs (networks which are controlled/operated by
> > > > > > > professional
> > > > > > > spammers or worse, such as the "Russian Business Network" (RBN),
> > > > > > > which died in end-2007) sometimes run networks located in "good"
> > > > > > > countries such as US or NL. Blocking them by GeoIP is not an
> > > > > > > option because of many false-positives. AS based rules may help
> > > > > > > here.      
> > > > > > 
> > > > > > The US is practically unblockable on the GeoIP filter, because too
> > > > > > much
> > > > > > is
> > > > > > hosted in the US (at least according to the database by businesses
> > > > > > that
> > > > > > have
> > > > > > their HQ there).    
> > > > > 
> > > > > Indeed, and there are several ASs over there the internet would be a
> > > > > better
> > > > > place without. But I guess that applies to most other countries,
> > > > > too.    
> > > > > > 
> > > > > > So this would be a good extension to blocking more granular.
> > > > > >     
> > > > > > > Since the data behind this can be extracted from BGP feeds,
> > > > > > > no external databases (such as MaxMind) are required.      
> > > > > > 
> > > > > > If we would use a BGP feed, we will only have the networks in the
> > > > > > database
> > > > > > that
> > > > > > are currently announced. Wouldn't scraping the WHOIS database be
> > > > > > better?    
> > > > > 
> > > > > Good idea. For example, RIPE has one here:
> > > > > http://ftp.ripe.net/ripe/dbase/split/inet(6)num.gz    
> > > > 
> > > > And where is the link to the ASN?  
> > > 
> > > Sorry, I used the wrong link while researching for the GeoIP stuff. Here
> > > is
> > > the correct one: https://ftp.ripe.net/ripe/dbase/ripe.db.gz
> > > 
> > > As far as I know, the link between "as-block" and an IP range can be done
> > > via the "mnt-by" or the "origin" tags.   
> > 
> > Okay.
> > 
> > > >   
> > > > > > 
> > > > > > Why not MaxMind? Not that I am in favour of that, but I am
> > > > > > interested
> > > > > > why it
> > > > > > is
> > > > > > not an option.    
> > > > > 
> > > > > Well, technically, it is an option. But since there is some trouble
> > > > > with
> > > > > their GeoIP database, as mentioned somewhere else some time before,
> > > > > I'd
> > > > > like to move away from them.    
> > > > 
> > > > What trouble?  
> > > 
> > > Well, some of them are mentioned above already. Here are some more:
> > > 
> > > (i) There is no changelog or release note. Once in a month, the GeoIP
> > > database
> > > is updated - and there is no (simple) way of telling which changes were
> > > made.
> > > 
> > > For example, the IP ranges belonging to Cloudflare were recently deleted
> > > from
> > > this database. As a result, some websites became unreachable from today to
> > > tomorrow (see bug #11482 for details) since the iptables module permits
> > > access
> > > in case an IP is not listed (which is the correct behaviour).
> > > 
> > > Not sure how much went wrong here in productive networks, but it certainly
> > > is an issue only MaxMind can fix.  
> > 
> > That is a problem that every blacklist has. Same goes for the proxy URL
> > blacklists.
> > 
> > It is hard to track what is on there and probably most people don't really
> > care
> > much. But in case of disputing if something should be on it or not it is
> > never
> > good to have a commercial company to make that decision for you.
> > 
> > I suppose we can be much more "objective" about this here with the AS
> > because we
> > just export information from the RIRs.
> 
> I agree.
> > 
> > > (ii) The database update is performed silently. In case of troubles, this
> > > procedure might not be obvious. On the other hand, if people install a
> > > Core
> > > Update, they are in maintenance mode, scheduled downtime, or whatever and
> > > are somewhat prepared for trouble.  
> > 
> > Yes. Saying what?
> 
> Saying that people are aware that updating might cause problems, so they are
> prepared to test and fix something afterwards. In case a new database version
> causes network trouble because some firewall rules do not apply anymore, they
> can react to this instantly without fixing a productive system.
> 
> If the database is updated silently at a random time, trouble occur out of
> nowhere and might break things since the firewall is actively used.
> 
> Because of that, I like updating the AS database via a Core Update better.
> > 
> > > (iii) The quality of the GeoIP database by MaxMind we use is poor.
> > > For example, allocations to countries are incorrect (especially in case of
> > > "A1 - Anonymous Proxies") even if the correct data is known. Much worse,
> > > sometimes the allocations are incorrect without being obvious (anycast
> > > networks
> > > are assigned to "US", or simply dropped in case of Cloudflare). It seems
> > > like they are only scraping WHOIS, too.  
> > 
> > I don't have much resources to verify the quality of it too much. I was
> > hoping
> > they would do a better job at this.
> 
> Not sure what they are doing with their commercial database editions, but the
> free one is not that good indeed. For example, all AOL IP ranges are listed in
> US, even if they are used in Brazil or other countries.
> > 
> > If they just get this from the RIRs databases, we can do this ourselves.
> 
> I think so but currently doing some research how to build this thing.
> > 
> > > There should be an "A3 - Anycast Networks" section for companies like
> > > Google,
> > > similar to "EU" or "AP" which also only indicate a continent without going
> > > down
> > > to the country level. A3 would be a step further.  
> > 
> > I don't really appreciate the A* categories because they are just a random
> > collection of what didn't fit anywhere else...
> 
> But they are actually needed since proxies, satellite and anycast network _do_
> not fit into anything else. We also have EU for networks somewhere in Europe,
> and there is AP for locations in Asia/Pacific. I did not really get your point
> here. :-)

My point is that blocking those categories is a very very vage guess. Like a box
of chocolates. You never know what you are gonna get.

> > 
> > > Note that I am not talking about their chargeable products or the city
> > > databases here.  
> > > Because of these issues, I'd like to build our own GeoIP database, but
> > > that
> > > is off-topic here, and will be mentioned in another mail. ;-)  
> > 
> > Oooookay...
> > 
> > > >   
> > > > > And since we can build our own database here, I do not see why we
> > > > > should
> > > > > rely on 3rd party data.    
> > > > > >     
> > > > > > > Unfortunately, my programming skills are too low for implementing
> > > > > > > this feature. Thereof, if it is decided to do this, I will need
> > > > > > > some help here. :-)      
> > > > > > 
> > > > > > *raises hand*    
> > > > > 
> > > > > Thank you very much. :-)    
> > > > > >     
> > > > > > > Technically, this is similar to the GeoIP firewall stuff (just
> > > > > > > another database), so I assume most of the work done there can
> > > > > > > just be copied.      
> > > > > > 
> > > > > > The GeoIP block uses an iptables extension which parses the
> > > > > > database. We
> > > > > > wouldn't use that here but would either build something with ipset
> > > > > > or
> > > > > > similar.    
> > > > > 
> > > > > Hmmm, not sure how well this performs: If there is only a small AS
> > > > > with
> > > > > few IP ranges to parse, ipset would be sufficient.
> > > > > 
> > > > > However, if we have large networks (Level 3, Hurricane Electric, DTAG,
> > > > > ...),
> > > > > ipset might become slow and cause many iptables rules. And if we
> > > > > update
> > > > > the database (whomever created it), we need to reload the firewall
> > > > > rules.
> > > > > Using a database lookup module similar to GeoIP avoids that issues
> > > > > here,
> > > > > as
> > > > > far as I am concerned.    
> > > > 
> > > > I mean it is an option to build an iptables extension, but I am not sure
> > > > if
> > > > that
> > > > is the best thing.
> > > > 
> > > > ipset is O(1) for each lookup. So it doesn't matter how large the AS is.
> > > > iptables rules would be O(n) which is not very nice for larger ASNs, but
> > > > we
> > > > should have a look since I do not expect ASNs to announce hundreds of
> > > > prefixes.
> > > > The average might be way less and I have no idea what to expect as the
> > > > worst-
> > > > case performance. That needs investigation.  
> > > 
> > > All right.
> > > 
> > > @Everybody else: Comments?
> > > 
> > > Best regards,
> > > Peter Müller  
> > > > 
> > > > Best,
> > > > -Michael
> > > >   
> > > > > 
> > > > > Best regards,
> > > > > Peter Müller    
> > > > > >     
> > > > > > > Any thoughts on this idea?
> > > > > > > 
> > > > > > > Best regards,
> > > > > > > Peter Müller      
> > > > > > 
> > > > > > -Michael    
> > > > > 
> > > > >     
> > > 
> > >   
> 
> Best regards,
> Peter Müller

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]