unbound startup

unbound startup

[Tapani Tarvainen]

[-- Attachment #1: Type: text/plain, Size: 3562 bytes --]

Hi all,

I'm only starting to use IPFire, so apologies if I'm missing some
conventions here. I'm fairly experienced with Linux and firewalls in
general, however.

I found some operations in IPFire unexpectedly slow, in particular
Edit Hosts: every operation, whether changing a host name or just
disabling one, took about five minutes to complete.

The machine in question is rather slow but not *that* slow
(1GHz Via C3, 1GB RAM).

What's more, during that time DNS is broken: ipfire responds to
queries but gives wrong answers, specifically NXDOMAIN for hosts that
do exist. And that was causing problems with my internal mail server.


So I looked at the source.

It turns out that every operation in Edit Hosts triggers
unbound restart, and that's where the time goes:

# time /etc/init.d/unbound restart
[...]

real    4m7.531s
user    1m55.320s
sys     0m7.760s

Looking at the script, this is where it spends most of the time:


update_hosts() {
	local enabled address hostname domainname

	while IFS="," read -r enabled address hostname domainname; do
		[ "${enabled}" = "on" ] || continue

		# Build FQDN
		local fqdn="${hostname}.${domainname}"

		unbound-control -q local_data "${fqdn} ${LOCAL_TTL} IN A ${address}"

		# Skip reverse resolution if the address equals the GREEN address
		[ "${address}" = "${GREEN_ADDRESS}" ] && continue

		# Add RDNS
		address=$(ip_address_revptr ${address})
		unbound-control -q local_data "${address} ${LOCAL_TTL} IN PTR ${fqdn}"
	done < /var/ipfire/main/hosts
}


I have roughly 150 entries in hosts list, so that ends up calling
unbound-control about 300 times. And there's a race condition between
the time unbound is started and when those entries are added (at
different times).

I made the following simple change:


update_hosts() {
	local enabled address hostname domainname

	while IFS="," read -r enabled address hostname domainname; do
		[ "${enabled}" = "on" ] || continue

		# Build FQDN
		local fqdn="${hostname}.${domainname}"

		echo "${fqdn} ${LOCAL_TTL} IN A ${address}"

		# Skip reverse resolution if the address equals the GREEN address
		[ "${address}" = "${GREEN_ADDRESS}" ] && continue

		# Add RDNS
		address=$(ip_address_revptr ${address})
		echo "${address} ${LOCAL_TTL} IN PTR ${fqdn}"
	done < /var/ipfire/main/hosts |	unbound-control -q local_datas
}


Result:

# time /etc/init.d/unbound restart
[...]

real    0m15.568s
user    0m4.827s
sys     0m1.403s


So it saves in my case four minutes at every hosts change, every dhcp change,
every boot. Still not blazingly fast but already tolerably so.


That is small and obvious enough change that perhaps it could be
considered for next upgrade (121)?

I can submit it as a patch if that helps.


It's still not a good fix though: it doesn't remove the race condition
window, only shortens it. Rather than using unbound-control to add
local entries it'd be better to put them to a file and include that in
unbound.conf (put the file or two in /etc/unbound/local.d/) so they'd
all take effect immediately when unbound is started. This would be a
bit bigger change but probably not much: if people think it'd be
useful I could give it a go.


Of course it's not actually necessary to restart unbound for every
change in hosts or dhcp at all, they could be effected by making
individual changes with unbound-control, but that would be a much
bigger change (I haven't looked at that part of the code in detail
enough to judge how big). But improving the startup time would be
useful even if that's done at some point.


-- 
Tapani Tarvainen

Re: unbound startup

[Tapani Tarvainen]

[-- Attachment #1: Type: text/plain, Size: 1171 bytes --]

With a bit closer look it seems this is more serious than I thought:
it's not only a bug but a security bug.

Under certain circumstances restarting unbound (which as noted happens
with every Edit Hosts &c) can lead to loss of data as well as result
in data leaks outside the firewall.

As it is now, at unbound startup there's a time window when it gives
wrong answers to DNS queries. NXDOMAIN is bad enough and can lead to
data loss in several circumstances, but as it starts forwarders before
populating local hosts it can also return wrong answers in split DNS
situations, that is, return external IP when it should return the
internal one. This is obviously bad if exernal DNS server is
compromised or spoofed, but even when it isn't, connections intended
to intranet machines could go outside when they shouldn't.

Exploiting this deliberately is not all that simple and all really bad
cases I can think of require split DNS setup and knowledge or ability
to guess when unbound is restarted, but some attacks could be set up
to wait for it.

If this list is not the right place for discussing about bugs, please
redirect wherever appropriate.

-- 
Tapani Tarvainen

Re: unbound startup

[Tapani Tarvainen]

[-- Attachment #1: Type: text/plain, Size: 315 bytes --]

On Tue, May 29, 2018 at 02:57:11PM +0300, Tapani Tarvainen (ipfire(a)tapanitarvainen.fi) wrote:

> With a bit closer look it seems this is more serious than I thought:
> it's not only a bug but a security bug.

Submitted to Bugzilla: https://bugzilla.ipfire.org/show_bug.cgi?id=11743

-- 
Tapani Tarvainen

Re: unbound startup

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 1644 bytes --]

On Tue, 2018-05-29 at 14:57 +0300, Tapani Tarvainen wrote:
> With a bit closer look it seems this is more serious than I thought:
> it's not only a bug but a security bug.

I do not consider this a security issue. This might be an information leak
though and cause some unintended behaviour, but that is about it.

> Under certain circumstances restarting unbound (which as noted happens
> with every Edit Hosts &c) can lead to loss of data as well as result
> in data leaks outside the firewall.
> 
> As it is now, at unbound startup there's a time window when it gives
> wrong answers to DNS queries. NXDOMAIN is bad enough and can lead to
> data loss in several circumstances, but as it starts forwarders before
> populating local hosts it can also return wrong answers in split DNS
> situations, that is, return external IP when it should return the
> internal one. This is obviously bad if exernal DNS server is
> compromised or spoofed, but even when it isn't, connections intended
> to intranet machines could go outside when they shouldn't.

Your application should not rely on getting a response from the DNS servers. And
if there is any important data to be sent to somewhere else it should try again.

> Exploiting this deliberately is not all that simple and all really bad
> cases I can think of require split DNS setup and knowledge or ability
> to guess when unbound is restarted, but some attacks could be set up
> to wait for it.
> 
> If this list is not the right place for discussing about bugs, please
> redirect wherever appropriate.

This is the right place. It's the dev list.

-Michael

Re: unbound startup

[Tapani Tarvainen]

[-- Attachment #1: Type: text/plain, Size: 1980 bytes --]

On Tue, May 29, 2018 at 08:30:41PM +0100, Michael Tremer (michael.tremer(a)ipfire.org) wrote:

> I do not consider this a security issue. This might be an information leak
> though and cause some unintended behaviour, but that is about it.
[...]
> Your application should not rely on getting a response from the DNS servers.

Of course not. But not getting a response is different than getting
*wrong* response. And that's what's happening here.

Someone or something attempting to access intranet server could be
directed to an external one instead. I consider that a security issue.

Some sample scenarios:

Assume company X has website www.example.com and their intranet with
same name but different IP at www.example.com/intra/ (not uncommon setup).
If aomeone breaks in the external server they could exploit this bug to
break inside the firewall, e.g., by faking intranet login page they could
collect passwords.

Anybody able to hijack or spoof the external DNS servers could of
course do similar attack also with differently named intranet hosts
(intra.example.com could go to an external machine ran by blackhats).

In email setup with some addresses in @example.com being handled
differently inside the firewall could lead to sensitive messages being
delivered to wrong recipient. E.g., boss(a)example.com only existing
internally and external mail server having wildcard *@example.com
going to some spam filtering company.

That last one does not even require any actual malicious attack.

I don't think it's a catastrophic or even very bad bug, unlikely to be
exploitable in what's probably typical IPFire usage scenario, but it
is a security issue nonetheless.

Unfortunately due to travels I won't be able to do more about this for
a couple of weeks, but I'll leave a box running the quick fix I put
in bugzilla and absent any problems may submit it as a patch later,
or perhaps write a cleaner fix.

-- 
Tapani Tarvainen

Re: unbound startup

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 4779 bytes --]

Hello,

On Mon, 2018-05-28 at 09:17 +0300, Tapani Tarvainen wrote:
> Hi all,
> 
> I'm only starting to use IPFire, so apologies if I'm missing some
> conventions here. I'm fairly experienced with Linux and firewalls in
> general, however.

Welcome :)

> I found some operations in IPFire unexpectedly slow, in particular
> Edit Hosts: every operation, whether changing a host name or just
> disabling one, took about five minutes to complete.

Yes, this is not implemented really well. I was a bit under a time constraint
and so far nobody has complained that this was an issue.

The restart script is called and will import every single entry one at a time.

> The machine in question is rather slow but not *that* slow
> (1GHz Via C3, 1GB RAM).
> 
> What's more, during that time DNS is broken: ipfire responds to
> queries but gives wrong answers, specifically NXDOMAIN for hosts that
> do exist. And that was causing problems with my internal mail server.

Exist in the local configuration or where?

> So I looked at the source.
> 
> It turns out that every operation in Edit Hosts triggers
> unbound restart, and that's where the time goes:
> 
> # time /etc/init.d/unbound restart
> [...]
> 
> real    4m7.531s
> user    1m55.320s
> sys     0m7.760s
> 
> Looking at the script, this is where it spends most of the time:
> 
> 
> update_hosts() {
> 	local enabled address hostname domainname
> 
> 	while IFS="," read -r enabled address hostname domainname; do
> 		[ "${enabled}" = "on" ] || continue
> 
> 		# Build FQDN
> 		local fqdn="${hostname}.${domainname}"
> 
> 		unbound-control -q local_data "${fqdn} ${LOCAL_TTL} IN A
> ${address}"
> 
> 		# Skip reverse resolution if the address equals the GREEN
> address
> 		[ "${address}" = "${GREEN_ADDRESS}" ] && continue
> 
> 		# Add RDNS
> 		address=$(ip_address_revptr ${address})
> 		unbound-control -q local_data "${address} ${LOCAL_TTL} IN PTR
> ${fqdn}"
> 	done < /var/ipfire/main/hosts
> }
> 
> 
> I have roughly 150 entries in hosts list, so that ends up calling
> unbound-control about 300 times. And there's a race condition between
> the time unbound is started and when those entries are added (at
> different times).
> 
> I made the following simple change:
> 
> 
> update_hosts() {
> 	local enabled address hostname domainname
> 
> 	while IFS="," read -r enabled address hostname domainname; do
> 		[ "${enabled}" = "on" ] || continue
> 
> 		# Build FQDN
> 		local fqdn="${hostname}.${domainname}"
> 
> 		echo "${fqdn} ${LOCAL_TTL} IN A ${address}"
> 
> 		# Skip reverse resolution if the address equals the GREEN
> address
> 		[ "${address}" = "${GREEN_ADDRESS}" ] && continue
> 
> 		# Add RDNS
> 		address=$(ip_address_revptr ${address})
> 		echo "${address} ${LOCAL_TTL} IN PTR ${fqdn}"
> 	done < /var/ipfire/main/hosts |	unbound-control -q local_datas
> }

It would have helped to send this as a patch, so the changes are easier to spot.

> Result:
> 
> # time /etc/init.d/unbound restart
> [...]
> 
> real    0m15.568s
> user    0m4.827s
> sys     0m1.403s
> 
> 
> So it saves in my case four minutes at every hosts change, every dhcp change,
> every boot. Still not blazingly fast but already tolerably so.
> 
> 
> That is small and obvious enough change that perhaps it could be
> considered for next upgrade (121)?

Yes, please send this as a patch:

  https://wiki.ipfire.org/devel/submit-patches

> 
> I can submit it as a patch if that helps.
> 
> 
> It's still not a good fix though: it doesn't remove the race condition
> window, only shortens it. Rather than using unbound-control to add
> local entries it'd be better to put them to a file and include that in
> unbound.conf (put the file or two in /etc/unbound/local.d/) so they'd
> all take effect immediately when unbound is started. This would be a
> bit bigger change but probably not much: if people think it'd be
> useful I could give it a go.

If you would want to go ahead and implement that solution that wouldn't be a bad
idea either.

> Of course it's not actually necessary to restart unbound for every
> change in hosts or dhcp at all, they could be effected by making
> individual changes with unbound-control, but that would be a much
> bigger change (I haven't looked at that part of the code in detail
> enough to judge how big). But improving the startup time would be
> useful even if that's done at some point.

It would be best just to change the entry that was touched, but it would be
quite complicated to figure out what has changed and since I thought that most
people would only have a few entries here it didn't look like it was worth
implementing this.

Best,
-Michael