Hi all,
I'm only starting to use IPFire, so apologies if I'm missing some conventions here. I'm fairly experienced with Linux and firewalls in general, however.
I found some operations in IPFire unexpectedly slow, in particular Edit Hosts: every operation, whether changing a host name or just disabling one, took about five minutes to complete.
The machine in question is rather slow but not *that* slow (1GHz Via C3, 1GB RAM).
What's more, during that time DNS is broken: ipfire responds to queries but gives wrong answers, specifically NXDOMAIN for hosts that do exist. And that was causing problems with my internal mail server.
So I looked at the source.
It turns out that every operation in Edit Hosts triggers unbound restart, and that's where the time goes:
# time /etc/init.d/unbound restart [...]
real 4m7.531s user 1m55.320s sys 0m7.760s
Looking at the script, this is where it spends most of the time:
update_hosts() { local enabled address hostname domainname
while IFS="," read -r enabled address hostname domainname; do [ "${enabled}" = "on" ] || continue
# Build FQDN local fqdn="${hostname}.${domainname}"
unbound-control -q local_data "${fqdn} ${LOCAL_TTL} IN A ${address}"
# Skip reverse resolution if the address equals the GREEN address [ "${address}" = "${GREEN_ADDRESS}" ] && continue
# Add RDNS address=$(ip_address_revptr ${address}) unbound-control -q local_data "${address} ${LOCAL_TTL} IN PTR ${fqdn}" done < /var/ipfire/main/hosts }
I have roughly 150 entries in hosts list, so that ends up calling unbound-control about 300 times. And there's a race condition between the time unbound is started and when those entries are added (at different times).
I made the following simple change:
update_hosts() { local enabled address hostname domainname
while IFS="," read -r enabled address hostname domainname; do [ "${enabled}" = "on" ] || continue
# Build FQDN local fqdn="${hostname}.${domainname}"
echo "${fqdn} ${LOCAL_TTL} IN A ${address}"
# Skip reverse resolution if the address equals the GREEN address [ "${address}" = "${GREEN_ADDRESS}" ] && continue
# Add RDNS address=$(ip_address_revptr ${address}) echo "${address} ${LOCAL_TTL} IN PTR ${fqdn}" done < /var/ipfire/main/hosts | unbound-control -q local_datas }
Result:
# time /etc/init.d/unbound restart [...]
real 0m15.568s user 0m4.827s sys 0m1.403s
So it saves in my case four minutes at every hosts change, every dhcp change, every boot. Still not blazingly fast but already tolerably so.
That is small and obvious enough change that perhaps it could be considered for next upgrade (121)?
I can submit it as a patch if that helps.
It's still not a good fix though: it doesn't remove the race condition window, only shortens it. Rather than using unbound-control to add local entries it'd be better to put them to a file and include that in unbound.conf (put the file or two in /etc/unbound/local.d/) so they'd all take effect immediately when unbound is started. This would be a bit bigger change but probably not much: if people think it'd be useful I could give it a go.
Of course it's not actually necessary to restart unbound for every change in hosts or dhcp at all, they could be effected by making individual changes with unbound-control, but that would be a much bigger change (I haven't looked at that part of the code in detail enough to judge how big). But improving the startup time would be useful even if that's done at some point.
With a bit closer look it seems this is more serious than I thought: it's not only a bug but a security bug.
Under certain circumstances restarting unbound (which as noted happens with every Edit Hosts &c) can lead to loss of data as well as result in data leaks outside the firewall.
As it is now, at unbound startup there's a time window when it gives wrong answers to DNS queries. NXDOMAIN is bad enough and can lead to data loss in several circumstances, but as it starts forwarders before populating local hosts it can also return wrong answers in split DNS situations, that is, return external IP when it should return the internal one. This is obviously bad if exernal DNS server is compromised or spoofed, but even when it isn't, connections intended to intranet machines could go outside when they shouldn't.
Exploiting this deliberately is not all that simple and all really bad cases I can think of require split DNS setup and knowledge or ability to guess when unbound is restarted, but some attacks could be set up to wait for it.
If this list is not the right place for discussing about bugs, please redirect wherever appropriate.
On Tue, May 29, 2018 at 02:57:11PM +0300, Tapani Tarvainen (ipfire@tapanitarvainen.fi) wrote:
With a bit closer look it seems this is more serious than I thought: it's not only a bug but a security bug.
Submitted to Bugzilla: https://bugzilla.ipfire.org/show_bug.cgi?id=11743
On Tue, 2018-05-29 at 14:57 +0300, Tapani Tarvainen wrote:
With a bit closer look it seems this is more serious than I thought: it's not only a bug but a security bug.
I do not consider this a security issue. This might be an information leak though and cause some unintended behaviour, but that is about it.
Under certain circumstances restarting unbound (which as noted happens with every Edit Hosts &c) can lead to loss of data as well as result in data leaks outside the firewall.
As it is now, at unbound startup there's a time window when it gives wrong answers to DNS queries. NXDOMAIN is bad enough and can lead to data loss in several circumstances, but as it starts forwarders before populating local hosts it can also return wrong answers in split DNS situations, that is, return external IP when it should return the internal one. This is obviously bad if exernal DNS server is compromised or spoofed, but even when it isn't, connections intended to intranet machines could go outside when they shouldn't.
Your application should not rely on getting a response from the DNS servers. And if there is any important data to be sent to somewhere else it should try again.
Exploiting this deliberately is not all that simple and all really bad cases I can think of require split DNS setup and knowledge or ability to guess when unbound is restarted, but some attacks could be set up to wait for it.
If this list is not the right place for discussing about bugs, please redirect wherever appropriate.
This is the right place. It's the dev list.
-Michael
On Tue, May 29, 2018 at 08:30:41PM +0100, Michael Tremer (michael.tremer@ipfire.org) wrote:
I do not consider this a security issue. This might be an information leak though and cause some unintended behaviour, but that is about it.
[...]
Your application should not rely on getting a response from the DNS servers.
Of course not. But not getting a response is different than getting *wrong* response. And that's what's happening here.
Someone or something attempting to access intranet server could be directed to an external one instead. I consider that a security issue.
Some sample scenarios:
Assume company X has website www.example.com and their intranet with same name but different IP at www.example.com/intra/ (not uncommon setup). If aomeone breaks in the external server they could exploit this bug to break inside the firewall, e.g., by faking intranet login page they could collect passwords.
Anybody able to hijack or spoof the external DNS servers could of course do similar attack also with differently named intranet hosts (intra.example.com could go to an external machine ran by blackhats).
In email setup with some addresses in @example.com being handled differently inside the firewall could lead to sensitive messages being delivered to wrong recipient. E.g., boss@example.com only existing internally and external mail server having wildcard *@example.com going to some spam filtering company.
That last one does not even require any actual malicious attack.
I don't think it's a catastrophic or even very bad bug, unlikely to be exploitable in what's probably typical IPFire usage scenario, but it is a security issue nonetheless.
Unfortunately due to travels I won't be able to do more about this for a couple of weeks, but I'll leave a box running the quick fix I put in bugzilla and absent any problems may submit it as a patch later, or perhaps write a cleaner fix.
Hello,
On Mon, 2018-05-28 at 09:17 +0300, Tapani Tarvainen wrote:
Hi all,
I'm only starting to use IPFire, so apologies if I'm missing some conventions here. I'm fairly experienced with Linux and firewalls in general, however.
Welcome :)
I found some operations in IPFire unexpectedly slow, in particular Edit Hosts: every operation, whether changing a host name or just disabling one, took about five minutes to complete.
Yes, this is not implemented really well. I was a bit under a time constraint and so far nobody has complained that this was an issue.
The restart script is called and will import every single entry one at a time.
The machine in question is rather slow but not *that* slow (1GHz Via C3, 1GB RAM).
What's more, during that time DNS is broken: ipfire responds to queries but gives wrong answers, specifically NXDOMAIN for hosts that do exist. And that was causing problems with my internal mail server.
Exist in the local configuration or where?
So I looked at the source.
It turns out that every operation in Edit Hosts triggers unbound restart, and that's where the time goes:
# time /etc/init.d/unbound restart [...]
real 4m7.531s user 1m55.320s sys 0m7.760s
Looking at the script, this is where it spends most of the time:
update_hosts() { local enabled address hostname domainname
while IFS="," read -r enabled address hostname domainname; do [ "${enabled}" = "on" ] || continue
# Build FQDN local fqdn="${hostname}.${domainname}" unbound-control -q local_data "${fqdn} ${LOCAL_TTL} IN A${address}"
# Skip reverse resolution if the address equals the GREENaddress [ "${address}" = "${GREEN_ADDRESS}" ] && continue
# Add RDNS address=$(ip_address_revptr ${address}) unbound-control -q local_data "${address} ${LOCAL_TTL} IN PTR${fqdn}" done < /var/ipfire/main/hosts }
I have roughly 150 entries in hosts list, so that ends up calling unbound-control about 300 times. And there's a race condition between the time unbound is started and when those entries are added (at different times).
I made the following simple change:
update_hosts() { local enabled address hostname domainname
while IFS="," read -r enabled address hostname domainname; do [ "${enabled}" = "on" ] || continue
# Build FQDN local fqdn="${hostname}.${domainname}" echo "${fqdn} ${LOCAL_TTL} IN A ${address}" # Skip reverse resolution if the address equals the GREENaddress [ "${address}" = "${GREEN_ADDRESS}" ] && continue
# Add RDNS address=$(ip_address_revptr ${address}) echo "${address} ${LOCAL_TTL} IN PTR ${fqdn}"done < /var/ipfire/main/hosts | unbound-control -q local_datas }
It would have helped to send this as a patch, so the changes are easier to spot.
Result:
# time /etc/init.d/unbound restart [...]
real 0m15.568s user 0m4.827s sys 0m1.403s
So it saves in my case four minutes at every hosts change, every dhcp change, every boot. Still not blazingly fast but already tolerably so.
That is small and obvious enough change that perhaps it could be considered for next upgrade (121)?
Yes, please send this as a patch:
https://wiki.ipfire.org/devel/submit-patches
I can submit it as a patch if that helps.
It's still not a good fix though: it doesn't remove the race condition window, only shortens it. Rather than using unbound-control to add local entries it'd be better to put them to a file and include that in unbound.conf (put the file or two in /etc/unbound/local.d/) so they'd all take effect immediately when unbound is started. This would be a bit bigger change but probably not much: if people think it'd be useful I could give it a go.
If you would want to go ahead and implement that solution that wouldn't be a bad idea either.
Of course it's not actually necessary to restart unbound for every change in hosts or dhcp at all, they could be effected by making individual changes with unbound-control, but that would be a much bigger change (I haven't looked at that part of the code in detail enough to judge how big). But improving the startup time would be useful even if that's done at some point.
It would be best just to change the entry that was touched, but it would be quite complicated to figure out what has changed and since I thought that most people would only have a few entries here it didn't look like it was worth implementing this.
Best, -Michael
-
Michael Tremer -
Tapani Tarvainen