Hi all,
**** This is a repost of an email incorrectly submitted. ****
Please bear with me as I am new to IPFire and not at all used to this style of development, so any guidance would be appreciated.
There is a bug, https://bugzilla.ipfire.org/show_bug.cgi?id=13254, where unbound-dhcp-leases-bridge restarts unbound a lot when it is totally unnecessary. It appears to be because when a lease is renewed, the script gets triggered, created a new /etc/unbound/dhcp-leases.conf then restarts Unbound to read the file. Generally this seems to be unnecessary as, with a renewed lease, the old and new /etc/unbound/dhcp-leases.conf files are the same. With 5 leases in that file (one for a machine not active) I am getting 3-4 restarts an hour of Unbound when I have a min/max lease time of 60/120min.
Looking at the code, it is fairly easy to fix. The current code creates a temp file then copies it into place then restarts unbound. All I am doing is doing a file comparison before the copy and skipping the restart if the files are the same.
There were a couple of gotchas because setting the file attributes and copying the file were done inside the "with" block for generating the temporary file. This meant a file comparison always returned False as the temp file was still open and so never the same is the old file.
I moved those two statements outside the "with". This forced me to change the fchmod to chmod.
It could be argued that the file copy should not be done if the files are not different, but it would take an extra "if" and you'd have to remember to delete the temp file. If required, I can make that change.
Also, one small thing I noticed once is that the old and new dhcp-leases.conf files could occasionally contain the same leases but in a different order. I have been unable to reproduce, but to sidestep it, instead of stepping through the leases variable directly, I sort it and step through that. It should make the resulting file completely deterministic and make the file comparison more effective.
My patch is:
From 73873d4944944a2f02317a73074a6894726f36f7 Mon Sep 17 00:00:00 2001 From: Nick Howitt nick@howitts.co.uk Date: Sun, 24 Mar 2024 15:17:19 +0000 Subject: [PATCH] Stop unbound-dhcp-leases-bridge from restarting unbound every time it write dhcp-leases.conf as it is very often unchanged and does not require a restart.
--- config/unbound/unbound-dhcp-leases-bridge | 28 +++++++++++++++-------- 1 file changed, 19 insertions(+), 9 deletions(-)
diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge index e9f022aff..d22772066 100644 --- a/config/unbound/unbound-dhcp-leases-bridge +++ b/config/unbound/unbound-dhcp-leases-bridge @@ -22,6 +22,7 @@ import argparse import datetime import daemon +import filecmp import functools import ipaddress import logging @@ -516,26 +517,35 @@ class UnboundConfigWriter(object):
def update_dhcp_leases(self, leases): # Write out all leases - self.write_dhcp_leases(leases) + if self.write_dhcp_leases(leases):
- log.debug("Reloading Unbound...") + log.debug("Reloading Unbound...")
- # Reload the configuration without dropping the cache - self._control("reload_keep_cache") + # Reload the configuration without dropping the cache + self._control("reload_keep_cache") + + else: + + log.debug("Not reloading Unbound. Leases not changed.")
def write_dhcp_leases(self, leases): log.debug("Writing DHCP leases...")
with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: - for l in leases: + for l in sorted(leases): for rr in l.rrset: f.write("local-data: "%s"\n" % " ".join(rr))
- # Make file readable for everyone - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) + filecmp.clear_cache() + RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) + + # Make file readable for everyone + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) + + # Move the file to its destination + os.rename(f.name, self.path)
- # Move the file to its destination - os.rename(f.name, self.path) + return RequireUnboundReload
def _control(self, *args): command = ["unbound-control"]
If there is a risk that the /etc/unbound/dhcp-leases.conf does not exist, for example on a clean install (I don't know), the filecmp.cmp line needs to be wrapped in a try...except to stop it blowing up:
try: RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) except: RequireUnboundReload = True
This will force the file to be to be generated, even if it is only copying in a blank temp file, and unbound will restart. It will only ever happen once.
On 24/03/2024 15:49, Nick Howitt wrote:
Hi all,
**** This is a repost of an email incorrectly submitted. ****
Please bear with me as I am new to IPFire and not at all used to this style of development, so any guidance would be appreciated.
There is a bug, https://bugzilla.ipfire.org/show_bug.cgi?id=13254, where unbound-dhcp-leases-bridge restarts unbound a lot when it is totally unnecessary. It appears to be because when a lease is renewed, the script gets triggered, created a new /etc/unbound/dhcp-leases.conf then restarts Unbound to read the file. Generally this seems to be unnecessary as, with a renewed lease, the old and new /etc/unbound/dhcp-leases.conf files are the same. With 5 leases in that file (one for a machine not active) I am getting 3-4 restarts an hour of Unbound when I have a min/max lease time of 60/120min.
Looking at the code, it is fairly easy to fix. The current code creates a temp file then copies it into place then restarts unbound. All I am doing is doing a file comparison before the copy and skipping the restart if the files are the same.
There were a couple of gotchas because setting the file attributes and copying the file were done inside the "with" block for generating the temporary file. This meant a file comparison always returned False as the temp file was still open and so never the same is the old file.
I moved those two statements outside the "with". This forced me to change the fchmod to chmod.
It could be argued that the file copy should not be done if the files are not different, but it would take an extra "if" and you'd have to remember to delete the temp file. If required, I can make that change.
Also, one small thing I noticed once is that the old and new dhcp-leases.conf files could occasionally contain the same leases but in a different order. I have been unable to reproduce, but to sidestep it, instead of stepping through the leases variable directly, I sort it and step through that. It should make the resulting file completely deterministic and make the file comparison more effective.
My patch is:
From 73873d4944944a2f02317a73074a6894726f36f7 Mon Sep 17 00:00:00 2001 From: Nick Howitt nick@howitts.co.uk Date: Sun, 24 Mar 2024 15:17:19 +0000 Subject: [PATCH] Stop unbound-dhcp-leases-bridge from restarting unbound every time it write dhcp-leases.conf as it is very often unchanged and does not require a restart.
config/unbound/unbound-dhcp-leases-bridge | 28 +++++++++++++++-------- 1 file changed, 19 insertions(+), 9 deletions(-)
diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge index e9f022aff..d22772066 100644 --- a/config/unbound/unbound-dhcp-leases-bridge +++ b/config/unbound/unbound-dhcp-leases-bridge @@ -22,6 +22,7 @@ import argparse import datetime import daemon +import filecmp import functools import ipaddress import logging @@ -516,26 +517,35 @@ class UnboundConfigWriter(object):
def update_dhcp_leases(self, leases): # Write out all leases - self.write_dhcp_leases(leases) + if self.write_dhcp_leases(leases):
- log.debug("Reloading Unbound...") + log.debug("Reloading Unbound...")
- # Reload the configuration without dropping the cache - self._control("reload_keep_cache") + # Reload the configuration without dropping the cache + self._control("reload_keep_cache")
+ else:
+ log.debug("Not reloading Unbound. Leases not changed.")
def write_dhcp_leases(self, leases): log.debug("Writing DHCP leases...")
with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: - for l in leases: + for l in sorted(leases): for rr in l.rrset: f.write("local-data: "%s"\n" % " ".join(rr))
- # Make file readable for everyone - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) + filecmp.clear_cache() + RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False)
+ # Make file readable for everyone + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)
+ # Move the file to its destination + os.rename(f.name, self.path)
- # Move the file to its destination - os.rename(f.name, self.path) + return RequireUnboundReload
def _control(self, *args): command = ["unbound-control"]
Hello,
I don’t think that the risk is very high, but I would account for this problem.
With my proposed changes from my previous email this should not be a large problem at all. When opening /etc/unbound/dhcp-leases.conf with mode “r+” you can catch the Exception and skip the comparison and open the file again with “w”.
On 25 Mar 2024, at 10:10, Nick Howitt nick@howitts.co.uk wrote:
If there is a risk that the /etc/unbound/dhcp-leases.conf does not exist, for example on a clean install (I don't know), the filecmp.cmp line needs to be wrapped in a try...except to stop it blowing up:
try: RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) except: RequireUnboundReload = True
I generally don’t like blanco exception catches. You will never know about the daemons that you are hiding.
This will force the file to be to be generated, even if it is only copying in a blank temp file, and unbound will restart. It will only ever happen once.
On 24/03/2024 15:49, Nick Howitt wrote:
Hi all, **** This is a repost of an email incorrectly submitted. **** Please bear with me as I am new to IPFire and not at all used to this style of development, so any guidance would be appreciated. There is a bug, https://bugzilla.ipfire.org/show_bug.cgi?id=13254, where unbound-dhcp-leases-bridge restarts unbound a lot when it is totally unnecessary. It appears to be because when a lease is renewed, the script gets triggered, created a new /etc/unbound/dhcp-leases.conf then restarts Unbound to read the file. Generally this seems to be unnecessary as, with a renewed lease, the old and new /etc/unbound/dhcp-leases.conf files are the same. With 5 leases in that file (one for a machine not active) I am getting 3-4 restarts an hour of Unbound when I have a min/max lease time of 60/120min. Looking at the code, it is fairly easy to fix. The current code creates a temp file then copies it into place then restarts unbound. All I am doing is doing a file comparison before the copy and skipping the restart if the files are the same. There were a couple of gotchas because setting the file attributes and copying the file were done inside the "with" block for generating the temporary file. This meant a file comparison always returned False as the temp file was still open and so never the same is the old file. I moved those two statements outside the "with". This forced me to change the fchmod to chmod. It could be argued that the file copy should not be done if the files are not different, but it would take an extra "if" and you'd have to remember to delete the temp file. If required, I can make that change. Also, one small thing I noticed once is that the old and new dhcp-leases.conf files could occasionally contain the same leases but in a different order. I have been unable to reproduce, but to sidestep it, instead of stepping through the leases variable directly, I sort it and step through that. It should make the resulting file completely deterministic and make the file comparison more effective. My patch is: From 73873d4944944a2f02317a73074a6894726f36f7 Mon Sep 17 00:00:00 2001 From: Nick Howitt nick@howitts.co.uk Date: Sun, 24 Mar 2024 15:17:19 +0000 Subject: [PATCH] Stop unbound-dhcp-leases-bridge from restarting unbound every time it write dhcp-leases.conf as it is very often unchanged and does not require a restart.
config/unbound/unbound-dhcp-leases-bridge | 28 +++++++++++++++-------- 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge index e9f022aff..d22772066 100644 --- a/config/unbound/unbound-dhcp-leases-bridge +++ b/config/unbound/unbound-dhcp-leases-bridge @@ -22,6 +22,7 @@ import argparse import datetime import daemon +import filecmp import functools import ipaddress import logging @@ -516,26 +517,35 @@ class UnboundConfigWriter(object): def update_dhcp_leases(self, leases): # Write out all leases
self.write_dhcp_leases(leases)
if self.write_dhcp_leases(leases):
log.debug("Reloading Unbound...")
log.debug("Reloading Unbound...")
# Reload the configuration without dropping the cacheself._control("reload_keep_cache")
# Reload the configuration without dropping the cacheself._control("reload_keep_cache")else:log.debug("Not reloading Unbound. Leases not changed.")
for l in leases:
for l in sorted(leases): for rr in l.rrset: f.write("local-data: \"%s\"\n" % " ".join(rr))
# Make file readable for everyoneos.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)
filecmp.clear_cache()RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False)# Make file readable for everyoneos.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)# Move the file to its destinationos.rename(f.name, self.path)
# Move the file to its destinationos.rename(f.name, self.path)
return RequireUnboundReload
On 25/03/2024 11:31, Michael Tremer wrote:
Hello,
I don’t think that the risk is very high, but I would account for this problem.
With my proposed changes from my previous email this should not be a large problem at all. When opening /etc/unbound/dhcp-leases.conf with mode “r+” you can catch the Exception and skip the comparison and open the file again with “w”.
On 25 Mar 2024, at 10:10, Nick Howitt nick@howitts.co.uk wrote:
If there is a risk that the /etc/unbound/dhcp-leases.conf does not exist, for example on a clean install (I don't know), the filecmp.cmp line needs to be wrapped in a try...except to stop it blowing up:
try: RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) except: RequireUnboundReload = True
I generally don’t like blanco exception catches. You will never know about the daemons that you are hiding.
This will force the file to be to be generated, even if it is only copying in a blank temp file, and unbound will restart. It will only ever happen once.
On 24/03/2024 15:49, Nick Howitt wrote:
Hi all, **** This is a repost of an email incorrectly submitted. **** Please bear with me as I am new to IPFire and not at all used to this style of development, so any guidance would be appreciated. There is a bug, https://bugzilla.ipfire.org/show_bug.cgi?id=13254, where unbound-dhcp-leases-bridge restarts unbound a lot when it is totally unnecessary. It appears to be because when a lease is renewed, the script gets triggered, created a new /etc/unbound/dhcp-leases.conf then restarts Unbound to read the file. Generally this seems to be unnecessary as, with a renewed lease, the old and new /etc/unbound/dhcp-leases.conf files are the same. With 5 leases in that file (one for a machine not active) I am getting 3-4 restarts an hour of Unbound when I have a min/max lease time of 60/120min. Looking at the code, it is fairly easy to fix. The current code creates a temp file then copies it into place then restarts unbound. All I am doing is doing a file comparison before the copy and skipping the restart if the files are the same. There were a couple of gotchas because setting the file attributes and copying the file were done inside the "with" block for generating the temporary file. This meant a file comparison always returned False as the temp file was still open and so never the same is the old file. I moved those two statements outside the "with". This forced me to change the fchmod to chmod. It could be argued that the file copy should not be done if the files are not different, but it would take an extra "if" and you'd have to remember to delete the temp file. If required, I can make that change. Also, one small thing I noticed once is that the old and new dhcp-leases.conf files could occasionally contain the same leases but in a different order. I have been unable to reproduce, but to sidestep it, instead of stepping through the leases variable directly, I sort it and step through that. It should make the resulting file completely deterministic and make the file comparison more effective. My patch is: From 73873d4944944a2f02317a73074a6894726f36f7 Mon Sep 17 00:00:00 2001 From: Nick Howitt nick@howitts.co.uk Date: Sun, 24 Mar 2024 15:17:19 +0000 Subject: [PATCH] Stop unbound-dhcp-leases-bridge from restarting unbound every time it write dhcp-leases.conf as it is very often unchanged and does not require a restart.
config/unbound/unbound-dhcp-leases-bridge | 28 +++++++++++++++-------- 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge index e9f022aff..d22772066 100644 --- a/config/unbound/unbound-dhcp-leases-bridge +++ b/config/unbound/unbound-dhcp-leases-bridge @@ -22,6 +22,7 @@ import argparse import datetime import daemon +import filecmp import functools import ipaddress import logging @@ -516,26 +517,35 @@ class UnboundConfigWriter(object): def update_dhcp_leases(self, leases): # Write out all leases
self.write_dhcp_leases(leases)
if self.write_dhcp_leases(leases):
log.debug("Reloading Unbound...")
log.debug("Reloading Unbound...")
# Reload the configuration without dropping the cacheself._control("reload_keep_cache")
# Reload the configuration without dropping the cacheself._control("reload_keep_cache")else:log.debug("Not reloading Unbound. Leases not changed.") def write_dhcp_leases(self, leases): log.debug("Writing DHCP leases...") with tempfile.NamedTemporaryFile(mode="w", delete=False) as f:
for l in leases:
for l in sorted(leases): for rr in l.rrset: f.write("local-data: \"%s\"\n" % " ".join(rr))
# Make file readable for everyoneos.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)
filecmp.clear_cache()RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False)# Make file readable for everyoneos.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)# Move the file to its destinationos.rename(f.name, self.path)
# Move the file to its destinationos.rename(f.name, self.path)
return RequireUnboundReload def _control(self, *args): command = ["unbound-control"]
Resubmitting the patch.
Following my previous reply, I would like to stick with filecmp as I am using it with "shallow=False" to avoid doing the comparison just with stat() calls. Note that I also flush the filecmp cache before using the command so it is running with the latest file data. Talking about file descriptors, atomic file replacement etc., scares me.
I have modified the earlier try/except I proposed to catch only FileNotFoundError, and I have changed CamelCase to snake_case.
--- config/unbound/unbound-dhcp-leases-bridge | 31 ++++++++++++++++------- 1 file changed, 22 insertions(+), 9 deletions(-)
diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge index e9f022aff..a9ec8e1e2 100644 --- a/config/unbound/unbound-dhcp-leases-bridge +++ b/config/unbound/unbound-dhcp-leases-bridge @@ -22,6 +22,7 @@ import argparse import datetime import daemon +import filecmp import functools import ipaddress import logging @@ -516,26 +517,38 @@ class UnboundConfigWriter(object):
def update_dhcp_leases(self, leases): # Write out all leases - self.write_dhcp_leases(leases) + if self.write_dhcp_leases(leases):
- log.debug("Reloading Unbound...") + log.debug("Reloading Unbound...")
- # Reload the configuration without dropping the cache - self._control("reload_keep_cache") + # Reload the configuration without dropping the cache + self._control("reload_keep_cache") + + else: + + log.debug("Not reloading Unbound. Leases not changed.")
def write_dhcp_leases(self, leases): log.debug("Writing DHCP leases...")
with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: - for l in leases: + for l in sorted(leases): for rr in l.rrset: f.write("local-data: "%s"\n" % " ".join(rr))
- # Make file readable for everyone - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) + filecmp.clear_cache() + try: + require_unbound_reload = not filecmp.cmp(f.name, self.path, shallow=False) + except FileNotFoundError: + require_unbound_reload = True + + # Make file readable for everyone + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) + + # Move the file to its destination + os.rename(f.name, self.path)
- # Move the file to its destination - os.rename(f.name, self.path) + return require_unbound_reload
def _control(self, *args): command = ["unbound-control"]
On 25 Mar 2024, at 13:11, Nick Howitt nick@howitts.co.uk wrote:
On 25/03/2024 11:31, Michael Tremer wrote:
Hello, I don’t think that the risk is very high, but I would account for this problem. With my proposed changes from my previous email this should not be a large problem at all. When opening /etc/unbound/dhcp-leases.conf with mode “r+” you can catch the Exception and skip the comparison and open the file again with “w”.
On 25 Mar 2024, at 10:10, Nick Howitt nick@howitts.co.uk wrote:
If there is a risk that the /etc/unbound/dhcp-leases.conf does not exist, for example on a clean install (I don't know), the filecmp.cmp line needs to be wrapped in a try...except to stop it blowing up:
try: RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) except: RequireUnboundReload = True
I generally don’t like blanco exception catches. You will never know about the daemons that you are hiding.
This will force the file to be to be generated, even if it is only copying in a blank temp file, and unbound will restart. It will only ever happen once.
On 24/03/2024 15:49, Nick Howitt wrote:
Hi all, **** This is a repost of an email incorrectly submitted. **** Please bear with me as I am new to IPFire and not at all used to this style of development, so any guidance would be appreciated. There is a bug, https://bugzilla.ipfire.org/show_bug.cgi?id=13254, where unbound-dhcp-leases-bridge restarts unbound a lot when it is totally unnecessary. It appears to be because when a lease is renewed, the script gets triggered, created a new /etc/unbound/dhcp-leases.conf then restarts Unbound to read the file. Generally this seems to be unnecessary as, with a renewed lease, the old and new /etc/unbound/dhcp-leases.conf files are the same. With 5 leases in that file (one for a machine not active) I am getting 3-4 restarts an hour of Unbound when I have a min/max lease time of 60/120min. Looking at the code, it is fairly easy to fix. The current code creates a temp file then copies it into place then restarts unbound. All I am doing is doing a file comparison before the copy and skipping the restart if the files are the same. There were a couple of gotchas because setting the file attributes and copying the file were done inside the "with" block for generating the temporary file. This meant a file comparison always returned False as the temp file was still open and so never the same is the old file. I moved those two statements outside the "with". This forced me to change the fchmod to chmod. It could be argued that the file copy should not be done if the files are not different, but it would take an extra "if" and you'd have to remember to delete the temp file. If required, I can make that change. Also, one small thing I noticed once is that the old and new dhcp-leases.conf files could occasionally contain the same leases but in a different order. I have been unable to reproduce, but to sidestep it, instead of stepping through the leases variable directly, I sort it and step through that. It should make the resulting file completely deterministic and make the file comparison more effective. My patch is: From 73873d4944944a2f02317a73074a6894726f36f7 Mon Sep 17 00:00:00 2001 From: Nick Howitt nick@howitts.co.uk Date: Sun, 24 Mar 2024 15:17:19 +0000 Subject: [PATCH] Stop unbound-dhcp-leases-bridge from restarting unbound every time it write dhcp-leases.conf as it is very often unchanged and does not require a restart.
config/unbound/unbound-dhcp-leases-bridge | 28 +++++++++++++++-------- 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge index e9f022aff..d22772066 100644 --- a/config/unbound/unbound-dhcp-leases-bridge +++ b/config/unbound/unbound-dhcp-leases-bridge @@ -22,6 +22,7 @@ import argparse import datetime import daemon +import filecmp import functools import ipaddress import logging @@ -516,26 +517,35 @@ class UnboundConfigWriter(object): def update_dhcp_leases(self, leases): # Write out all leases
self.write_dhcp_leases(leases)
if self.write_dhcp_leases(leases):
log.debug("Reloading Unbound...")
log.debug("Reloading Unbound...")
# Reload the configuration without dropping the cacheself._control("reload_keep_cache")
# Reload the configuration without dropping the cacheself._control("reload_keep_cache")else:log.debug("Not reloading Unbound. Leases not changed.")
for l in leases:
for l in sorted(leases): for rr in l.rrset: f.write("local-data: \"%s\"\n" % " ".join(rr))
# Make file readable for everyoneos.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)
filecmp.clear_cache()RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False)# Make file readable for everyoneos.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)# Move the file to its destinationos.rename(f.name, self.path)
# Move the file to its destinationos.rename(f.name, self.path)
return RequireUnboundReloadResubmitting the patch.
Following my previous reply, I would like to stick with filecmp as I am using it with "shallow=False" to avoid doing the comparison just with stat() calls. Note that I also flush the filecmp cache before using the command so it is running with the latest file data. Talking about file descriptors, atomic file replacement etc., scares me.
File descriptors are great. They avoid race-conditions and all sorts of things.
I don’t think that module is the best choice really if you have to hack around and flush the cache. Why would it even cache things in the first place? I don’t see where the cache helps.
I have modified the earlier try/except I proposed to catch only FileNotFoundError, and I have changed CamelCase to snake_case.
config/unbound/unbound-dhcp-leases-bridge | 31 ++++++++++++++++------- 1 file changed, 22 insertions(+), 9 deletions(-)
diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge index e9f022aff..a9ec8e1e2 100644 --- a/config/unbound/unbound-dhcp-leases-bridge +++ b/config/unbound/unbound-dhcp-leases-bridge @@ -22,6 +22,7 @@ import argparse import datetime import daemon +import filecmp import functools import ipaddress import logging @@ -516,26 +517,38 @@ class UnboundConfigWriter(object):
def update_dhcp_leases(self, leases): # Write out all leases
- self.write_dhcp_leases(leases)
- if self.write_dhcp_leases(leases):
- log.debug("Reloading Unbound...")
- log.debug("Reloading Unbound...")
- # Reload the configuration without dropping the cache
- self._control("reload_keep_cache")
- # Reload the configuration without dropping the cache
- self._control("reload_keep_cache")
- else:
- log.debug("Not reloading Unbound. Leases not changed.")
def write_dhcp_leases(self, leases): log.debug("Writing DHCP leases...")
with tempfile.NamedTemporaryFile(mode="w", delete=False) as f:
- for l in leases:
- for l in sorted(leases):
for rr in l.rrset: f.write("local-data: "%s"\n" % " ".join(rr))
- # Make file readable for everyone
- os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)
- filecmp.clear_cache()
- try:
- require_unbound_reload = not filecmp.cmp(f.name, self.path, shallow=False)
- except FileNotFoundError:
- require_unbound_reload = True
If you want to go this route, you will have to flush the temporary file first. Otherwise you might have some data left in the buffer that has not been written (because the file is still open). You will then compare incomplete output. This would not become a problem if you used the file descriptors.
Catching FileNotFound would work for me.
- # Make file readable for everyone
- os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)
- # Move the file to its destination
- os.rename(f.name, self.path)
- # Move the file to its destination
- os.rename(f.name, self.path)
- return require_unbound_reload
def _control(self, *args): command = ["unbound-control"] -- 2.43.0
Nick
On 26/03/2024 11:10, Michael Tremer wrote:
On 25 Mar 2024, at 13:11, Nick Howitt nick@howitts.co.uk wrote:
On 25/03/2024 11:31, Michael Tremer wrote:
Hello, I don’t think that the risk is very high, but I would account for this problem. With my proposed changes from my previous email this should not be a large problem at all. When opening /etc/unbound/dhcp-leases.conf with mode “r+” you can catch the Exception and skip the comparison and open the file again with “w”.
On 25 Mar 2024, at 10:10, Nick Howitt nick@howitts.co.uk wrote:
If there is a risk that the /etc/unbound/dhcp-leases.conf does not exist, for example on a clean install (I don't know), the filecmp.cmp line needs to be wrapped in a try...except to stop it blowing up:
try: RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) except: RequireUnboundReload = True
I generally don’t like blanco exception catches. You will never know about the daemons that you are hiding.
This will force the file to be to be generated, even if it is only copying in a blank temp file, and unbound will restart. It will only ever happen once.
On 24/03/2024 15:49, Nick Howitt wrote:
Hi all, **** This is a repost of an email incorrectly submitted. **** Please bear with me as I am new to IPFire and not at all used to this style of development, so any guidance would be appreciated. There is a bug, https://bugzilla.ipfire.org/show_bug.cgi?id=13254, where unbound-dhcp-leases-bridge restarts unbound a lot when it is totally unnecessary. It appears to be because when a lease is renewed, the script gets triggered, created a new /etc/unbound/dhcp-leases.conf then restarts Unbound to read the file. Generally this seems to be unnecessary as, with a renewed lease, the old and new /etc/unbound/dhcp-leases.conf files are the same. With 5 leases in that file (one for a machine not active) I am getting 3-4 restarts an hour of Unbound when I have a min/max lease time of 60/120min. Looking at the code, it is fairly easy to fix. The current code creates a temp file then copies it into place then restarts unbound. All I am doing is doing a file comparison before the copy and skipping the restart if the files are the same. There were a couple of gotchas because setting the file attributes and copying the file were done inside the "with" block for generating the temporary file. This meant a file comparison always returned False as the temp file was still open and so never the same is the old file. I moved those two statements outside the "with". This forced me to change the fchmod to chmod. It could be argued that the file copy should not be done if the files are not different, but it would take an extra "if" and you'd have to remember to delete the temp file. If required, I can make that change. Also, one small thing I noticed once is that the old and new dhcp-leases.conf files could occasionally contain the same leases but in a different order. I have been unable to reproduce, but to sidestep it, instead of stepping through the leases variable directly, I sort it and step through that. It should make the resulting file completely deterministic and make the file comparison more effective. My patch is: From 73873d4944944a2f02317a73074a6894726f36f7 Mon Sep 17 00:00:00 2001 From: Nick Howitt nick@howitts.co.uk Date: Sun, 24 Mar 2024 15:17:19 +0000 Subject: [PATCH] Stop unbound-dhcp-leases-bridge from restarting unbound every time it write dhcp-leases.conf as it is very often unchanged and does not require a restart.
config/unbound/unbound-dhcp-leases-bridge | 28 +++++++++++++++-------- 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge index e9f022aff..d22772066 100644 --- a/config/unbound/unbound-dhcp-leases-bridge +++ b/config/unbound/unbound-dhcp-leases-bridge @@ -22,6 +22,7 @@ import argparse import datetime import daemon +import filecmp import functools import ipaddress import logging @@ -516,26 +517,35 @@ class UnboundConfigWriter(object): def update_dhcp_leases(self, leases): # Write out all leases
self.write_dhcp_leases(leases)
if self.write_dhcp_leases(leases):
log.debug("Reloading Unbound...")
log.debug("Reloading Unbound...")
# Reload the configuration without dropping the cacheself._control("reload_keep_cache")
# Reload the configuration without dropping the cacheself._control("reload_keep_cache")else:log.debug("Not reloading Unbound. Leases not changed.") def write_dhcp_leases(self, leases): log.debug("Writing DHCP leases...") with tempfile.NamedTemporaryFile(mode="w", delete=False) as f:
for l in leases:
for l in sorted(leases): for rr in l.rrset: f.write("local-data: \"%s\"\n" % " ".join(rr))
# Make file readable for everyoneos.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)
filecmp.clear_cache()RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False)# Make file readable for everyoneos.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)# Move the file to its destinationos.rename(f.name, self.path)
# Move the file to its destinationos.rename(f.name, self.path)
return RequireUnboundReload def _control(self, *args): command = ["unbound-control"]Resubmitting the patch.
Following my previous reply, I would like to stick with filecmp as I am using it with "shallow=False" to avoid doing the comparison just with stat() calls. Note that I also flush the filecmp cache before using the command so it is running with the latest file data. Talking about file descriptors, atomic file replacement etc., scares me.
File descriptors are great. They avoid race-conditions and all sorts of things.
But I have never used them so I don't intend to go that way. Too much for me to get my brain round and a limited ability to test.
I don’t think that module is the best choice really if you have to hack around and flush the cache. Why would it even cache things in the first place? I don’t see where the cache helps.
From the filecmp.clear_cache documentation:
Clear the filecmp cache. This may be useful if a file is compared so quickly after it is modified that it is within the mtime resolution of the underlying filesystem.
Now this makes me believe that the clear_cache is only relevant when using shallow=True, which I'm not and I don't know how to check the underlying code. It is used to make sure the stat() info is updated, but we are not using the stat() info. I guess we could as easily add a one second sleep instead as that is way longer than the mtime resolution.
The module is chosen because all the references I read about comparing files preferred this one-liner for comparing files, although other methods were given and there were lots of discussions about large files (GB+) because if you chomp them it may be much quicker to say they are not the same when you find the first block that does not match. Having said this, I don't know how filecmp works, but we only have a small file. At the same time, I am not a regular programmer so simple one-liners are good for me.
I have modified the earlier try/except I proposed to catch only FileNotFoundError, and I have changed CamelCase to snake_case.
config/unbound/unbound-dhcp-leases-bridge | 31 ++++++++++++++++------- 1 file changed, 22 insertions(+), 9 deletions(-)
diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge index e9f022aff..a9ec8e1e2 100644 --- a/config/unbound/unbound-dhcp-leases-bridge +++ b/config/unbound/unbound-dhcp-leases-bridge @@ -22,6 +22,7 @@ import argparse import datetime import daemon +import filecmp import functools import ipaddress import logging @@ -516,26 +517,38 @@ class UnboundConfigWriter(object):
def update_dhcp_leases(self, leases): # Write out all leases
- self.write_dhcp_leases(leases)
- if self.write_dhcp_leases(leases):
- log.debug("Reloading Unbound...")
- log.debug("Reloading Unbound...")
- # Reload the configuration without dropping the cache
- self._control("reload_keep_cache")
- # Reload the configuration without dropping the cache
- self._control("reload_keep_cache")
- else:
- log.debug("Not reloading Unbound. Leases not changed.")
def write_dhcp_leases(self, leases): log.debug("Writing DHCP leases...")
with tempfile.NamedTemporaryFile(mode="w", delete=False) as f:
- for l in leases:
- for l in sorted(leases):
for rr in l.rrset: f.write("local-data: "%s"\n" % " ".join(rr))
- # Make file readable for everyone
- os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)
- filecmp.clear_cache()
- try:
- require_unbound_reload = not filecmp.cmp(f.name, self.path, shallow=False)
- except FileNotFoundError:
- require_unbound_reload = True
If you want to go this route, you will have to flush the temporary file first. Otherwise you might have some data left in the buffer that has not been written (because the file is still open). You will then compare incomplete output. This would not become a problem if you used the file descriptors.
The temp file is closed at the point of doing the filecmp, and the filecmp cache has been flushed. How do I flush the temporary file further. Reading up, is it just having the last statement in the with block as f.flush() with the same indent as "for l in sorted(leases):"
Catching FileNotFound would work for me.
Good
- # Make file readable for everyone
- os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)
- # Move the file to its destination
- os.rename(f.name, self.path)
- # Move the file to its destination
- os.rename(f.name, self.path)
- return require_unbound_reload
def _control(self, *args): command = ["unbound-control"] -- 2.43.0
Nick
Hello,
On 27 Mar 2024, at 15:28, Nick Howitt nick@howitts.co.uk wrote:
On 26/03/2024 11:10, Michael Tremer wrote:
On 25 Mar 2024, at 13:11, Nick Howitt nick@howitts.co.uk wrote:
On 25/03/2024 11:31, Michael Tremer wrote:
Hello, I don’t think that the risk is very high, but I would account for this problem. With my proposed changes from my previous email this should not be a large problem at all. When opening /etc/unbound/dhcp-leases.conf with mode “r+” you can catch the Exception and skip the comparison and open the file again with “w”.
On 25 Mar 2024, at 10:10, Nick Howitt nick@howitts.co.uk wrote:
If there is a risk that the /etc/unbound/dhcp-leases.conf does not exist, for example on a clean install (I don't know), the filecmp.cmp line needs to be wrapped in a try...except to stop it blowing up:
try: RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) except: RequireUnboundReload = True
I generally don’t like blanco exception catches. You will never know about the daemons that you are hiding.
This will force the file to be to be generated, even if it is only copying in a blank temp file, and unbound will restart. It will only ever happen once.
On 24/03/2024 15:49, Nick Howitt wrote:
Hi all, **** This is a repost of an email incorrectly submitted. **** Please bear with me as I am new to IPFire and not at all used to this style of development, so any guidance would be appreciated. There is a bug, https://bugzilla.ipfire.org/show_bug.cgi?id=13254, where unbound-dhcp-leases-bridge restarts unbound a lot when it is totally unnecessary. It appears to be because when a lease is renewed, the script gets triggered, created a new /etc/unbound/dhcp-leases.conf then restarts Unbound to read the file. Generally this seems to be unnecessary as, with a renewed lease, the old and new /etc/unbound/dhcp-leases.conf files are the same. With 5 leases in that file (one for a machine not active) I am getting 3-4 restarts an hour of Unbound when I have a min/max lease time of 60/120min. Looking at the code, it is fairly easy to fix. The current code creates a temp file then copies it into place then restarts unbound. All I am doing is doing a file comparison before the copy and skipping the restart if the files are the same. There were a couple of gotchas because setting the file attributes and copying the file were done inside the "with" block for generating the temporary file. This meant a file comparison always returned False as the temp file was still open and so never the same is the old file. I moved those two statements outside the "with". This forced me to change the fchmod to chmod. It could be argued that the file copy should not be done if the files are not different, but it would take an extra "if" and you'd have to remember to delete the temp file. If required, I can make that change. Also, one small thing I noticed once is that the old and new dhcp-leases.conf files could occasionally contain the same leases but in a different order. I have been unable to reproduce, but to sidestep it, instead of stepping through the leases variable directly, I sort it and step through that. It should make the resulting file completely deterministic and make the file comparison more effective. My patch is: From 73873d4944944a2f02317a73074a6894726f36f7 Mon Sep 17 00:00:00 2001 From: Nick Howitt nick@howitts.co.uk Date: Sun, 24 Mar 2024 15:17:19 +0000 Subject: [PATCH] Stop unbound-dhcp-leases-bridge from restarting unbound every time it write dhcp-leases.conf as it is very often unchanged and does not require a restart.
config/unbound/unbound-dhcp-leases-bridge | 28 +++++++++++++++-------- 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge index e9f022aff..d22772066 100644 --- a/config/unbound/unbound-dhcp-leases-bridge +++ b/config/unbound/unbound-dhcp-leases-bridge @@ -22,6 +22,7 @@ import argparse import datetime import daemon +import filecmp import functools import ipaddress import logging @@ -516,26 +517,35 @@ class UnboundConfigWriter(object): def update_dhcp_leases(self, leases): # Write out all leases
self.write_dhcp_leases(leases)
if self.write_dhcp_leases(leases):
log.debug("Reloading Unbound...")
log.debug("Reloading Unbound...")
# Reload the configuration without dropping the cacheself._control("reload_keep_cache")
# Reload the configuration without dropping the cacheself._control("reload_keep_cache")else:log.debug("Not reloading Unbound. Leases not changed.")
for l in leases:
for l in sorted(leases): for rr in l.rrset: f.write("local-data: \"%s\"\n" % " ".join(rr))
# Make file readable for everyoneos.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)
filecmp.clear_cache()RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False)# Make file readable for everyoneos.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)# Move the file to its destinationos.rename(f.name, self.path)
# Move the file to its destinationos.rename(f.name, self.path)
return RequireUnboundReloadResubmitting the patch.
Following my previous reply, I would like to stick with filecmp as I am using it with "shallow=False" to avoid doing the comparison just with stat() calls. Note that I also flush the filecmp cache before using the command so it is running with the latest file data. Talking about file descriptors, atomic file replacement etc., scares me.
File descriptors are great. They avoid race-conditions and all sorts of things.
But I have never used them so I don't intend to go that way. Too much for me to get my brain round and a limited ability to test.
Well, I pretty much gave you the entire solution… There was nothing left to do apart from copying and pasting the snippets.
I don’t think that module is the best choice really if you have to hack around and flush the cache. Why would it even cache things in the first place? I don’t see where the cache helps.
From the filecmp.clear_cache documentation:
Clear the filecmp cache. This may be useful if a file is compared so quickly after it is modified that it is within the mtime resolution of the underlying filesystem.
Now this makes me believe that the clear_cache is only relevant when using shallow=True, which I'm not and I don't know how to check the underlying code. It is used to make sure the stat() info is updated, but we are not using the stat() info. I guess we could as easily add a one second sleep instead as that is way longer than the mtime resolution.
The function is here and it is rather simple: https://github.com/python/cpython/blob/3.12/Lib/filecmp.py#L30
It will always stat() both files and compares three items in case shallow is False: The type of the file, the size and mtime. If the file is not a regular file, the function will exist rather early. So in our case, that is all work the function is doing for thing. It boils down to something very similar to what I suggested to use to compare the files.
On top of that, there is a cache which will always be used but will never return a match for us because the temporary file at least will always have another mtime.
You could clear the cache after every call, but I would rather not have our code messy and care about implementation details of some Python module.
The module is chosen because all the references I read about comparing files preferred this one-liner for comparing files, although other methods were given and there were lots of discussions about large files (GB+) because if you chomp them it may be much quicker to say they are not the same when you find the first block that does not match. Having said this, I don't know how filecmp works, but we only have a small file. At the same time, I am not a regular programmer so simple one-liners are good for me.
You would always want to read both files block for block and compare the blocks. There is no way to implement this significantly faster.
I have modified the earlier try/except I proposed to catch only FileNotFoundError, and I have changed CamelCase to snake_case.
config/unbound/unbound-dhcp-leases-bridge | 31 ++++++++++++++++------- 1 file changed, 22 insertions(+), 9 deletions(-)
diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge index e9f022aff..a9ec8e1e2 100644 --- a/config/unbound/unbound-dhcp-leases-bridge +++ b/config/unbound/unbound-dhcp-leases-bridge @@ -22,6 +22,7 @@ import argparse import datetime import daemon +import filecmp import functools import ipaddress import logging @@ -516,26 +517,38 @@ class UnboundConfigWriter(object):
def update_dhcp_leases(self, leases): # Write out all leases
- self.write_dhcp_leases(leases)
- if self.write_dhcp_leases(leases):
- log.debug("Reloading Unbound...")
- log.debug("Reloading Unbound...")
- # Reload the configuration without dropping the cache
- self._control("reload_keep_cache")
- # Reload the configuration without dropping the cache
- self._control("reload_keep_cache")
- else:
- log.debug("Not reloading Unbound. Leases not changed.")
def write_dhcp_leases(self, leases): log.debug("Writing DHCP leases...")
with tempfile.NamedTemporaryFile(mode="w", delete=False) as f:
- for l in leases:
- for l in sorted(leases):
for rr in l.rrset: f.write("local-data: "%s"\n" % " ".join(rr))
- # Make file readable for everyone
- os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)
- filecmp.clear_cache()
- try:
- require_unbound_reload = not filecmp.cmp(f.name, self.path, shallow=False)
- except FileNotFoundError:
- require_unbound_reload = True
If you want to go this route, you will have to flush the temporary file first. Otherwise you might have some data left in the buffer that has not been written (because the file is still open). You will then compare incomplete output. This would not become a problem if you used the file descriptors.
The temp file is closed at the point of doing the filecmp, and the filecmp cache has been flushed. How do I flush the temporary file further. Reading up, is it just having the last statement in the with block as f.flush() with the same indent as "for l in sorted(leases):"
If you have left the with block you will already have flushed the buffers and closed the file. It looks like I misread the indentation in the patch then.
Catching FileNotFound would work for me.
Good
- # Make file readable for everyone
- os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)
- # Move the file to its destination
- os.rename(f.name, self.path)
- # Move the file to its destination
- os.rename(f.name, self.path)
- return require_unbound_reload
def _control(self, *args): command = ["unbound-control"] -- 2.43.0
Nick
Next version.
On 28/03/2024 10:41, Michael Tremer wrote:
Hello,
On 27 Mar 2024, at 15:28, Nick Howitt nick@howitts.co.uk wrote:
On 26/03/2024 11:10, Michael Tremer wrote:
On 25 Mar 2024, at 13:11, Nick Howitt nick@howitts.co.uk wrote:
On 25/03/2024 11:31, Michael Tremer wrote:
Hello, I don’t think that the risk is very high, but I would account for this problem. With my proposed changes from my previous email this should not be a large problem at all. When opening /etc/unbound/dhcp-leases.conf with mode “r+” you can catch the Exception and skip the comparison and open the file again with “w”.
On 25 Mar 2024, at 10:10, Nick Howitt nick@howitts.co.uk wrote:
If there is a risk that the /etc/unbound/dhcp-leases.conf does not exist, for example on a clean install (I don't know), the filecmp.cmp line needs to be wrapped in a try...except to stop it blowing up:
try: RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) except: RequireUnboundReload = True
I generally don’t like blanco exception catches. You will never know about the daemons that you are hiding.
This will force the file to be to be generated, even if it is only copying in a blank temp file, and unbound will restart. It will only ever happen once.
On 24/03/2024 15:49, Nick Howitt wrote: > Hi all, > **** This is a repost of an email incorrectly submitted. **** > Please bear with me as I am new to IPFire and not at all used to this style of development, so any guidance would be appreciated. > There is a bug, https://bugzilla.ipfire.org/show_bug.cgi?id=13254, where unbound-dhcp-leases-bridge restarts unbound a lot when it is totally unnecessary. It appears to be because when a lease is renewed, the script gets triggered, created a new /etc/unbound/dhcp-leases.conf then restarts Unbound to read the file. Generally this seems to be unnecessary as, with a renewed lease, the old and new /etc/unbound/dhcp-leases.conf files are the same. With 5 leases in that file (one for a machine not active) I am getting 3-4 restarts an hour of Unbound when I have a min/max lease time of 60/120min. > Looking at the code, it is fairly easy to fix. The current code creates a temp file then copies it into place then restarts unbound. All I am doing is doing a file comparison before the copy and skipping the restart if the files are the same. > There were a couple of gotchas because setting the file attributes and copying the file were done inside the "with" block for generating the temporary file. This meant a file comparison always returned False as the temp file was still open and so never the same is the old file. > I moved those two statements outside the "with". This forced me to change the fchmod to chmod. > It could be argued that the file copy should not be done if the files are not different, but it would take an extra "if" and you'd have to remember to delete the temp file. If required, I can make that change. > Also, one small thing I noticed once is that the old and new dhcp-leases.conf files could occasionally contain the same leases but in a different order. I have been unable to reproduce, but to sidestep it, instead of stepping through the leases variable directly, I sort it and step through that. It should make the resulting file completely deterministic and make the file comparison more effective. > My patch is: > From 73873d4944944a2f02317a73074a6894726f36f7 Mon Sep 17 00:00:00 2001 > From: Nick Howitt nick@howitts.co.uk > Date: Sun, 24 Mar 2024 15:17:19 +0000 > Subject: [PATCH] Stop unbound-dhcp-leases-bridge from restarting unbound every > time it write dhcp-leases.conf as it is very often unchanged and does not > require a restart. > --- > config/unbound/unbound-dhcp-leases-bridge | 28 +++++++++++++++-------- > 1 file changed, 19 insertions(+), 9 deletions(-) > diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge > index e9f022aff..d22772066 100644 > --- a/config/unbound/unbound-dhcp-leases-bridge > +++ b/config/unbound/unbound-dhcp-leases-bridge > @@ -22,6 +22,7 @@ > import argparse > import datetime > import daemon > +import filecmp > import functools > import ipaddress > import logging > @@ -516,26 +517,35 @@ class UnboundConfigWriter(object): > def update_dhcp_leases(self, leases): > # Write out all leases > - self.write_dhcp_leases(leases) > + if self.write_dhcp_leases(leases): > - log.debug("Reloading Unbound...") > + log.debug("Reloading Unbound...") > - # Reload the configuration without dropping the cache > - self._control("reload_keep_cache") > + # Reload the configuration without dropping the cache > + self._control("reload_keep_cache") > + > + else: > + > + log.debug("Not reloading Unbound. Leases not changed.") > def write_dhcp_leases(self, leases): > log.debug("Writing DHCP leases...") > with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: > - for l in leases: > + for l in sorted(leases): > for rr in l.rrset: > f.write("local-data: "%s"\n" % " ".join(rr)) > - # Make file readable for everyone > - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) > + filecmp.clear_cache() > + RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) > + > + # Make file readable for everyone > + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) > + > + # Move the file to its destination > + os.rename(f.name, self.path) > - # Move the file to its destination > - os.rename(f.name, self.path) > + return RequireUnboundReload > def _control(self, *args): > command = ["unbound-control"]
Resubmitting the patch.
Following my previous reply, I would like to stick with filecmp as I am using it with "shallow=False" to avoid doing the comparison just with stat() calls. Note that I also flush the filecmp cache before using the command so it is running with the latest file data. Talking about file descriptors, atomic file replacement etc., scares me.
File descriptors are great. They avoid race-conditions and all sorts of things.
But I have never used them so I don't intend to go that way. Too much for me to get my brain round and a limited ability to test.
Well, I pretty much gave you the entire solution… There was nothing left to do apart from copying and pasting the snippets.
Ok I've tried to use your version but struggled to get it going. If I used it as was and called compare(file1_name, file2_name) it failed as "compare" was not defined. I could call it using self.compare but then it failed with "TypeError: UnboundConfigWriter.compare() takes 2 positional arguments but 3 were given". I changed the function to compare(self, f1, f2). Then it failed with "AttributeError: 'str' object has no attribute 'seek'", so I changed it again to accept file names then open the files before doing the seeks. I really hope it is OK as I am out of my depth when defining functions with "self" as an argument. Please can you check it? If it is not OK, can I revert to filecmp?
I have made one futher change, the sorted(leases) line was not working and not sorting the object ever, and I saw lines changing places in the resulting leases file. I have changed it to "sorted(leases, key=lambda x: x.ipaddr)". It definitely sorts this time.
I don’t think that module is the best choice really if you have to hack around and flush the cache. Why would it even cache things in the first place? I don’t see where the cache helps.
From the filecmp.clear_cache documentation:
Clear the filecmp cache. This may be useful if a file is compared so quickly after it is modified that it is within the mtime resolution of the underlying filesystem.
Now this makes me believe that the clear_cache is only relevant when using shallow=True, which I'm not and I don't know how to check the underlying code. It is used to make sure the stat() info is updated, but we are not using the stat() info. I guess we could as easily add a one second sleep instead as that is way longer than the mtime resolution.
The function is here and it is rather simple: https://github.com/python/cpython/blob/3.12/Lib/filecmp.py#L30
It will always stat() both files and compares three items in case shallow is False: The type of the file, the size and mtime. If the file is not a regular file, the function will exist rather early. So in our case, that is all work the function is doing for thing. It boils down to something very similar to what I suggested to use to compare the files.
On top of that, there is a cache which will always be used but will never return a match for us because the temporary file at least will always have another mtime.
You could clear the cache after every call, but I would rather not have our code messy and care about implementation details of some Python module.
The module is chosen because all the references I read about comparing files preferred this one-liner for comparing files, although other methods were given and there were lots of discussions about large files (GB+) because if you chomp them it may be much quicker to say they are not the same when you find the first block that does not match. Having said this, I don't know how filecmp works, but we only have a small file. At the same time, I am not a regular programmer so simple one-liners are good for me.
You would always want to read both files block for block and compare the blocks. There is no way to implement this significantly faster.
I have modified the earlier try/except I proposed to catch only FileNotFoundError, and I have changed CamelCase to snake_case.
config/unbound/unbound-dhcp-leases-bridge | 31 ++++++++++++++++------- 1 file changed, 22 insertions(+), 9 deletions(-)
diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge index e9f022aff..a9ec8e1e2 100644 --- a/config/unbound/unbound-dhcp-leases-bridge +++ b/config/unbound/unbound-dhcp-leases-bridge @@ -22,6 +22,7 @@ import argparse import datetime import daemon +import filecmp import functools import ipaddress import logging @@ -516,26 +517,38 @@ class UnboundConfigWriter(object):
def update_dhcp_leases(self, leases): # Write out all leases
- self.write_dhcp_leases(leases)
- if self.write_dhcp_leases(leases):
- log.debug("Reloading Unbound...")
- log.debug("Reloading Unbound...")
- # Reload the configuration without dropping the cache
- self._control("reload_keep_cache")
- # Reload the configuration without dropping the cache
- self._control("reload_keep_cache")
- else:
- log.debug("Not reloading Unbound. Leases not changed.")
def write_dhcp_leases(self, leases): log.debug("Writing DHCP leases...")
with tempfile.NamedTemporaryFile(mode="w", delete=False) as f:
- for l in leases:
- for l in sorted(leases):
for rr in l.rrset: f.write("local-data: "%s"\n" % " ".join(rr))
- # Make file readable for everyone
- os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)
- filecmp.clear_cache()
- try:
- require_unbound_reload = not filecmp.cmp(f.name, self.path, shallow=False)
- except FileNotFoundError:
- require_unbound_reload = True
If you want to go this route, you will have to flush the temporary file first. Otherwise you might have some data left in the buffer that has not been written (because the file is still open). You will then compare incomplete output. This would not become a problem if you used the file descriptors.
The temp file is closed at the point of doing the filecmp, and the filecmp cache has been flushed. How do I flush the temporary file further. Reading up, is it just having the last statement in the with block as f.flush() with the same indent as "for l in sorted(leases):"
If you have left the with block you will already have flushed the buffers and closed the file. It looks like I misread the indentation in the patch then.
Catching FileNotFound would work for me.
Good
- # Make file readable for everyone
- os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)
- # Move the file to its destination
- os.rename(f.name, self.path)
- # Move the file to its destination
- os.rename(f.name, self.path)
- return require_unbound_reload
def _control(self, *args): command = ["unbound-control"] -- 2.43.0
Nick
--- config/unbound/unbound-dhcp-leases-bridge | 50 +++++++++++++++++++---- 1 file changed, 41 insertions(+), 9 deletions(-)
diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge index e9f022aff..80d9523ea 100644 --- a/config/unbound/unbound-dhcp-leases-bridge +++ b/config/unbound/unbound-dhcp-leases-bridge @@ -516,26 +516,37 @@ class UnboundConfigWriter(object):
def update_dhcp_leases(self, leases): # Write out all leases - self.write_dhcp_leases(leases) + if self.write_dhcp_leases(leases):
- log.debug("Reloading Unbound...") + log.debug("Reloading Unbound...")
- # Reload the configuration without dropping the cache - self._control("reload_keep_cache") + # Reload the configuration without dropping the cache + self._control("reload_keep_cache") + + else: + + log.debug("Not reloading Unbound. Leases not changed.")
def write_dhcp_leases(self, leases): log.debug("Writing DHCP leases...")
with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: - for l in leases: + for l in sorted(leases, key=lambda x: x.ipaddr): for rr in l.rrset: f.write("local-data: "%s"\n" % " ".join(rr))
- # Make file readable for everyone - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) + try: + require_unbound_reload = not self.compare(f.name, self.path) + except FileNotFoundError: + require_unbound_reload = True + + # Make file readable for everyone + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) + + # Move the file to its destination + os.rename(f.name, self.path)
- # Move the file to its destination - os.rename(f.name, self.path) + return require_unbound_reload
def _control(self, *args): command = ["unbound-control"] @@ -551,6 +562,27 @@ class UnboundConfigWriter(object):
raise e
+ def compare(self, if1, if2): + buffersize = 16 * 1024 + + f1 = open(if1, "r") + f2 = open(if2, "r") + + # Reset file handles + f1.seek(0) + f2.seek(0) + + while True: + b1 = f1.read(buffersize) + b2 = f2.read(buffersize) + + if not b1 == b2: + return False + + elif not b1 or not b2: + break + + return True
if __name__ == "__main__": parser = argparse.ArgumentParser(description="Bridge for DHCP Leases and Unbound DNS")
Bump
On 28/03/2024 17:25, Nick Howitt wrote:
Next version.
On 28/03/2024 10:41, Michael Tremer wrote:
Hello,
On 27 Mar 2024, at 15:28, Nick Howitt nick@howitts.co.uk wrote:
On 26/03/2024 11:10, Michael Tremer wrote:
On 25 Mar 2024, at 13:11, Nick Howitt nick@howitts.co.uk wrote:
On 25/03/2024 11:31, Michael Tremer wrote:
Hello, I don’t think that the risk is very high, but I would account for this problem. With my proposed changes from my previous email this should not be a large problem at all. When opening /etc/unbound/dhcp-leases.conf with mode “r+” you can catch the Exception and skip the comparison and open the file again with “w”. > On 25 Mar 2024, at 10:10, Nick Howitt nick@howitts.co.uk wrote: > > If there is a risk that the /etc/unbound/dhcp-leases.conf does > not exist, for example on a clean install (I don't know), the > filecmp.cmp line needs to be wrapped in a try...except to stop > it blowing up: > > try: > RequireUnboundReload = not filecmp.cmp(f.name, self.path, > shallow=False) > except: > RequireUnboundReload = True I generally don’t like blanco exception catches. You will never know about the daemons that you are hiding. > This will force the file to be to be generated, even if it is > only copying in a blank temp file, and unbound will restart. It > will only ever happen once. > > > On 24/03/2024 15:49, Nick Howitt wrote: >> Hi all, >> **** This is a repost of an email incorrectly submitted. **** >> Please bear with me as I am new to IPFire and not at all used >> to this style of development, so any guidance would be >> appreciated. >> There is a bug, >> https://bugzilla.ipfire.org/show_bug.cgi?id=13254, where >> unbound-dhcp-leases-bridge restarts unbound a lot when it is >> totally unnecessary. It appears to be because when a lease is >> renewed, the script gets triggered, created a new >> /etc/unbound/dhcp-leases.conf then restarts Unbound to read the >> file. Generally this seems to be unnecessary as, with a renewed >> lease, the old and new /etc/unbound/dhcp-leases.conf files are >> the same. With 5 leases in that file (one for a machine not >> active) I am getting 3-4 restarts an hour of Unbound when I >> have a min/max lease time of 60/120min. >> Looking at the code, it is fairly easy to fix. The current code >> creates a temp file then copies it into place then restarts >> unbound. All I am doing is doing a file comparison before the >> copy and skipping the restart if the files are the same. >> There were a couple of gotchas because setting the file >> attributes and copying the file were done inside the "with" >> block for generating the temporary file. This meant a file >> comparison always returned False as the temp file was still >> open and so never the same is the old file. >> I moved those two statements outside the "with". This forced me >> to change the fchmod to chmod. >> It could be argued that the file copy should not be done if the >> files are not different, but it would take an extra "if" and >> you'd have to remember to delete the temp file. If required, I >> can make that change. >> Also, one small thing I noticed once is that the old and new >> dhcp-leases.conf files could occasionally contain the same >> leases but in a different order. I have been unable to >> reproduce, but to sidestep it, instead of stepping through the >> leases variable directly, I sort it and step through that. It >> should make the resulting file completely deterministic and >> make the file comparison more effective. >> My patch is: >> From 73873d4944944a2f02317a73074a6894726f36f7 Mon Sep 17 >> 00:00:00 2001 >> From: Nick Howitt nick@howitts.co.uk >> Date: Sun, 24 Mar 2024 15:17:19 +0000 >> Subject: [PATCH] Stop unbound-dhcp-leases-bridge from >> restarting unbound every >> time it write dhcp-leases.conf as it is very often unchanged >> and does not >> require a restart. >> --- >> config/unbound/unbound-dhcp-leases-bridge | 28 >> +++++++++++++++-------- >> 1 file changed, 19 insertions(+), 9 deletions(-) >> diff --git a/config/unbound/unbound-dhcp-leases-bridge >> b/config/unbound/unbound-dhcp-leases-bridge >> index e9f022aff..d22772066 100644 >> --- a/config/unbound/unbound-dhcp-leases-bridge >> +++ b/config/unbound/unbound-dhcp-leases-bridge >> @@ -22,6 +22,7 @@ >> import argparse >> import datetime >> import daemon >> +import filecmp >> import functools >> import ipaddress >> import logging >> @@ -516,26 +517,35 @@ class UnboundConfigWriter(object): >> def update_dhcp_leases(self, leases): >> # Write out all leases >> - self.write_dhcp_leases(leases) >> + if self.write_dhcp_leases(leases): >> - log.debug("Reloading Unbound...") >> + log.debug("Reloading Unbound...") >> - # Reload the configuration without dropping the cache >> - self._control("reload_keep_cache") >> + # Reload the configuration without dropping the cache >> + self._control("reload_keep_cache") >> + >> + else: >> + >> + log.debug("Not reloading Unbound. Leases not >> changed.") >> def write_dhcp_leases(self, leases): >> log.debug("Writing DHCP leases...") >> with tempfile.NamedTemporaryFile(mode="w", >> delete=False) as f: >> - for l in leases: >> + for l in sorted(leases): >> for rr in l.rrset: >> f.write("local-data: "%s"\n" % " >> ".join(rr)) >> - # Make file readable for everyone >> - os.fchmod(f.fileno(), >> stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >> + filecmp.clear_cache() >> + RequireUnboundReload = not filecmp.cmp(f.name, >> self.path, shallow=False) >> + >> + # Make file readable for everyone >> + os.chmod(f.name, >> stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >> + >> + # Move the file to its destination >> + os.rename(f.name, self.path) >> - # Move the file to its destination >> - os.rename(f.name, self.path) >> + return RequireUnboundReload >> def _control(self, *args): >> command = ["unbound-control"]
Resubmitting the patch.
Following my previous reply, I would like to stick with filecmp as I am using it with "shallow=False" to avoid doing the comparison just with stat() calls. Note that I also flush the filecmp cache before using the command so it is running with the latest file data. Talking about file descriptors, atomic file replacement etc., scares me.
File descriptors are great. They avoid race-conditions and all sorts of things.
But I have never used them so I don't intend to go that way. Too much for me to get my brain round and a limited ability to test.
Well, I pretty much gave you the entire solution… There was nothing left to do apart from copying and pasting the snippets.
Ok I've tried to use your version but struggled to get it going. If I used it as was and called compare(file1_name, file2_name) it failed as "compare" was not defined. I could call it using self.compare but then it failed with "TypeError: UnboundConfigWriter.compare() takes 2 positional arguments but 3 were given". I changed the function to compare(self, f1, f2). Then it failed with "AttributeError: 'str' object has no attribute 'seek'", so I changed it again to accept file names then open the files before doing the seeks. I really hope it is OK as I am out of my depth when defining functions with "self" as an argument. Please can you check it? If it is not OK, can I revert to filecmp?
I have made one futher change, the sorted(leases) line was not working and not sorting the object ever, and I saw lines changing places in the resulting leases file. I have changed it to "sorted(leases, key=lambda x: x.ipaddr)". It definitely sorts this time.
I don’t think that module is the best choice really if you have to hack around and flush the cache. Why would it even cache things in the first place? I don’t see where the cache helps.
From the filecmp.clear_cache documentation:
Clear the filecmp cache. This may be useful if a file is compared so quickly after it is modified that it is within the mtime resolution of the underlying filesystem.
Now this makes me believe that the clear_cache is only relevant when using shallow=True, which I'm not and I don't know how to check the underlying code. It is used to make sure the stat() info is updated, but we are not using the stat() info. I guess we could as easily add a one second sleep instead as that is way longer than the mtime resolution.
The function is here and it is rather simple: https://github.com/python/cpython/blob/3.12/Lib/filecmp.py#L30
It will always stat() both files and compares three items in case shallow is False: The type of the file, the size and mtime. If the file is not a regular file, the function will exist rather early. So in our case, that is all work the function is doing for thing. It boils down to something very similar to what I suggested to use to compare the files.
On top of that, there is a cache which will always be used but will never return a match for us because the temporary file at least will always have another mtime.
You could clear the cache after every call, but I would rather not have our code messy and care about implementation details of some Python module.
The module is chosen because all the references I read about comparing files preferred this one-liner for comparing files, although other methods were given and there were lots of discussions about large files (GB+) because if you chomp them it may be much quicker to say they are not the same when you find the first block that does not match. Having said this, I don't know how filecmp works, but we only have a small file. At the same time, I am not a regular programmer so simple one-liners are good for me.
You would always want to read both files block for block and compare the blocks. There is no way to implement this significantly faster.
I have modified the earlier try/except I proposed to catch only FileNotFoundError, and I have changed CamelCase to snake_case.
config/unbound/unbound-dhcp-leases-bridge | 31 ++++++++++++++++------- 1 file changed, 22 insertions(+), 9 deletions(-)
diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge index e9f022aff..a9ec8e1e2 100644 --- a/config/unbound/unbound-dhcp-leases-bridge +++ b/config/unbound/unbound-dhcp-leases-bridge @@ -22,6 +22,7 @@ import argparse import datetime import daemon +import filecmp import functools import ipaddress import logging @@ -516,26 +517,38 @@ class UnboundConfigWriter(object):
def update_dhcp_leases(self, leases): # Write out all leases
- self.write_dhcp_leases(leases)
- if self.write_dhcp_leases(leases):
- log.debug("Reloading Unbound...")
- log.debug("Reloading Unbound...")
- # Reload the configuration without dropping the cache
- self._control("reload_keep_cache")
- # Reload the configuration without dropping the cache
- self._control("reload_keep_cache")
- else:
- log.debug("Not reloading Unbound. Leases not changed.")
def write_dhcp_leases(self, leases): log.debug("Writing DHCP leases...")
with tempfile.NamedTemporaryFile(mode="w", delete=False) as f:
- for l in leases:
- for l in sorted(leases):
for rr in l.rrset: f.write("local-data: "%s"\n" % " ".join(rr))
- # Make file readable for everyone
- os.fchmod(f.fileno(),
stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)
- filecmp.clear_cache()
- try:
- require_unbound_reload = not filecmp.cmp(f.name, self.path,
shallow=False)
- except FileNotFoundError:
- require_unbound_reload = True
If you want to go this route, you will have to flush the temporary file first. Otherwise you might have some data left in the buffer that has not been written (because the file is still open). You will then compare incomplete output. This would not become a problem if you used the file descriptors.
The temp file is closed at the point of doing the filecmp, and the filecmp cache has been flushed. How do I flush the temporary file further. Reading up, is it just having the last statement in the with block as f.flush() with the same indent as "for l in sorted(leases):"
If you have left the with block you will already have flushed the buffers and closed the file. It looks like I misread the indentation in the patch then.
Catching FileNotFound would work for me.
Good
- # Make file readable for everyone
- os.chmod(f.name,
stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)
- # Move the file to its destination
- os.rename(f.name, self.path)
- # Move the file to its destination
- os.rename(f.name, self.path)
- return require_unbound_reload
def _control(self, *args): command = ["unbound-control"] -- 2.43.0
Nick
config/unbound/unbound-dhcp-leases-bridge | 50 +++++++++++++++++++---- 1 file changed, 41 insertions(+), 9 deletions(-)
diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge index e9f022aff..80d9523ea 100644 --- a/config/unbound/unbound-dhcp-leases-bridge +++ b/config/unbound/unbound-dhcp-leases-bridge @@ -516,26 +516,37 @@ class UnboundConfigWriter(object):
def update_dhcp_leases(self, leases): # Write out all leases - self.write_dhcp_leases(leases) + if self.write_dhcp_leases(leases):
- log.debug("Reloading Unbound...") + log.debug("Reloading Unbound...")
- # Reload the configuration without dropping the cache - self._control("reload_keep_cache") + # Reload the configuration without dropping the cache + self._control("reload_keep_cache")
+ else:
+ log.debug("Not reloading Unbound. Leases not changed.")
def write_dhcp_leases(self, leases): log.debug("Writing DHCP leases...")
with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: - for l in leases: + for l in sorted(leases, key=lambda x: x.ipaddr): for rr in l.rrset: f.write("local-data: "%s"\n" % " ".join(rr))
- # Make file readable for everyone - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) + try: + require_unbound_reload = not self.compare(f.name, self.path) + except FileNotFoundError: + require_unbound_reload = True
+ # Make file readable for everyone + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)
+ # Move the file to its destination + os.rename(f.name, self.path)
- # Move the file to its destination - os.rename(f.name, self.path) + return require_unbound_reload
def _control(self, *args): command = ["unbound-control"] @@ -551,6 +562,27 @@ class UnboundConfigWriter(object):
raise e
+ def compare(self, if1, if2): + buffersize = 16 * 1024
+ f1 = open(if1, "r") + f2 = open(if2, "r")
+ # Reset file handles + f1.seek(0) + f2.seek(0)
+ while True: + b1 = f1.read(buffersize) + b2 = f2.read(buffersize)
+ if not b1 == b2: + return False
+ elif not b1 or not b2: + break
+ return True
if __name__ == "__main__": parser = argparse.ArgumentParser(description="Bridge for DHCP Leases and Unbound DNS")
Please can I bump this again to request a review of my patch submitted on 28th March? Thanks, Nick.
On 10/04/2024 12:04, Nick Howitt wrote:
Bump
On 28/03/2024 17:25, Nick Howitt wrote:
Next version.
On 28/03/2024 10:41, Michael Tremer wrote:
Hello,
On 27 Mar 2024, at 15:28, Nick Howitt nick@howitts.co.uk wrote:
On 26/03/2024 11:10, Michael Tremer wrote:
On 25 Mar 2024, at 13:11, Nick Howitt nick@howitts.co.uk wrote:
On 25/03/2024 11:31, Michael Tremer wrote: > Hello, > I don’t think that the risk is very high, but I would account > for this problem. > With my proposed changes from my previous email this should not > be a large problem at all. When opening > /etc/unbound/dhcp-leases.conf with mode “r+” you can catch the > Exception and skip the comparison and open the file again with “w”. >> On 25 Mar 2024, at 10:10, Nick Howitt nick@howitts.co.uk wrote: >> >> If there is a risk that the /etc/unbound/dhcp-leases.conf does >> not exist, for example on a clean install (I don't know), the >> filecmp.cmp line needs to be wrapped in a try...except to stop >> it blowing up: >> >> try: >> RequireUnboundReload = not filecmp.cmp(f.name, self.path, >> shallow=False) >> except: >> RequireUnboundReload = True > I generally don’t like blanco exception catches. You will never > know about the daemons that you are hiding. >> This will force the file to be to be generated, even if it is >> only copying in a blank temp file, and unbound will restart. It >> will only ever happen once. >> >> >> On 24/03/2024 15:49, Nick Howitt wrote: >>> Hi all, >>> **** This is a repost of an email incorrectly submitted. **** >>> Please bear with me as I am new to IPFire and not at all used >>> to this style of development, so any guidance would be >>> appreciated. >>> There is a bug, >>> https://bugzilla.ipfire.org/show_bug.cgi?id=13254, where >>> unbound-dhcp-leases-bridge restarts unbound a lot when it is >>> totally unnecessary. It appears to be because when a lease is >>> renewed, the script gets triggered, created a new >>> /etc/unbound/dhcp-leases.conf then restarts Unbound to read >>> the file. Generally this seems to be unnecessary as, with a >>> renewed lease, the old and new /etc/unbound/dhcp-leases.conf >>> files are the same. With 5 leases in that file (one for a >>> machine not active) I am getting 3-4 restarts an hour of >>> Unbound when I have a min/max lease time of 60/120min. >>> Looking at the code, it is fairly easy to fix. The current >>> code creates a temp file then copies it into place then >>> restarts unbound. All I am doing is doing a file comparison >>> before the copy and skipping the restart if the files are the >>> same. >>> There were a couple of gotchas because setting the file >>> attributes and copying the file were done inside the "with" >>> block for generating the temporary file. This meant a file >>> comparison always returned False as the temp file was still >>> open and so never the same is the old file. >>> I moved those two statements outside the "with". This forced >>> me to change the fchmod to chmod. >>> It could be argued that the file copy should not be done if >>> the files are not different, but it would take an extra "if" >>> and you'd have to remember to delete the temp file. If >>> required, I can make that change. >>> Also, one small thing I noticed once is that the old and new >>> dhcp-leases.conf files could occasionally contain the same >>> leases but in a different order. I have been unable to >>> reproduce, but to sidestep it, instead of stepping through the >>> leases variable directly, I sort it and step through that. It >>> should make the resulting file completely deterministic and >>> make the file comparison more effective. >>> My patch is: >>> From 73873d4944944a2f02317a73074a6894726f36f7 Mon Sep 17 >>> 00:00:00 2001 >>> From: Nick Howitt nick@howitts.co.uk >>> Date: Sun, 24 Mar 2024 15:17:19 +0000 >>> Subject: [PATCH] Stop unbound-dhcp-leases-bridge from >>> restarting unbound every >>> time it write dhcp-leases.conf as it is very often unchanged >>> and does not >>> require a restart. >>> --- >>> config/unbound/unbound-dhcp-leases-bridge | 28 >>> +++++++++++++++-------- >>> 1 file changed, 19 insertions(+), 9 deletions(-) >>> diff --git a/config/unbound/unbound-dhcp-leases-bridge >>> b/config/unbound/unbound-dhcp-leases-bridge >>> index e9f022aff..d22772066 100644 >>> --- a/config/unbound/unbound-dhcp-leases-bridge >>> +++ b/config/unbound/unbound-dhcp-leases-bridge >>> @@ -22,6 +22,7 @@ >>> import argparse >>> import datetime >>> import daemon >>> +import filecmp >>> import functools >>> import ipaddress >>> import logging >>> @@ -516,26 +517,35 @@ class UnboundConfigWriter(object): >>> def update_dhcp_leases(self, leases): >>> # Write out all leases >>> - self.write_dhcp_leases(leases) >>> + if self.write_dhcp_leases(leases): >>> - log.debug("Reloading Unbound...") >>> + log.debug("Reloading Unbound...") >>> - # Reload the configuration without dropping the cache >>> - self._control("reload_keep_cache") >>> + # Reload the configuration without dropping the >>> cache >>> + self._control("reload_keep_cache") >>> + >>> + else: >>> + >>> + log.debug("Not reloading Unbound. Leases not >>> changed.") >>> def write_dhcp_leases(self, leases): >>> log.debug("Writing DHCP leases...") >>> with tempfile.NamedTemporaryFile(mode="w", >>> delete=False) as f: >>> - for l in leases: >>> + for l in sorted(leases): >>> for rr in l.rrset: >>> f.write("local-data: "%s"\n" % " >>> ".join(rr)) >>> - # Make file readable for everyone >>> - os.fchmod(f.fileno(), >>> stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>> + filecmp.clear_cache() >>> + RequireUnboundReload = not filecmp.cmp(f.name, >>> self.path, shallow=False) >>> + >>> + # Make file readable for everyone >>> + os.chmod(f.name, >>> stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>> + >>> + # Move the file to its destination >>> + os.rename(f.name, self.path) >>> - # Move the file to its destination >>> - os.rename(f.name, self.path) >>> + return RequireUnboundReload >>> def _control(self, *args): >>> command = ["unbound-control"] Resubmitting the patch.
Following my previous reply, I would like to stick with filecmp as I am using it with "shallow=False" to avoid doing the comparison just with stat() calls. Note that I also flush the filecmp cache before using the command so it is running with the latest file data. Talking about file descriptors, atomic file replacement etc., scares me.
File descriptors are great. They avoid race-conditions and all sorts of things.
But I have never used them so I don't intend to go that way. Too much for me to get my brain round and a limited ability to test.
Well, I pretty much gave you the entire solution… There was nothing left to do apart from copying and pasting the snippets.
Ok I've tried to use your version but struggled to get it going. If I used it as was and called compare(file1_name, file2_name) it failed as "compare" was not defined. I could call it using self.compare but then it failed with "TypeError: UnboundConfigWriter.compare() takes 2 positional arguments but 3 were given". I changed the function to compare(self, f1, f2). Then it failed with "AttributeError: 'str' object has no attribute 'seek'", so I changed it again to accept file names then open the files before doing the seeks. I really hope it is OK as I am out of my depth when defining functions with "self" as an argument. Please can you check it? If it is not OK, can I revert to filecmp?
I have made one futher change, the sorted(leases) line was not working and not sorting the object ever, and I saw lines changing places in the resulting leases file. I have changed it to "sorted(leases, key=lambda x: x.ipaddr)". It definitely sorts this time.
I don’t think that module is the best choice really if you have to hack around and flush the cache. Why would it even cache things in the first place? I don’t see where the cache helps.
From the filecmp.clear_cache documentation:
Clear the filecmp cache. This may be useful if a file is compared so quickly after it is modified that it is within the mtime resolution of the underlying filesystem.
Now this makes me believe that the clear_cache is only relevant when using shallow=True, which I'm not and I don't know how to check the underlying code. It is used to make sure the stat() info is updated, but we are not using the stat() info. I guess we could as easily add a one second sleep instead as that is way longer than the mtime resolution.
The function is here and it is rather simple: https://github.com/python/cpython/blob/3.12/Lib/filecmp.py#L30
It will always stat() both files and compares three items in case shallow is False: The type of the file, the size and mtime. If the file is not a regular file, the function will exist rather early. So in our case, that is all work the function is doing for thing. It boils down to something very similar to what I suggested to use to compare the files.
On top of that, there is a cache which will always be used but will never return a match for us because the temporary file at least will always have another mtime.
You could clear the cache after every call, but I would rather not have our code messy and care about implementation details of some Python module.
The module is chosen because all the references I read about comparing files preferred this one-liner for comparing files, although other methods were given and there were lots of discussions about large files (GB+) because if you chomp them it may be much quicker to say they are not the same when you find the first block that does not match. Having said this, I don't know how filecmp works, but we only have a small file. At the same time, I am not a regular programmer so simple one-liners are good for me.
You would always want to read both files block for block and compare the blocks. There is no way to implement this significantly faster.
I have modified the earlier try/except I proposed to catch only FileNotFoundError, and I have changed CamelCase to snake_case.
config/unbound/unbound-dhcp-leases-bridge | 31 ++++++++++++++++------- 1 file changed, 22 insertions(+), 9 deletions(-)
diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge index e9f022aff..a9ec8e1e2 100644 --- a/config/unbound/unbound-dhcp-leases-bridge +++ b/config/unbound/unbound-dhcp-leases-bridge @@ -22,6 +22,7 @@ import argparse import datetime import daemon +import filecmp import functools import ipaddress import logging @@ -516,26 +517,38 @@ class UnboundConfigWriter(object):
def update_dhcp_leases(self, leases): # Write out all leases
- self.write_dhcp_leases(leases)
- if self.write_dhcp_leases(leases):
- log.debug("Reloading Unbound...")
- log.debug("Reloading Unbound...")
- # Reload the configuration without dropping the cache
- self._control("reload_keep_cache")
- # Reload the configuration without dropping the cache
- self._control("reload_keep_cache")
- else:
- log.debug("Not reloading Unbound. Leases not changed.")
def write_dhcp_leases(self, leases): log.debug("Writing DHCP leases...")
with tempfile.NamedTemporaryFile(mode="w", delete=False) as f:
- for l in leases:
- for l in sorted(leases):
for rr in l.rrset: f.write("local-data: "%s"\n" % " ".join(rr))
- # Make file readable for everyone
- os.fchmod(f.fileno(),
stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)
- filecmp.clear_cache()
- try:
- require_unbound_reload = not filecmp.cmp(f.name, self.path,
shallow=False)
- except FileNotFoundError:
- require_unbound_reload = True
If you want to go this route, you will have to flush the temporary file first. Otherwise you might have some data left in the buffer that has not been written (because the file is still open). You will then compare incomplete output. This would not become a problem if you used the file descriptors.
The temp file is closed at the point of doing the filecmp, and the filecmp cache has been flushed. How do I flush the temporary file further. Reading up, is it just having the last statement in the with block as f.flush() with the same indent as "for l in sorted(leases):"
If you have left the with block you will already have flushed the buffers and closed the file. It looks like I misread the indentation in the patch then.
Catching FileNotFound would work for me.
Good
- # Make file readable for everyone
- os.chmod(f.name,
stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)
- # Move the file to its destination
- os.rename(f.name, self.path)
- # Move the file to its destination
- os.rename(f.name, self.path)
- return require_unbound_reload
def _control(self, *args): command = ["unbound-control"] -- 2.43.0
Nick
config/unbound/unbound-dhcp-leases-bridge | 50 +++++++++++++++++++---- 1 file changed, 41 insertions(+), 9 deletions(-)
diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge index e9f022aff..80d9523ea 100644 --- a/config/unbound/unbound-dhcp-leases-bridge +++ b/config/unbound/unbound-dhcp-leases-bridge @@ -516,26 +516,37 @@ class UnboundConfigWriter(object):
def update_dhcp_leases(self, leases): # Write out all leases - self.write_dhcp_leases(leases) + if self.write_dhcp_leases(leases):
- log.debug("Reloading Unbound...") + log.debug("Reloading Unbound...")
- # Reload the configuration without dropping the cache - self._control("reload_keep_cache") + # Reload the configuration without dropping the cache + self._control("reload_keep_cache")
+ else:
+ log.debug("Not reloading Unbound. Leases not changed.")
def write_dhcp_leases(self, leases): log.debug("Writing DHCP leases...")
with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: - for l in leases: + for l in sorted(leases, key=lambda x: x.ipaddr): for rr in l.rrset: f.write("local-data: "%s"\n" % " ".join(rr))
- # Make file readable for everyone - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) + try: + require_unbound_reload = not self.compare(f.name, self.path) + except FileNotFoundError: + require_unbound_reload = True
+ # Make file readable for everyone + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)
+ # Move the file to its destination + os.rename(f.name, self.path)
- # Move the file to its destination - os.rename(f.name, self.path) + return require_unbound_reload
def _control(self, *args): command = ["unbound-control"] @@ -551,6 +562,27 @@ class UnboundConfigWriter(object):
raise e
+ def compare(self, if1, if2): + buffersize = 16 * 1024
+ f1 = open(if1, "r") + f2 = open(if2, "r")
+ # Reset file handles + f1.seek(0) + f2.seek(0)
+ while True: + b1 = f1.read(buffersize) + b2 = f2.read(buffersize)
+ if not b1 == b2: + return False
+ elif not b1 or not b2: + break
+ return True
if __name__ == "__main__": parser = argparse.ArgumentParser(description="Bridge for DHCP Leases and Unbound DNS")
Hello Nick,
On 25 Apr 2024, at 13:43, Nick Howitt nick@howitts.co.uk wrote:
Please can I bump this again to request a review of my patch submitted on 28th March? Thanks, Nick.
It is a little bit difficult to test your proposed changes if testers have to copy and paste them out of an email.
I have just submitted a patch set to this list that breaks down all the changes that you proposed and I cleaned up the code so that we can use the filecmp function without any extra steps. We can simply jump out of the write function and return False to prevent Unbound being reloaded.
We will also avoid replacing the file all the time as that causes lots of unnecessary disk I/O. I am not saying it makes a significant difference, but flash has a limited write count that we are still wasting.
Please review and you if you are happy, please add the appropriate Git tag.
Best, -Michael
On 10/04/2024 12:04, Nick Howitt wrote:
Bump
On 28/03/2024 17:25, Nick Howitt wrote:
Next version.
On 28/03/2024 10:41, Michael Tremer wrote:
Hello,
On 27 Mar 2024, at 15:28, Nick Howitt nick@howitts.co.uk wrote:
On 26/03/2024 11:10, Michael Tremer wrote:
> On 25 Mar 2024, at 13:11, Nick Howitt nick@howitts.co.uk wrote: > > > > On 25/03/2024 11:31, Michael Tremer wrote: >> Hello, >> I don’t think that the risk is very high, but I would account for this problem. >> With my proposed changes from my previous email this should not be a large problem at all. When opening /etc/unbound/dhcp-leases.conf with mode “r+” you can catch the Exception and skip the comparison and open the file again with “w”. >>> On 25 Mar 2024, at 10:10, Nick Howitt nick@howitts.co.uk wrote: >>> >>> If there is a risk that the /etc/unbound/dhcp-leases.conf does not exist, for example on a clean install (I don't know), the filecmp.cmp line needs to be wrapped in a try...except to stop it blowing up: >>> >>> try: >>> RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) >>> except: >>> RequireUnboundReload = True >> I generally don’t like blanco exception catches. You will never know about the daemons that you are hiding. >>> This will force the file to be to be generated, even if it is only copying in a blank temp file, and unbound will restart. It will only ever happen once. >>> >>> >>> On 24/03/2024 15:49, Nick Howitt wrote: >>>> Hi all, >>>> **** This is a repost of an email incorrectly submitted. **** >>>> Please bear with me as I am new to IPFire and not at all used to this style of development, so any guidance would be appreciated. >>>> There is a bug, https://bugzilla.ipfire.org/show_bug.cgi?id=13254, where unbound-dhcp-leases-bridge restarts unbound a lot when it is totally unnecessary. It appears to be because when a lease is renewed, the script gets triggered, created a new /etc/unbound/dhcp-leases.conf then restarts Unbound to read the file. Generally this seems to be unnecessary as, with a renewed lease, the old and new /etc/unbound/dhcp-leases.conf files are the same. With 5 leases in that file (one for a machine not active) I am getting 3-4 restarts an hour of Unbound when I have a min/max lease time of 60/120min. >>>> Looking at the code, it is fairly easy to fix. The current code creates a temp file then copies it into place then restarts unbound. All I am doing is doing a file comparison before the copy and skipping the restart if the files are the same. >>>> There were a couple of gotchas because setting the file attributes and copying the file were done inside the "with" block for generating the temporary file. This meant a file comparison always returned False as the temp file was still open and so never the same is the old file. >>>> I moved those two statements outside the "with". This forced me to change the fchmod to chmod. >>>> It could be argued that the file copy should not be done if the files are not different, but it would take an extra "if" and you'd have to remember to delete the temp file. If required, I can make that change. >>>> Also, one small thing I noticed once is that the old and new dhcp-leases.conf files could occasionally contain the same leases but in a different order. I have been unable to reproduce, but to sidestep it, instead of stepping through the leases variable directly, I sort it and step through that. It should make the resulting file completely deterministic and make the file comparison more effective. >>>> My patch is: >>>> From 73873d4944944a2f02317a73074a6894726f36f7 Mon Sep 17 00:00:00 2001 >>>> From: Nick Howitt nick@howitts.co.uk >>>> Date: Sun, 24 Mar 2024 15:17:19 +0000 >>>> Subject: [PATCH] Stop unbound-dhcp-leases-bridge from restarting unbound every >>>> time it write dhcp-leases.conf as it is very often unchanged and does not >>>> require a restart. >>>> --- >>>> config/unbound/unbound-dhcp-leases-bridge | 28 +++++++++++++++-------- >>>> 1 file changed, 19 insertions(+), 9 deletions(-) >>>> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >>>> index e9f022aff..d22772066 100644 >>>> --- a/config/unbound/unbound-dhcp-leases-bridge >>>> +++ b/config/unbound/unbound-dhcp-leases-bridge >>>> @@ -22,6 +22,7 @@ >>>> import argparse >>>> import datetime >>>> import daemon >>>> +import filecmp >>>> import functools >>>> import ipaddress >>>> import logging >>>> @@ -516,26 +517,35 @@ class UnboundConfigWriter(object): >>>> def update_dhcp_leases(self, leases): >>>> # Write out all leases >>>> - self.write_dhcp_leases(leases) >>>> + if self.write_dhcp_leases(leases): >>>> - log.debug("Reloading Unbound...") >>>> + log.debug("Reloading Unbound...") >>>> - # Reload the configuration without dropping the cache >>>> - self._control("reload_keep_cache") >>>> + # Reload the configuration without dropping the cache >>>> + self._control("reload_keep_cache") >>>> + >>>> + else: >>>> + >>>> + log.debug("Not reloading Unbound. Leases not changed.") >>>> def write_dhcp_leases(self, leases): >>>> log.debug("Writing DHCP leases...") >>>> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >>>> - for l in leases: >>>> + for l in sorted(leases): >>>> for rr in l.rrset: >>>> f.write("local-data: "%s"\n" % " ".join(rr)) >>>> - # Make file readable for everyone >>>> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>> + filecmp.clear_cache() >>>> + RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) >>>> + >>>> + # Make file readable for everyone >>>> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>> + >>>> + # Move the file to its destination >>>> + os.rename(f.name, self.path) >>>> - # Move the file to its destination >>>> - os.rename(f.name, self.path) >>>> + return RequireUnboundReload >>>> def _control(self, *args): >>>> command = ["unbound-control"] > Resubmitting the patch. > > Following my previous reply, I would like to stick with filecmp as I am using it with "shallow=False" to avoid doing the comparison just with stat() calls. Note that I also flush the filecmp cache before using the command so it is running with the latest file data. Talking about file descriptors, atomic file replacement etc., scares me. File descriptors are great. They avoid race-conditions and all sorts of things.
But I have never used them so I don't intend to go that way. Too much for me to get my brain round and a limited ability to test.
Well, I pretty much gave you the entire solution… There was nothing left to do apart from copying and pasting the snippets.
Ok I've tried to use your version but struggled to get it going. If I used it as was and called compare(file1_name, file2_name) it failed as "compare" was not defined. I could call it using self.compare but then it failed with "TypeError: UnboundConfigWriter.compare() takes 2 positional arguments but 3 were given". I changed the function to compare(self, f1, f2). Then it failed with "AttributeError: 'str' object has no attribute 'seek'", so I changed it again to accept file names then open the files before doing the seeks. I really hope it is OK as I am out of my depth when defining functions with "self" as an argument. Please can you check it? If it is not OK, can I revert to filecmp?
I have made one futher change, the sorted(leases) line was not working and not sorting the object ever, and I saw lines changing places in the resulting leases file. I have changed it to "sorted(leases, key=lambda x: x.ipaddr)". It definitely sorts this time.
I don’t think that module is the best choice really if you have to hack around and flush the cache. Why would it even cache things in the first place? I don’t see where the cache helps.
From the filecmp.clear_cache documentation:
Clear the filecmp cache. This may be useful if a file is compared so quickly after it is modified that it is within the mtime resolution of the underlying filesystem.
Now this makes me believe that the clear_cache is only relevant when using shallow=True, which I'm not and I don't know how to check the underlying code. It is used to make sure the stat() info is updated, but we are not using the stat() info. I guess we could as easily add a one second sleep instead as that is way longer than the mtime resolution.
The function is here and it is rather simple: https://github.com/python/cpython/blob/3.12/Lib/filecmp.py#L30
It will always stat() both files and compares three items in case shallow is False: The type of the file, the size and mtime. If the file is not a regular file, the function will exist rather early. So in our case, that is all work the function is doing for thing. It boils down to something very similar to what I suggested to use to compare the files.
On top of that, there is a cache which will always be used but will never return a match for us because the temporary file at least will always have another mtime.
You could clear the cache after every call, but I would rather not have our code messy and care about implementation details of some Python module.
The module is chosen because all the references I read about comparing files preferred this one-liner for comparing files, although other methods were given and there were lots of discussions about large files (GB+) because if you chomp them it may be much quicker to say they are not the same when you find the first block that does not match. Having said this, I don't know how filecmp works, but we only have a small file. At the same time, I am not a regular programmer so simple one-liners are good for me.
You would always want to read both files block for block and compare the blocks. There is no way to implement this significantly faster.
> I have modified the earlier try/except I proposed to catch only FileNotFoundError, and I have changed CamelCase to snake_case. > > --- > config/unbound/unbound-dhcp-leases-bridge | 31 ++++++++++++++++------- > 1 file changed, 22 insertions(+), 9 deletions(-) > > diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge > index e9f022aff..a9ec8e1e2 100644 > --- a/config/unbound/unbound-dhcp-leases-bridge > +++ b/config/unbound/unbound-dhcp-leases-bridge > @@ -22,6 +22,7 @@ > import argparse > import datetime > import daemon > +import filecmp > import functools > import ipaddress > import logging > @@ -516,26 +517,38 @@ class UnboundConfigWriter(object): > > def update_dhcp_leases(self, leases): > # Write out all leases > - self.write_dhcp_leases(leases) > + if self.write_dhcp_leases(leases): > > - log.debug("Reloading Unbound...") > + log.debug("Reloading Unbound...") > > - # Reload the configuration without dropping the cache > - self._control("reload_keep_cache") > + # Reload the configuration without dropping the cache > + self._control("reload_keep_cache") > + > + else: > + > + log.debug("Not reloading Unbound. Leases not changed.") > > def write_dhcp_leases(self, leases): > log.debug("Writing DHCP leases...") > > with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: > - for l in leases: > + for l in sorted(leases): > for rr in l.rrset: > f.write("local-data: "%s"\n" % " ".join(rr)) > > - # Make file readable for everyone > - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) > + filecmp.clear_cache() > + try: > + require_unbound_reload = not filecmp.cmp(f.name, self.path, shallow=False) > + except FileNotFoundError: > + require_unbound_reload = True If you want to go this route, you will have to flush the temporary file first. Otherwise you might have some data left in the buffer that has not been written (because the file is still open). You will then compare incomplete output. This would not become a problem if you used the file descriptors.
The temp file is closed at the point of doing the filecmp, and the filecmp cache has been flushed. How do I flush the temporary file further. Reading up, is it just having the last statement in the with block as f.flush() with the same indent as "for l in sorted(leases):"
If you have left the with block you will already have flushed the buffers and closed the file. It looks like I misread the indentation in the patch then.
Catching FileNotFound would work for me.
Good
> + > + # Make file readable for everyone > + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) > + > + # Move the file to its destination > + os.rename(f.name, self.path) > > - # Move the file to its destination > - os.rename(f.name, self.path) > + return require_unbound_reload > > def _control(self, *args): > command = ["unbound-control"] > -- > 2.43.0 > > > Nick
config/unbound/unbound-dhcp-leases-bridge | 50 +++++++++++++++++++---- 1 file changed, 41 insertions(+), 9 deletions(-)
diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge index e9f022aff..80d9523ea 100644 --- a/config/unbound/unbound-dhcp-leases-bridge +++ b/config/unbound/unbound-dhcp-leases-bridge @@ -516,26 +516,37 @@ class UnboundConfigWriter(object):
def update_dhcp_leases(self, leases): # Write out all leases
self.write_dhcp_leases(leases)
if self.write_dhcp_leases(leases):
log.debug("Reloading Unbound...")
log.debug("Reloading Unbound...")
# Reload the configuration without dropping the cacheself._control("reload_keep_cache")
# Reload the configuration without dropping the cacheself._control("reload_keep_cache")else:log.debug("Not reloading Unbound. Leases not changed.")def write_dhcp_leases(self, leases): log.debug("Writing DHCP leases...")
with tempfile.NamedTemporaryFile(mode="w", delete=False) as f:
for l in leases:
for l in sorted(leases, key=lambda x: x.ipaddr): for rr in l.rrset: f.write("local-data: \"%s\"\n" % " ".join(rr))
# Make file readable for everyoneos.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)
try:require_unbound_reload = not self.compare(f.name, self.path)except FileNotFoundError:require_unbound_reload = True# Make file readable for everyoneos.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)# Move the file to its destinationos.rename(f.name, self.path)
# Move the file to its destinationos.rename(f.name, self.path)
return require_unbound_reloaddef _control(self, *args): command = ["unbound-control"]
@@ -551,6 +562,27 @@ class UnboundConfigWriter(object):
raise e
- def compare(self, if1, if2):
buffersize = 16 * 1024f1 = open(if1, "r")f2 = open(if2, "r")# Reset file handlesf1.seek(0)f2.seek(0)while True:b1 = f1.read(buffersize)b2 = f2.read(buffersize)if not b1 == b2:return Falseelif not b1 or not b2:breakreturn Trueif __name__ == "__main__": parser = argparse.ArgumentParser(description="Bridge for DHCP Leases and Unbound DNS")
On 26/04/2024 16:12, Michael Tremer wrote:
Hello Nick,
On 25 Apr 2024, at 13:43, Nick Howittnick@howitts.co.uk wrote:
Please can I bump this again to request a review of my patch submitted on 28th March? Thanks, Nick.
It is a little bit difficult to test your proposed changes if testers have to copy and paste them out of an email.
Confused. I was told I had to submit the diff in an email with [PATCH] in the title and not to attach a file. Please can you tell me how I should be operating?
I have just submitted a patch set to this list that breaks down all the changes that you proposed and I cleaned up the code so that we can use the filecmp function without any extra steps. We can simply jump out of the write function and return False to prevent Unbound being reloaded.
I thought you didn't want me to use filecmp and pushed me to use your file compare routine.
We will also avoid replacing the file all the time as that causes lots of unnecessary disk I/O. I am not saying it makes a significant difference, but flash has a limited write count that we are still wasting.
OK, but my changes were no worse than before, either doing a delete or rename whereas before it always did a rename. The "unnecessary I/O" was already there.
Please review and you if you are happy, please add the appropriate Git tag.
Sorry but I don't know what you mean by "adding the appropriate Git tag". I am only using Git locally and have no repo access.
How can I pull your changes? Otherwise I'll need to patch my set up manually.
I also have another problem. I have temporarily had to remove my IPF gateway for a business reason and don't know when I'll be able to add it back in. I'll have to see if I can insert it in my LAN somehow. At the same time it is also running Jon's mods which I'll have to undo.
With respect to Jon's mods, I've asked a couple of times on bugzilla, but is there any point in his routine maintaining /etc/unbound/dhcp-leases.conf. As far as I see the file is only read once when unbound starts, so it only has to be valid when unbound starts and not at any other point in time. To this effect he can remove the file maintenance and I attached a modified copy unbound-dhcp-leases-bridge to his repo which generates a new dhcp-leases.conf and exits, removing all the daemonising bit and the inode watcher. This file can be run as part of the unbound init routine just before the line that actually starts or reloads unbound. It also has the advantage of cutting down a load of disk I/O, which is important as you've just pointed out.
Also on his patches, I had a significantly faster way of determining the domain name (in bash) and he and I were looking for your feedback on if we'd be allowed to use it.
Regards,
Nick
Best, -Michael
On 10/04/2024 12:04, Nick Howitt wrote:
Bump
On 28/03/2024 17:25, Nick Howitt wrote:
Next version.
On 28/03/2024 10:41, Michael Tremer wrote:
Hello,
On 27 Mar 2024, at 15:28, Nick Howittnick@howitts.co.uk wrote:
On 26/03/2024 11:10, Michael Tremer wrote: > >> On 25 Mar 2024, at 13:11, Nick Howittnick@howitts.co.uk wrote: >> >> >> >> On 25/03/2024 11:31, Michael Tremer wrote: >>> Hello, >>> I don’t think that the risk is very high, but I would account for this problem. >>> With my proposed changes from my previous email this should not be a large problem at all. When opening /etc/unbound/dhcp-leases.conf with mode “r+” you can catch the Exception and skip the comparison and open the file again with “w”. >>>> On 25 Mar 2024, at 10:10, Nick Howittnick@howitts.co.uk wrote: >>>> >>>> If there is a risk that the /etc/unbound/dhcp-leases.conf does not exist, for example on a clean install (I don't know), the filecmp.cmp line needs to be wrapped in a try...except to stop it blowing up: >>>> >>>> try: >>>> RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) >>>> except: >>>> RequireUnboundReload = True >>> I generally don’t like blanco exception catches. You will never know about the daemons that you are hiding. >>>> This will force the file to be to be generated, even if it is only copying in a blank temp file, and unbound will restart. It will only ever happen once. >>>> >>>> >>>> On 24/03/2024 15:49, Nick Howitt wrote: >>>>> Hi all, >>>>> **** This is a repost of an email incorrectly submitted. **** >>>>> Please bear with me as I am new to IPFire and not at all used to this style of development, so any guidance would be appreciated. >>>>> There is a bug,https://bugzilla.ipfire.org/show_bug.cgi?id=13254, where unbound-dhcp-leases-bridge restarts unbound a lot when it is totally unnecessary. It appears to be because when a lease is renewed, the script gets triggered, created a new /etc/unbound/dhcp-leases.conf then restarts Unbound to read the file. Generally this seems to be unnecessary as, with a renewed lease, the old and new /etc/unbound/dhcp-leases.conf files are the same. With 5 leases in that file (one for a machine not active) I am getting 3-4 restarts an hour of Unbound when I have a min/max lease time of 60/120min. >>>>> Looking at the code, it is fairly easy to fix. The current code creates a temp file then copies it into place then restarts unbound. All I am doing is doing a file comparison before the copy and skipping the restart if the files are the same. >>>>> There were a couple of gotchas because setting the file attributes and copying the file were done inside the "with" block for generating the temporary file. This meant a file comparison always returned False as the temp file was still open and so never the same is the old file. >>>>> I moved those two statements outside the "with". This forced me to change the fchmod to chmod. >>>>> It could be argued that the file copy should not be done if the files are not different, but it would take an extra "if" and you'd have to remember to delete the temp file. If required, I can make that change. >>>>> Also, one small thing I noticed once is that the old and new dhcp-leases.conf files could occasionally contain the same leases but in a different order. I have been unable to reproduce, but to sidestep it, instead of stepping through the leases variable directly, I sort it and step through that. It should make the resulting file completely deterministic and make the file comparison more effective. >>>>> My patch is: >>>>> From 73873d4944944a2f02317a73074a6894726f36f7 Mon Sep 17 00:00:00 2001 >>>>> From: Nick Howittnick@howitts.co.uk >>>>> Date: Sun, 24 Mar 2024 15:17:19 +0000 >>>>> Subject: [PATCH] Stop unbound-dhcp-leases-bridge from restarting unbound every >>>>> time it write dhcp-leases.conf as it is very often unchanged and does not >>>>> require a restart. >>>>> --- >>>>> config/unbound/unbound-dhcp-leases-bridge | 28 +++++++++++++++-------- >>>>> 1 file changed, 19 insertions(+), 9 deletions(-) >>>>> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >>>>> index e9f022aff..d22772066 100644 >>>>> --- a/config/unbound/unbound-dhcp-leases-bridge >>>>> +++ b/config/unbound/unbound-dhcp-leases-bridge >>>>> @@ -22,6 +22,7 @@ >>>>> import argparse >>>>> import datetime >>>>> import daemon >>>>> +import filecmp >>>>> import functools >>>>> import ipaddress >>>>> import logging >>>>> @@ -516,26 +517,35 @@ class UnboundConfigWriter(object): >>>>> def update_dhcp_leases(self, leases): >>>>> # Write out all leases >>>>> - self.write_dhcp_leases(leases) >>>>> + if self.write_dhcp_leases(leases): >>>>> - log.debug("Reloading Unbound...") >>>>> + log.debug("Reloading Unbound...") >>>>> - # Reload the configuration without dropping the cache >>>>> - self._control("reload_keep_cache") >>>>> + # Reload the configuration without dropping the cache >>>>> + self._control("reload_keep_cache") >>>>> + >>>>> + else: >>>>> + >>>>> + log.debug("Not reloading Unbound. Leases not changed.") >>>>> def write_dhcp_leases(self, leases): >>>>> log.debug("Writing DHCP leases...") >>>>> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >>>>> - for l in leases: >>>>> + for l in sorted(leases): >>>>> for rr in l.rrset: >>>>> f.write("local-data: "%s"\n" % " ".join(rr)) >>>>> - # Make file readable for everyone >>>>> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>> + filecmp.clear_cache() >>>>> + RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) >>>>> + >>>>> + # Make file readable for everyone >>>>> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>> + >>>>> + # Move the file to its destination >>>>> + os.rename(f.name, self.path) >>>>> - # Move the file to its destination >>>>> - os.rename(f.name, self.path) >>>>> + return RequireUnboundReload >>>>> def _control(self, *args): >>>>> command = ["unbound-control"] >> Resubmitting the patch. >> >> Following my previous reply, I would like to stick with filecmp as I am using it with "shallow=False" to avoid doing the comparison just with stat() calls. Note that I also flush the filecmp cache before using the command so it is running with the latest file data. Talking about file descriptors, atomic file replacement etc., scares me. > File descriptors are great. They avoid race-conditions and all sorts of things. But I have never used them so I don't intend to go that way. Too much for me to get my brain round and a limited ability to test.
Well, I pretty much gave you the entire solution… There was nothing left to do apart from copying and pasting the snippets.
Ok I've tried to use your version but struggled to get it going. If I used it as was and called compare(file1_name, file2_name) it failed as "compare" was not defined. I could call it using self.compare but then it failed with "TypeError: UnboundConfigWriter.compare() takes 2 positional arguments but 3 were given". I changed the function to compare(self, f1, f2). Then it failed with "AttributeError: 'str' object has no attribute 'seek'", so I changed it again to accept file names then open the files before doing the seeks. I really hope it is OK as I am out of my depth when defining functions with "self" as an argument. Please can you check it? If it is not OK, can I revert to filecmp?
I have made one futher change, the sorted(leases) line was not working and not sorting the object ever, and I saw lines changing places in the resulting leases file. I have changed it to "sorted(leases, key=lambda x: x.ipaddr)". It definitely sorts this time.
> I don’t think that module is the best choice really if you have to hack around and flush the cache. Why would it even cache things in the first place? I don’t see where the cache helps. From the filecmp.clear_cache documentation:
Clear the filecmp cache. This may be useful if a file is compared so quickly after it is modified that it is within the mtime resolution of the underlying filesystem.Now this makes me believe that the clear_cache is only relevant when using shallow=True, which I'm not and I don't know how to check the underlying code. It is used to make sure the stat() info is updated, but we are not using the stat() info. I guess we could as easily add a one second sleep instead as that is way longer than the mtime resolution.
The function is here and it is rather simple:https://github.com/python/cpython/blob/3.12/Lib/filecmp.py#L30
It will always stat() both files and compares three items in case shallow is False: The type of the file, the size and mtime. If the file is not a regular file, the function will exist rather early. So in our case, that is all work the function is doing for thing. It boils down to something very similar to what I suggested to use to compare the files.
On top of that, there is a cache which will always be used but will never return a match for us because the temporary file at least will always have another mtime.
You could clear the cache after every call, but I would rather not have our code messy and care about implementation details of some Python module.
The module is chosen because all the references I read about comparing files preferred this one-liner for comparing files, although other methods were given and there were lots of discussions about large files (GB+) because if you chomp them it may be much quicker to say they are not the same when you find the first block that does not match. Having said this, I don't know how filecmp works, but we only have a small file. At the same time, I am not a regular programmer so simple one-liners are good for me.
You would always want to read both files block for block and compare the blocks. There is no way to implement this significantly faster.
>> I have modified the earlier try/except I proposed to catch only FileNotFoundError, and I have changed CamelCase to snake_case. >> >> --- >> config/unbound/unbound-dhcp-leases-bridge | 31 ++++++++++++++++------- >> 1 file changed, 22 insertions(+), 9 deletions(-) >> >> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >> index e9f022aff..a9ec8e1e2 100644 >> --- a/config/unbound/unbound-dhcp-leases-bridge >> +++ b/config/unbound/unbound-dhcp-leases-bridge >> @@ -22,6 +22,7 @@ >> import argparse >> import datetime >> import daemon >> +import filecmp >> import functools >> import ipaddress >> import logging >> @@ -516,26 +517,38 @@ class UnboundConfigWriter(object): >> >> def update_dhcp_leases(self, leases): >> # Write out all leases >> - self.write_dhcp_leases(leases) >> + if self.write_dhcp_leases(leases): >> >> - log.debug("Reloading Unbound...") >> + log.debug("Reloading Unbound...") >> >> - # Reload the configuration without dropping the cache >> - self._control("reload_keep_cache") >> + # Reload the configuration without dropping the cache >> + self._control("reload_keep_cache") >> + >> + else: >> + >> + log.debug("Not reloading Unbound. Leases not changed.") >> >> def write_dhcp_leases(self, leases): >> log.debug("Writing DHCP leases...") >> >> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >> - for l in leases: >> + for l in sorted(leases): >> for rr in l.rrset: >> f.write("local-data: "%s"\n" % " ".join(rr)) >> >> - # Make file readable for everyone >> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >> + filecmp.clear_cache() >> + try: >> + require_unbound_reload = not filecmp.cmp(f.name, self.path, shallow=False) >> + except FileNotFoundError: >> + require_unbound_reload = True > If you want to go this route, you will have to flush the temporary file first. Otherwise you might have some data left in the buffer that has not been written (because the file is still open). You will then compare incomplete output. This would not become a problem if you used the file descriptors. The temp file is closed at the point of doing the filecmp, and the filecmp cache has been flushed. How do I flush the temporary file further. Reading up, is it just having the last statement in the with block as f.flush() with the same indent as "for l in sorted(leases):"
If you have left the with block you will already have flushed the buffers and closed the file. It looks like I misread the indentation in the patch then.
> Catching FileNotFound would work for me. Good >> + >> + # Make file readable for everyone >> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >> + >> + # Move the file to its destination >> + os.rename(f.name, self.path) >> >> - # Move the file to its destination >> - os.rename(f.name, self.path) >> + return require_unbound_reload >> >> def _control(self, *args): >> command = ["unbound-control"] >> -- >> 2.43.0 >> >> >> Nick
config/unbound/unbound-dhcp-leases-bridge | 50 +++++++++++++++++++---- 1 file changed, 41 insertions(+), 9 deletions(-)
diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge index e9f022aff..80d9523ea 100644 --- a/config/unbound/unbound-dhcp-leases-bridge +++ b/config/unbound/unbound-dhcp-leases-bridge @@ -516,26 +516,37 @@ class UnboundConfigWriter(object):
def update_dhcp_leases(self, leases): # Write out all leases
self.write_dhcp_leases(leases)
if self.write_dhcp_leases(leases):
log.debug("Reloading Unbound...")
log.debug("Reloading Unbound...")
# Reload the configuration without dropping the cacheself._control("reload_keep_cache")
# Reload the configuration without dropping the cacheself._control("reload_keep_cache")else:log.debug("Not reloading Unbound. Leases not changed.") def write_dhcp_leases(self, leases): log.debug("Writing DHCP leases...") with tempfile.NamedTemporaryFile(mode="w", delete=False) as f:
for l in leases:
for l in sorted(leases, key=lambda x: x.ipaddr): for rr in l.rrset: f.write("local-data: \"%s\"\n" % " ".join(rr))
# Make file readable for everyoneos.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)
try:require_unbound_reload = not self.compare(f.name, self.path)except FileNotFoundError:require_unbound_reload = True# Make file readable for everyoneos.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)# Move the file to its destinationos.rename(f.name, self.path)
# Move the file to its destinationos.rename(f.name, self.path)
return require_unbound_reload def _control(self, *args): command = ["unbound-control"]@@ -551,6 +562,27 @@ class UnboundConfigWriter(object):
raise e
def compare(self, if1, if2):
buffersize = 16 * 1024f1 = open(if1, "r")f2 = open(if2, "r")# Reset file handlesf1.seek(0)f2.seek(0)while True:b1 = f1.read(buffersize)b2 = f2.read(buffersize)if not b1 == b2:return Falseelif not b1 or not b2:breakreturn Trueif __name__ == "__main__": parser = argparse.ArgumentParser(description="Bridge for DHCP Leases and Unbound DNS")
Hello Nick,
On 26 Apr 2024, at 17:48, Nick Howitt nick@howitts.co.uk wrote:
On 26/04/2024 16:12, Michael Tremer wrote:
Hello Nick,
On 25 Apr 2024, at 13:43, Nick Howitt nick@howitts.co.uk wrote:
Please can I bump this again to request a review of my patch submitted on 28th March? Thanks, Nick.
It is a little bit difficult to test your proposed changes if testers have to copy and paste them out of an email.
Confused. I was told I had to submit the diff in an email with [PATCH] in the title and not to attach a file. Please can you tell me how I should be operating?
No worries, we are going to sort this out.
So the first email in the thread was a correct patch. It was just the patch in the format that Git uses and it was properly parsed by Patchwork. Later in the same thread, the patch was just copied and pasted which was impossible for Patchwork to pick up. There is nothing wrong with using bits of code in the conversation, but it would have been nice to have the final version submitted once again.
I have just submitted a patch set to this list that breaks down all the changes that you proposed and I cleaned up the code so that we can use the filecmp function without any extra steps. We can simply jump out of the write function and return False to prevent Unbound being reloaded.
I thought you didn't want me to use filecmp and pushed me to use your file compare routine.
Yeah, I would have preferred that too, but I thought using the module might be quicker now.
We will also avoid replacing the file all the time as that causes lots of unnecessary disk I/O. I am not saying it makes a significant difference, but flash has a limited write count that we are still wasting.
OK, but my changes were no worse than before, either doing a delete or rename whereas before it always did a rename. The "unnecessary I/O" was already there.
Please review and you if you are happy, please add the appropriate Git tag.
Sorry but I don't know what you mean by "adding the appropriate Git tag". I am only using Git locally and have no repo access.
https://www.ipfire.org/docs/devel/git/tags
How can I pull your changes? Otherwise I'll need to patch my set up manually.
You can apply the patch to your local repository using “git am”:
https://git-scm.com/docs/git-am
I also have another problem. I have temporarily had to remove my IPF gateway for a business reason and don't know when I'll be able to add it back in. I'll have to see if I can insert it in my LAN somehow. At the same time it is also running Jon's mods which I'll have to undo.
With respect to Jon's mods, I've asked a couple of times on bugzilla, but is there any point in his routine maintaining /etc/unbound/dhcp-leases.conf. As far as I see the file is only read once when unbound starts, so it only has to be valid when unbound starts and not at any other point in time. To this effect he can remove the file maintenance and I attached a modified copy unbound-dhcp-leases-bridge to his repo which generates a new dhcp-leases.conf and exits, removing all the daemonising bit and the inode watcher. This file can be run as part of the unbound init routine just before the line that actually starts or reloads unbound. It also has the advantage of cutting down a load of disk I/O, which is important as you've just pointed out.
I don’t think that we can fully remove the daemon part, because I don’t think that there is a way to get a script (shell or otherwise) fast enough to perform the job. I also think that we benefit a lot from keeping some state in memory, whereas the script will have to parse the old file first, generate a new file and then compare the result. That will take a long time.
Do we needed the leases file generated at startup? Yes, I think so, too. DNS is quite essential for the network and for local services to be able to start. So it has to come up very early. If it comes up with proper configuration we avoid any other problems where hosts might resolve names too early and get a different result compared to a couple of seconds later when the zone is fully initialised.
So all in all, I think the route I would most likely look at is changing the daemon to open a socket. We write minimal scripts which are called by dhcpd that simply pipe their data into the socket which the daemon will then parse. With all available leases in memory already, we can check if a lease has changed or if it has been renewed without any further changes. If nothing has changed, we don’t need to do anything. In case of a change, we will have to write the file again and have Unbound reload, or we even call unbound-control and make the one change without a reload. Doing that in the daemon should give us more flexibility and we avoid any performance penalties as the daemon could run multiple commands concurrently which dhcpd can’t do.
Also on his patches, I had a significantly faster way of determining the domain name (in bash) and he and I were looking for your feedback on if we'd be allowed to use it.
Anything is allowed. It just has to be fast. And for that, I believe, calling any other binary will be a problem.
-Michael
Regards,
Nick
Best, -Michael
On 10/04/2024 12:04, Nick Howitt wrote:
Bump
On 28/03/2024 17:25, Nick Howitt wrote:
Next version.
On 28/03/2024 10:41, Michael Tremer wrote:
Hello,
> On 27 Mar 2024, at 15:28, Nick Howitt nick@howitts.co.uk wrote: > > > > On 26/03/2024 11:10, Michael Tremer wrote: > >> >> >>> On 25 Mar 2024, at 13:11, Nick Howitt nick@howitts.co.uk wrote: >>> >>> >>> >>> On 25/03/2024 11:31, Michael Tremer wrote: >>> >>>> Hello, >>>> I don’t think that the risk is very high, but I would account for this problem. >>>> With my proposed changes from my previous email this should not be a large problem at all. When opening /etc/unbound/dhcp-leases.conf with mode “r+” you can catch the Exception and skip the comparison and open the file again with “w”. >>>> >>>>> On 25 Mar 2024, at 10:10, Nick Howitt nick@howitts.co.uk wrote: >>>>> >>>>> If there is a risk that the /etc/unbound/dhcp-leases.conf does not exist, for example on a clean install (I don't know), the filecmp.cmp line needs to be wrapped in a try...except to stop it blowing up: >>>>> >>>>> try: >>>>> RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) >>>>> except: >>>>> RequireUnboundReload = True >>>>> >>>> I generally don’t like blanco exception catches. You will never know about the daemons that you are hiding. >>>> >>>>> This will force the file to be to be generated, even if it is only copying in a blank temp file, and unbound will restart. It will only ever happen once. >>>>> >>>>> >>>>> On 24/03/2024 15:49, Nick Howitt wrote: >>>>> >>>>>> Hi all, >>>>>> **** This is a repost of an email incorrectly submitted. **** >>>>>> Please bear with me as I am new to IPFire and not at all used to this style of development, so any guidance would be appreciated. >>>>>> There is a bug, https://bugzilla.ipfire.org/show_bug.cgi?id=13254, where unbound-dhcp-leases-bridge restarts unbound a lot when it is totally unnecessary. It appears to be because when a lease is renewed, the script gets triggered, created a new /etc/unbound/dhcp-leases.conf then restarts Unbound to read the file. Generally this seems to be unnecessary as, with a renewed lease, the old and new /etc/unbound/dhcp-leases.conf files are the same. With 5 leases in that file (one for a machine not active) I am getting 3-4 restarts an hour of Unbound when I have a min/max lease time of 60/120min. >>>>>> Looking at the code, it is fairly easy to fix. The current code creates a temp file then copies it into place then restarts unbound. All I am doing is doing a file comparison before the copy and skipping the restart if the files are the same. >>>>>> There were a couple of gotchas because setting the file attributes and copying the file were done inside the "with" block for generating the temporary file. This meant a file comparison always returned False as the temp file was still open and so never the same is the old file. >>>>>> I moved those two statements outside the "with". This forced me to change the fchmod to chmod. >>>>>> It could be argued that the file copy should not be done if the files are not different, but it would take an extra "if" and you'd have to remember to delete the temp file. If required, I can make that change. >>>>>> Also, one small thing I noticed once is that the old and new dhcp-leases.conf files could occasionally contain the same leases but in a different order. I have been unable to reproduce, but to sidestep it, instead of stepping through the leases variable directly, I sort it and step through that. It should make the resulting file completely deterministic and make the file comparison more effective. >>>>>> My patch is: >>>>>> From 73873d4944944a2f02317a73074a6894726f36f7 Mon Sep 17 00:00:00 2001 >>>>>> From: Nick Howitt nick@howitts.co.uk >>>>>> Date: Sun, 24 Mar 2024 15:17:19 +0000 >>>>>> Subject: [PATCH] Stop unbound-dhcp-leases-bridge from restarting unbound every >>>>>> time it write dhcp-leases.conf as it is very often unchanged and does not >>>>>> require a restart. >>>>>> --- >>>>>> config/unbound/unbound-dhcp-leases-bridge | 28 +++++++++++++++-------- >>>>>> 1 file changed, 19 insertions(+), 9 deletions(-) >>>>>> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >>>>>> index e9f022aff..d22772066 100644 >>>>>> --- a/config/unbound/unbound-dhcp-leases-bridge >>>>>> +++ b/config/unbound/unbound-dhcp-leases-bridge >>>>>> @@ -22,6 +22,7 @@ >>>>>> import argparse >>>>>> import datetime >>>>>> import daemon >>>>>> +import filecmp >>>>>> import functools >>>>>> import ipaddress >>>>>> import logging >>>>>> @@ -516,26 +517,35 @@ class UnboundConfigWriter(object): >>>>>> def update_dhcp_leases(self, leases): >>>>>> # Write out all leases >>>>>> - self.write_dhcp_leases(leases) >>>>>> + if self.write_dhcp_leases(leases): >>>>>> - log.debug("Reloading Unbound...") >>>>>> + log.debug("Reloading Unbound...") >>>>>> - # Reload the configuration without dropping the cache >>>>>> - self._control("reload_keep_cache") >>>>>> + # Reload the configuration without dropping the cache >>>>>> + self._control("reload_keep_cache") >>>>>> + >>>>>> + else: >>>>>> + >>>>>> + log.debug("Not reloading Unbound. Leases not changed.") >>>>>> def write_dhcp_leases(self, leases): >>>>>> log.debug("Writing DHCP leases...") >>>>>> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >>>>>> - for l in leases: >>>>>> + for l in sorted(leases): >>>>>> for rr in l.rrset: >>>>>> f.write("local-data: "%s"\n" % " ".join(rr)) >>>>>> - # Make file readable for everyone >>>>>> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>> + filecmp.clear_cache() >>>>>> + RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) >>>>>> + >>>>>> + # Make file readable for everyone >>>>>> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>> + >>>>>> + # Move the file to its destination >>>>>> + os.rename(f.name, self.path) >>>>>> - # Move the file to its destination >>>>>> - os.rename(f.name, self.path) >>>>>> + return RequireUnboundReload >>>>>> def _control(self, *args): >>>>>> command = ["unbound-control"] >>>>>> >>> Resubmitting the patch. >>> >>> Following my previous reply, I would like to stick with filecmp as I am using it with "shallow=False" to avoid doing the comparison just with stat() calls. Note that I also flush the filecmp cache before using the command so it is running with the latest file data. Talking about file descriptors, atomic file replacement etc., scares me. >>> >> File descriptors are great. They avoid race-conditions and all sorts of things. >> > But I have never used them so I don't intend to go that way. Too much for me to get my brain round and a limited ability to test. > Well, I pretty much gave you the entire solution… There was nothing left to do apart from copying and pasting the snippets.
Ok I've tried to use your version but struggled to get it going. If I used it as was and called compare(file1_name, file2_name) it failed as "compare" was not defined. I could call it using self.compare but then it failed with "TypeError: UnboundConfigWriter.compare() takes 2 positional arguments but 3 were given". I changed the function to compare(self, f1, f2). Then it failed with "AttributeError: 'str' object has no attribute 'seek'", so I changed it again to accept file names then open the files before doing the seeks. I really hope it is OK as I am out of my depth when defining functions with "self" as an argument. Please can you check it? If it is not OK, can I revert to filecmp?
I have made one futher change, the sorted(leases) line was not working and not sorting the object ever, and I saw lines changing places in the resulting leases file. I have changed it to "sorted(leases, key=lambda x: x.ipaddr)". It definitely sorts this time.
>> I don’t think that module is the best choice really if you have to hack around and flush the cache. Why would it even cache things in the first place? I don’t see where the cache helps. >> > From the filecmp.clear_cache documentation: > > Clear the filecmp cache. This may be useful if a file is compared so > quickly after it is modified that it is within the mtime resolution > of the underlying filesystem. > > Now this makes me believe that the clear_cache is only relevant when using shallow=True, which I'm not and I don't know how to check the underlying code. It is used to make sure the stat() info is updated, but we are not using the stat() info. I guess we could as easily add a one second sleep instead as that is way longer than the mtime resolution. > The function is here and it is rather simple: https://github.com/python/cpython/blob/3.12/Lib/filecmp.py#L30
It will always stat() both files and compares three items in case shallow is False: The type of the file, the size and mtime. If the file is not a regular file, the function will exist rather early. So in our case, that is all work the function is doing for thing. It boils down to something very similar to what I suggested to use to compare the files.
On top of that, there is a cache which will always be used but will never return a match for us because the temporary file at least will always have another mtime.
You could clear the cache after every call, but I would rather not have our code messy and care about implementation details of some Python module.
> The module is chosen because all the references I read about comparing files preferred this one-liner for comparing files, although other methods were given and there were lots of discussions about large files (GB+) because if you chomp them it may be much quicker to say they are not the same when you find the first block that does not match. Having said this, I don't know how filecmp works, but we only have a small file. At the same time, I am not a regular programmer so simple one-liners are good for me. > You would always want to read both files block for block and compare the blocks. There is no way to implement this significantly faster.
>>> I have modified the earlier try/except I proposed to catch only FileNotFoundError, and I have changed CamelCase to snake_case. >>> >>> --- >>> config/unbound/unbound-dhcp-leases-bridge | 31 ++++++++++++++++------- >>> 1 file changed, 22 insertions(+), 9 deletions(-) >>> >>> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >>> index e9f022aff..a9ec8e1e2 100644 >>> --- a/config/unbound/unbound-dhcp-leases-bridge >>> +++ b/config/unbound/unbound-dhcp-leases-bridge >>> @@ -22,6 +22,7 @@ >>> import argparse >>> import datetime >>> import daemon >>> +import filecmp >>> import functools >>> import ipaddress >>> import logging >>> @@ -516,26 +517,38 @@ class UnboundConfigWriter(object): >>> >>> def update_dhcp_leases(self, leases): >>> # Write out all leases >>> - self.write_dhcp_leases(leases) >>> + if self.write_dhcp_leases(leases): >>> >>> - log.debug("Reloading Unbound...") >>> + log.debug("Reloading Unbound...") >>> >>> - # Reload the configuration without dropping the cache >>> - self._control("reload_keep_cache") >>> + # Reload the configuration without dropping the cache >>> + self._control("reload_keep_cache") >>> + >>> + else: >>> + >>> + log.debug("Not reloading Unbound. Leases not changed.") >>> >>> def write_dhcp_leases(self, leases): >>> log.debug("Writing DHCP leases...") >>> >>> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >>> - for l in leases: >>> + for l in sorted(leases): >>> for rr in l.rrset: >>> f.write("local-data: "%s"\n" % " ".join(rr)) >>> >>> - # Make file readable for everyone >>> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>> + filecmp.clear_cache() >>> + try: >>> + require_unbound_reload = not filecmp.cmp(f.name, self.path, shallow=False) >>> + except FileNotFoundError: >>> + require_unbound_reload = True >>> >> If you want to go this route, you will have to flush the temporary file first. Otherwise you might have some data left in the buffer that has not been written (because the file is still open). You will then compare incomplete output. This would not become a problem if you used the file descriptors. >> > The temp file is closed at the point of doing the filecmp, and the filecmp cache has been flushed. How do I flush the temporary file further. Reading up, is it just having the last statement in the with block as f.flush() with the same indent as "for l in sorted(leases):" > If you have left the with block you will already have flushed the buffers and closed the file. It looks like I misread the indentation in the patch then.
>> Catching FileNotFound would work for me. >> > Good > >> >>> + >>> + # Make file readable for everyone >>> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>> + >>> + # Move the file to its destination >>> + os.rename(f.name, self.path) >>> >>> - # Move the file to its destination >>> - os.rename(f.name, self.path) >>> + return require_unbound_reload >>> >>> def _control(self, *args): >>> command = ["unbound-control"] >>> -- >>> 2.43.0 >>> >>> >>> Nick >>>
config/unbound/unbound-dhcp-leases-bridge | 50 +++++++++++++++++++---- 1 file changed, 41 insertions(+), 9 deletions(-)
diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge index e9f022aff..80d9523ea 100644 --- a/config/unbound/unbound-dhcp-leases-bridge +++ b/config/unbound/unbound-dhcp-leases-bridge @@ -516,26 +516,37 @@ class UnboundConfigWriter(object):
def update_dhcp_leases(self, leases): # Write out all leases
- self.write_dhcp_leases(leases)
- if self.write_dhcp_leases(leases):
- log.debug("Reloading Unbound...")
- log.debug("Reloading Unbound...")
- # Reload the configuration without dropping the cache
- self._control("reload_keep_cache")
- # Reload the configuration without dropping the cache
- self._control("reload_keep_cache")
- else:
- log.debug("Not reloading Unbound. Leases not changed.")
def write_dhcp_leases(self, leases): log.debug("Writing DHCP leases...")
with tempfile.NamedTemporaryFile(mode="w", delete=False) as f:
- for l in leases:
- for l in sorted(leases, key=lambda x: x.ipaddr):
for rr in l.rrset: f.write("local-data: "%s"\n" % " ".join(rr))
- # Make file readable for everyone
- os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)
- try:
- require_unbound_reload = not self.compare(f.name, self.path)
- except FileNotFoundError:
- require_unbound_reload = True
- # Make file readable for everyone
- os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)
- # Move the file to its destination
- os.rename(f.name, self.path)
- # Move the file to its destination
- os.rename(f.name, self.path)
- return require_unbound_reload
def _control(self, *args): command = ["unbound-control"] @@ -551,6 +562,27 @@ class UnboundConfigWriter(object):
raise e
- def compare(self, if1, if2):
- buffersize = 16 * 1024
- f1 = open(if1, "r")
- f2 = open(if2, "r")
- # Reset file handles
- f1.seek(0)
- f2.seek(0)
- while True:
- b1 = f1.read(buffersize)
- b2 = f2.read(buffersize)
- if not b1 == b2:
- return False
- elif not b1 or not b2:
- break
- return True
if __name__ == "__main__": parser = argparse.ArgumentParser(description="Bridge for DHCP Leases and Unbound DNS")
On 27/04/2024 09:35, Michael Tremer wrote:
Hello Nick,
On 26 Apr 2024, at 17:48, Nick Howittnick@howitts.co.uk wrote:
On 26/04/2024 16:12, Michael Tremer wrote:
Hello Nick,
On 25 Apr 2024, at 13:43, Nick Howittnick@howitts.co.uk wrote:
Please can I bump this again to request a review of my patch submitted on 28th March? Thanks, Nick.
It is a little bit difficult to test your proposed changes if testers have to copy and paste them out of an email.
Confused. I was told I had to submit the diff in an email with [PATCH] in the title and not to attach a file. Please can you tell me how I should be operating?
No worries, we are going to sort this out.
So the first email in the thread was a correct patch. It was just the patch in the format that Git uses and it was properly parsed by Patchwork. Later in the same thread, the patch was just copied and pasted which was impossible for Patchwork to pick up. There is nothing wrong with using bits of code in the conversation, but it would have been nice to have the final version submitted once again.
I have just submitted a patch set to this list that breaks down all the changes that you proposed and I cleaned up the code so that we can use the filecmp function without any extra steps. We can simply jump out of the write function and return False to prevent Unbound being reloaded.
I thought you didn't want me to use filecmp and pushed me to use your file compare routine.
Yeah, I would have preferred that too, but I thought using the module might be quicker now.
We will also avoid replacing the file all the time as that causes lots of unnecessary disk I/O. I am not saying it makes a significant difference, but flash has a limited write count that we are still wasting.
OK, but my changes were no worse than before, either doing a delete or rename whereas before it always did a rename. The "unnecessary I/O" was already there.
Please review and you if you are happy, please add the appropriate Git tag.
Sorry but I don't know what you mean by "adding the appropriate Git tag". I am only using Git locally and have no repo access.
Thanks.
How can I pull your changes? Otherwise I'll need to patch my set up manually.
You can apply the patch to your local repository using “git am”:
Thanks again.
I also have another problem. I have temporarily had to remove my IPF gateway for a business reason and don't know when I'll be able to add it back in. I'll have to see if I can insert it in my LAN somehow. At the same time it is also running Jon's mods which I'll have to undo.
With respect to Jon's mods, I've asked a couple of times on bugzilla, but is there any point in his routine maintaining /etc/unbound/dhcp-leases.conf. As far as I see the file is only read once when unbound starts, so it only has to be valid when unbound starts and not at any other point in time. To this effect he can remove the file maintenance and I attached a modified copy unbound-dhcp-leases-bridge to his repo which generates a new dhcp-leases.conf and exits, removing all the daemonising bit and the inode watcher. This file can be run as part of the unbound init routine just before the line that actually starts or reloads unbound. It also has the advantage of cutting down a load of disk I/O, which is important as you've just pointed out.
I don’t think that we can fully remove the daemon part, because I don’t think that there is a way to get a script (shell or otherwise) fast enough to perform the job. I also think that we benefit a lot from keeping some state in memory, whereas the script will have to parse the old file first, generate a new file and then compare the result. That will take a long time.
I had Jon's script down to 70ms after I removed the dhcp-leases file updates, used my fast method for deriving the domain name and removed Jon's initial setting up bit which is not needed once the changes are accepted. This is on a 2c VM running at 2GHz. I thought the goal was < 100ms which has been done comfortably.
FWIW, if you no longer maintain dhcp-leases in the Jon's script you never read the old file or do a compare. All you do is read the hosts and static-leases files and nothing is written to file.
Do we needed the leases file generated at startup? Yes, I think so, too. DNS is quite essential for the network and for local services to be able to start. So it has to come up very early. If it comes up with proper configuration we avoid any other problems where hosts might resolve names too early and get a different result compared to a couple of seconds later when the zone is fully initialised.
My cut-down version of unbound-dhcp-leases-bridge is at https://bugzilla.ipfire.org/attachment.cgi?id=1516 on https://bugzilla.ipfire.org/show_bug.cgi?id=13254. On my test system it takes 219ms to run so delays unbound starting up minimally.
At the same time there is a gain in the dhcp start up as the unbound-dhcp-leases-bridge is no longer needed and it therefore avoids triggering an unbound restart which happens every time unbound-dhcp-leases-bridge is run.
If we are generating dhcp-leases.conf at start up, then there is no point in maintaining it while unbound is running as the maintained file will never be read.
So all in all, I think the route I would most likely look at is changing the daemon to open a socket. We write minimal scripts which are called by dhcpd that simply pipe their data into the socket which the daemon will then parse. With all available leases in memory already, we can check if a lease has changed or if it has been renewed without any further changes. If nothing has changed, we don’t need to do anything. In case of a change, we will have to write the file again and have Unbound reload, or we even call unbound-control and make the one change without a reload. Doing that in the daemon should give us more flexibility and we avoid any performance penalties as the daemon could run multiple commands concurrently which dhcpd can’t do.
Sockets are beyond me, I am afraid.
Part of the goal of all this is to avoid reloading unbound and part of my goal is to avoid writing dhcp-leases.conf unless unbound is restarted. I am not sure why you are still mentioning writing a file and restarting/reloading unbound.
Also on his patches, I had a significantly faster way of determining the domain name (in bash) and he and I were looking for your feedback on if we'd be allowed to use it.
Anything is allowed. It just has to be fast. And for that, I believe, calling any other binary will be a problem.
OK. I was proposing to read the DOMAINNAME with:
DOMAINNAME=${HOSTNAME#*.*}
which is many times faster than the conventional route:
eval $(/usr/local/bin/readhash /var/ipfire/main/settings)
If there is any concern with bash not having the full environment when the program is called, the shebang could be changed to:
#!/bin/bash -l
Nick
-Michael
Regards,
Nick
Best, -Michael
On 10/04/2024 12:04, Nick Howitt wrote:
Bump
On 28/03/2024 17:25, Nick Howitt wrote:
Next version.
On 28/03/2024 10:41, Michael Tremer wrote:
> Hello, > > >> On 27 Mar 2024, at 15:28, Nick Howittnick@howitts.co.uk wrote: >> >> >> >> On 26/03/2024 11:10, Michael Tremer wrote: >> >>> >>>> On 25 Mar 2024, at 13:11, Nick Howittnick@howitts.co.uk wrote: >>>> >>>> >>>> >>>> On 25/03/2024 11:31, Michael Tremer wrote: >>>> >>>>> Hello, >>>>> I don’t think that the risk is very high, but I would account for this problem. >>>>> With my proposed changes from my previous email this should not be a large problem at all. When opening /etc/unbound/dhcp-leases.conf with mode “r+” you can catch the Exception and skip the comparison and open the file again with “w”. >>>>> >>>>>> On 25 Mar 2024, at 10:10, Nick Howittnick@howitts.co.uk wrote: >>>>>> >>>>>> If there is a risk that the /etc/unbound/dhcp-leases.conf does not exist, for example on a clean install (I don't know), the filecmp.cmp line needs to be wrapped in a try...except to stop it blowing up: >>>>>> >>>>>> try: >>>>>> RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) >>>>>> except: >>>>>> RequireUnboundReload = True >>>>>> >>>>> I generally don’t like blanco exception catches. You will never know about the daemons that you are hiding. >>>>> >>>>>> This will force the file to be to be generated, even if it is only copying in a blank temp file, and unbound will restart. It will only ever happen once. >>>>>> >>>>>> >>>>>> On 24/03/2024 15:49, Nick Howitt wrote: >>>>>> >>>>>>> Hi all, >>>>>>> **** This is a repost of an email incorrectly submitted. **** >>>>>>> Please bear with me as I am new to IPFire and not at all used to this style of development, so any guidance would be appreciated. >>>>>>> There is a bug,https://bugzilla.ipfire.org/show_bug.cgi?id=13254, where unbound-dhcp-leases-bridge restarts unbound a lot when it is totally unnecessary. It appears to be because when a lease is renewed, the script gets triggered, created a new /etc/unbound/dhcp-leases.conf then restarts Unbound to read the file. Generally this seems to be unnecessary as, with a renewed lease, the old and new /etc/unbound/dhcp-leases.conf files are the same. With 5 leases in that file (one for a machine not active) I am getting 3-4 restarts an hour of Unbound when I have a min/max lease time of 60/120min. >>>>>>> Looking at the code, it is fairly easy to fix. The current code creates a temp file then copies it into place then restarts unbound. All I am doing is doing a file comparison before the copy and skipping the restart if the files are the same. >>>>>>> There were a couple of gotchas because setting the file attributes and copying the file were done inside the "with" block for generating the temporary file. This meant a file comparison always returned False as the temp file was still open and so never the same is the old file. >>>>>>> I moved those two statements outside the "with". This forced me to change the fchmod to chmod. >>>>>>> It could be argued that the file copy should not be done if the files are not different, but it would take an extra "if" and you'd have to remember to delete the temp file. If required, I can make that change. >>>>>>> Also, one small thing I noticed once is that the old and new dhcp-leases.conf files could occasionally contain the same leases but in a different order. I have been unable to reproduce, but to sidestep it, instead of stepping through the leases variable directly, I sort it and step through that. It should make the resulting file completely deterministic and make the file comparison more effective. >>>>>>> My patch is: >>>>>>> From 73873d4944944a2f02317a73074a6894726f36f7 Mon Sep 17 00:00:00 2001 >>>>>>> From: Nick Howittnick@howitts.co.uk >>>>>>> Date: Sun, 24 Mar 2024 15:17:19 +0000 >>>>>>> Subject: [PATCH] Stop unbound-dhcp-leases-bridge from restarting unbound every >>>>>>> time it write dhcp-leases.conf as it is very often unchanged and does not >>>>>>> require a restart. >>>>>>> --- >>>>>>> config/unbound/unbound-dhcp-leases-bridge | 28 +++++++++++++++-------- >>>>>>> 1 file changed, 19 insertions(+), 9 deletions(-) >>>>>>> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >>>>>>> index e9f022aff..d22772066 100644 >>>>>>> --- a/config/unbound/unbound-dhcp-leases-bridge >>>>>>> +++ b/config/unbound/unbound-dhcp-leases-bridge >>>>>>> @@ -22,6 +22,7 @@ >>>>>>> import argparse >>>>>>> import datetime >>>>>>> import daemon >>>>>>> +import filecmp >>>>>>> import functools >>>>>>> import ipaddress >>>>>>> import logging >>>>>>> @@ -516,26 +517,35 @@ class UnboundConfigWriter(object): >>>>>>> def update_dhcp_leases(self, leases): >>>>>>> # Write out all leases >>>>>>> - self.write_dhcp_leases(leases) >>>>>>> + if self.write_dhcp_leases(leases): >>>>>>> - log.debug("Reloading Unbound...") >>>>>>> + log.debug("Reloading Unbound...") >>>>>>> - # Reload the configuration without dropping the cache >>>>>>> - self._control("reload_keep_cache") >>>>>>> + # Reload the configuration without dropping the cache >>>>>>> + self._control("reload_keep_cache") >>>>>>> + >>>>>>> + else: >>>>>>> + >>>>>>> + log.debug("Not reloading Unbound. Leases not changed.") >>>>>>> def write_dhcp_leases(self, leases): >>>>>>> log.debug("Writing DHCP leases...") >>>>>>> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >>>>>>> - for l in leases: >>>>>>> + for l in sorted(leases): >>>>>>> for rr in l.rrset: >>>>>>> f.write("local-data: "%s"\n" % " ".join(rr)) >>>>>>> - # Make file readable for everyone >>>>>>> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>> + filecmp.clear_cache() >>>>>>> + RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) >>>>>>> + >>>>>>> + # Make file readable for everyone >>>>>>> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>> + >>>>>>> + # Move the file to its destination >>>>>>> + os.rename(f.name, self.path) >>>>>>> - # Move the file to its destination >>>>>>> - os.rename(f.name, self.path) >>>>>>> + return RequireUnboundReload >>>>>>> def _control(self, *args): >>>>>>> command = ["unbound-control"] >>>>>>> >>>> Resubmitting the patch. >>>> >>>> Following my previous reply, I would like to stick with filecmp as I am using it with "shallow=False" to avoid doing the comparison just with stat() calls. Note that I also flush the filecmp cache before using the command so it is running with the latest file data. Talking about file descriptors, atomic file replacement etc., scares me. >>>> >>> File descriptors are great. They avoid race-conditions and all sorts of things. >>> >> But I have never used them so I don't intend to go that way. Too much for me to get my brain round and a limited ability to test. >> > Well, I pretty much gave you the entire solution… There was nothing left to do apart from copying and pasting the snippets. > > Ok I've tried to use your version but struggled to get it going. If I used it as was and called compare(file1_name, file2_name) it failed as "compare" was not defined. I could call it using self.compare but then it failed with "TypeError: UnboundConfigWriter.compare() takes 2 positional arguments but 3 were given". I changed the function to compare(self, f1, f2). Then it failed with "AttributeError: 'str' object has no attribute 'seek'", so I changed it again to accept file names then open the files before doing the seeks. I really hope it is OK as I am out of my depth when defining functions with "self" as an argument. Please can you check it? If it is not OK, can I revert to filecmp?
I have made one futher change, the sorted(leases) line was not working and not sorting the object ever, and I saw lines changing places in the resulting leases file. I have changed it to "sorted(leases, key=lambda x: x.ipaddr)". It definitely sorts this time.
>>> I don’t think that module is the best choice really if you have to hack around and flush the cache. Why would it even cache things in the first place? I don’t see where the cache helps. >>> >> From the filecmp.clear_cache documentation: >> >> Clear the filecmp cache. This may be useful if a file is compared so >> quickly after it is modified that it is within the mtime resolution >> of the underlying filesystem. >> >> Now this makes me believe that the clear_cache is only relevant when using shallow=True, which I'm not and I don't know how to check the underlying code. It is used to make sure the stat() info is updated, but we are not using the stat() info. I guess we could as easily add a one second sleep instead as that is way longer than the mtime resolution. >> > The function is here and it is rather simple:https://github.com/python/cpython/blob/3.12/Lib/filecmp.py#L30 > > It will always stat() both files and compares three items in case shallow is False: The type of the file, the size and mtime. If the file is not a regular file, the function will exist rather early. So in our case, that is all work the function is doing for thing. It boils down to something very similar to what I suggested to use to compare the files. > > On top of that, there is a cache which will always be used but will never return a match for us because the temporary file at least will always have another mtime. > > You could clear the cache after every call, but I would rather not have our code messy and care about implementation details of some Python module. > > >> The module is chosen because all the references I read about comparing files preferred this one-liner for comparing files, although other methods were given and there were lots of discussions about large files (GB+) because if you chomp them it may be much quicker to say they are not the same when you find the first block that does not match. Having said this, I don't know how filecmp works, but we only have a small file. At the same time, I am not a regular programmer so simple one-liners are good for me. >> > You would always want to read both files block for block and compare the blocks. There is no way to implement this significantly faster. > > >>>> I have modified the earlier try/except I proposed to catch only FileNotFoundError, and I have changed CamelCase to snake_case. >>>> >>>> --- >>>> config/unbound/unbound-dhcp-leases-bridge | 31 ++++++++++++++++------- >>>> 1 file changed, 22 insertions(+), 9 deletions(-) >>>> >>>> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >>>> index e9f022aff..a9ec8e1e2 100644 >>>> --- a/config/unbound/unbound-dhcp-leases-bridge >>>> +++ b/config/unbound/unbound-dhcp-leases-bridge >>>> @@ -22,6 +22,7 @@ >>>> import argparse >>>> import datetime >>>> import daemon >>>> +import filecmp >>>> import functools >>>> import ipaddress >>>> import logging >>>> @@ -516,26 +517,38 @@ class UnboundConfigWriter(object): >>>> >>>> def update_dhcp_leases(self, leases): >>>> # Write out all leases >>>> - self.write_dhcp_leases(leases) >>>> + if self.write_dhcp_leases(leases): >>>> >>>> - log.debug("Reloading Unbound...") >>>> + log.debug("Reloading Unbound...") >>>> >>>> - # Reload the configuration without dropping the cache >>>> - self._control("reload_keep_cache") >>>> + # Reload the configuration without dropping the cache >>>> + self._control("reload_keep_cache") >>>> + >>>> + else: >>>> + >>>> + log.debug("Not reloading Unbound. Leases not changed.") >>>> >>>> def write_dhcp_leases(self, leases): >>>> log.debug("Writing DHCP leases...") >>>> >>>> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >>>> - for l in leases: >>>> + for l in sorted(leases): >>>> for rr in l.rrset: >>>> f.write("local-data: "%s"\n" % " ".join(rr)) >>>> >>>> - # Make file readable for everyone >>>> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>> + filecmp.clear_cache() >>>> + try: >>>> + require_unbound_reload = not filecmp.cmp(f.name, self.path, shallow=False) >>>> + except FileNotFoundError: >>>> + require_unbound_reload = True >>>> >>> If you want to go this route, you will have to flush the temporary file first. Otherwise you might have some data left in the buffer that has not been written (because the file is still open). You will then compare incomplete output. This would not become a problem if you used the file descriptors. >>> >> The temp file is closed at the point of doing the filecmp, and the filecmp cache has been flushed. How do I flush the temporary file further. Reading up, is it just having the last statement in the with block as f.flush() with the same indent as "for l in sorted(leases):" >> > If you have left the with block you will already have flushed the buffers and closed the file. It looks like I misread the indentation in the patch then. > > >>> Catching FileNotFound would work for me. >>> >> Good >> >>>> + >>>> + # Make file readable for everyone >>>> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>> + >>>> + # Move the file to its destination >>>> + os.rename(f.name, self.path) >>>> >>>> - # Move the file to its destination >>>> - os.rename(f.name, self.path) >>>> + return require_unbound_reload >>>> >>>> def _control(self, *args): >>>> command = ["unbound-control"] >>>> -- >>>> 2.43.0 >>>> >>>> >>>> Nick >>>>
>
config/unbound/unbound-dhcp-leases-bridge | 50 +++++++++++++++++++---- 1 file changed, 41 insertions(+), 9 deletions(-)
diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge index e9f022aff..80d9523ea 100644 --- a/config/unbound/unbound-dhcp-leases-bridge +++ b/config/unbound/unbound-dhcp-leases-bridge @@ -516,26 +516,37 @@ class UnboundConfigWriter(object):
def update_dhcp_leases(self, leases): # Write out all leases
- self.write_dhcp_leases(leases)
- if self.write_dhcp_leases(leases):
- log.debug("Reloading Unbound...")
- log.debug("Reloading Unbound...")
- # Reload the configuration without dropping the cache
- self._control("reload_keep_cache")
- # Reload the configuration without dropping the cache
- self._control("reload_keep_cache")
- else:
- log.debug("Not reloading Unbound. Leases not changed.")
def write_dhcp_leases(self, leases): log.debug("Writing DHCP leases...")
with tempfile.NamedTemporaryFile(mode="w", delete=False) as f:
- for l in leases:
- for l in sorted(leases, key=lambda x: x.ipaddr):
for rr in l.rrset: f.write("local-data: "%s"\n" % " ".join(rr))
- # Make file readable for everyone
- os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)
- try:
- require_unbound_reload = not self.compare(f.name, self.path)
- except FileNotFoundError:
- require_unbound_reload = True
- # Make file readable for everyone
- os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH)
- # Move the file to its destination
- os.rename(f.name, self.path)
- # Move the file to its destination
- os.rename(f.name, self.path)
- return require_unbound_reload
def _control(self, *args): command = ["unbound-control"] @@ -551,6 +562,27 @@ class UnboundConfigWriter(object):
raise e
- def compare(self, if1, if2):
- buffersize = 16 * 1024
- f1 = open(if1, "r")
- f2 = open(if2, "r")
- # Reset file handles
- f1.seek(0)
- f2.seek(0)
- while True:
- b1 = f1.read(buffersize)
- b2 = f2.read(buffersize)
- if not b1 == b2:
- return False
- elif not b1 or not b2:
- break
- return True
if __name__ == "__main__": parser = argparse.ArgumentParser(description="Bridge for DHCP Leases and Unbound DNS")
On 27/04/2024 10:53, Nick Howitt wrote:
On 27/04/2024 09:35, Michael Tremer wrote:
Hello Nick,
On 26 Apr 2024, at 17:48, Nick Howittnick@howitts.co.uk wrote:
On 26/04/2024 16:12, Michael Tremer wrote:
Hello Nick,
On 25 Apr 2024, at 13:43, Nick Howittnick@howitts.co.uk wrote:
Please can I bump this again to request a review of my patch submitted on 28th March? Thanks, Nick.
It is a little bit difficult to test your proposed changes if testers have to copy and paste them out of an email.
Confused. I was told I had to submit the diff in an email with [PATCH] in the title and not to attach a file. Please can you tell me how I should be operating?
No worries, we are going to sort this out.
So the first email in the thread was a correct patch. It was just the patch in the format that Git uses and it was properly parsed by Patchwork. Later in the same thread, the patch was just copied and pasted which was impossible for Patchwork to pick up. There is nothing wrong with using bits of code in the conversation, but it would have been nice to have the final version submitted once again.
I have just submitted a patch set to this list that breaks down all the changes that you proposed and I cleaned up the code so that we can use the filecmp function without any extra steps. We can simply jump out of the write function and return False to prevent Unbound being reloaded.
I thought you didn't want me to use filecmp and pushed me to use your file compare routine.
Yeah, I would have preferred that too, but I thought using the module might be quicker now.
We will also avoid replacing the file all the time as that causes lots of unnecessary disk I/O. I am not saying it makes a significant difference, but flash has a limited write count that we are still wasting.
OK, but my changes were no worse than before, either doing a delete or rename whereas before it always did a rename. The "unnecessary I/O" was already there.
Please review and you if you are happy, please add the appropriate Git tag.
Sorry but I don't know what you mean by "adding the appropriate Git tag". I am only using Git locally and have no repo access.
I tested the patches yesterday and today and I am happy with them. I hope I have signed them off correctly.
FWIW I hand patched the program as it was easier than learning something new in this case as the patch set is so small.
One thing not mentioned in the bug report. Not only is unbound restarting on renewals which this fixes, it also restarts on new leases **and** expiring leases. For big work environments, I would have thought the best workaround is to set the default DHCP lease to 24h. It would help during the week to lessen the restarts but not on Monday mornings. It may also need a bigger address pool to cope with devices randomising MAC addresses.
Roll on Jon's fix as it would get round all these restarts. I use it modified not to update the dhcp-leases.conf. and another couple of tweaks. Are you able to test it, but it does mean piecing it together?
Thanks.
How can I pull your changes? Otherwise I'll need to patch my set up manually.
You can apply the patch to your local repository using “git am”:
Thanks again.
I also have another problem. I have temporarily had to remove my IPF gateway for a business reason and don't know when I'll be able to add it back in. I'll have to see if I can insert it in my LAN somehow. At the same time it is also running Jon's mods which I'll have to undo.
With respect to Jon's mods, I've asked a couple of times on bugzilla, but is there any point in his routine maintaining /etc/unbound/dhcp-leases.conf. As far as I see the file is only read once when unbound starts, so it only has to be valid when unbound starts and not at any other point in time. To this effect he can remove the file maintenance and I attached a modified copy unbound-dhcp-leases-bridge to his repo which generates a new dhcp-leases.conf and exits, removing all the daemonising bit and the inode watcher. This file can be run as part of the unbound init routine just before the line that actually starts or reloads unbound. It also has the advantage of cutting down a load of disk I/O, which is important as you've just pointed out.
I don’t think that we can fully remove the daemon part, because I don’t think that there is a way to get a script (shell or otherwise) fast enough to perform the job. I also think that we benefit a lot from keeping some state in memory, whereas the script will have to parse the old file first, generate a new file and then compare the result. That will take a long time.
I had Jon's script down to 70ms after I removed the dhcp-leases file updates, used my fast method for deriving the domain name and removed Jon's initial setting up bit which is not needed once the changes are accepted. This is on a 2c VM running at 2GHz. I thought the goal was < 100ms which has been done comfortably.
FWIW, if you no longer maintain dhcp-leases in the Jon's script you never read the old file or do a compare. All you do is read the hosts and static-leases files and nothing is written to file.
Do we needed the leases file generated at startup? Yes, I think so, too. DNS is quite essential for the network and for local services to be able to start. So it has to come up very early. If it comes up with proper configuration we avoid any other problems where hosts might resolve names too early and get a different result compared to a couple of seconds later when the zone is fully initialised.
My cut-down version of unbound-dhcp-leases-bridge is at https://bugzilla.ipfire.org/attachment.cgi?id=1516 on https://bugzilla.ipfire.org/show_bug.cgi?id=13254. On my test system it takes 219ms to run so delays unbound starting up minimally.
At the same time there is a gain in the dhcp start up as the unbound-dhcp-leases-bridge is no longer needed and it therefore avoids triggering an unbound restart which happens every time unbound-dhcp-leases-bridge is run.
If we are generating dhcp-leases.conf at start up, then there is no point in maintaining it while unbound is running as the maintained file will never be read.
So all in all, I think the route I would most likely look at is changing the daemon to open a socket. We write minimal scripts which are called by dhcpd that simply pipe their data into the socket which the daemon will then parse. With all available leases in memory already, we can check if a lease has changed or if it has been renewed without any further changes. If nothing has changed, we don’t need to do anything. In case of a change, we will have to write the file again and have Unbound reload, or we even call unbound-control and make the one change without a reload. Doing that in the daemon should give us more flexibility and we avoid any performance penalties as the daemon could run multiple commands concurrently which dhcpd can’t do.
Sockets are beyond me, I am afraid.
Part of the goal of all this is to avoid reloading unbound and part of my goal is to avoid writing dhcp-leases.conf unless unbound is restarted. I am not sure why you are still mentioning writing a file and restarting/reloading unbound.
Also on his patches, I had a significantly faster way of determining the domain name (in bash) and he and I were looking for your feedback on if we'd be allowed to use it.
Anything is allowed. It just has to be fast. And for that, I believe, calling any other binary will be a problem.
OK. I was proposing to read the DOMAINNAME with:
DOMAINNAME=${HOSTNAME#*.*}which is many times faster than the conventional route:
eval $(/usr/local/bin/readhash /var/ipfire/main/settings)If there is any concern with bash not having the full environment when the program is called, the shebang could be changed to:
#!/bin/bash -lNick
-Michael
Regards,
Nick
Best, -Michael
On 10/04/2024 12:04, Nick Howitt wrote:
Bump
On 28/03/2024 17:25, Nick Howitt wrote:
> Next version. > > On 28/03/2024 10:41, Michael Tremer wrote: > >> Hello, >> >> >>> On 27 Mar 2024, at 15:28, Nick Howittnick@howitts.co.uk wrote: >>> >>> >>> >>> On 26/03/2024 11:10, Michael Tremer wrote: >>> >>>>> On 25 Mar 2024, at 13:11, Nick Howittnick@howitts.co.uk wrote: >>>>> >>>>> >>>>> >>>>> On 25/03/2024 11:31, Michael Tremer wrote: >>>>> >>>>>> Hello, >>>>>> I don’t think that the risk is very high, but I would account for this problem. >>>>>> With my proposed changes from my previous email this should not be a large problem at all. When opening /etc/unbound/dhcp-leases.conf with mode “r+” you can catch the Exception and skip the comparison and open the file again with “w”. >>>>>> >>>>>>> On 25 Mar 2024, at 10:10, Nick Howittnick@howitts.co.uk wrote: >>>>>>> >>>>>>> If there is a risk that the /etc/unbound/dhcp-leases.conf does not exist, for example on a clean install (I don't know), the filecmp.cmp line needs to be wrapped in a try...except to stop it blowing up: >>>>>>> >>>>>>> try: >>>>>>> RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) >>>>>>> except: >>>>>>> RequireUnboundReload = True >>>>>>> >>>>>> I generally don’t like blanco exception catches. You will never know about the daemons that you are hiding. >>>>>> >>>>>>> This will force the file to be to be generated, even if it is only copying in a blank temp file, and unbound will restart. It will only ever happen once. >>>>>>> >>>>>>> >>>>>>> On 24/03/2024 15:49, Nick Howitt wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> **** This is a repost of an email incorrectly submitted. **** >>>>>>>> Please bear with me as I am new to IPFire and not at all used to this style of development, so any guidance would be appreciated. >>>>>>>> There is a bug,https://bugzilla.ipfire.org/show_bug.cgi?id=13254, where unbound-dhcp-leases-bridge restarts unbound a lot when it is totally unnecessary. It appears to be because when a lease is renewed, the script gets triggered, created a new /etc/unbound/dhcp-leases.conf then restarts Unbound to read the file. Generally this seems to be unnecessary as, with a renewed lease, the old and new /etc/unbound/dhcp-leases.conf files are the same. With 5 leases in that file (one for a machine not active) I am getting 3-4 restarts an hour of Unbound when I have a min/max lease time of 60/120min. >>>>>>>> Looking at the code, it is fairly easy to fix. The current code creates a temp file then copies it into place then restarts unbound. All I am doing is doing a file comparison before the copy and skipping the restart if the files are the same. >>>>>>>> There were a couple of gotchas because setting the file attributes and copying the file were done inside the "with" block for generating the temporary file. This meant a file comparison always returned False as the temp file was still open and so never the same is the old file. >>>>>>>> I moved those two statements outside the "with". This forced me to change the fchmod to chmod. >>>>>>>> It could be argued that the file copy should not be done if the files are not different, but it would take an extra "if" and you'd have to remember to delete the temp file. If required, I can make that change. >>>>>>>> Also, one small thing I noticed once is that the old and new dhcp-leases.conf files could occasionally contain the same leases but in a different order. I have been unable to reproduce, but to sidestep it, instead of stepping through the leases variable directly, I sort it and step through that. It should make the resulting file completely deterministic and make the file comparison more effective. >>>>>>>> My patch is: >>>>>>>> >From 73873d4944944a2f02317a73074a6894726f36f7 Mon Sep 17 00:00:00 2001 >>>>>>>> From: Nick Howittnick@howitts.co.uk >>>>>>>> Date: Sun, 24 Mar 2024 15:17:19 +0000 >>>>>>>> Subject: [PATCH] Stop unbound-dhcp-leases-bridge from restarting unbound every >>>>>>>> time it write dhcp-leases.conf as it is very often unchanged and does not >>>>>>>> require a restart. >>>>>>>> --- >>>>>>>> config/unbound/unbound-dhcp-leases-bridge | 28 +++++++++++++++-------- >>>>>>>> 1 file changed, 19 insertions(+), 9 deletions(-) >>>>>>>> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>> index e9f022aff..d22772066 100644 >>>>>>>> --- a/config/unbound/unbound-dhcp-leases-bridge >>>>>>>> +++ b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>> @@ -22,6 +22,7 @@ >>>>>>>> import argparse >>>>>>>> import datetime >>>>>>>> import daemon >>>>>>>> +import filecmp >>>>>>>> import functools >>>>>>>> import ipaddress >>>>>>>> import logging >>>>>>>> @@ -516,26 +517,35 @@ class UnboundConfigWriter(object): >>>>>>>> def update_dhcp_leases(self, leases): >>>>>>>> # Write out all leases >>>>>>>> - self.write_dhcp_leases(leases) >>>>>>>> + if self.write_dhcp_leases(leases): >>>>>>>> - log.debug("Reloading Unbound...") >>>>>>>> + log.debug("Reloading Unbound...") >>>>>>>> - # Reload the configuration without dropping the cache >>>>>>>> - self._control("reload_keep_cache") >>>>>>>> + # Reload the configuration without dropping the cache >>>>>>>> + self._control("reload_keep_cache") >>>>>>>> + >>>>>>>> + else: >>>>>>>> + >>>>>>>> + log.debug("Not reloading Unbound. Leases not changed.") >>>>>>>> def write_dhcp_leases(self, leases): >>>>>>>> log.debug("Writing DHCP leases...") >>>>>>>> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >>>>>>>> - for l in leases: >>>>>>>> + for l in sorted(leases): >>>>>>>> for rr in l.rrset: >>>>>>>> f.write("local-data: "%s"\n" % " ".join(rr)) >>>>>>>> - # Make file readable for everyone >>>>>>>> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>> + filecmp.clear_cache() >>>>>>>> + RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) >>>>>>>> + >>>>>>>> + # Make file readable for everyone >>>>>>>> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>> + >>>>>>>> + # Move the file to its destination >>>>>>>> + os.rename(f.name, self.path) >>>>>>>> - # Move the file to its destination >>>>>>>> - os.rename(f.name, self.path) >>>>>>>> + return RequireUnboundReload >>>>>>>> def _control(self, *args): >>>>>>>> command = ["unbound-control"] >>>>>>>> >>>>> Resubmitting the patch. >>>>> >>>>> Following my previous reply, I would like to stick with filecmp as I am using it with "shallow=False" to avoid doing the comparison just with stat() calls. Note that I also flush the filecmp cache before using the command so it is running with the latest file data. Talking about file descriptors, atomic file replacement etc., scares me. >>>>> >>>> File descriptors are great. They avoid race-conditions and all sorts of things. >>>> >>> But I have never used them so I don't intend to go that way. Too much for me to get my brain round and a limited ability to test. >>> >> Well, I pretty much gave you the entire solution… There was nothing left to do apart from copying and pasting the snippets. >> >> > Ok I've tried to use your version but struggled to get it going. If I used it as was and called compare(file1_name, file2_name) it failed as "compare" was not defined. > I could call it using self.compare but then it failed with "TypeError: UnboundConfigWriter.compare() takes 2 positional arguments but 3 were given". I changed the function to compare(self, f1, f2). > Then it failed with "AttributeError: 'str' object has no attribute 'seek'", so I changed it again to accept file names then open the files before doing the seeks. > I really hope it is OK as I am out of my depth when defining functions with "self" as an argument. Please can you check it? If it is not OK, can I revert to filecmp? > > I have made one futher change, the sorted(leases) line was not working and not sorting the object ever, and I saw lines changing places in the resulting leases file. I have changed it to "sorted(leases, key=lambda x: x.ipaddr)". It definitely sorts this time. > > >>>> I don’t think that module is the best choice really if you have to hack around and flush the cache. Why would it even cache things in the first place? I don’t see where the cache helps. >>>> >>> From the filecmp.clear_cache documentation: >>> >>> Clear the filecmp cache. This may be useful if a file is compared so >>> quickly after it is modified that it is within the mtime resolution >>> of the underlying filesystem. >>> >>> Now this makes me believe that the clear_cache is only relevant when using shallow=True, which I'm not and I don't know how to check the underlying code. It is used to make sure the stat() info is updated, but we are not using the stat() info. I guess we could as easily add a one second sleep instead as that is way longer than the mtime resolution. >>> >> The function is here and it is rather simple:https://github.com/python/cpython/blob/3.12/Lib/filecmp.py#L30 >> >> It will always stat() both files and compares three items in case shallow is False: The type of the file, the size and mtime. If the file is not a regular file, the function will exist rather early. So in our case, that is all work the function is doing for thing. It boils down to something very similar to what I suggested to use to compare the files. >> >> On top of that, there is a cache which will always be used but will never return a match for us because the temporary file at least will always have another mtime. >> >> You could clear the cache after every call, but I would rather not have our code messy and care about implementation details of some Python module. >> >> >>> The module is chosen because all the references I read about comparing files preferred this one-liner for comparing files, although other methods were given and there were lots of discussions about large files (GB+) because if you chomp them it may be much quicker to say they are not the same when you find the first block that does not match. Having said this, I don't know how filecmp works, but we only have a small file. At the same time, I am not a regular programmer so simple one-liners are good for me. >>> >> You would always want to read both files block for block and compare the blocks. There is no way to implement this significantly faster. >> >> >>>>> I have modified the earlier try/except I proposed to catch only FileNotFoundError, and I have changed CamelCase to snake_case. >>>>> >>>>> --- >>>>> config/unbound/unbound-dhcp-leases-bridge | 31 ++++++++++++++++------- >>>>> 1 file changed, 22 insertions(+), 9 deletions(-) >>>>> >>>>> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >>>>> index e9f022aff..a9ec8e1e2 100644 >>>>> --- a/config/unbound/unbound-dhcp-leases-bridge >>>>> +++ b/config/unbound/unbound-dhcp-leases-bridge >>>>> @@ -22,6 +22,7 @@ >>>>> import argparse >>>>> import datetime >>>>> import daemon >>>>> +import filecmp >>>>> import functools >>>>> import ipaddress >>>>> import logging >>>>> @@ -516,26 +517,38 @@ class UnboundConfigWriter(object): >>>>> >>>>> def update_dhcp_leases(self, leases): >>>>> # Write out all leases >>>>> - self.write_dhcp_leases(leases) >>>>> + if self.write_dhcp_leases(leases): >>>>> >>>>> - log.debug("Reloading Unbound...") >>>>> + log.debug("Reloading Unbound...") >>>>> >>>>> - # Reload the configuration without dropping the cache >>>>> - self._control("reload_keep_cache") >>>>> + # Reload the configuration without dropping the cache >>>>> + self._control("reload_keep_cache") >>>>> + >>>>> + else: >>>>> + >>>>> + log.debug("Not reloading Unbound. Leases not changed.") >>>>> >>>>> def write_dhcp_leases(self, leases): >>>>> log.debug("Writing DHCP leases...") >>>>> >>>>> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >>>>> - for l in leases: >>>>> + for l in sorted(leases): >>>>> for rr in l.rrset: >>>>> f.write("local-data: "%s"\n" % " ".join(rr)) >>>>> >>>>> - # Make file readable for everyone >>>>> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>> + filecmp.clear_cache() >>>>> + try: >>>>> + require_unbound_reload = not filecmp.cmp(f.name, self.path, shallow=False) >>>>> + except FileNotFoundError: >>>>> + require_unbound_reload = True >>>>> >>>> If you want to go this route, you will have to flush the temporary file first. Otherwise you might have some data left in the buffer that has not been written (because the file is still open). You will then compare incomplete output. This would not become a problem if you used the file descriptors. >>>> >>> The temp file is closed at the point of doing the filecmp, and the filecmp cache has been flushed. How do I flush the temporary file further. Reading up, is it just having the last statement in the with block as f.flush() with the same indent as "for l in sorted(leases):" >>> >> If you have left the with block you will already have flushed the buffers and closed the file. It looks like I misread the indentation in the patch then. >> >> >>>> Catching FileNotFound would work for me. >>>> >>> Good >>> >>>>> + >>>>> + # Make file readable for everyone >>>>> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>> + >>>>> + # Move the file to its destination >>>>> + os.rename(f.name, self.path) >>>>> >>>>> - # Move the file to its destination >>>>> - os.rename(f.name, self.path) >>>>> + return require_unbound_reload >>>>> >>>>> def _control(self, *args): >>>>> command = ["unbound-control"] >>>>> -- >>>>> 2.43.0 >>>>> >>>>> >>>>> Nick >>>>> > --- > config/unbound/unbound-dhcp-leases-bridge | 50 +++++++++++++++++++---- > 1 file changed, 41 insertions(+), 9 deletions(-) > > diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge > index e9f022aff..80d9523ea 100644 > --- a/config/unbound/unbound-dhcp-leases-bridge > +++ b/config/unbound/unbound-dhcp-leases-bridge > @@ -516,26 +516,37 @@ class UnboundConfigWriter(object): > > def update_dhcp_leases(self, leases): > # Write out all leases > - self.write_dhcp_leases(leases) > + if self.write_dhcp_leases(leases): > > - log.debug("Reloading Unbound...") > + log.debug("Reloading Unbound...") > > - # Reload the configuration without dropping the cache > - self._control("reload_keep_cache") > + # Reload the configuration without dropping the cache > + self._control("reload_keep_cache") > + > + else: > + > + log.debug("Not reloading Unbound. Leases not changed.") > > def write_dhcp_leases(self, leases): > log.debug("Writing DHCP leases...") > > with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: > - for l in leases: > + for l in sorted(leases, key=lambda x: x.ipaddr): > for rr in l.rrset: > f.write("local-data: "%s"\n" % " ".join(rr)) > > - # Make file readable for everyone > - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) > + try: > + require_unbound_reload = not self.compare(f.name, self.path) > + except FileNotFoundError: > + require_unbound_reload = True > + > + # Make file readable for everyone > + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) > + > + # Move the file to its destination > + os.rename(f.name, self.path) > > - # Move the file to its destination > - os.rename(f.name, self.path) > + return require_unbound_reload > > def _control(self, *args): > command = ["unbound-control"] > @@ -551,6 +562,27 @@ class UnboundConfigWriter(object): > > raise e > > + def compare(self, if1, if2): > + buffersize = 16 * 1024 > + > + f1 = open(if1, "r") > + f2 = open(if2, "r") > + > + # Reset file handles > + f1.seek(0) > + f2.seek(0) > + > + while True: > + b1 = f1.read(buffersize) > + b2 = f2.read(buffersize) > + > + if not b1 == b2: > + return False > + > + elif not b1 or not b2: > + break > + > + return True > > if __name__ == "__main__": > parser = argparse.ArgumentParser(description="Bridge for DHCP Leases and Unbound DNS") >
Hello Nick,
On 30 Apr 2024, at 12:58, Nick Howitt nick@howitts.co.uk wrote:
On 27/04/2024 10:53, Nick Howitt wrote:
On 27/04/2024 09:35, Michael Tremer wrote:
Hello Nick,
On 26 Apr 2024, at 17:48, Nick Howitt nick@howitts.co.uk wrote:
On 26/04/2024 16:12, Michael Tremer wrote:
Hello Nick,
On 25 Apr 2024, at 13:43, Nick Howitt nick@howitts.co.uk wrote:
Please can I bump this again to request a review of my patch submitted on 28th March? Thanks, Nick.
It is a little bit difficult to test your proposed changes if testers have to copy and paste them out of an email.
Confused. I was told I had to submit the diff in an email with [PATCH] in the title and not to attach a file. Please can you tell me how I should be operating?
No worries, we are going to sort this out.
So the first email in the thread was a correct patch. It was just the patch in the format that Git uses and it was properly parsed by Patchwork. Later in the same thread, the patch was just copied and pasted which was impossible for Patchwork to pick up. There is nothing wrong with using bits of code in the conversation, but it would have been nice to have the final version submitted once again.
I have just submitted a patch set to this list that breaks down all the changes that you proposed and I cleaned up the code so that we can use the filecmp function without any extra steps. We can simply jump out of the write function and return False to prevent Unbound being reloaded.
I thought you didn't want me to use filecmp and pushed me to use your file compare routine.
Yeah, I would have preferred that too, but I thought using the module might be quicker now.
We will also avoid replacing the file all the time as that causes lots of unnecessary disk I/O. I am not saying it makes a significant difference, but flash has a limited write count that we are still wasting.
OK, but my changes were no worse than before, either doing a delete or rename whereas before it always did a rename. The "unnecessary I/O" was already there.
Please review and you if you are happy, please add the appropriate Git tag.
Sorry but I don't know what you mean by "adding the appropriate Git tag". I am only using Git locally and have no repo access.
I tested the patches yesterday and today and I am happy with them. I hope I have signed them off correctly.
Perfect. Thank you very much.
FWIW I hand patched the program as it was easier than learning something new in this case as the patch set is so small.
Haha, okay :)
One thing not mentioned in the bug report. Not only is unbound restarting on renewals which this fixes, it also restarts on new leases **and** expiring leases.
I think this is the correct way though, but I agree it creates extra work.
For big work environments, I would have thought the best workaround is to set the default DHCP lease to 24h. It would help during the week to lessen the restarts but not on Monday mornings. It may also need a bigger address pool to cope with devices randomising MAC addresses.
It depends on where you deploy this. In any office/home environment 24h is probably a good lease time. Maybe even 12 or 10 is a good option. I have rarely seen lease times of a week or so recently; in fact the default seems to be closer to an hour. That should however not be a problem for us any more because we don’t reload Unbound if a client simply renews their lease. Only if the join or leave the network.
Roll on Jon's fix as it would get round all these restarts. I use it modified not to update the dhcp-leases.conf. and another couple of tweaks. Are you able to test it, but it does mean piecing it together?
To be honest, I kind of lost track where we are with that now. I did look at a couple of the references from your last email, but didn’t make it very far, yet.
-Michael
Thanks.
How can I pull your changes? Otherwise I'll need to patch my set up manually.
You can apply the patch to your local repository using “git am”:
Thanks again.
I also have another problem. I have temporarily had to remove my IPF gateway for a business reason and don't know when I'll be able to add it back in. I'll have to see if I can insert it in my LAN somehow. At the same time it is also running Jon's mods which I'll have to undo.
With respect to Jon's mods, I've asked a couple of times on bugzilla, but is there any point in his routine maintaining /etc/unbound/dhcp-leases.conf. As far as I see the file is only read once when unbound starts, so it only has to be valid when unbound starts and not at any other point in time. To this effect he can remove the file maintenance and I attached a modified copy unbound-dhcp-leases-bridge to his repo which generates a new dhcp-leases.conf and exits, removing all the daemonising bit and the inode watcher. This file can be run as part of the unbound init routine just before the line that actually starts or reloads unbound. It also has the advantage of cutting down a load of disk I/O, which is important as you've just pointed out.
I don’t think that we can fully remove the daemon part, because I don’t think that there is a way to get a script (shell or otherwise) fast enough to perform the job. I also think that we benefit a lot from keeping some state in memory, whereas the script will have to parse the old file first, generate a new file and then compare the result. That will take a long time.
I had Jon's script down to 70ms after I removed the dhcp-leases file updates, used my fast method for deriving the domain name and removed Jon's initial setting up bit which is not needed once the changes are accepted. This is on a 2c VM running at 2GHz. I thought the goal was < 100ms which has been done comfortably.
FWIW, if you no longer maintain dhcp-leases in the Jon's script you never read the old file or do a compare. All you do is read the hosts and static-leases files and nothing is written to file.
Do we needed the leases file generated at startup? Yes, I think so, too. DNS is quite essential for the network and for local services to be able to start. So it has to come up very early. If it comes up with proper configuration we avoid any other problems where hosts might resolve names too early and get a different result compared to a couple of seconds later when the zone is fully initialised.
My cut-down version of unbound-dhcp-leases-bridge is at https://bugzilla.ipfire.org/attachment.cgi?id=1516 on https://bugzilla.ipfire.org/show_bug.cgi?id=13254. On my test system it takes 219ms to run so delays unbound starting up minimally.
At the same time there is a gain in the dhcp start up as the unbound-dhcp-leases-bridge is no longer needed and it therefore avoids triggering an unbound restart which happens every time unbound-dhcp-leases-bridge is run.
If we are generating dhcp-leases.conf at start up, then there is no point in maintaining it while unbound is running as the maintained file will never be read.
So all in all, I think the route I would most likely look at is changing the daemon to open a socket. We write minimal scripts which are called by dhcpd that simply pipe their data into the socket which the daemon will then parse. With all available leases in memory already, we can check if a lease has changed or if it has been renewed without any further changes. If nothing has changed, we don’t need to do anything. In case of a change, we will have to write the file again and have Unbound reload, or we even call unbound-control and make the one change without a reload. Doing that in the daemon should give us more flexibility and we avoid any performance penalties as the daemon could run multiple commands concurrently which dhcpd can’t do.
Sockets are beyond me, I am afraid.
Part of the goal of all this is to avoid reloading unbound and part of my goal is to avoid writing dhcp-leases.conf unless unbound is restarted. I am not sure why you are still mentioning writing a file and restarting/reloading unbound.
Also on his patches, I had a significantly faster way of determining the domain name (in bash) and he and I were looking for your feedback on if we'd be allowed to use it.
Anything is allowed. It just has to be fast. And for that, I believe, calling any other binary will be a problem.
OK. I was proposing to read the DOMAINNAME with: DOMAINNAME=${HOSTNAME#*.*} which is many times faster than the conventional route: eval $(/usr/local/bin/readhash /var/ipfire/main/settings) If there is any concern with bash not having the full environment when the program is called, the shebang could be changed to: #!/bin/bash -l Nick
-Michael
Regards,
Nick
Best, -Michael
On 10/04/2024 12:04, Nick Howitt wrote:
> Bump > > On 28/03/2024 17:25, Nick Howitt wrote: > > >> Next version. >> >> On 28/03/2024 10:41, Michael Tremer wrote: >> >> >>> Hello, >>> >>> >>> >>>> On 27 Mar 2024, at 15:28, Nick Howitt nick@howitts.co.uk wrote: >>>> >>>> >>>> >>>> On 26/03/2024 11:10, Michael Tremer wrote: >>>> >>>> >>>>>> On 25 Mar 2024, at 13:11, Nick Howitt nick@howitts.co.uk wrote: >>>>>> >>>>>> >>>>>> >>>>>> On 25/03/2024 11:31, Michael Tremer wrote: >>>>>> >>>>>> >>>>>>> Hello, >>>>>>> I don’t think that the risk is very high, but I would account for this problem. >>>>>>> With my proposed changes from my previous email this should not be a large problem at all. When opening /etc/unbound/dhcp-leases.conf with mode “r+” you can catch the Exception and skip the comparison and open the file again with “w”. >>>>>>> >>>>>>> >>>>>>>> On 25 Mar 2024, at 10:10, Nick Howitt nick@howitts.co.uk wrote: >>>>>>>> >>>>>>>> If there is a risk that the /etc/unbound/dhcp-leases.conf does not exist, for example on a clean install (I don't know), the filecmp.cmp line needs to be wrapped in a try...except to stop it blowing up: >>>>>>>> >>>>>>>> try: >>>>>>>> RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) >>>>>>>> except: >>>>>>>> RequireUnboundReload = True >>>>>>>> >>>>>>>> >>>>>>> I generally don’t like blanco exception catches. You will never know about the daemons that you are hiding. >>>>>>> >>>>>>> >>>>>>>> This will force the file to be to be generated, even if it is only copying in a blank temp file, and unbound will restart. It will only ever happen once. >>>>>>>> >>>>>>>> >>>>>>>> On 24/03/2024 15:49, Nick Howitt wrote: >>>>>>>> >>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> **** This is a repost of an email incorrectly submitted. **** >>>>>>>>> Please bear with me as I am new to IPFire and not at all used to this style of development, so any guidance would be appreciated. >>>>>>>>> There is a bug, https://bugzilla.ipfire.org/show_bug.cgi?id=13254, where unbound-dhcp-leases-bridge restarts unbound a lot when it is totally unnecessary. It appears to be because when a lease is renewed, the script gets triggered, created a new /etc/unbound/dhcp-leases.conf then restarts Unbound to read the file. Generally this seems to be unnecessary as, with a renewed lease, the old and new /etc/unbound/dhcp-leases.conf files are the same. With 5 leases in that file (one for a machine not active) I am getting 3-4 restarts an hour of Unbound when I have a min/max lease time of 60/120min. >>>>>>>>> Looking at the code, it is fairly easy to fix. The current code creates a temp file then copies it into place then restarts unbound. All I am doing is doing a file comparison before the copy and skipping the restart if the files are the same. >>>>>>>>> There were a couple of gotchas because setting the file attributes and copying the file were done inside the "with" block for generating the temporary file. This meant a file comparison always returned False as the temp file was still open and so never the same is the old file. >>>>>>>>> I moved those two statements outside the "with". This forced me to change the fchmod to chmod. >>>>>>>>> It could be argued that the file copy should not be done if the files are not different, but it would take an extra "if" and you'd have to remember to delete the temp file. If required, I can make that change. >>>>>>>>> Also, one small thing I noticed once is that the old and new dhcp-leases.conf files could occasionally contain the same leases but in a different order. I have been unable to reproduce, but to sidestep it, instead of stepping through the leases variable directly, I sort it and step through that. It should make the resulting file completely deterministic and make the file comparison more effective. >>>>>>>>> My patch is: >>>>>>>>> >From 73873d4944944a2f02317a73074a6894726f36f7 Mon Sep 17 00:00:00 2001 >>>>>>>>> From: Nick Howitt nick@howitts.co.uk >>>>>>>>> Date: Sun, 24 Mar 2024 15:17:19 +0000 >>>>>>>>> Subject: [PATCH] Stop unbound-dhcp-leases-bridge from restarting unbound every >>>>>>>>> time it write dhcp-leases.conf as it is very often unchanged and does not >>>>>>>>> require a restart. >>>>>>>>> --- >>>>>>>>> config/unbound/unbound-dhcp-leases-bridge | 28 +++++++++++++++-------- >>>>>>>>> 1 file changed, 19 insertions(+), 9 deletions(-) >>>>>>>>> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>> index e9f022aff..d22772066 100644 >>>>>>>>> --- a/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>> +++ b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>> @@ -22,6 +22,7 @@ >>>>>>>>> import argparse >>>>>>>>> import datetime >>>>>>>>> import daemon >>>>>>>>> +import filecmp >>>>>>>>> import functools >>>>>>>>> import ipaddress >>>>>>>>> import logging >>>>>>>>> @@ -516,26 +517,35 @@ class UnboundConfigWriter(object): >>>>>>>>> def update_dhcp_leases(self, leases): >>>>>>>>> # Write out all leases >>>>>>>>> - self.write_dhcp_leases(leases) >>>>>>>>> + if self.write_dhcp_leases(leases): >>>>>>>>> - log.debug("Reloading Unbound...") >>>>>>>>> + log.debug("Reloading Unbound...") >>>>>>>>> - # Reload the configuration without dropping the cache >>>>>>>>> - self._control("reload_keep_cache") >>>>>>>>> + # Reload the configuration without dropping the cache >>>>>>>>> + self._control("reload_keep_cache") >>>>>>>>> + >>>>>>>>> + else: >>>>>>>>> + >>>>>>>>> + log.debug("Not reloading Unbound. Leases not changed.") >>>>>>>>> def write_dhcp_leases(self, leases): >>>>>>>>> log.debug("Writing DHCP leases...") >>>>>>>>> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >>>>>>>>> - for l in leases: >>>>>>>>> + for l in sorted(leases): >>>>>>>>> for rr in l.rrset: >>>>>>>>> f.write("local-data: "%s"\n" % " ".join(rr)) >>>>>>>>> - # Make file readable for everyone >>>>>>>>> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>>> + filecmp.clear_cache() >>>>>>>>> + RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) >>>>>>>>> + >>>>>>>>> + # Make file readable for everyone >>>>>>>>> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>>> + >>>>>>>>> + # Move the file to its destination >>>>>>>>> + os.rename(f.name, self.path) >>>>>>>>> - # Move the file to its destination >>>>>>>>> - os.rename(f.name, self.path) >>>>>>>>> + return RequireUnboundReload >>>>>>>>> def _control(self, *args): >>>>>>>>> command = ["unbound-control"] >>>>>>>>> >>>>>>>>> >>>>>> Resubmitting the patch. >>>>>> >>>>>> Following my previous reply, I would like to stick with filecmp as I am using it with "shallow=False" to avoid doing the comparison just with stat() calls. Note that I also flush the filecmp cache before using the command so it is running with the latest file data. Talking about file descriptors, atomic file replacement etc., scares me. >>>>>> >>>>>> >>>>> File descriptors are great. They avoid race-conditions and all sorts of things. >>>>> >>>>> >>>> But I have never used them so I don't intend to go that way. Too much for me to get my brain round and a limited ability to test. >>>> >>>> >>> Well, I pretty much gave you the entire solution… There was nothing left to do apart from copying and pasting the snippets. >>> >>> >>> >> Ok I've tried to use your version but struggled to get it going. If I used it as was and called compare(file1_name, file2_name) it failed as "compare" was not defined. >> I could call it using self.compare but then it failed with "TypeError: UnboundConfigWriter.compare() takes 2 positional arguments but 3 were given". I changed the function to compare(self, f1, f2). >> Then it failed with "AttributeError: 'str' object has no attribute 'seek'", so I changed it again to accept file names then open the files before doing the seeks. >> I really hope it is OK as I am out of my depth when defining functions with "self" as an argument. Please can you check it? If it is not OK, can I revert to filecmp? >> >> I have made one futher change, the sorted(leases) line was not working and not sorting the object ever, and I saw lines changing places in the resulting leases file. I have changed it to "sorted(leases, key=lambda x: x.ipaddr)". It definitely sorts this time. >> >> >> >>>>> I don’t think that module is the best choice really if you have to hack around and flush the cache. Why would it even cache things in the first place? I don’t see where the cache helps. >>>>> >>>>> >>>> From the filecmp.clear_cache documentation: >>>> >>>> Clear the filecmp cache. This may be useful if a file is compared so >>>> quickly after it is modified that it is within the mtime resolution >>>> of the underlying filesystem. >>>> >>>> Now this makes me believe that the clear_cache is only relevant when using shallow=True, which I'm not and I don't know how to check the underlying code. It is used to make sure the stat() info is updated, but we are not using the stat() info. I guess we could as easily add a one second sleep instead as that is way longer than the mtime resolution. >>>> >>>> >>> The function is here and it is rather simple: https://github.com/python/cpython/blob/3.12/Lib/filecmp.py#L30 >>> >>> It will always stat() both files and compares three items in case shallow is False: The type of the file, the size and mtime. If the file is not a regular file, the function will exist rather early. So in our case, that is all work the function is doing for thing. It boils down to something very similar to what I suggested to use to compare the files. >>> >>> On top of that, there is a cache which will always be used but will never return a match for us because the temporary file at least will always have another mtime. >>> >>> You could clear the cache after every call, but I would rather not have our code messy and care about implementation details of some Python module. >>> >>> >>> >>>> The module is chosen because all the references I read about comparing files preferred this one-liner for comparing files, although other methods were given and there were lots of discussions about large files (GB+) because if you chomp them it may be much quicker to say they are not the same when you find the first block that does not match. Having said this, I don't know how filecmp works, but we only have a small file. At the same time, I am not a regular programmer so simple one-liners are good for me. >>>> >>>> >>> You would always want to read both files block for block and compare the blocks. There is no way to implement this significantly faster. >>> >>> >>> >>>>>> I have modified the earlier try/except I proposed to catch only FileNotFoundError, and I have changed CamelCase to snake_case. >>>>>> >>>>>> --- >>>>>> config/unbound/unbound-dhcp-leases-bridge | 31 ++++++++++++++++------- >>>>>> 1 file changed, 22 insertions(+), 9 deletions(-) >>>>>> >>>>>> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >>>>>> index e9f022aff..a9ec8e1e2 100644 >>>>>> --- a/config/unbound/unbound-dhcp-leases-bridge >>>>>> +++ b/config/unbound/unbound-dhcp-leases-bridge >>>>>> @@ -22,6 +22,7 @@ >>>>>> import argparse >>>>>> import datetime >>>>>> import daemon >>>>>> +import filecmp >>>>>> import functools >>>>>> import ipaddress >>>>>> import logging >>>>>> @@ -516,26 +517,38 @@ class UnboundConfigWriter(object): >>>>>> >>>>>> def update_dhcp_leases(self, leases): >>>>>> # Write out all leases >>>>>> - self.write_dhcp_leases(leases) >>>>>> + if self.write_dhcp_leases(leases): >>>>>> >>>>>> - log.debug("Reloading Unbound...") >>>>>> + log.debug("Reloading Unbound...") >>>>>> >>>>>> - # Reload the configuration without dropping the cache >>>>>> - self._control("reload_keep_cache") >>>>>> + # Reload the configuration without dropping the cache >>>>>> + self._control("reload_keep_cache") >>>>>> + >>>>>> + else: >>>>>> + >>>>>> + log.debug("Not reloading Unbound. Leases not changed.") >>>>>> >>>>>> def write_dhcp_leases(self, leases): >>>>>> log.debug("Writing DHCP leases...") >>>>>> >>>>>> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >>>>>> - for l in leases: >>>>>> + for l in sorted(leases): >>>>>> for rr in l.rrset: >>>>>> f.write("local-data: "%s"\n" % " ".join(rr)) >>>>>> >>>>>> - # Make file readable for everyone >>>>>> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>> + filecmp.clear_cache() >>>>>> + try: >>>>>> + require_unbound_reload = not filecmp.cmp(f.name, self.path, shallow=False) >>>>>> + except FileNotFoundError: >>>>>> + require_unbound_reload = True >>>>>> >>>>>> >>>>> If you want to go this route, you will have to flush the temporary file first. Otherwise you might have some data left in the buffer that has not been written (because the file is still open). You will then compare incomplete output. This would not become a problem if you used the file descriptors. >>>>> >>>>> >>>> The temp file is closed at the point of doing the filecmp, and the filecmp cache has been flushed. How do I flush the temporary file further. Reading up, is it just having the last statement in the with block as f.flush() with the same indent as "for l in sorted(leases):" >>>> >>>> >>> If you have left the with block you will already have flushed the buffers and closed the file. It looks like I misread the indentation in the patch then. >>> >>> >>> >>>>> Catching FileNotFound would work for me. >>>>> >>>>> >>>> Good >>>> >>>> >>>>>> + >>>>>> + # Make file readable for everyone >>>>>> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>> + >>>>>> + # Move the file to its destination >>>>>> + os.rename(f.name, self.path) >>>>>> >>>>>> - # Move the file to its destination >>>>>> - os.rename(f.name, self.path) >>>>>> + return require_unbound_reload >>>>>> >>>>>> def _control(self, *args): >>>>>> command = ["unbound-control"] >>>>>> -- >>>>>> 2.43.0 >>>>>> >>>>>> >>>>>> Nick >>>>>> >>>>>> >> --- >> config/unbound/unbound-dhcp-leases-bridge | 50 +++++++++++++++++++---- >> 1 file changed, 41 insertions(+), 9 deletions(-) >> >> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >> index e9f022aff..80d9523ea 100644 >> --- a/config/unbound/unbound-dhcp-leases-bridge >> +++ b/config/unbound/unbound-dhcp-leases-bridge >> @@ -516,26 +516,37 @@ class UnboundConfigWriter(object): >> >> def update_dhcp_leases(self, leases): >> # Write out all leases >> - self.write_dhcp_leases(leases) >> + if self.write_dhcp_leases(leases): >> >> - log.debug("Reloading Unbound...") >> + log.debug("Reloading Unbound...") >> >> - # Reload the configuration without dropping the cache >> - self._control("reload_keep_cache") >> + # Reload the configuration without dropping the cache >> + self._control("reload_keep_cache") >> + >> + else: >> + >> + log.debug("Not reloading Unbound. Leases not changed.") >> >> def write_dhcp_leases(self, leases): >> log.debug("Writing DHCP leases...") >> >> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >> - for l in leases: >> + for l in sorted(leases, key=lambda x: x.ipaddr): >> for rr in l.rrset: >> f.write("local-data: "%s"\n" % " ".join(rr)) >> >> - # Make file readable for everyone >> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >> + try: >> + require_unbound_reload = not self.compare(f.name, self.path) >> + except FileNotFoundError: >> + require_unbound_reload = True >> + >> + # Make file readable for everyone >> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >> + >> + # Move the file to its destination >> + os.rename(f.name, self.path) >> >> - # Move the file to its destination >> - os.rename(f.name, self.path) >> + return require_unbound_reload >> >> def _control(self, *args): >> command = ["unbound-control"] >> @@ -551,6 +562,27 @@ class UnboundConfigWriter(object): >> >> raise e >> >> + def compare(self, if1, if2): >> + buffersize = 16 * 1024 >> + >> + f1 = open(if1, "r") >> + f2 = open(if2, "r") >> + >> + # Reset file handles >> + f1.seek(0) >> + f2.seek(0) >> + >> + while True: >> + b1 = f1.read(buffersize) >> + b2 = f2.read(buffersize) >> + >> + if not b1 == b2: >> + return False >> + >> + elif not b1 or not b2: >> + break >> + >> + return True >> >> if __name__ == "__main__": >> parser = argparse.ArgumentParser(description="Bridge for DHCP Leases and Unbound DNS") >> >>
To be honest, I kind of lost track where we are with that now.
Michael,
If you are referring to my question: Are you OK with moving forward using the DCHP on-commit, on-release, on-expiry?? (like the proof-of-concept)
Jon
On Apr 30, 2024, at 7:01 AM, Michael Tremer michael.tremer@ipfire.org wrote:
Hello Nick,
On 30 Apr 2024, at 12:58, Nick Howitt nick@howitts.co.uk wrote:
On 27/04/2024 10:53, Nick Howitt wrote:
On 27/04/2024 09:35, Michael Tremer wrote:
Hello Nick,
On 26 Apr 2024, at 17:48, Nick Howitt nick@howitts.co.uk wrote:
On 26/04/2024 16:12, Michael Tremer wrote:
Hello Nick,
> On 25 Apr 2024, at 13:43, Nick Howitt nick@howitts.co.uk wrote: > > Please can I bump this again to request a review of my patch submitted on 28th March? Thanks, Nick. > > It is a little bit difficult to test your proposed changes if testers have to copy and paste them out of an email.
Confused. I was told I had to submit the diff in an email with [PATCH] in the title and not to attach a file. Please can you tell me how I should be operating?
No worries, we are going to sort this out.
So the first email in the thread was a correct patch. It was just the patch in the format that Git uses and it was properly parsed by Patchwork. Later in the same thread, the patch was just copied and pasted which was impossible for Patchwork to pick up. There is nothing wrong with using bits of code in the conversation, but it would have been nice to have the final version submitted once again.
I have just submitted a patch set to this list that breaks down all the changes that you proposed and I cleaned up the code so that we can use the filecmp function without any extra steps. We can simply jump out of the write function and return False to prevent Unbound being reloaded.
I thought you didn't want me to use filecmp and pushed me to use your file compare routine.
Yeah, I would have preferred that too, but I thought using the module might be quicker now.
We will also avoid replacing the file all the time as that causes lots of unnecessary disk I/O. I am not saying it makes a significant difference, but flash has a limited write count that we are still wasting.
OK, but my changes were no worse than before, either doing a delete or rename whereas before it always did a rename. The "unnecessary I/O" was already there.
Please review and you if you are happy, please add the appropriate Git tag.
Sorry but I don't know what you mean by "adding the appropriate Git tag". I am only using Git locally and have no repo access.
I tested the patches yesterday and today and I am happy with them. I hope I have signed them off correctly.
Perfect. Thank you very much.
FWIW I hand patched the program as it was easier than learning something new in this case as the patch set is so small.
Haha, okay :)
One thing not mentioned in the bug report. Not only is unbound restarting on renewals which this fixes, it also restarts on new leases **and** expiring leases.
I think this is the correct way though, but I agree it creates extra work.
For big work environments, I would have thought the best workaround is to set the default DHCP lease to 24h. It would help during the week to lessen the restarts but not on Monday mornings. It may also need a bigger address pool to cope with devices randomising MAC addresses.
It depends on where you deploy this. In any office/home environment 24h is probably a good lease time. Maybe even 12 or 10 is a good option. I have rarely seen lease times of a week or so recently; in fact the default seems to be closer to an hour. That should however not be a problem for us any more because we don’t reload Unbound if a client simply renews their lease. Only if the join or leave the network.
Roll on Jon's fix as it would get round all these restarts. I use it modified not to update the dhcp-leases.conf. and another couple of tweaks. Are you able to test it, but it does mean piecing it together?
To be honest, I kind of lost track where we are with that now. I did look at a couple of the references from your last email, but didn’t make it very far, yet.
-Michael
Thanks.
How can I pull your changes? Otherwise I'll need to patch my set up manually.
You can apply the patch to your local repository using “git am”:
Thanks again.
I also have another problem. I have temporarily had to remove my IPF gateway for a business reason and don't know when I'll be able to add it back in. I'll have to see if I can insert it in my LAN somehow. At the same time it is also running Jon's mods which I'll have to undo.
With respect to Jon's mods, I've asked a couple of times on bugzilla, but is there any point in his routine maintaining /etc/unbound/dhcp-leases.conf. As far as I see the file is only read once when unbound starts, so it only has to be valid when unbound starts and not at any other point in time. To this effect he can remove the file maintenance and I attached a modified copy unbound-dhcp-leases-bridge to his repo which generates a new dhcp-leases.conf and exits, removing all the daemonising bit and the inode watcher. This file can be run as part of the unbound init routine just before the line that actually starts or reloads unbound. It also has the advantage of cutting down a load of disk I/O, which is important as you've just pointed out.
I don’t think that we can fully remove the daemon part, because I don’t think that there is a way to get a script (shell or otherwise) fast enough to perform the job. I also think that we benefit a lot from keeping some state in memory, whereas the script will have to parse the old file first, generate a new file and then compare the result. That will take a long time.
I had Jon's script down to 70ms after I removed the dhcp-leases file updates, used my fast method for deriving the domain name and removed Jon's initial setting up bit which is not needed once the changes are accepted. This is on a 2c VM running at 2GHz. I thought the goal was < 100ms which has been done comfortably.
FWIW, if you no longer maintain dhcp-leases in the Jon's script you never read the old file or do a compare. All you do is read the hosts and static-leases files and nothing is written to file.
Do we needed the leases file generated at startup? Yes, I think so, too. DNS is quite essential for the network and for local services to be able to start. So it has to come up very early. If it comes up with proper configuration we avoid any other problems where hosts might resolve names too early and get a different result compared to a couple of seconds later when the zone is fully initialised.
My cut-down version of unbound-dhcp-leases-bridge is at https://bugzilla.ipfire.org/attachment.cgi?id=1516 on https://bugzilla.ipfire.org/show_bug.cgi?id=13254. On my test system it takes 219ms to run so delays unbound starting up minimally.
At the same time there is a gain in the dhcp start up as the unbound-dhcp-leases-bridge is no longer needed and it therefore avoids triggering an unbound restart which happens every time unbound-dhcp-leases-bridge is run.
If we are generating dhcp-leases.conf at start up, then there is no point in maintaining it while unbound is running as the maintained file will never be read.
So all in all, I think the route I would most likely look at is changing the daemon to open a socket. We write minimal scripts which are called by dhcpd that simply pipe their data into the socket which the daemon will then parse. With all available leases in memory already, we can check if a lease has changed or if it has been renewed without any further changes. If nothing has changed, we don’t need to do anything. In case of a change, we will have to write the file again and have Unbound reload, or we even call unbound-control and make the one change without a reload. Doing that in the daemon should give us more flexibility and we avoid any performance penalties as the daemon could run multiple commands concurrently which dhcpd can’t do.
Sockets are beyond me, I am afraid.
Part of the goal of all this is to avoid reloading unbound and part of my goal is to avoid writing dhcp-leases.conf unless unbound is restarted. I am not sure why you are still mentioning writing a file and restarting/reloading unbound.
Also on his patches, I had a significantly faster way of determining the domain name (in bash) and he and I were looking for your feedback on if we'd be allowed to use it.
Anything is allowed. It just has to be fast. And for that, I believe, calling any other binary will be a problem.
OK. I was proposing to read the DOMAINNAME with: DOMAINNAME=${HOSTNAME#*.*} which is many times faster than the conventional route: eval $(/usr/local/bin/readhash /var/ipfire/main/settings) If there is any concern with bash not having the full environment when the program is called, the shebang could be changed to: #!/bin/bash -l Nick
-Michael
Regards,
Nick
Best, -Michael
> On 10/04/2024 12:04, Nick Howitt wrote: > > >> Bump >> >> On 28/03/2024 17:25, Nick Howitt wrote: >> >> >>> Next version. >>> >>> On 28/03/2024 10:41, Michael Tremer wrote: >>> >>> >>>> Hello, >>>> >>>> >>>> >>>>> On 27 Mar 2024, at 15:28, Nick Howitt nick@howitts.co.uk wrote: >>>>> >>>>> >>>>> >>>>> On 26/03/2024 11:10, Michael Tremer wrote: >>>>> >>>>> >>>>>>> On 25 Mar 2024, at 13:11, Nick Howitt nick@howitts.co.uk wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 25/03/2024 11:31, Michael Tremer wrote: >>>>>>> >>>>>>> >>>>>>>> Hello, >>>>>>>> I don’t think that the risk is very high, but I would account for this problem. >>>>>>>> With my proposed changes from my previous email this should not be a large problem at all. When opening /etc/unbound/dhcp-leases.conf with mode “r+” you can catch the Exception and skip the comparison and open the file again with “w”. >>>>>>>> >>>>>>>> >>>>>>>>> On 25 Mar 2024, at 10:10, Nick Howitt nick@howitts.co.uk wrote: >>>>>>>>> >>>>>>>>> If there is a risk that the /etc/unbound/dhcp-leases.conf does not exist, for example on a clean install (I don't know), the filecmp.cmp line needs to be wrapped in a try...except to stop it blowing up: >>>>>>>>> >>>>>>>>> try: >>>>>>>>> RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) >>>>>>>>> except: >>>>>>>>> RequireUnboundReload = True >>>>>>>>> >>>>>>>>> >>>>>>>> I generally don’t like blanco exception catches. You will never know about the daemons that you are hiding. >>>>>>>> >>>>>>>> >>>>>>>>> This will force the file to be to be generated, even if it is only copying in a blank temp file, and unbound will restart. It will only ever happen once. >>>>>>>>> >>>>>>>>> >>>>>>>>> On 24/03/2024 15:49, Nick Howitt wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> **** This is a repost of an email incorrectly submitted. **** >>>>>>>>>> Please bear with me as I am new to IPFire and not at all used to this style of development, so any guidance would be appreciated. >>>>>>>>>> There is a bug, https://bugzilla.ipfire.org/show_bug.cgi?id=13254, where unbound-dhcp-leases-bridge restarts unbound a lot when it is totally unnecessary. It appears to be because when a lease is renewed, the script gets triggered, created a new /etc/unbound/dhcp-leases.conf then restarts Unbound to read the file. Generally this seems to be unnecessary as, with a renewed lease, the old and new /etc/unbound/dhcp-leases.conf files are the same. With 5 leases in that file (one for a machine not active) I am getting 3-4 restarts an hour of Unbound when I have a min/max lease time of 60/120min. >>>>>>>>>> Looking at the code, it is fairly easy to fix. The current code creates a temp file then copies it into place then restarts unbound. All I am doing is doing a file comparison before the copy and skipping the restart if the files are the same. >>>>>>>>>> There were a couple of gotchas because setting the file attributes and copying the file were done inside the "with" block for generating the temporary file. This meant a file comparison always returned False as the temp file was still open and so never the same is the old file. >>>>>>>>>> I moved those two statements outside the "with". This forced me to change the fchmod to chmod. >>>>>>>>>> It could be argued that the file copy should not be done if the files are not different, but it would take an extra "if" and you'd have to remember to delete the temp file. If required, I can make that change. >>>>>>>>>> Also, one small thing I noticed once is that the old and new dhcp-leases.conf files could occasionally contain the same leases but in a different order. I have been unable to reproduce, but to sidestep it, instead of stepping through the leases variable directly, I sort it and step through that. It should make the resulting file completely deterministic and make the file comparison more effective. >>>>>>>>>> My patch is: >>>>>>>>>>> From 73873d4944944a2f02317a73074a6894726f36f7 Mon Sep 17 00:00:00 2001 >>>>>>>>>> From: Nick Howitt nick@howitts.co.uk >>>>>>>>>> Date: Sun, 24 Mar 2024 15:17:19 +0000 >>>>>>>>>> Subject: [PATCH] Stop unbound-dhcp-leases-bridge from restarting unbound every >>>>>>>>>> time it write dhcp-leases.conf as it is very often unchanged and does not >>>>>>>>>> require a restart. >>>>>>>>>> --- >>>>>>>>>> config/unbound/unbound-dhcp-leases-bridge | 28 +++++++++++++++-------- >>>>>>>>>> 1 file changed, 19 insertions(+), 9 deletions(-) >>>>>>>>>> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>>> index e9f022aff..d22772066 100644 >>>>>>>>>> --- a/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>>> +++ b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>>> @@ -22,6 +22,7 @@ >>>>>>>>>> import argparse >>>>>>>>>> import datetime >>>>>>>>>> import daemon >>>>>>>>>> +import filecmp >>>>>>>>>> import functools >>>>>>>>>> import ipaddress >>>>>>>>>> import logging >>>>>>>>>> @@ -516,26 +517,35 @@ class UnboundConfigWriter(object): >>>>>>>>>> def update_dhcp_leases(self, leases): >>>>>>>>>> # Write out all leases >>>>>>>>>> - self.write_dhcp_leases(leases) >>>>>>>>>> + if self.write_dhcp_leases(leases): >>>>>>>>>> - log.debug("Reloading Unbound...") >>>>>>>>>> + log.debug("Reloading Unbound...") >>>>>>>>>> - # Reload the configuration without dropping the cache >>>>>>>>>> - self._control("reload_keep_cache") >>>>>>>>>> + # Reload the configuration without dropping the cache >>>>>>>>>> + self._control("reload_keep_cache") >>>>>>>>>> + >>>>>>>>>> + else: >>>>>>>>>> + >>>>>>>>>> + log.debug("Not reloading Unbound. Leases not changed.") >>>>>>>>>> def write_dhcp_leases(self, leases): >>>>>>>>>> log.debug("Writing DHCP leases...") >>>>>>>>>> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >>>>>>>>>> - for l in leases: >>>>>>>>>> + for l in sorted(leases): >>>>>>>>>> for rr in l.rrset: >>>>>>>>>> f.write("local-data: "%s"\n" % " ".join(rr)) >>>>>>>>>> - # Make file readable for everyone >>>>>>>>>> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>>>> + filecmp.clear_cache() >>>>>>>>>> + RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) >>>>>>>>>> + >>>>>>>>>> + # Make file readable for everyone >>>>>>>>>> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>>>> + >>>>>>>>>> + # Move the file to its destination >>>>>>>>>> + os.rename(f.name, self.path) >>>>>>>>>> - # Move the file to its destination >>>>>>>>>> - os.rename(f.name, self.path) >>>>>>>>>> + return RequireUnboundReload >>>>>>>>>> def _control(self, *args): >>>>>>>>>> command = ["unbound-control"] >>>>>>>>>> >>>>>>>>>> >>>>>>> Resubmitting the patch. >>>>>>> >>>>>>> Following my previous reply, I would like to stick with filecmp as I am using it with "shallow=False" to avoid doing the comparison just with stat() calls. Note that I also flush the filecmp cache before using the command so it is running with the latest file data. Talking about file descriptors, atomic file replacement etc., scares me. >>>>>>> >>>>>>> >>>>>> File descriptors are great. They avoid race-conditions and all sorts of things. >>>>>> >>>>>> >>>>> But I have never used them so I don't intend to go that way. Too much for me to get my brain round and a limited ability to test. >>>>> >>>>> >>>> Well, I pretty much gave you the entire solution… There was nothing left to do apart from copying and pasting the snippets. >>>> >>>> >>>> >>> Ok I've tried to use your version but struggled to get it going. If I used it as was and called compare(file1_name, file2_name) it failed as "compare" was not defined. >>> I could call it using self.compare but then it failed with "TypeError: UnboundConfigWriter.compare() takes 2 positional arguments but 3 were given". I changed the function to compare(self, f1, f2). >>> Then it failed with "AttributeError: 'str' object has no attribute 'seek'", so I changed it again to accept file names then open the files before doing the seeks. >>> I really hope it is OK as I am out of my depth when defining functions with "self" as an argument. Please can you check it? If it is not OK, can I revert to filecmp? >>> >>> I have made one futher change, the sorted(leases) line was not working and not sorting the object ever, and I saw lines changing places in the resulting leases file. I have changed it to "sorted(leases, key=lambda x: x.ipaddr)". It definitely sorts this time. >>> >>> >>> >>>>>> I don’t think that module is the best choice really if you have to hack around and flush the cache. Why would it even cache things in the first place? I don’t see where the cache helps. >>>>>> >>>>>> >>>>> From the filecmp.clear_cache documentation: >>>>> >>>>> Clear the filecmp cache. This may be useful if a file is compared so >>>>> quickly after it is modified that it is within the mtime resolution >>>>> of the underlying filesystem. >>>>> >>>>> Now this makes me believe that the clear_cache is only relevant when using shallow=True, which I'm not and I don't know how to check the underlying code. It is used to make sure the stat() info is updated, but we are not using the stat() info. I guess we could as easily add a one second sleep instead as that is way longer than the mtime resolution. >>>>> >>>>> >>>> The function is here and it is rather simple: https://github.com/python/cpython/blob/3.12/Lib/filecmp.py#L30 >>>> >>>> It will always stat() both files and compares three items in case shallow is False: The type of the file, the size and mtime. If the file is not a regular file, the function will exist rather early. So in our case, that is all work the function is doing for thing. It boils down to something very similar to what I suggested to use to compare the files. >>>> >>>> On top of that, there is a cache which will always be used but will never return a match for us because the temporary file at least will always have another mtime. >>>> >>>> You could clear the cache after every call, but I would rather not have our code messy and care about implementation details of some Python module. >>>> >>>> >>>> >>>>> The module is chosen because all the references I read about comparing files preferred this one-liner for comparing files, although other methods were given and there were lots of discussions about large files (GB+) because if you chomp them it may be much quicker to say they are not the same when you find the first block that does not match. Having said this, I don't know how filecmp works, but we only have a small file. At the same time, I am not a regular programmer so simple one-liners are good for me. >>>>> >>>>> >>>> You would always want to read both files block for block and compare the blocks. There is no way to implement this significantly faster. >>>> >>>> >>>> >>>>>>> I have modified the earlier try/except I proposed to catch only FileNotFoundError, and I have changed CamelCase to snake_case. >>>>>>> >>>>>>> --- >>>>>>> config/unbound/unbound-dhcp-leases-bridge | 31 ++++++++++++++++------- >>>>>>> 1 file changed, 22 insertions(+), 9 deletions(-) >>>>>>> >>>>>>> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >>>>>>> index e9f022aff..a9ec8e1e2 100644 >>>>>>> --- a/config/unbound/unbound-dhcp-leases-bridge >>>>>>> +++ b/config/unbound/unbound-dhcp-leases-bridge >>>>>>> @@ -22,6 +22,7 @@ >>>>>>> import argparse >>>>>>> import datetime >>>>>>> import daemon >>>>>>> +import filecmp >>>>>>> import functools >>>>>>> import ipaddress >>>>>>> import logging >>>>>>> @@ -516,26 +517,38 @@ class UnboundConfigWriter(object): >>>>>>> >>>>>>> def update_dhcp_leases(self, leases): >>>>>>> # Write out all leases >>>>>>> - self.write_dhcp_leases(leases) >>>>>>> + if self.write_dhcp_leases(leases): >>>>>>> >>>>>>> - log.debug("Reloading Unbound...") >>>>>>> + log.debug("Reloading Unbound...") >>>>>>> >>>>>>> - # Reload the configuration without dropping the cache >>>>>>> - self._control("reload_keep_cache") >>>>>>> + # Reload the configuration without dropping the cache >>>>>>> + self._control("reload_keep_cache") >>>>>>> + >>>>>>> + else: >>>>>>> + >>>>>>> + log.debug("Not reloading Unbound. Leases not changed.") >>>>>>> >>>>>>> def write_dhcp_leases(self, leases): >>>>>>> log.debug("Writing DHCP leases...") >>>>>>> >>>>>>> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >>>>>>> - for l in leases: >>>>>>> + for l in sorted(leases): >>>>>>> for rr in l.rrset: >>>>>>> f.write("local-data: "%s"\n" % " ".join(rr)) >>>>>>> >>>>>>> - # Make file readable for everyone >>>>>>> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>> + filecmp.clear_cache() >>>>>>> + try: >>>>>>> + require_unbound_reload = not filecmp.cmp(f.name, self.path, shallow=False) >>>>>>> + except FileNotFoundError: >>>>>>> + require_unbound_reload = True >>>>>>> >>>>>>> >>>>>> If you want to go this route, you will have to flush the temporary file first. Otherwise you might have some data left in the buffer that has not been written (because the file is still open). You will then compare incomplete output. This would not become a problem if you used the file descriptors. >>>>>> >>>>>> >>>>> The temp file is closed at the point of doing the filecmp, and the filecmp cache has been flushed. How do I flush the temporary file further. Reading up, is it just having the last statement in the with block as f.flush() with the same indent as "for l in sorted(leases):" >>>>> >>>>> >>>> If you have left the with block you will already have flushed the buffers and closed the file. It looks like I misread the indentation in the patch then. >>>> >>>> >>>> >>>>>> Catching FileNotFound would work for me. >>>>>> >>>>>> >>>>> Good >>>>> >>>>> >>>>>>> + >>>>>>> + # Make file readable for everyone >>>>>>> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>> + >>>>>>> + # Move the file to its destination >>>>>>> + os.rename(f.name, self.path) >>>>>>> >>>>>>> - # Move the file to its destination >>>>>>> - os.rename(f.name, self.path) >>>>>>> + return require_unbound_reload >>>>>>> >>>>>>> def _control(self, *args): >>>>>>> command = ["unbound-control"] >>>>>>> -- >>>>>>> 2.43.0 >>>>>>> >>>>>>> >>>>>>> Nick >>>>>>> >>>>>>> >>> --- >>> config/unbound/unbound-dhcp-leases-bridge | 50 +++++++++++++++++++---- >>> 1 file changed, 41 insertions(+), 9 deletions(-) >>> >>> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >>> index e9f022aff..80d9523ea 100644 >>> --- a/config/unbound/unbound-dhcp-leases-bridge >>> +++ b/config/unbound/unbound-dhcp-leases-bridge >>> @@ -516,26 +516,37 @@ class UnboundConfigWriter(object): >>> >>> def update_dhcp_leases(self, leases): >>> # Write out all leases >>> - self.write_dhcp_leases(leases) >>> + if self.write_dhcp_leases(leases): >>> >>> - log.debug("Reloading Unbound...") >>> + log.debug("Reloading Unbound...") >>> >>> - # Reload the configuration without dropping the cache >>> - self._control("reload_keep_cache") >>> + # Reload the configuration without dropping the cache >>> + self._control("reload_keep_cache") >>> + >>> + else: >>> + >>> + log.debug("Not reloading Unbound. Leases not changed.") >>> >>> def write_dhcp_leases(self, leases): >>> log.debug("Writing DHCP leases...") >>> >>> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >>> - for l in leases: >>> + for l in sorted(leases, key=lambda x: x.ipaddr): >>> for rr in l.rrset: >>> f.write("local-data: "%s"\n" % " ".join(rr)) >>> >>> - # Make file readable for everyone >>> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>> + try: >>> + require_unbound_reload = not self.compare(f.name, self.path) >>> + except FileNotFoundError: >>> + require_unbound_reload = True >>> + >>> + # Make file readable for everyone >>> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>> + >>> + # Move the file to its destination >>> + os.rename(f.name, self.path) >>> >>> - # Move the file to its destination >>> - os.rename(f.name, self.path) >>> + return require_unbound_reload >>> >>> def _control(self, *args): >>> command = ["unbound-control"] >>> @@ -551,6 +562,27 @@ class UnboundConfigWriter(object): >>> >>> raise e >>> >>> + def compare(self, if1, if2): >>> + buffersize = 16 * 1024 >>> + >>> + f1 = open(if1, "r") >>> + f2 = open(if2, "r") >>> + >>> + # Reset file handles >>> + f1.seek(0) >>> + f2.seek(0) >>> + >>> + while True: >>> + b1 = f1.read(buffersize) >>> + b2 = f2.read(buffersize) >>> + >>> + if not b1 == b2: >>> + return False >>> + >>> + elif not b1 or not b2: >>> + break >>> + >>> + return True >>> >>> if __name__ == "__main__": >>> parser = argparse.ArgumentParser(description="Bridge for DHCP Leases and Unbound DNS") >>> >>>
Jon Murphy jon.murphy@ipfire.org
Just for reference, yes:
https://bugzilla.ipfire.org/show_bug.cgi?id=13254#c134
On 4 May 2024, at 03:16, jon jon.murphy@ipfire.org wrote:
To be honest, I kind of lost track where we are with that now.
Michael,
If you are referring to my question: Are you OK with moving forward using the DCHP on-commit, on-release, on-expiry?? (like the proof-of-concept)
Jon
On Apr 30, 2024, at 7:01 AM, Michael Tremer michael.tremer@ipfire.org wrote:
Hello Nick,
On 30 Apr 2024, at 12:58, Nick Howitt nick@howitts.co.uk wrote:
On 27/04/2024 10:53, Nick Howitt wrote:
On 27/04/2024 09:35, Michael Tremer wrote:
Hello Nick,
On 26 Apr 2024, at 17:48, Nick Howitt nick@howitts.co.uk wrote:
On 26/04/2024 16:12, Michael Tremer wrote:
> Hello Nick, > > > >> On 25 Apr 2024, at 13:43, Nick Howitt nick@howitts.co.uk wrote: >> >> Please can I bump this again to request a review of my patch submitted on 28th March? Thanks, Nick. >> >> > It is a little bit difficult to test your proposed changes if testers have to copy and paste them out of an email. > Confused. I was told I had to submit the diff in an email with [PATCH] in the title and not to attach a file. Please can you tell me how I should be operating?
No worries, we are going to sort this out.
So the first email in the thread was a correct patch. It was just the patch in the format that Git uses and it was properly parsed by Patchwork. Later in the same thread, the patch was just copied and pasted which was impossible for Patchwork to pick up. There is nothing wrong with using bits of code in the conversation, but it would have been nice to have the final version submitted once again.
> I have just submitted a patch set to this list that breaks down all the changes that you proposed and I cleaned up the code so that we can use the filecmp function without any extra steps. We can simply jump out of the write function and return False to prevent Unbound being reloaded. > I thought you didn't want me to use filecmp and pushed me to use your file compare routine.
Yeah, I would have preferred that too, but I thought using the module might be quicker now.
> We will also avoid replacing the file all the time as that causes lots of unnecessary disk I/O. I am not saying it makes a significant difference, but flash has a limited write count that we are still wasting. > OK, but my changes were no worse than before, either doing a delete or rename whereas before it always did a rename. The "unnecessary I/O" was already there.
> Please review and you if you are happy, please add the appropriate Git tag. > Sorry but I don't know what you mean by "adding the appropriate Git tag". I am only using Git locally and have no repo access.
I tested the patches yesterday and today and I am happy with them. I hope I have signed them off correctly.
Perfect. Thank you very much.
FWIW I hand patched the program as it was easier than learning something new in this case as the patch set is so small.
Haha, okay :)
One thing not mentioned in the bug report. Not only is unbound restarting on renewals which this fixes, it also restarts on new leases **and** expiring leases.
I think this is the correct way though, but I agree it creates extra work.
For big work environments, I would have thought the best workaround is to set the default DHCP lease to 24h. It would help during the week to lessen the restarts but not on Monday mornings. It may also need a bigger address pool to cope with devices randomising MAC addresses.
It depends on where you deploy this. In any office/home environment 24h is probably a good lease time. Maybe even 12 or 10 is a good option. I have rarely seen lease times of a week or so recently; in fact the default seems to be closer to an hour. That should however not be a problem for us any more because we don’t reload Unbound if a client simply renews their lease. Only if the join or leave the network.
Roll on Jon's fix as it would get round all these restarts. I use it modified not to update the dhcp-leases.conf. and another couple of tweaks. Are you able to test it, but it does mean piecing it together?
To be honest, I kind of lost track where we are with that now. I did look at a couple of the references from your last email, but didn’t make it very far, yet.
-Michael
Thanks.
How can I pull your changes? Otherwise I'll need to patch my set up manually.
You can apply the patch to your local repository using “git am”:
Thanks again.
I also have another problem. I have temporarily had to remove my IPF gateway for a business reason and don't know when I'll be able to add it back in. I'll have to see if I can insert it in my LAN somehow. At the same time it is also running Jon's mods which I'll have to undo.
With respect to Jon's mods, I've asked a couple of times on bugzilla, but is there any point in his routine maintaining /etc/unbound/dhcp-leases.conf. As far as I see the file is only read once when unbound starts, so it only has to be valid when unbound starts and not at any other point in time. To this effect he can remove the file maintenance and I attached a modified copy unbound-dhcp-leases-bridge to his repo which generates a new dhcp-leases.conf and exits, removing all the daemonising bit and the inode watcher. This file can be run as part of the unbound init routine just before the line that actually starts or reloads unbound. It also has the advantage of cutting down a load of disk I/O, which is important as you've just pointed out.
I don’t think that we can fully remove the daemon part, because I don’t think that there is a way to get a script (shell or otherwise) fast enough to perform the job. I also think that we benefit a lot from keeping some state in memory, whereas the script will have to parse the old file first, generate a new file and then compare the result. That will take a long time.
I had Jon's script down to 70ms after I removed the dhcp-leases file updates, used my fast method for deriving the domain name and removed Jon's initial setting up bit which is not needed once the changes are accepted. This is on a 2c VM running at 2GHz. I thought the goal was < 100ms which has been done comfortably.
FWIW, if you no longer maintain dhcp-leases in the Jon's script you never read the old file or do a compare. All you do is read the hosts and static-leases files and nothing is written to file.
Do we needed the leases file generated at startup? Yes, I think so, too. DNS is quite essential for the network and for local services to be able to start. So it has to come up very early. If it comes up with proper configuration we avoid any other problems where hosts might resolve names too early and get a different result compared to a couple of seconds later when the zone is fully initialised.
My cut-down version of unbound-dhcp-leases-bridge is at https://bugzilla.ipfire.org/attachment.cgi?id=1516 on https://bugzilla.ipfire.org/show_bug.cgi?id=13254. On my test system it takes 219ms to run so delays unbound starting up minimally.
At the same time there is a gain in the dhcp start up as the unbound-dhcp-leases-bridge is no longer needed and it therefore avoids triggering an unbound restart which happens every time unbound-dhcp-leases-bridge is run.
If we are generating dhcp-leases.conf at start up, then there is no point in maintaining it while unbound is running as the maintained file will never be read.
So all in all, I think the route I would most likely look at is changing the daemon to open a socket. We write minimal scripts which are called by dhcpd that simply pipe their data into the socket which the daemon will then parse. With all available leases in memory already, we can check if a lease has changed or if it has been renewed without any further changes. If nothing has changed, we don’t need to do anything. In case of a change, we will have to write the file again and have Unbound reload, or we even call unbound-control and make the one change without a reload. Doing that in the daemon should give us more flexibility and we avoid any performance penalties as the daemon could run multiple commands concurrently which dhcpd can’t do.
Sockets are beyond me, I am afraid.
Part of the goal of all this is to avoid reloading unbound and part of my goal is to avoid writing dhcp-leases.conf unless unbound is restarted. I am not sure why you are still mentioning writing a file and restarting/reloading unbound.
Also on his patches, I had a significantly faster way of determining the domain name (in bash) and he and I were looking for your feedback on if we'd be allowed to use it.
Anything is allowed. It just has to be fast. And for that, I believe, calling any other binary will be a problem.
OK. I was proposing to read the DOMAINNAME with: DOMAINNAME=${HOSTNAME#*.*} which is many times faster than the conventional route: eval $(/usr/local/bin/readhash /var/ipfire/main/settings) If there is any concern with bash not having the full environment when the program is called, the shebang could be changed to: #!/bin/bash -l Nick
-Michael
Regards,
Nick
> Best, > -Michael > > > >> On 10/04/2024 12:04, Nick Howitt wrote: >> >> >>> Bump >>> >>> On 28/03/2024 17:25, Nick Howitt wrote: >>> >>> >>>> Next version. >>>> >>>> On 28/03/2024 10:41, Michael Tremer wrote: >>>> >>>> >>>>> Hello, >>>>> >>>>> >>>>> >>>>>> On 27 Mar 2024, at 15:28, Nick Howitt nick@howitts.co.uk wrote: >>>>>> >>>>>> >>>>>> >>>>>> On 26/03/2024 11:10, Michael Tremer wrote: >>>>>> >>>>>> >>>>>>>> On 25 Mar 2024, at 13:11, Nick Howitt nick@howitts.co.uk wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 25/03/2024 11:31, Michael Tremer wrote: >>>>>>>> >>>>>>>> >>>>>>>>> Hello, >>>>>>>>> I don’t think that the risk is very high, but I would account for this problem. >>>>>>>>> With my proposed changes from my previous email this should not be a large problem at all. When opening /etc/unbound/dhcp-leases.conf with mode “r+” you can catch the Exception and skip the comparison and open the file again with “w”. >>>>>>>>> >>>>>>>>> >>>>>>>>>> On 25 Mar 2024, at 10:10, Nick Howitt nick@howitts.co.uk wrote: >>>>>>>>>> >>>>>>>>>> If there is a risk that the /etc/unbound/dhcp-leases.conf does not exist, for example on a clean install (I don't know), the filecmp.cmp line needs to be wrapped in a try...except to stop it blowing up: >>>>>>>>>> >>>>>>>>>> try: >>>>>>>>>> RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) >>>>>>>>>> except: >>>>>>>>>> RequireUnboundReload = True >>>>>>>>>> >>>>>>>>>> >>>>>>>>> I generally don’t like blanco exception catches. You will never know about the daemons that you are hiding. >>>>>>>>> >>>>>>>>> >>>>>>>>>> This will force the file to be to be generated, even if it is only copying in a blank temp file, and unbound will restart. It will only ever happen once. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 24/03/2024 15:49, Nick Howitt wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Hi all, >>>>>>>>>>> **** This is a repost of an email incorrectly submitted. **** >>>>>>>>>>> Please bear with me as I am new to IPFire and not at all used to this style of development, so any guidance would be appreciated. >>>>>>>>>>> There is a bug, https://bugzilla.ipfire.org/show_bug.cgi?id=13254, where unbound-dhcp-leases-bridge restarts unbound a lot when it is totally unnecessary. It appears to be because when a lease is renewed, the script gets triggered, created a new /etc/unbound/dhcp-leases.conf then restarts Unbound to read the file. Generally this seems to be unnecessary as, with a renewed lease, the old and new /etc/unbound/dhcp-leases.conf files are the same. With 5 leases in that file (one for a machine not active) I am getting 3-4 restarts an hour of Unbound when I have a min/max lease time of 60/120min. >>>>>>>>>>> Looking at the code, it is fairly easy to fix. The current code creates a temp file then copies it into place then restarts unbound. All I am doing is doing a file comparison before the copy and skipping the restart if the files are the same. >>>>>>>>>>> There were a couple of gotchas because setting the file attributes and copying the file were done inside the "with" block for generating the temporary file. This meant a file comparison always returned False as the temp file was still open and so never the same is the old file. >>>>>>>>>>> I moved those two statements outside the "with". This forced me to change the fchmod to chmod. >>>>>>>>>>> It could be argued that the file copy should not be done if the files are not different, but it would take an extra "if" and you'd have to remember to delete the temp file. If required, I can make that change. >>>>>>>>>>> Also, one small thing I noticed once is that the old and new dhcp-leases.conf files could occasionally contain the same leases but in a different order. I have been unable to reproduce, but to sidestep it, instead of stepping through the leases variable directly, I sort it and step through that. It should make the resulting file completely deterministic and make the file comparison more effective. >>>>>>>>>>> My patch is: >>>>>>>>>>>> From 73873d4944944a2f02317a73074a6894726f36f7 Mon Sep 17 00:00:00 2001 >>>>>>>>>>> From: Nick Howitt nick@howitts.co.uk >>>>>>>>>>> Date: Sun, 24 Mar 2024 15:17:19 +0000 >>>>>>>>>>> Subject: [PATCH] Stop unbound-dhcp-leases-bridge from restarting unbound every >>>>>>>>>>> time it write dhcp-leases.conf as it is very often unchanged and does not >>>>>>>>>>> require a restart. >>>>>>>>>>> --- >>>>>>>>>>> config/unbound/unbound-dhcp-leases-bridge | 28 +++++++++++++++-------- >>>>>>>>>>> 1 file changed, 19 insertions(+), 9 deletions(-) >>>>>>>>>>> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>>>> index e9f022aff..d22772066 100644 >>>>>>>>>>> --- a/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>>>> +++ b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>>>> @@ -22,6 +22,7 @@ >>>>>>>>>>> import argparse >>>>>>>>>>> import datetime >>>>>>>>>>> import daemon >>>>>>>>>>> +import filecmp >>>>>>>>>>> import functools >>>>>>>>>>> import ipaddress >>>>>>>>>>> import logging >>>>>>>>>>> @@ -516,26 +517,35 @@ class UnboundConfigWriter(object): >>>>>>>>>>> def update_dhcp_leases(self, leases): >>>>>>>>>>> # Write out all leases >>>>>>>>>>> - self.write_dhcp_leases(leases) >>>>>>>>>>> + if self.write_dhcp_leases(leases): >>>>>>>>>>> - log.debug("Reloading Unbound...") >>>>>>>>>>> + log.debug("Reloading Unbound...") >>>>>>>>>>> - # Reload the configuration without dropping the cache >>>>>>>>>>> - self._control("reload_keep_cache") >>>>>>>>>>> + # Reload the configuration without dropping the cache >>>>>>>>>>> + self._control("reload_keep_cache") >>>>>>>>>>> + >>>>>>>>>>> + else: >>>>>>>>>>> + >>>>>>>>>>> + log.debug("Not reloading Unbound. Leases not changed.") >>>>>>>>>>> def write_dhcp_leases(self, leases): >>>>>>>>>>> log.debug("Writing DHCP leases...") >>>>>>>>>>> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >>>>>>>>>>> - for l in leases: >>>>>>>>>>> + for l in sorted(leases): >>>>>>>>>>> for rr in l.rrset: >>>>>>>>>>> f.write("local-data: "%s"\n" % " ".join(rr)) >>>>>>>>>>> - # Make file readable for everyone >>>>>>>>>>> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>>>>> + filecmp.clear_cache() >>>>>>>>>>> + RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) >>>>>>>>>>> + >>>>>>>>>>> + # Make file readable for everyone >>>>>>>>>>> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>>>>> + >>>>>>>>>>> + # Move the file to its destination >>>>>>>>>>> + os.rename(f.name, self.path) >>>>>>>>>>> - # Move the file to its destination >>>>>>>>>>> - os.rename(f.name, self.path) >>>>>>>>>>> + return RequireUnboundReload >>>>>>>>>>> def _control(self, *args): >>>>>>>>>>> command = ["unbound-control"] >>>>>>>>>>> >>>>>>>>>>> >>>>>>>> Resubmitting the patch. >>>>>>>> >>>>>>>> Following my previous reply, I would like to stick with filecmp as I am using it with "shallow=False" to avoid doing the comparison just with stat() calls. Note that I also flush the filecmp cache before using the command so it is running with the latest file data. Talking about file descriptors, atomic file replacement etc., scares me. >>>>>>>> >>>>>>>> >>>>>>> File descriptors are great. They avoid race-conditions and all sorts of things. >>>>>>> >>>>>>> >>>>>> But I have never used them so I don't intend to go that way. Too much for me to get my brain round and a limited ability to test. >>>>>> >>>>>> >>>>> Well, I pretty much gave you the entire solution… There was nothing left to do apart from copying and pasting the snippets. >>>>> >>>>> >>>>> >>>> Ok I've tried to use your version but struggled to get it going. If I used it as was and called compare(file1_name, file2_name) it failed as "compare" was not defined. >>>> I could call it using self.compare but then it failed with "TypeError: UnboundConfigWriter.compare() takes 2 positional arguments but 3 were given". I changed the function to compare(self, f1, f2). >>>> Then it failed with "AttributeError: 'str' object has no attribute 'seek'", so I changed it again to accept file names then open the files before doing the seeks. >>>> I really hope it is OK as I am out of my depth when defining functions with "self" as an argument. Please can you check it? If it is not OK, can I revert to filecmp? >>>> >>>> I have made one futher change, the sorted(leases) line was not working and not sorting the object ever, and I saw lines changing places in the resulting leases file. I have changed it to "sorted(leases, key=lambda x: x.ipaddr)". It definitely sorts this time. >>>> >>>> >>>> >>>>>>> I don’t think that module is the best choice really if you have to hack around and flush the cache. Why would it even cache things in the first place? I don’t see where the cache helps. >>>>>>> >>>>>>> >>>>>> From the filecmp.clear_cache documentation: >>>>>> >>>>>> Clear the filecmp cache. This may be useful if a file is compared so >>>>>> quickly after it is modified that it is within the mtime resolution >>>>>> of the underlying filesystem. >>>>>> >>>>>> Now this makes me believe that the clear_cache is only relevant when using shallow=True, which I'm not and I don't know how to check the underlying code. It is used to make sure the stat() info is updated, but we are not using the stat() info. I guess we could as easily add a one second sleep instead as that is way longer than the mtime resolution. >>>>>> >>>>>> >>>>> The function is here and it is rather simple: https://github.com/python/cpython/blob/3.12/Lib/filecmp.py#L30 >>>>> >>>>> It will always stat() both files and compares three items in case shallow is False: The type of the file, the size and mtime. If the file is not a regular file, the function will exist rather early. So in our case, that is all work the function is doing for thing. It boils down to something very similar to what I suggested to use to compare the files. >>>>> >>>>> On top of that, there is a cache which will always be used but will never return a match for us because the temporary file at least will always have another mtime. >>>>> >>>>> You could clear the cache after every call, but I would rather not have our code messy and care about implementation details of some Python module. >>>>> >>>>> >>>>> >>>>>> The module is chosen because all the references I read about comparing files preferred this one-liner for comparing files, although other methods were given and there were lots of discussions about large files (GB+) because if you chomp them it may be much quicker to say they are not the same when you find the first block that does not match. Having said this, I don't know how filecmp works, but we only have a small file. At the same time, I am not a regular programmer so simple one-liners are good for me. >>>>>> >>>>>> >>>>> You would always want to read both files block for block and compare the blocks. There is no way to implement this significantly faster. >>>>> >>>>> >>>>> >>>>>>>> I have modified the earlier try/except I proposed to catch only FileNotFoundError, and I have changed CamelCase to snake_case. >>>>>>>> >>>>>>>> --- >>>>>>>> config/unbound/unbound-dhcp-leases-bridge | 31 ++++++++++++++++------- >>>>>>>> 1 file changed, 22 insertions(+), 9 deletions(-) >>>>>>>> >>>>>>>> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>> index e9f022aff..a9ec8e1e2 100644 >>>>>>>> --- a/config/unbound/unbound-dhcp-leases-bridge >>>>>>>> +++ b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>> @@ -22,6 +22,7 @@ >>>>>>>> import argparse >>>>>>>> import datetime >>>>>>>> import daemon >>>>>>>> +import filecmp >>>>>>>> import functools >>>>>>>> import ipaddress >>>>>>>> import logging >>>>>>>> @@ -516,26 +517,38 @@ class UnboundConfigWriter(object): >>>>>>>> >>>>>>>> def update_dhcp_leases(self, leases): >>>>>>>> # Write out all leases >>>>>>>> - self.write_dhcp_leases(leases) >>>>>>>> + if self.write_dhcp_leases(leases): >>>>>>>> >>>>>>>> - log.debug("Reloading Unbound...") >>>>>>>> + log.debug("Reloading Unbound...") >>>>>>>> >>>>>>>> - # Reload the configuration without dropping the cache >>>>>>>> - self._control("reload_keep_cache") >>>>>>>> + # Reload the configuration without dropping the cache >>>>>>>> + self._control("reload_keep_cache") >>>>>>>> + >>>>>>>> + else: >>>>>>>> + >>>>>>>> + log.debug("Not reloading Unbound. Leases not changed.") >>>>>>>> >>>>>>>> def write_dhcp_leases(self, leases): >>>>>>>> log.debug("Writing DHCP leases...") >>>>>>>> >>>>>>>> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >>>>>>>> - for l in leases: >>>>>>>> + for l in sorted(leases): >>>>>>>> for rr in l.rrset: >>>>>>>> f.write("local-data: "%s"\n" % " ".join(rr)) >>>>>>>> >>>>>>>> - # Make file readable for everyone >>>>>>>> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>> + filecmp.clear_cache() >>>>>>>> + try: >>>>>>>> + require_unbound_reload = not filecmp.cmp(f.name, self.path, shallow=False) >>>>>>>> + except FileNotFoundError: >>>>>>>> + require_unbound_reload = True >>>>>>>> >>>>>>>> >>>>>>> If you want to go this route, you will have to flush the temporary file first. Otherwise you might have some data left in the buffer that has not been written (because the file is still open). You will then compare incomplete output. This would not become a problem if you used the file descriptors. >>>>>>> >>>>>>> >>>>>> The temp file is closed at the point of doing the filecmp, and the filecmp cache has been flushed. How do I flush the temporary file further. Reading up, is it just having the last statement in the with block as f.flush() with the same indent as "for l in sorted(leases):" >>>>>> >>>>>> >>>>> If you have left the with block you will already have flushed the buffers and closed the file. It looks like I misread the indentation in the patch then. >>>>> >>>>> >>>>> >>>>>>> Catching FileNotFound would work for me. >>>>>>> >>>>>>> >>>>>> Good >>>>>> >>>>>> >>>>>>>> + >>>>>>>> + # Make file readable for everyone >>>>>>>> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>> + >>>>>>>> + # Move the file to its destination >>>>>>>> + os.rename(f.name, self.path) >>>>>>>> >>>>>>>> - # Move the file to its destination >>>>>>>> - os.rename(f.name, self.path) >>>>>>>> + return require_unbound_reload >>>>>>>> >>>>>>>> def _control(self, *args): >>>>>>>> command = ["unbound-control"] >>>>>>>> -- >>>>>>>> 2.43.0 >>>>>>>> >>>>>>>> >>>>>>>> Nick >>>>>>>> >>>>>>>> >>>> --- >>>> config/unbound/unbound-dhcp-leases-bridge | 50 +++++++++++++++++++---- >>>> 1 file changed, 41 insertions(+), 9 deletions(-) >>>> >>>> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >>>> index e9f022aff..80d9523ea 100644 >>>> --- a/config/unbound/unbound-dhcp-leases-bridge >>>> +++ b/config/unbound/unbound-dhcp-leases-bridge >>>> @@ -516,26 +516,37 @@ class UnboundConfigWriter(object): >>>> >>>> def update_dhcp_leases(self, leases): >>>> # Write out all leases >>>> - self.write_dhcp_leases(leases) >>>> + if self.write_dhcp_leases(leases): >>>> >>>> - log.debug("Reloading Unbound...") >>>> + log.debug("Reloading Unbound...") >>>> >>>> - # Reload the configuration without dropping the cache >>>> - self._control("reload_keep_cache") >>>> + # Reload the configuration without dropping the cache >>>> + self._control("reload_keep_cache") >>>> + >>>> + else: >>>> + >>>> + log.debug("Not reloading Unbound. Leases not changed.") >>>> >>>> def write_dhcp_leases(self, leases): >>>> log.debug("Writing DHCP leases...") >>>> >>>> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >>>> - for l in leases: >>>> + for l in sorted(leases, key=lambda x: x.ipaddr): >>>> for rr in l.rrset: >>>> f.write("local-data: "%s"\n" % " ".join(rr)) >>>> >>>> - # Make file readable for everyone >>>> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>> + try: >>>> + require_unbound_reload = not self.compare(f.name, self.path) >>>> + except FileNotFoundError: >>>> + require_unbound_reload = True >>>> + >>>> + # Make file readable for everyone >>>> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>> + >>>> + # Move the file to its destination >>>> + os.rename(f.name, self.path) >>>> >>>> - # Move the file to its destination >>>> - os.rename(f.name, self.path) >>>> + return require_unbound_reload >>>> >>>> def _control(self, *args): >>>> command = ["unbound-control"] >>>> @@ -551,6 +562,27 @@ class UnboundConfigWriter(object): >>>> >>>> raise e >>>> >>>> + def compare(self, if1, if2): >>>> + buffersize = 16 * 1024 >>>> + >>>> + f1 = open(if1, "r") >>>> + f2 = open(if2, "r") >>>> + >>>> + # Reset file handles >>>> + f1.seek(0) >>>> + f2.seek(0) >>>> + >>>> + while True: >>>> + b1 = f1.read(buffersize) >>>> + b2 = f2.read(buffersize) >>>> + >>>> + if not b1 == b2: >>>> + return False >>>> + >>>> + elif not b1 or not b2: >>>> + break >>>> + >>>> + return True >>>> >>>> if __name__ == "__main__": >>>> parser = argparse.ArgumentParser(description="Bridge for DHCP Leases and Unbound DNS") >>>> >>>>
Jon Murphy jon.murphy@ipfire.org
Concerning CU186, I may have stumbled across an issue. It took a few days for it to pop up. See the message log below from Jun 17 13:48:34 to the end. I experienced a flurry of Reloads and DNS stalled for ~ 15 minutes.
[root@ipfire ~] # grep --color "Reloading Unbound" /var/log/messages Jun 16 10:43:48 ipfire dhcp[3615]: Reloading Unbound... Jun 16 10:58:56 ipfire dhcp[3615]: Reloading Unbound... Jun 16 11:29:26 ipfire dhcp[3615]: Reloading Unbound... Jun 16 11:43:46 ipfire dhcp[3615]: Reloading Unbound... Jun 16 15:08:10 ipfire dhcp[3615]: Reloading Unbound... Jun 16 21:48:53 ipfire dhcp[3615]: Reloading Unbound... Jun 17 06:52:30 ipfire dhcp[3615]: Reloading Unbound... Jun 17 11:38:59 ipfire dhcp[3615]: Reloading Unbound... Jun 17 11:41:16 ipfire dhcp[3615]: Reloading Unbound... Jun 17 11:41:33 ipfire dhcp[3615]: Reloading Unbound... Jun 17 11:42:23 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:39:37 ipfire dhcp[3615]: Reloading Unbound...
Jun 17 13:48:34 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:48:39 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:49:47 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:50:47 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:52:00 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:52:43 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:53:43 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:54:43 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:55:49 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:56:55 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:58:07 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:58:44 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:59:44 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:00:44 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:01:52 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:02:52 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:04:10 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:04:53 ipfire dhcp[3615]: Reloading Unbound... [root@ipfire ~] #
It seems like a new "Reloading Unbound…" was arriving before the "start of service (unbound 1.19.3)" actually happened:
``` Jun 17 13:48:35 ipfire dhcpd: DHCPDISCOVER from 20:91:df:00:01:02 via blue0 Jun 17 13:48:36 ipfire dhcpd: DHCPOFFER on 192.168.65.138 to 20:91:df:00:01:02 (JiPhoneSE) via blue0 Jun 17 13:48:37 ipfire dhcpd: DHCPREQUEST for 192.168.65.138 (192.168.65.1) from 20:91:df:00:01:02 (JiPhoneSE) via blue0 Jun 17 13:48:37 ipfire dhcpd: DHCPACK on 192.168.65.138 to 20:91:df:00:01:02 (JiPhoneSE) via blue0 Jun 17 13:48:39 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:49:13 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:49:13 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0
Jun 17 13:49:36 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:49:36 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:49:36 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:49:36 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:49:36 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:49:36 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:49:42 ipfire dhcpd: DHCPREQUEST for 192.168.65.110 from a8:8f:d9:00:01:02 (iPhone-W) via blue0 Jun 17 13:49:42 ipfire dhcpd: DHCPACK on 192.168.65.110 to a8:8f:d9:00:01:02 (iPhone-W) via blue0 Jun 17 13:49:47 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:49:59 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:49:59 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:50:05 ipfire dhcpd: DHCPREQUEST for 192.168.65.103 from 00:07:a6:00:01:02 (LEVDS-Dimmer-7BCA) via blue0 Jun 17 13:50:05 ipfire dhcpd: DHCPACK on 192.168.65.103 to 00:07:a6:00:01:02 (LEVDS-Dimmer-7BCA) via blue0
Jun 17 13:50:37 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:50:37 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:50:37 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:50:37 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:50:37 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:50:37 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:50:44 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:50:44 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:50:47 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:51:30 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:51:30 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0
Jun 17 13:51:37 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:51:37 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:51:37 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:51:37 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:51:37 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:51:37 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:52:00 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:52:15 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:52:15 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0
Jun 17 13:52:37 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:52:37 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:52:37 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:52:37 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:52:37 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:52:37 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:52:43 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:52:44 ipfire dhcpd: DHCPREQUEST for 192.168.65.180 from 34:f3:9a:00:01:02 (deb12dell) via blue0 Jun 17 13:52:44 ipfire dhcpd: DHCPACK on 192.168.65.180 to 34:f3:9a:00:01:02 (deb12dell) via blue0 Jun 17 13:53:00 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:53:00 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0
Jun 17 13:53:38 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:53:38 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:53:38 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:53:38 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:53:38 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:53:38 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:53:43 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:53:46 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:53:46 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:54:31 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:54:31 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0
Jun 17 13:54:38 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:54:38 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:54:38 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:54:38 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:54:38 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:54:38 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:54:43 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:55:19 ipfire dhcpd: DHCPREQUEST for 192.168.65.10 from 74:ac:b9:00:01:02 via blue0 Jun 17 13:55:19 ipfire dhcpd: DHCPACK on 192.168.65.10 to 74:ac:b9:00:01:02 via blue0
Jun 17 13:55:22 ipfire dhcpd: DHCPDISCOVER from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:55:22 ipfire dhcpd: DHCPOFFER on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:55:22 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 (192.168.65.1) from 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:55:22 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0
Jun 17 13:55:38 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:55:38 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:55:38 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:55:38 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:55:38 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:55:38 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:55:49 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:55:55 ipfire dhcpd: DHCPREQUEST for 192.168.65.105 from 44:61:32:00:01:02 (ecobee) via blue0 Jun 17 13:55:55 ipfire dhcpd: DHCPACK on 192.168.65.105 to 44:61:32:00:01:02 (ecobee) via blue0 Jun 17 13:55:58 ipfire dhcpd: DHCPREQUEST for 192.168.60.250 from 9c:93:4e:00:01:02 via green0 Jun 17 13:55:58 ipfire dhcpd: DHCPACK on 192.168.60.250 to 9c:93:4e:00:01:02 via green0 Jun 17 13:56:02 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:56:02 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0
Jun 17 13:56:38 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:56:38 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:56:38 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:56:38 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:56:38 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:56:38 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:56:53 ipfire dhcpd: DHCPDISCOVER from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:56:53 ipfire dhcpd: DHCPOFFER on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:56:53 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 (192.168.65.1) from 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:56:53 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:56:55 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:57:33 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:57:33 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:57:34 ipfire dhcpd: DHCPREQUEST for 192.168.60.240 from 8c:dc:d4:00:01:02 via green0 Jun 17 13:57:34 ipfire dhcpd: DHCPACK on 192.168.60.240 to 8c:dc:d4:00:01:02 via green0
Jun 17 13:57:39 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:57:39 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:57:39 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:57:39 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:57:39 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:57:39 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:58:03 ipfire dhcpd: DHCPREQUEST for 192.168.65.160 from 14:88:e6:00:01:02 (SiPad) via blue0 Jun 17 13:58:03 ipfire dhcpd: DHCPACK on 192.168.65.160 to 14:88:e6:00:01:02 (SiPad) via blue0 Jun 17 13:58:07 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:58:08 ipfire dhcpd: DHCPREQUEST for 192.168.60.203 from dc:a6:32:00:01:02 via green0 Jun 17 13:58:08 ipfire dhcpd: DHCPACK on 192.168.60.203 to dc:a6:32:00:01:02 via green0 Jun 17 13:58:19 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:58:19 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0
Jun 17 13:58:39 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:58:39 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:58:39 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:58:39 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:58:39 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:58:39 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:58:44 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:59:04 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:59:04 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0
Jun 17 13:59:39 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:59:39 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:59:39 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:59:39 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:59:39 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:59:39 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:59:44 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:59:50 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:59:50 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 14:00:35 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 14:00:35 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0
Jun 17 14:00:39 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 14:00:39 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 14:00:39 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 14:00:39 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 14:00:39 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 14:00:39 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 14:00:44 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:01:21 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 14:01:21 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0
Jun 17 14:01:46 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 14:01:46 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 14:01:46 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 14:01:47 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 14:01:47 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 14:01:47 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 14:01:52 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:02:06 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 14:02:06 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0
Jun 17 14:02:47 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 14:02:47 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 14:02:47 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 14:02:47 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 14:02:47 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 14:02:47 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 14:02:51 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 14:02:52 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 14:02:52 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:03:19 ipfire dhcpd: DHCPDISCOVER from 5e:24:53:00:01:02 via blue0 Jun 17 14:03:20 ipfire dhcpd: DHCPOFFER on 192.168.65.153 to 5e:24:53:00:01:02 via blue0 Jun 17 14:03:21 ipfire dhcpd: DHCPREQUEST for 192.168.65.153 (192.168.65.1) from 5e:24:53:00:01:02 via blue0 Jun 17 14:03:21 ipfire dhcpd: DHCPACK on 192.168.65.153 to 5e:24:53:00:01:02 via blue0 Jun 17 14:03:38 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 14:03:38 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 14:03:44 ipfire dhcpd: DHCPREQUEST for 192.168.65.109 from ac:c9:06:00:01:02 (MBA1) via blue0 Jun 17 14:03:44 ipfire dhcpd: DHCPACK on 192.168.65.109 to ac:c9:06:00:01:02 (MBA1) via blue0
Jun 17 14:03:47 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 14:03:47 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 14:03:47 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 14:03:47 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 14:03:47 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 14:03:47 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 14:04:02 ipfire dhcpd: DHCPREQUEST for 192.168.65.109 from ac:c9:06:00:01:02 (MBA1) via blue0 Jun 17 14:04:02 ipfire dhcpd: DHCPACK on 192.168.65.109 to ac:c9:06:00:01:02 (MBA1) via blue0 Jun 17 14:04:10 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:04:23 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 14:04:23 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0
Jun 17 14:04:48 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 14:04:48 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 14:04:48 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 14:04:48 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 14:04:48 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 14:04:48 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 14:04:53 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:05:41 ipfire dhcpd: DHCPREQUEST for 192.168.65.106 from 40:c7:11:00:01:02 (SiPhone12) via blue0 Jun 17 14:05:41 ipfire dhcpd: DHCPACK on 192.168.65.106 to 40:c7:11:00:01:02 (SiPhone12) via blue0
Jun 17 14:05:49 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 14:05:49 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 14:05:49 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 14:05:49 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 14:05:49 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 14:05:49 ipfire unbound: [2118:0] notice: init module 2: iterator
Jun 17 14:06:50 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 14:06:56 ipfire dhcpd: DHCPREQUEST for 192.168.65.109 from ac:c9:06:00:01:02 (MBA1) via blue0 Jun 17 14:06:56 ipfire dhcpd: DHCPACK on 192.168.65.109 to ac:c9:06:00:01:02 (MBA1) via blue0 Jun 17 14:07:01 ipfire dhcpd: DHCPREQUEST for 192.168.65.108 from 04:c2:9b:00:01:02 (Aura-9413) via blue0 Jun 17 14:07:01 ipfire dhcpd: DHCPACK on 192.168.65.108 to 04:c2:9b:00:01:02 (Aura-9413) via blue0 Jun 17 14:10:02 ipfire dhcpd: DHCPREQUEST for 192.168.60.20 from 0c:4d:e9:00:01:02 via green0 Jun 17 14:10:02 ipfire dhcpd: DHCPACK on 192.168.60.20 to 0c:4d:e9:00:01:02 via green0
Things seem to return to normal after that….
Jon
On Apr 30, 2024, at 5:58 AM, Nick Howitt <nick@howitts.co.uk mailto:nick@howitts.co.uk> wrote:
On 27/04/2024 10:53, Nick Howitt wrote:
On 27/04/2024 09:35, Michael Tremer wrote:
Hello Nick,
On 26 Apr 2024, at 17:48, Nick Howitt nick@howitts.co.uk mailto:nick@howitts.co.uk wrote:
On 26/04/2024 16:12, Michael Tremer wrote:
Hello Nick,
On 25 Apr 2024, at 13:43, Nick Howitt nick@howitts.co.uk mailto:nick@howitts.co.uk wrote:
Please can I bump this again to request a review of my patch submitted on 28th March? Thanks, Nick.
It is a little bit difficult to test your proposed changes if testers have to copy and paste them out of an email.
Confused. I was told I had to submit the diff in an email with [PATCH] in the title and not to attach a file. Please can you tell me how I should be operating?
No worries, we are going to sort this out.
So the first email in the thread was a correct patch. It was just the patch in the format that Git uses and it was properly parsed by Patchwork. Later in the same thread, the patch was just copied and pasted which was impossible for Patchwork to pick up. There is nothing wrong with using bits of code in the conversation, but it would have been nice to have the final version submitted once again.
I have just submitted a patch set to this list that breaks down all the changes that you proposed and I cleaned up the code so that we can use the filecmp function without any extra steps. We can simply jump out of the write function and return False to prevent Unbound being reloaded.
I thought you didn't want me to use filecmp and pushed me to use your file compare routine.
Yeah, I would have preferred that too, but I thought using the module might be quicker now.
We will also avoid replacing the file all the time as that causes lots of unnecessary disk I/O. I am not saying it makes a significant difference, but flash has a limited write count that we are still wasting.
OK, but my changes were no worse than before, either doing a delete or rename whereas before it always did a rename. The "unnecessary I/O" was already there.
Please review and you if you are happy, please add the appropriate Git tag.
Sorry but I don't know what you mean by "adding the appropriate Git tag". I am only using Git locally and have no repo access.
https://www.ipfire.org/docs/devel/git/tags https://www.ipfire.org/docs/devel/git/tags
I tested the patches yesterday and today and I am happy with them. I hope I have signed them off correctly.
FWIW I hand patched the program as it was easier than learning something new in this case as the patch set is so small.
One thing not mentioned in the bug report. Not only is unbound restarting on renewals which this fixes, it also restarts on new leases **and** expiring leases. For big work environments, I would have thought the best workaround is to set the default DHCP lease to 24h. It would help during the week to lessen the restarts but not on Monday mornings. It may also need a bigger address pool to cope with devices randomising MAC addresses.
Roll on Jon's fix as it would get round all these restarts. I use it modified not to update the dhcp-leases.conf. and another couple of tweaks. Are you able to test it, but it does mean piecing it together?
Thanks.
How can I pull your changes? Otherwise I'll need to patch my set up manually.
You can apply the patch to your local repository using “git am”:
https://git-scm.com/docs/git-am https://git-scm.com/docs/git-amThanks again.
I also have another problem. I have temporarily had to remove my IPF gateway for a business reason and don't know when I'll be able to add it back in. I'll have to see if I can insert it in my LAN somehow. At the same time it is also running Jon's mods which I'll have to undo.
With respect to Jon's mods, I've asked a couple of times on bugzilla, but is there any point in his routine maintaining /etc/unbound/dhcp-leases.conf. As far as I see the file is only read once when unbound starts, so it only has to be valid when unbound starts and not at any other point in time. To this effect he can remove the file maintenance and I attached a modified copy unbound-dhcp-leases-bridge to his repo which generates a new dhcp-leases.conf and exits, removing all the daemonising bit and the inode watcher. This file can be run as part of the unbound init routine just before the line that actually starts or reloads unbound. It also has the advantage of cutting down a load of disk I/O, which is important as you've just pointed out.
I don’t think that we can fully remove the daemon part, because I don’t think that there is a way to get a script (shell or otherwise) fast enough to perform the job. I also think that we benefit a lot from keeping some state in memory, whereas the script will have to parse the old file first, generate a new file and then compare the result. That will take a long time.
I had Jon's script down to 70ms after I removed the dhcp-leases file updates, used my fast method for deriving the domain name and removed Jon's initial setting up bit which is not needed once the changes are accepted. This is on a 2c VM running at 2GHz. I thought the goal was < 100ms which has been done comfortably.
FWIW, if you no longer maintain dhcp-leases in the Jon's script you never read the old file or do a compare. All you do is read the hosts and static-leases files and nothing is written to file.
Do we needed the leases file generated at startup? Yes, I think so, too. DNS is quite essential for the network and for local services to be able to start. So it has to come up very early. If it comes up with proper configuration we avoid any other problems where hosts might resolve names too early and get a different result compared to a couple of seconds later when the zone is fully initialised.
My cut-down version of unbound-dhcp-leases-bridge is at https://bugzilla.ipfire.org/attachment.cgi?id=1516 https://bugzilla.ipfire.org/attachment.cgi?id=1516 on https://bugzilla.ipfire.org/show_bug.cgi?id=13254 https://bugzilla.ipfire.org/show_bug.cgi?id=13254. On my test system it takes 219ms to run so delays unbound starting up minimally.
At the same time there is a gain in the dhcp start up as the unbound-dhcp-leases-bridge is no longer needed and it therefore avoids triggering an unbound restart which happens every time unbound-dhcp-leases-bridge is run.
If we are generating dhcp-leases.conf at start up, then there is no point in maintaining it while unbound is running as the maintained file will never be read.
So all in all, I think the route I would most likely look at is changing the daemon to open a socket. We write minimal scripts which are called by dhcpd that simply pipe their data into the socket which the daemon will then parse. With all available leases in memory already, we can check if a lease has changed or if it has been renewed without any further changes. If nothing has changed, we don’t need to do anything. In case of a change, we will have to write the file again and have Unbound reload, or we even call unbound-control and make the one change without a reload. Doing that in the daemon should give us more flexibility and we avoid any performance penalties as the daemon could run multiple commands concurrently which dhcpd can’t do.
Sockets are beyond me, I am afraid.
Part of the goal of all this is to avoid reloading unbound and part of my goal is to avoid writing dhcp-leases.conf unless unbound is restarted. I am not sure why you are still mentioning writing a file and restarting/reloading unbound.
Also on his patches, I had a significantly faster way of determining the domain name (in bash) and he and I were looking for your feedback on if we'd be allowed to use it.
Anything is allowed. It just has to be fast. And for that, I believe, calling any other binary will be a problem.
OK. I was proposing to read the DOMAINNAME with: DOMAINNAME=${HOSTNAME#*.*} which is many times faster than the conventional route: eval $(/usr/local/bin/readhash /var/ipfire/main/settings) If there is any concern with bash not having the full environment when the program is called, the shebang could be changed to: #!/bin/bash -l Nick
-Michael
Regards,
Nick
Best, -Michael
On 10/04/2024 12:04, Nick Howitt wrote:
> Bump > > On 28/03/2024 17:25, Nick Howitt wrote: > >> Next version. >> >> On 28/03/2024 10:41, Michael Tremer wrote: >> >>> Hello, >>> >>> >>>> On 27 Mar 2024, at 15:28, Nick Howitt nick@howitts.co.uk mailto:nick@howitts.co.uk wrote: >>>> >>>> >>>> >>>> On 26/03/2024 11:10, Michael Tremer wrote: >>>> >>>>>> On 25 Mar 2024, at 13:11, Nick Howitt nick@howitts.co.uk mailto:nick@howitts.co.uk wrote: >>>>>> >>>>>> >>>>>> >>>>>> On 25/03/2024 11:31, Michael Tremer wrote: >>>>>> >>>>>>> Hello, >>>>>>> I don’t think that the risk is very high, but I would account for this problem. >>>>>>> With my proposed changes from my previous email this should not be a large problem at all. When opening /etc/unbound/dhcp-leases.conf with mode “r+” you can catch the Exception and skip the comparison and open the file again with “w”. >>>>>>> >>>>>>>> On 25 Mar 2024, at 10:10, Nick Howitt nick@howitts.co.uk mailto:nick@howitts.co.uk wrote: >>>>>>>> >>>>>>>> If there is a risk that the /etc/unbound/dhcp-leases.conf does not exist, for example on a clean install (I don't know), the filecmp.cmp line needs to be wrapped in a try...except to stop it blowing up: >>>>>>>> >>>>>>>> try: >>>>>>>> RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) >>>>>>>> except: >>>>>>>> RequireUnboundReload = True >>>>>>>> >>>>>>> I generally don’t like blanco exception catches. You will never know about the daemons that you are hiding. >>>>>>> >>>>>>>> This will force the file to be to be generated, even if it is only copying in a blank temp file, and unbound will restart. It will only ever happen once. >>>>>>>> >>>>>>>> >>>>>>>> On 24/03/2024 15:49, Nick Howitt wrote: >>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> **** This is a repost of an email incorrectly submitted. **** >>>>>>>>> Please bear with me as I am new to IPFire and not at all used to this style of development, so any guidance would be appreciated. >>>>>>>>> There is a bug, https://bugzilla.ipfire.org/show_bug.cgi?id=13254 https://bugzilla.ipfire.org/show_bug.cgi?id=13254, where unbound-dhcp-leases-bridge restarts unbound a lot when it is totally unnecessary. It appears to be because when a lease is renewed, the script gets triggered, created a new /etc/unbound/dhcp-leases.conf then restarts Unbound to read the file. Generally this seems to be unnecessary as, with a renewed lease, the old and new /etc/unbound/dhcp-leases.conf files are the same. With 5 leases in that file (one for a machine not active) I am getting 3-4 restarts an hour of Unbound when I have a min/max lease time of 60/120min. >>>>>>>>> Looking at the code, it is fairly easy to fix. The current code creates a temp file then copies it into place then restarts unbound. All I am doing is doing a file comparison before the copy and skipping the restart if the files are the same. >>>>>>>>> There were a couple of gotchas because setting the file attributes and copying the file were done inside the "with" block for generating the temporary file. This meant a file comparison always returned False as the temp file was still open and so never the same is the old file. >>>>>>>>> I moved those two statements outside the "with". This forced me to change the fchmod to chmod. >>>>>>>>> It could be argued that the file copy should not be done if the files are not different, but it would take an extra "if" and you'd have to remember to delete the temp file. If required, I can make that change. >>>>>>>>> Also, one small thing I noticed once is that the old and new dhcp-leases.conf files could occasionally contain the same leases but in a different order. I have been unable to reproduce, but to sidestep it, instead of stepping through the leases variable directly, I sort it and step through that. It should make the resulting file completely deterministic and make the file comparison more effective. >>>>>>>>> My patch is: >>>>>>>>> >From 73873d4944944a2f02317a73074a6894726f36f7 Mon Sep 17 00:00:00 2001 >>>>>>>>> From: Nick Howitt nick@howitts.co.uk mailto:nick@howitts.co.uk >>>>>>>>> Date: Sun, 24 Mar 2024 15:17:19 +0000 >>>>>>>>> Subject: [PATCH] Stop unbound-dhcp-leases-bridge from restarting unbound every >>>>>>>>> time it write dhcp-leases.conf as it is very often unchanged and does not >>>>>>>>> require a restart. >>>>>>>>> --- >>>>>>>>> config/unbound/unbound-dhcp-leases-bridge | 28 +++++++++++++++-------- >>>>>>>>> 1 file changed, 19 insertions(+), 9 deletions(-) >>>>>>>>> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>> index e9f022aff..d22772066 100644 >>>>>>>>> --- a/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>> +++ b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>> @@ -22,6 +22,7 @@ >>>>>>>>> import argparse >>>>>>>>> import datetime >>>>>>>>> import daemon >>>>>>>>> +import filecmp >>>>>>>>> import functools >>>>>>>>> import ipaddress >>>>>>>>> import logging >>>>>>>>> @@ -516,26 +517,35 @@ class UnboundConfigWriter(object): >>>>>>>>> def update_dhcp_leases(self, leases): >>>>>>>>> # Write out all leases >>>>>>>>> - self.write_dhcp_leases(leases) >>>>>>>>> + if self.write_dhcp_leases(leases): >>>>>>>>> - log.debug("Reloading Unbound...") >>>>>>>>> + log.debug("Reloading Unbound...") >>>>>>>>> - # Reload the configuration without dropping the cache >>>>>>>>> - self._control("reload_keep_cache") >>>>>>>>> + # Reload the configuration without dropping the cache >>>>>>>>> + self._control("reload_keep_cache") >>>>>>>>> + >>>>>>>>> + else: >>>>>>>>> + >>>>>>>>> + log.debug("Not reloading Unbound. Leases not changed.") >>>>>>>>> def write_dhcp_leases(self, leases): >>>>>>>>> log.debug("Writing DHCP leases...") >>>>>>>>> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >>>>>>>>> - for l in leases: >>>>>>>>> + for l in sorted(leases): >>>>>>>>> for rr in l.rrset: >>>>>>>>> f.write("local-data: "%s"\n" % " ".join(rr)) >>>>>>>>> - # Make file readable for everyone >>>>>>>>> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>>> + filecmp.clear_cache() >>>>>>>>> + RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) >>>>>>>>> + >>>>>>>>> + # Make file readable for everyone >>>>>>>>> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>>> + >>>>>>>>> + # Move the file to its destination >>>>>>>>> + os.rename(f.name, self.path) >>>>>>>>> - # Move the file to its destination >>>>>>>>> - os.rename(f.name, self.path) >>>>>>>>> + return RequireUnboundReload >>>>>>>>> def _control(self, *args): >>>>>>>>> command = ["unbound-control"] >>>>>>>>> >>>>>> Resubmitting the patch. >>>>>> >>>>>> Following my previous reply, I would like to stick with filecmp as I am using it with "shallow=False" to avoid doing the comparison just with stat() calls. Note that I also flush the filecmp cache before using the command so it is running with the latest file data. Talking about file descriptors, atomic file replacement etc., scares me. >>>>>> >>>>> File descriptors are great. They avoid race-conditions and all sorts of things. >>>>> >>>> But I have never used them so I don't intend to go that way. Too much for me to get my brain round and a limited ability to test. >>>> >>> Well, I pretty much gave you the entire solution… There was nothing left to do apart from copying and pasting the snippets. >>> >>> >> Ok I've tried to use your version but struggled to get it going. If I used it as was and called compare(file1_name, file2_name) it failed as "compare" was not defined. >> I could call it using self.compare but then it failed with "TypeError: UnboundConfigWriter.compare() takes 2 positional arguments but 3 were given". I changed the function to compare(self, f1, f2). >> Then it failed with "AttributeError: 'str' object has no attribute 'seek'", so I changed it again to accept file names then open the files before doing the seeks. >> I really hope it is OK as I am out of my depth when defining functions with "self" as an argument. Please can you check it? If it is not OK, can I revert to filecmp? >> >> I have made one futher change, the sorted(leases) line was not working and not sorting the object ever, and I saw lines changing places in the resulting leases file. I have changed it to "sorted(leases, key=lambda x: x.ipaddr)". It definitely sorts this time. >> >> >>>>> I don’t think that module is the best choice really if you have to hack around and flush the cache. Why would it even cache things in the first place? I don’t see where the cache helps. >>>>> >>>> From the filecmp.clear_cache documentation: >>>> >>>> Clear the filecmp cache. This may be useful if a file is compared so >>>> quickly after it is modified that it is within the mtime resolution >>>> of the underlying filesystem. >>>> >>>> Now this makes me believe that the clear_cache is only relevant when using shallow=True, which I'm not and I don't know how to check the underlying code. It is used to make sure the stat() info is updated, but we are not using the stat() info. I guess we could as easily add a one second sleep instead as that is way longer than the mtime resolution. >>>> >>> The function is here and it is rather simple: https://github.com/python/cpython/blob/3.12/Lib/filecmp.py#L30 https://github.com/python/cpython/blob/3.12/Lib/filecmp.py#L30 >>> >>> It will always stat() both files and compares three items in case shallow is False: The type of the file, the size and mtime. If the file is not a regular file, the function will exist rather early. So in our case, that is all work the function is doing for thing. It boils down to something very similar to what I suggested to use to compare the files. >>> >>> On top of that, there is a cache which will always be used but will never return a match for us because the temporary file at least will always have another mtime. >>> >>> You could clear the cache after every call, but I would rather not have our code messy and care about implementation details of some Python module. >>> >>> >>>> The module is chosen because all the references I read about comparing files preferred this one-liner for comparing files, although other methods were given and there were lots of discussions about large files (GB+) because if you chomp them it may be much quicker to say they are not the same when you find the first block that does not match. Having said this, I don't know how filecmp works, but we only have a small file. At the same time, I am not a regular programmer so simple one-liners are good for me. >>>> >>> You would always want to read both files block for block and compare the blocks. There is no way to implement this significantly faster. >>> >>> >>>>>> I have modified the earlier try/except I proposed to catch only FileNotFoundError, and I have changed CamelCase to snake_case. >>>>>> >>>>>> --- >>>>>> config/unbound/unbound-dhcp-leases-bridge | 31 ++++++++++++++++------- >>>>>> 1 file changed, 22 insertions(+), 9 deletions(-) >>>>>> >>>>>> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >>>>>> index e9f022aff..a9ec8e1e2 100644 >>>>>> --- a/config/unbound/unbound-dhcp-leases-bridge >>>>>> +++ b/config/unbound/unbound-dhcp-leases-bridge >>>>>> @@ -22,6 +22,7 @@ >>>>>> import argparse >>>>>> import datetime >>>>>> import daemon >>>>>> +import filecmp >>>>>> import functools >>>>>> import ipaddress >>>>>> import logging >>>>>> @@ -516,26 +517,38 @@ class UnboundConfigWriter(object): >>>>>> >>>>>> def update_dhcp_leases(self, leases): >>>>>> # Write out all leases >>>>>> - self.write_dhcp_leases(leases) >>>>>> + if self.write_dhcp_leases(leases): >>>>>> >>>>>> - log.debug("Reloading Unbound...") >>>>>> + log.debug("Reloading Unbound...") >>>>>> >>>>>> - # Reload the configuration without dropping the cache >>>>>> - self._control("reload_keep_cache") >>>>>> + # Reload the configuration without dropping the cache >>>>>> + self._control("reload_keep_cache") >>>>>> + >>>>>> + else: >>>>>> + >>>>>> + log.debug("Not reloading Unbound. Leases not changed.") >>>>>> >>>>>> def write_dhcp_leases(self, leases): >>>>>> log.debug("Writing DHCP leases...") >>>>>> >>>>>> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >>>>>> - for l in leases: >>>>>> + for l in sorted(leases): >>>>>> for rr in l.rrset: >>>>>> f.write("local-data: "%s"\n" % " ".join(rr)) >>>>>> >>>>>> - # Make file readable for everyone >>>>>> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>> + filecmp.clear_cache() >>>>>> + try: >>>>>> + require_unbound_reload = not filecmp.cmp(f.name, self.path, shallow=False) >>>>>> + except FileNotFoundError: >>>>>> + require_unbound_reload = True >>>>>> >>>>> If you want to go this route, you will have to flush the temporary file first. Otherwise you might have some data left in the buffer that has not been written (because the file is still open). You will then compare incomplete output. This would not become a problem if you used the file descriptors. >>>>> >>>> The temp file is closed at the point of doing the filecmp, and the filecmp cache has been flushed. How do I flush the temporary file further. Reading up, is it just having the last statement in the with block as f.flush() with the same indent as "for l in sorted(leases):" >>>> >>> If you have left the with block you will already have flushed the buffers and closed the file. It looks like I misread the indentation in the patch then. >>> >>> >>>>> Catching FileNotFound would work for me. >>>>> >>>> Good >>>> >>>>>> + >>>>>> + # Make file readable for everyone >>>>>> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>> + >>>>>> + # Move the file to its destination >>>>>> + os.rename(f.name, self.path) >>>>>> >>>>>> - # Move the file to its destination >>>>>> - os.rename(f.name, self.path) >>>>>> + return require_unbound_reload >>>>>> >>>>>> def _control(self, *args): >>>>>> command = ["unbound-control"] >>>>>> -- >>>>>> 2.43.0 >>>>>> >>>>>> >>>>>> Nick >>>>>> >> --- >> config/unbound/unbound-dhcp-leases-bridge | 50 +++++++++++++++++++---- >> 1 file changed, 41 insertions(+), 9 deletions(-) >> >> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >> index e9f022aff..80d9523ea 100644 >> --- a/config/unbound/unbound-dhcp-leases-bridge >> +++ b/config/unbound/unbound-dhcp-leases-bridge >> @@ -516,26 +516,37 @@ class UnboundConfigWriter(object): >> >> def update_dhcp_leases(self, leases): >> # Write out all leases >> - self.write_dhcp_leases(leases) >> + if self.write_dhcp_leases(leases): >> >> - log.debug("Reloading Unbound...") >> + log.debug("Reloading Unbound...") >> >> - # Reload the configuration without dropping the cache >> - self._control("reload_keep_cache") >> + # Reload the configuration without dropping the cache >> + self._control("reload_keep_cache") >> + >> + else: >> + >> + log.debug("Not reloading Unbound. Leases not changed.") >> >> def write_dhcp_leases(self, leases): >> log.debug("Writing DHCP leases...") >> >> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >> - for l in leases: >> + for l in sorted(leases, key=lambda x: x.ipaddr): >> for rr in l.rrset: >> f.write("local-data: "%s"\n" % " ".join(rr)) >> >> - # Make file readable for everyone >> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >> + try: >> + require_unbound_reload = not self.compare(f.name, self.path) >> + except FileNotFoundError: >> + require_unbound_reload = True >> + >> + # Make file readable for everyone >> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >> + >> + # Move the file to its destination >> + os.rename(f.name, self.path) >> >> - # Move the file to its destination >> - os.rename(f.name, self.path) >> + return require_unbound_reload >> >> def _control(self, *args): >> command = ["unbound-control"] >> @@ -551,6 +562,27 @@ class UnboundConfigWriter(object): >> >> raise e >> >> + def compare(self, if1, if2): >> + buffersize = 16 * 1024 >> + >> + f1 = open(if1, "r") >> + f2 = open(if2, "r") >> + >> + # Reset file handles >> + f1.seek(0) >> + f2.seek(0) >> + >> + while True: >> + b1 = f1.read(buffersize) >> + b2 = f2.read(buffersize) >> + >> + if not b1 == b2: >> + return False >> + >> + elif not b1 or not b2: >> + break >> + >> + return True >> >> if __name__ == "__main__": >> parser = argparse.ArgumentParser(description="Bridge for DHCP Leases and Unbound DNS") >>
Jon Murphy jon.murphy@ipfire.org
Are you running with Michael's updates? I know that does a reload as dhcp starts.
We did have some correspondence on that approach and concluded it had an issue but I can't remember the details or find the correspondence. IIRC there was a significant weakness in the approach and concluded it would be better to regenerate the dhcp-leases.conf file every time unbound restarted, and at that point, the code Michael has in his unbound-dhcp-leases-script which leads up to the reload can be completely removed. This would mean using my cutdown unbound-dhcp-leases script during the unbound restart and paring back his unbound-dhcp-leases-script. This would sidestep your problem.
I had been hoping for an update along these lines but it looks like other things are taking his time. I could set up a repo locally with all the changes, but would not be able to create a multipart patch set in the way that is required, especially as new files are created. It would end up as updates to both the unbound app and the dhcp app.
Nick
On 18/06/2024 04:27, jon wrote:
Concerning CU186, I may have stumbled across an issue. It took a few days for it to pop up. See the message log below from Jun 17 13:48:34 to the end. I experienced a flurry of Reloads and DNS stalled for ~ 15 minutes.
[*root@ipfire*~] # grep --color "Reloading Unbound" /var/log/messages Jun 16 10:43:48 ipfire dhcp[3615]: *Reloading Unbound*... Jun 16 10:58:56 ipfire dhcp[3615]: *Reloading Unbound*... Jun 16 11:29:26 ipfire dhcp[3615]: *Reloading Unbound*... Jun 16 11:43:46 ipfire dhcp[3615]: *Reloading Unbound*... Jun 16 15:08:10 ipfire dhcp[3615]: *Reloading Unbound*... Jun 16 21:48:53 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 06:52:30 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 11:38:59 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 11:41:16 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 11:41:33 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 11:42:23 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 13:39:37 ipfire dhcp[3615]: *Reloading Unbound*...
Jun 17 13:48:34 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 13:48:39 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 13:49:47 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 13:50:47 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 13:52:00 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 13:52:43 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 13:53:43 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 13:54:43 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 13:55:49 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 13:56:55 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 13:58:07 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 13:58:44 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 13:59:44 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 14:00:44 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 14:01:52 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 14:02:52 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 14:04:10 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 14:04:53 ipfire dhcp[3615]: *Reloading Unbound*... [*root@ipfire*~] #
It seems like a new "Reloading Unbound…" was arriving before the "start of service (unbound 1.19.3)" actually happened:
Jun 17 13:48:35 ipfire dhcpd: DHCPDISCOVER from 20:91:df:00:01:02 via blue0 Jun 17 13:48:36 ipfire dhcpd: DHCPOFFER on 192.168.65.138 to 20:91:df:00:01:02 (JiPhoneSE) via blue0 Jun 17 13:48:37 ipfire dhcpd: DHCPREQUEST for 192.168.65.138 (192.168.65.1) from 20:91:df:00:01:02 (JiPhoneSE) via blue0 Jun 17 13:48:37 ipfire dhcpd: DHCPACK on 192.168.65.138 to 20:91:df:00:01:02 (JiPhoneSE) via blue0 Jun 17 13:48:39 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:49:13 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:49:13 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:49:36 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:49:36 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:49:36 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:49:36 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:49:36 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:49:36 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:49:42 ipfire dhcpd: DHCPREQUEST for 192.168.65.110 from a8:8f:d9:00:01:02 (iPhone-W) via blue0 Jun 17 13:49:42 ipfire dhcpd: DHCPACK on 192.168.65.110 to a8:8f:d9:00:01:02 (iPhone-W) via blue0 Jun 17 13:49:47 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:49:59 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:49:59 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:50:05 ipfire dhcpd: DHCPREQUEST for 192.168.65.103 from 00:07:a6:00:01:02 (LEVDS-Dimmer-7BCA) via blue0 Jun 17 13:50:05 ipfire dhcpd: DHCPACK on 192.168.65.103 to 00:07:a6:00:01:02 (LEVDS-Dimmer-7BCA) via blue0 Jun 17 13:50:37 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:50:37 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:50:37 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:50:37 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:50:37 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:50:37 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:50:44 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:50:44 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:50:47 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:51:30 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:51:30 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:51:37 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:51:37 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:51:37 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:51:37 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:51:37 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:51:37 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:52:00 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:52:15 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:52:15 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:52:37 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:52:37 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:52:37 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:52:37 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:52:37 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:52:37 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:52:43 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:52:44 ipfire dhcpd: DHCPREQUEST for 192.168.65.180 from 34:f3:9a:00:01:02 (deb12dell) via blue0 Jun 17 13:52:44 ipfire dhcpd: DHCPACK on 192.168.65.180 to 34:f3:9a:00:01:02 (deb12dell) via blue0 Jun 17 13:53:00 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:53:00 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:53:38 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:53:38 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:53:38 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:53:38 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:53:38 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:53:38 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:53:43 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:53:46 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:53:46 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:54:31 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:54:31 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:54:38 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:54:38 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:54:38 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:54:38 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:54:38 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:54:38 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:54:43 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:55:19 ipfire dhcpd: DHCPREQUEST for 192.168.65.10 from 74:ac:b9:00:01:02 via blue0 Jun 17 13:55:19 ipfire dhcpd: DHCPACK on 192.168.65.10 to 74:ac:b9:00:01:02 via blue0 Jun 17 13:55:22 ipfire dhcpd: DHCPDISCOVER from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:55:22 ipfire dhcpd: DHCPOFFER on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:55:22 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 (192.168.65.1) from 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:55:22 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:55:38 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:55:38 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:55:38 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:55:38 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:55:38 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:55:38 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:55:49 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:55:55 ipfire dhcpd: DHCPREQUEST for 192.168.65.105 from 44:61:32:00:01:02 (ecobee) via blue0 Jun 17 13:55:55 ipfire dhcpd: DHCPACK on 192.168.65.105 to 44:61:32:00:01:02 (ecobee) via blue0 Jun 17 13:55:58 ipfire dhcpd: DHCPREQUEST for 192.168.60.250 from 9c:93:4e:00:01:02 via green0 Jun 17 13:55:58 ipfire dhcpd: DHCPACK on 192.168.60.250 to 9c:93:4e:00:01:02 via green0 Jun 17 13:56:02 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:56:02 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:56:38 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:56:38 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:56:38 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:56:38 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:56:38 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:56:38 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:56:53 ipfire dhcpd: DHCPDISCOVER from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:56:53 ipfire dhcpd: DHCPOFFER on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:56:53 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 (192.168.65.1) from 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:56:53 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:56:55 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:57:33 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:57:33 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:57:34 ipfire dhcpd: DHCPREQUEST for 192.168.60.240 from 8c:dc:d4:00:01:02 via green0 Jun 17 13:57:34 ipfire dhcpd: DHCPACK on 192.168.60.240 to 8c:dc:d4:00:01:02 via green0 Jun 17 13:57:39 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:57:39 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:57:39 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:57:39 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:57:39 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:57:39 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:58:03 ipfire dhcpd: DHCPREQUEST for 192.168.65.160 from 14:88:e6:00:01:02 (SiPad) via blue0 Jun 17 13:58:03 ipfire dhcpd: DHCPACK on 192.168.65.160 to 14:88:e6:00:01:02 (SiPad) via blue0 Jun 17 13:58:07 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:58:08 ipfire dhcpd: DHCPREQUEST for 192.168.60.203 from dc:a6:32:00:01:02 via green0 Jun 17 13:58:08 ipfire dhcpd: DHCPACK on 192.168.60.203 to dc:a6:32:00:01:02 via green0 Jun 17 13:58:19 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:58:19 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:58:39 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:58:39 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:58:39 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:58:39 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:58:39 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:58:39 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:58:44 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:59:04 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:59:04 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:59:39 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:59:39 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:59:39 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:59:39 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:59:39 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:59:39 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:59:44 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:59:50 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:59:50 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 14:00:35 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 14:00:35 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 14:00:39 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 14:00:39 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 14:00:39 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 14:00:39 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 14:00:39 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 14:00:39 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 14:00:44 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:01:21 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 14:01:21 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 14:01:46 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 14:01:46 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 14:01:46 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 14:01:47 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 14:01:47 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 14:01:47 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 14:01:52 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:02:06 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 14:02:06 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 14:02:47 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 14:02:47 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 14:02:47 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 14:02:47 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 14:02:47 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 14:02:47 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 14:02:51 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 14:02:52 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 14:02:52 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:03:19 ipfire dhcpd: DHCPDISCOVER from 5e:24:53:00:01:02 via blue0 Jun 17 14:03:20 ipfire dhcpd: DHCPOFFER on 192.168.65.153 to 5e:24:53:00:01:02 via blue0 Jun 17 14:03:21 ipfire dhcpd: DHCPREQUEST for 192.168.65.153 (192.168.65.1) from 5e:24:53:00:01:02 via blue0 Jun 17 14:03:21 ipfire dhcpd: DHCPACK on 192.168.65.153 to 5e:24:53:00:01:02 via blue0 Jun 17 14:03:38 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 14:03:38 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 14:03:44 ipfire dhcpd: DHCPREQUEST for 192.168.65.109 from ac:c9:06:00:01:02 (MBA1) via blue0 Jun 17 14:03:44 ipfire dhcpd: DHCPACK on 192.168.65.109 to ac:c9:06:00:01:02 (MBA1) via blue0 Jun 17 14:03:47 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 14:03:47 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 14:03:47 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 14:03:47 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 14:03:47 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 14:03:47 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 14:04:02 ipfire dhcpd: DHCPREQUEST for 192.168.65.109 from ac:c9:06:00:01:02 (MBA1) via blue0 Jun 17 14:04:02 ipfire dhcpd: DHCPACK on 192.168.65.109 to ac:c9:06:00:01:02 (MBA1) via blue0 Jun 17 14:04:10 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:04:23 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 14:04:23 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 14:04:48 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 14:04:48 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 14:04:48 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 14:04:48 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 14:04:48 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 14:04:48 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 14:04:53 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:05:41 ipfire dhcpd: DHCPREQUEST for 192.168.65.106 from 40:c7:11:00:01:02 (SiPhone12) via blue0 Jun 17 14:05:41 ipfire dhcpd: DHCPACK on 192.168.65.106 to 40:c7:11:00:01:02 (SiPhone12) via blue0 Jun 17 14:05:49 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 14:05:49 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 14:05:49 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 14:05:49 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 14:05:49 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 14:05:49 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 14:06:50 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 14:06:56 ipfire dhcpd: DHCPREQUEST for 192.168.65.109 from ac:c9:06:00:01:02 (MBA1) via blue0 Jun 17 14:06:56 ipfire dhcpd: DHCPACK on 192.168.65.109 to ac:c9:06:00:01:02 (MBA1) via blue0 Jun 17 14:07:01 ipfire dhcpd: DHCPREQUEST for 192.168.65.108 from 04:c2:9b:00:01:02 (Aura-9413) via blue0 Jun 17 14:07:01 ipfire dhcpd: DHCPACK on 192.168.65.108 to 04:c2:9b:00:01:02 (Aura-9413) via blue0 Jun 17 14:10:02 ipfire dhcpd: DHCPREQUEST for 192.168.60.20 from 0c:4d:e9:00:01:02 via green0 Jun 17 14:10:02 ipfire dhcpd: DHCPACK on 192.168.60.20 to 0c:4d:e9:00:01:02 via green0 Things seem to return to normal after that…. Jon > On Apr 30, 2024, at 5:58 AM, Nick Howitt <nick@howitts.co.uk> wrote: > > > > On 27/04/2024 10:53, Nick Howitt wrote: >> >> >> On 27/04/2024 09:35, Michael Tremer wrote: >>> Hello Nick, >>> >>>> On 26 Apr 2024, at 17:48, Nick Howitt<nick@howitts.co.uk> wrote: >>>> >>>> >>>> >>>> On 26/04/2024 16:12, Michael Tremer wrote: >>>>> Hello Nick, >>>>> >>>>> >>>>>> On 25 Apr 2024, at 13:43, Nick Howitt<nick@howitts.co.uk> wrote: >>>>>> >>>>>> Please can I bump this again to request a review of my patch submitted on 28th March? Thanks, Nick. >>>>>> >>>>> It is a little bit difficult to test your proposed changes if testers have to copy and paste them out of an email. >>>> Confused. I was told I had to submit the diff in an email with [PATCH] in the title and not to attach a file. Please can you tell me how I should be operating? >>> No worries, we are going to sort this out. >>> >>> So the first email in the thread was a correct patch. It was just the patch in the format that Git uses and it was properly parsed by Patchwork. Later in the same thread, the patch was just copied and pasted which was impossible for Patchwork to pick up. There is nothing wrong with using bits of code in the conversation, but it would have been nice to have the final version submitted once again. >>> >>>>> I have just submitted a patch set to this list that breaks down all the changes that you proposed and I cleaned up the code so that we can use the filecmp function without any extra steps. We can simply jump out of the write function and return False to prevent Unbound being reloaded. >>>> I thought you didn't want me to use filecmp and pushed me to use your file compare routine. >>> Yeah, I would have preferred that too, but I thought using the module might be quicker now. >>> >>>>> We will also avoid replacing the file all the time as that causes lots of unnecessary disk I/O. I am not saying it makes a significant difference, but flash has a limited write count that we are still wasting. >>>> OK, but my changes were no worse than before, either doing a delete or rename whereas before it always did a rename. The "unnecessary I/O" was already there. >>>>> Please review and you if you are happy, please add the appropriate Git tag. >>>> Sorry but I don't know what you mean by "adding the appropriate Git tag". I am only using Git locally and have no repo access. >>> https://www.ipfire.org/docs/devel/git/tags > > I tested the patches yesterday and today and I am happy with them. I > hope I have signed them off correctly. > > FWIW I hand patched the program as it was easier than learning > something new in this case as the patch set is so small. > > One thing not mentioned in the bug report. Not only is unbound > restarting on renewals which this fixes, it also restarts on new > leases **and** expiring leases. For big work environments, I would > have thought the best workaround is to set the default DHCP lease to > 24h. It would help during the week to lessen the restarts but not on > Monday mornings. It may also need a bigger address pool to cope with > devices randomising MAC addresses. > > Roll on Jon's fix as it would get round all these restarts. I use it > modified not to update the dhcp-leases.conf. and another couple of > tweaks. Are you able to test it, but it does mean piecing it together? > >> Thanks. >>>> How can I pull your changes? Otherwise I'll need to patch my set up manually. >>> You can apply the patch to your local repository using “git am”: >>> >>> https://git-scm.com/docs/git-am >> Thanks again. >>>> I also have another problem. I have temporarily had to remove my IPF gateway for a business reason and don't know when I'll be able to add it back in. I'll have to see if I can insert it in my LAN somehow. At the same time it is also running Jon's mods which I'll have to undo. >>>> >>>> With respect to Jon's mods, I've asked a couple of times on bugzilla, but is there any point in his routine maintaining /etc/unbound/dhcp-leases.conf. As far as I see the file is only read once when unbound starts, so it only has to be valid when unbound starts and not at any other point in time. To this effect he can remove the file maintenance and I attached a modified copy unbound-dhcp-leases-bridge to his repo which generates a new dhcp-leases.conf and exits, removing all the daemonising bit and the inode watcher. This file can be run as part of the unbound init routine just before the line that actually starts or reloads unbound. It also has the advantage of cutting down a load of disk I/O, which is important as you've just pointed out. >>> I don’t think that we can fully remove the daemon part, because I don’t think that there is a way to get a script (shell or otherwise) fast enough to perform the job. I also think that we benefit a lot from keeping some state in memory, whereas the script will have to parse the old file first, generate a new file and then compare the result. That will take a long time. >> I had Jon's script down to 70ms after I removed the dhcp-leases file >> updates, used my fast method for deriving the domain name and >> removed Jon's initial setting up bit which is not needed once the >> changes are accepted. This is on a 2c VM running at 2GHz. I thought >> the goal was < 100ms which has been done comfortably. >> >> FWIW, if you no longer maintain dhcp-leases in the Jon's script you >> never read the old file or do a compare. All you do is read the >> hosts and static-leases files and nothing is written to file. >>> Do we needed the leases file generated at startup? Yes, I think so, too. DNS is quite essential for the network and for local services to be able to start. So it has to come up very early. If it comes up with proper configuration we avoid any other problems where hosts might resolve names too early and get a different result compared to a couple of seconds later when the zone is fully initialised. >> My cut-down version of unbound-dhcp-leases-bridge is at >> https://bugzilla.ipfire.org/attachment.cgi?id=1516 on >> https://bugzilla.ipfire.org/show_bug.cgi?id=13254. On my test system >> it takes 219ms to run so delays unbound starting up minimally. >> >> At the same time there is a gain in the dhcp start up as the >> unbound-dhcp-leases-bridge is no longer needed and it therefore >> avoids triggering an unbound restart which happens every time >> unbound-dhcp-leases-bridge is run. >> >> If we are generating dhcp-leases.conf at start up, then there is no >> point in maintaining it while unbound is running as the maintained >> file will never be read. >>> So all in all, I think the route I would most likely look at is changing the daemon to open a socket. We write minimal scripts which are called by dhcpd that simply pipe their data into the socket which the daemon will then parse. With all available leases in memory already, we can check if a lease has changed or if it has been renewed without any further changes. If nothing has changed, we don’t need to do anything. In case of a change, we will have to write the file again and have Unbound reload, or we even call unbound-control and make the one change without a reload. Doing that in the daemon should give us more flexibility and we avoid any performance penalties as the daemon could run multiple commands concurrently which dhcpd can’t do. >> Sockets are beyond me, I am afraid. >> >> Part of the goal of all this is to avoid reloading unbound and part >> of my goal is to avoid writing dhcp-leases.conf unless unbound is >> restarted. I am not sure why you are still mentioning writing a file >> and restarting/reloading unbound. >>>> Also on his patches, I had a significantly faster way of determining the domain name (in bash) and he and I were looking for your feedback on if we'd be allowed to use it. >>> Anything is allowed. It just has to be fast. And for that, I believe, calling any other binary will be a problem. >> OK. I was proposing to read the DOMAINNAME with: >> >> DOMAINNAME=${HOSTNAME#*.*} >> >> which is many times faster than the conventional route: >> >> eval $(/usr/local/bin/readhash /var/ipfire/main/settings) >> >> If there is any concern with bash not having the full environment >> when the program is called, the shebang could be changed to: >> >> #!/bin/bash -l >> >> Nick >>> -Michael >>> >>>> Regards, >>>> >>>> Nick >>>>> Best, >>>>> -Michael >>>>> >>>>> >>>>>> On 10/04/2024 12:04, Nick Howitt wrote: >>>>>> >>>>>>> Bump >>>>>>> >>>>>>> On 28/03/2024 17:25, Nick Howitt wrote: >>>>>>> >>>>>>>> Next version. >>>>>>>> >>>>>>>> On 28/03/2024 10:41, Michael Tremer wrote: >>>>>>>> >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> >>>>>>>>>> On 27 Mar 2024, at 15:28, Nick Howitt<nick@howitts.co.uk> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 26/03/2024 11:10, Michael Tremer wrote: >>>>>>>>>> >>>>>>>>>>>> On 25 Mar 2024, at 13:11, Nick Howitt<nick@howitts.co.uk> wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 25/03/2024 11:31, Michael Tremer wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hello, >>>>>>>>>>>>> I don’t think that the risk is very high, but I would account for this problem. >>>>>>>>>>>>> With my proposed changes from my previous email this should not be a large problem at all. When opening /etc/unbound/dhcp-leases.conf with mode “r+” you can catch the Exception and skip the comparison and open the file again with “w”. >>>>>>>>>>>>> >>>>>>>>>>>>>> On 25 Mar 2024, at 10:10, Nick Howitt<nick@howitts.co.uk> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> If there is a risk that the /etc/unbound/dhcp-leases.conf does not exist, for example on a clean install (I don't know), the filecmp.cmp line needs to be wrapped in a try...except to stop it blowing up: >>>>>>>>>>>>>> >>>>>>>>>>>>>> try: >>>>>>>>>>>>>> RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) >>>>>>>>>>>>>> except: >>>>>>>>>>>>>> RequireUnboundReload = True >>>>>>>>>>>>>> >>>>>>>>>>>>> I generally don’t like blanco exception catches. You will never know about the daemons that you are hiding. >>>>>>>>>>>>> >>>>>>>>>>>>>> This will force the file to be to be generated, even if it is only copying in a blank temp file, and unbound will restart. It will only ever happen once. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24/03/2024 15:49, Nick Howitt wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>> **** This is a repost of an email incorrectly submitted. **** >>>>>>>>>>>>>>> Please bear with me as I am new to IPFire and not at all used to this style of development, so any guidance would be appreciated. >>>>>>>>>>>>>>> There is a bug,https://bugzilla.ipfire.org/show_bug.cgi?id=13254, where unbound-dhcp-leases-bridge restarts unbound a lot when it is totally unnecessary. It appears to be because when a lease is renewed, the script gets triggered, created a new /etc/unbound/dhcp-leases.conf then restarts Unbound to read the file. Generally this seems to be unnecessary as, with a renewed lease, the old and new /etc/unbound/dhcp-leases.conf files are the same. With 5 leases in that file (one for a machine not active) I am getting 3-4 restarts an hour of Unbound when I have a min/max lease time of 60/120min. >>>>>>>>>>>>>>> Looking at the code, it is fairly easy to fix. The current code creates a temp file then copies it into place then restarts unbound. All I am doing is doing a file comparison before the copy and skipping the restart if the files are the same. >>>>>>>>>>>>>>> There were a couple of gotchas because setting the file attributes and copying the file were done inside the "with" block for generating the temporary file. This meant a file comparison always returned False as the temp file was still open and so never the same is the old file. >>>>>>>>>>>>>>> I moved those two statements outside the "with". This forced me to change the fchmod to chmod. >>>>>>>>>>>>>>> It could be argued that the file copy should not be done if the files are not different, but it would take an extra "if" and you'd have to remember to delete the temp file. If required, I can make that change. >>>>>>>>>>>>>>> Also, one small thing I noticed once is that the old and new dhcp-leases.conf files could occasionally contain the same leases but in a different order. I have been unable to reproduce, but to sidestep it, instead of stepping through the leases variable directly, I sort it and step through that. It should make the resulting file completely deterministic and make the file comparison more effective. >>>>>>>>>>>>>>> My patch is: >>>>>>>>>>>>>>> >From 73873d4944944a2f02317a73074a6894726f36f7 Mon Sep 17 00:00:00 2001 >>>>>>>>>>>>>>> From: Nick Howitt<nick@howitts.co.uk> >>>>>>>>>>>>>>> Date: Sun, 24 Mar 2024 15:17:19 +0000 >>>>>>>>>>>>>>> Subject: [PATCH] Stop unbound-dhcp-leases-bridge from restarting unbound every >>>>>>>>>>>>>>> time it write dhcp-leases.conf as it is very often unchanged and does not >>>>>>>>>>>>>>> require a restart. >>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>> config/unbound/unbound-dhcp-leases-bridge | 28 +++++++++++++++-------- >>>>>>>>>>>>>>> 1 file changed, 19 insertions(+), 9 deletions(-) >>>>>>>>>>>>>>> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>>>>>>>> index e9f022aff..d22772066 100644 >>>>>>>>>>>>>>> --- a/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>>>>>>>> +++ b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>>>>>>>> @@ -22,6 +22,7 @@ >>>>>>>>>>>>>>> import argparse >>>>>>>>>>>>>>> import datetime >>>>>>>>>>>>>>> import daemon >>>>>>>>>>>>>>> +import filecmp >>>>>>>>>>>>>>> import functools >>>>>>>>>>>>>>> import ipaddress >>>>>>>>>>>>>>> import logging >>>>>>>>>>>>>>> @@ -516,26 +517,35 @@ class UnboundConfigWriter(object): >>>>>>>>>>>>>>> def update_dhcp_leases(self, leases): >>>>>>>>>>>>>>> # Write out all leases >>>>>>>>>>>>>>> - self.write_dhcp_leases(leases) >>>>>>>>>>>>>>> + if self.write_dhcp_leases(leases): >>>>>>>>>>>>>>> - log.debug("Reloading Unbound...") >>>>>>>>>>>>>>> + log.debug("Reloading Unbound...") >>>>>>>>>>>>>>> - # Reload the configuration without dropping the cache >>>>>>>>>>>>>>> - self._control("reload_keep_cache") >>>>>>>>>>>>>>> + # Reload the configuration without dropping the cache >>>>>>>>>>>>>>> + self._control("reload_keep_cache") >>>>>>>>>>>>>>> + >>>>>>>>>>>>>>> + else: >>>>>>>>>>>>>>> + >>>>>>>>>>>>>>> + log.debug("Not reloading Unbound. Leases not changed.") >>>>>>>>>>>>>>> def write_dhcp_leases(self, leases): >>>>>>>>>>>>>>> log.debug("Writing DHCP leases...") >>>>>>>>>>>>>>> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >>>>>>>>>>>>>>> - for l in leases: >>>>>>>>>>>>>>> + for l in sorted(leases): >>>>>>>>>>>>>>> for rr in l.rrset: >>>>>>>>>>>>>>> f.write("local-data: \"%s\"\n" % " ".join(rr)) >>>>>>>>>>>>>>> - # Make file readable for everyone >>>>>>>>>>>>>>> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>>>>>>>>> + filecmp.clear_cache() >>>>>>>>>>>>>>> + RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) >>>>>>>>>>>>>>> + >>>>>>>>>>>>>>> + # Make file readable for everyone >>>>>>>>>>>>>>> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>>>>>>>>> + >>>>>>>>>>>>>>> + # Move the file to its destination >>>>>>>>>>>>>>> + os.rename(f.name, self.path) >>>>>>>>>>>>>>> - # Move the file to its destination >>>>>>>>>>>>>>> - os.rename(f.name, self.path) >>>>>>>>>>>>>>> + return RequireUnboundReload >>>>>>>>>>>>>>> def _control(self, *args): >>>>>>>>>>>>>>> command = ["unbound-control"] >>>>>>>>>>>>>>> >>>>>>>>>>>> Resubmitting the patch. >>>>>>>>>>>> >>>>>>>>>>>> Following my previous reply, I would like to stick with filecmp as I am using it with "shallow=False" to avoid doing the comparison just with stat() calls. Note that I also flush the filecmp cache before using the command so it is running with the latest file data. Talking about file descriptors, atomic file replacement etc., scares me. >>>>>>>>>>>> >>>>>>>>>>> File descriptors are great. They avoid race-conditions and all sorts of things. >>>>>>>>>>> >>>>>>>>>> But I have never used them so I don't intend to go that way. Too much for me to get my brain round and a limited ability to test. >>>>>>>>>> >>>>>>>>> Well, I pretty much gave you the entire solution… There was nothing left to do apart from copying and pasting the snippets. >>>>>>>>> >>>>>>>>> >>>>>>>> Ok I've tried to use your version but struggled to get it going. If I used it as was and called compare(file1_name, file2_name) it failed as "compare" was not defined. >>>>>>>> I could call it using self.compare but then it failed with "TypeError: UnboundConfigWriter.compare() takes 2 positional arguments but 3 were given". I changed the function to compare(self, f1, f2). >>>>>>>> Then it failed with "AttributeError: 'str' object has no attribute 'seek'", so I changed it again to accept file names then open the files before doing the seeks. >>>>>>>> I really hope it is OK as I am out of my depth when defining functions with "self" as an argument. Please can you check it? If it is not OK, can I revert to filecmp? >>>>>>>> >>>>>>>> I have made one futher change, the sorted(leases) line was not working and not sorting the object ever, and I saw lines changing places in the resulting leases file. I have changed it to "sorted(leases, key=lambda x: x.ipaddr)". It definitely sorts this time. >>>>>>>> >>>>>>>> >>>>>>>>>>> I don’t think that module is the best choice really if you have to hack around and flush the cache. Why would it even cache things in the first place? I don’t see where the cache helps. >>>>>>>>>>> >>>>>>>>>> From the filecmp.clear_cache documentation: >>>>>>>>>> >>>>>>>>>> Clear the filecmp cache. This may be useful if a file is compared so >>>>>>>>>> quickly after it is modified that it is within the mtime resolution >>>>>>>>>> of the underlying filesystem. >>>>>>>>>> >>>>>>>>>> Now this makes me believe that the clear_cache is only relevant when using shallow=True, which I'm not and I don't know how to check the underlying code. It is used to make sure the stat() info is updated, but we are not using the stat() info. I guess we could as easily add a one second sleep instead as that is way longer than the mtime resolution. >>>>>>>>>> >>>>>>>>> The function is here and it is rather simple:https://github.com/python/cpython/blob/3.12/Lib/filecmp.py#L30 >>>>>>>>> >>>>>>>>> It will always stat() both files and compares three items in case shallow is False: The type of the file, the size and mtime. If the file is not a regular file, the function will exist rather early. So in our case, that is all work the function is doing for thing. It boils down to something very similar to what I suggested to use to compare the files. >>>>>>>>> >>>>>>>>> On top of that, there is a cache which will always be used but will never return a match for us because the temporary file at least will always have another mtime. >>>>>>>>> >>>>>>>>> You could clear the cache after every call, but I would rather not have our code messy and care about implementation details of some Python module. >>>>>>>>> >>>>>>>>> >>>>>>>>>> The module is chosen because all the references I read about comparing files preferred this one-liner for comparing files, although other methods were given and there were lots of discussions about large files (GB+) because if you chomp them it may be much quicker to say they are not the same when you find the first block that does not match. Having said this, I don't know how filecmp works, but we only have a small file. At the same time, I am not a regular programmer so simple one-liners are good for me. >>>>>>>>>> >>>>>>>>> You would always want to read both files block for block and compare the blocks. There is no way to implement this significantly faster. >>>>>>>>> >>>>>>>>> >>>>>>>>>>>> I have modified the earlier try/except I proposed to catch only FileNotFoundError, and I have changed CamelCase to snake_case. >>>>>>>>>>>> >>>>>>>>>>>> --- >>>>>>>>>>>> config/unbound/unbound-dhcp-leases-bridge | 31 ++++++++++++++++------- >>>>>>>>>>>> 1 file changed, 22 insertions(+), 9 deletions(-) >>>>>>>>>>>> >>>>>>>>>>>> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>>>>> index e9f022aff..a9ec8e1e2 100644 >>>>>>>>>>>> --- a/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>>>>> +++ b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>>>>> @@ -22,6 +22,7 @@ >>>>>>>>>>>> import argparse >>>>>>>>>>>> import datetime >>>>>>>>>>>> import daemon >>>>>>>>>>>> +import filecmp >>>>>>>>>>>> import functools >>>>>>>>>>>> import ipaddress >>>>>>>>>>>> import logging >>>>>>>>>>>> @@ -516,26 +517,38 @@ class UnboundConfigWriter(object): >>>>>>>>>>>> >>>>>>>>>>>> def update_dhcp_leases(self, leases): >>>>>>>>>>>> # Write out all leases >>>>>>>>>>>> - self.write_dhcp_leases(leases) >>>>>>>>>>>> + if self.write_dhcp_leases(leases): >>>>>>>>>>>> >>>>>>>>>>>> - log.debug("Reloading Unbound...") >>>>>>>>>>>> + log.debug("Reloading Unbound...") >>>>>>>>>>>> >>>>>>>>>>>> - # Reload the configuration without dropping the cache >>>>>>>>>>>> - self._control("reload_keep_cache") >>>>>>>>>>>> + # Reload the configuration without dropping the cache >>>>>>>>>>>> + self._control("reload_keep_cache") >>>>>>>>>>>> + >>>>>>>>>>>> + else: >>>>>>>>>>>> + >>>>>>>>>>>> + log.debug("Not reloading Unbound. Leases not changed.") >>>>>>>>>>>> >>>>>>>>>>>> def write_dhcp_leases(self, leases): >>>>>>>>>>>> log.debug("Writing DHCP leases...") >>>>>>>>>>>> >>>>>>>>>>>> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >>>>>>>>>>>> - for l in leases: >>>>>>>>>>>> + for l in sorted(leases): >>>>>>>>>>>> for rr in l.rrset: >>>>>>>>>>>> f.write("local-data: \"%s\"\n" % " ".join(rr)) >>>>>>>>>>>> >>>>>>>>>>>> - # Make file readable for everyone >>>>>>>>>>>> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>>>>>> + filecmp.clear_cache() >>>>>>>>>>>> + try: >>>>>>>>>>>> + require_unbound_reload = not filecmp.cmp(f.name, self.path, shallow=False) >>>>>>>>>>>> + except FileNotFoundError: >>>>>>>>>>>> + require_unbound_reload = True >>>>>>>>>>>> >>>>>>>>>>> If you want to go this route, you will have to flush the temporary file first. Otherwise you might have some data left in the buffer that has not been written (because the file is still open). You will then compare incomplete output. This would not become a problem if you used the file descriptors. >>>>>>>>>>> >>>>>>>>>> The temp file is closed at the point of doing the filecmp, and the filecmp cache has been flushed. How do I flush the temporary file further. Reading up, is it just having the last statement in the with block as f.flush() with the same indent as "for l in sorted(leases):" >>>>>>>>>> >>>>>>>>> If you have left the with block you will already have flushed the buffers and closed the file. It looks like I misread the indentation in the patch then. >>>>>>>>> >>>>>>>>> >>>>>>>>>>> Catching FileNotFound would work for me. >>>>>>>>>>> >>>>>>>>>> Good >>>>>>>>>> >>>>>>>>>>>> + >>>>>>>>>>>> + # Make file readable for everyone >>>>>>>>>>>> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>>>>>> + >>>>>>>>>>>> + # Move the file to its destination >>>>>>>>>>>> + os.rename(f.name, self.path) >>>>>>>>>>>> >>>>>>>>>>>> - # Move the file to its destination >>>>>>>>>>>> - os.rename(f.name, self.path) >>>>>>>>>>>> + return require_unbound_reload >>>>>>>>>>>> >>>>>>>>>>>> def _control(self, *args): >>>>>>>>>>>> command = ["unbound-control"] >>>>>>>>>>>> -- >>>>>>>>>>>> 2.43.0 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Nick >>>>>>>>>>>> >>>>>>>> --- >>>>>>>> config/unbound/unbound-dhcp-leases-bridge | 50 +++++++++++++++++++---- >>>>>>>> 1 file changed, 41 insertions(+), 9 deletions(-) >>>>>>>> >>>>>>>> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>> index e9f022aff..80d9523ea 100644 >>>>>>>> --- a/config/unbound/unbound-dhcp-leases-bridge >>>>>>>> +++ b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>> @@ -516,26 +516,37 @@ class UnboundConfigWriter(object): >>>>>>>> >>>>>>>> def update_dhcp_leases(self, leases): >>>>>>>> # Write out all leases >>>>>>>> - self.write_dhcp_leases(leases) >>>>>>>> + if self.write_dhcp_leases(leases): >>>>>>>> >>>>>>>> - log.debug("Reloading Unbound...") >>>>>>>> + log.debug("Reloading Unbound...") >>>>>>>> >>>>>>>> - # Reload the configuration without dropping the cache >>>>>>>> - self._control("reload_keep_cache") >>>>>>>> + # Reload the configuration without dropping the cache >>>>>>>> + self._control("reload_keep_cache") >>>>>>>> + >>>>>>>> + else: >>>>>>>> + >>>>>>>> + log.debug("Not reloading Unbound. Leases not changed.") >>>>>>>> >>>>>>>> def write_dhcp_leases(self, leases): >>>>>>>> log.debug("Writing DHCP leases...") >>>>>>>> >>>>>>>> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >>>>>>>> - for l in leases: >>>>>>>> + for l in sorted(leases, key=lambda x: x.ipaddr): >>>>>>>> for rr in l.rrset: >>>>>>>> f.write("local-data: \"%s\"\n" % " ".join(rr)) >>>>>>>> >>>>>>>> - # Make file readable for everyone >>>>>>>> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>> + try: >>>>>>>> + require_unbound_reload = not self.compare(f.name, self.path) >>>>>>>> + except FileNotFoundError: >>>>>>>> + require_unbound_reload = True >>>>>>>> + >>>>>>>> + # Make file readable for everyone >>>>>>>> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>> + >>>>>>>> + # Move the file to its destination >>>>>>>> + os.rename(f.name, self.path) >>>>>>>> >>>>>>>> - # Move the file to its destination >>>>>>>> - os.rename(f.name, self.path) >>>>>>>> + return require_unbound_reload >>>>>>>> >>>>>>>> def _control(self, *args): >>>>>>>> command = ["unbound-control"] >>>>>>>> @@ -551,6 +562,27 @@ class UnboundConfigWriter(object): >>>>>>>> >>>>>>>> raise e >>>>>>>> >>>>>>>> + def compare(self, if1, if2): >>>>>>>> + buffersize = 16 * 1024 >>>>>>>> + >>>>>>>> + f1 = open(if1, "r") >>>>>>>> + f2 = open(if2, "r") >>>>>>>> + >>>>>>>> + # Reset file handles >>>>>>>> + f1.seek(0) >>>>>>>> + f2.seek(0) >>>>>>>> + >>>>>>>> + while True: >>>>>>>> + b1 = f1.read(buffersize) >>>>>>>> + b2 = f2.read(buffersize) >>>>>>>> + >>>>>>>> + if not b1 == b2: >>>>>>>> + return False >>>>>>>> + >>>>>>>> + elif not b1 or not b2: >>>>>>>> + break >>>>>>>> + >>>>>>>> + return True >>>>>>>> >>>>>>>> if __name__ == "__main__": >>>>>>>> parser = argparse.ArgumentParser(description="Bridge for DHCP Leases and Unbound DNS") >>>>>>>> >> > Jon Murphy jon.murphy@ipfire.org
Found the discussion on bugzilla from comment 150 (https://bugzilla.ipfire.org/show_bug.cgi?id=13254#c150) through to comment 153.
On 18/06/2024 09:57, Nick Howitt wrote:
Are you running with Michael's updates? I know that does a reload as dhcp starts.
We did have some correspondence on that approach and concluded it had an issue but I can't remember the details or find the correspondence. IIRC there was a significant weakness in the approach and concluded it would be better to regenerate the dhcp-leases.conf file every time unbound restarted, and at that point, the code Michael has in his unbound-dhcp-leases-script which leads up to the reload can be completely removed. This would mean using my cutdown unbound-dhcp-leases script during the unbound restart and paring back his unbound-dhcp-leases-script. This would sidestep your problem.
I had been hoping for an update along these lines but it looks like other things are taking his time. I could set up a repo locally with all the changes, but would not be able to create a multipart patch set in the way that is required, especially as new files are created. It would end up as updates to both the unbound app and the dhcp app.
Nick
On 18/06/2024 04:27, jon wrote:
Concerning CU186, I may have stumbled across an issue. It took a few days for it to pop up. See the message log below from Jun 17 13:48:34 to the end. I experienced a flurry of Reloads and DNS stalled for ~ 15 minutes.
[*root@ipfire*~] # grep --color "Reloading Unbound" /var/log/messages Jun 16 10:43:48 ipfire dhcp[3615]: *Reloading Unbound*... Jun 16 10:58:56 ipfire dhcp[3615]: *Reloading Unbound*... Jun 16 11:29:26 ipfire dhcp[3615]: *Reloading Unbound*... Jun 16 11:43:46 ipfire dhcp[3615]: *Reloading Unbound*... Jun 16 15:08:10 ipfire dhcp[3615]: *Reloading Unbound*... Jun 16 21:48:53 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 06:52:30 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 11:38:59 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 11:41:16 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 11:41:33 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 11:42:23 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 13:39:37 ipfire dhcp[3615]: *Reloading Unbound*...
Jun 17 13:48:34 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 13:48:39 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 13:49:47 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 13:50:47 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 13:52:00 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 13:52:43 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 13:53:43 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 13:54:43 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 13:55:49 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 13:56:55 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 13:58:07 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 13:58:44 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 13:59:44 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 14:00:44 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 14:01:52 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 14:02:52 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 14:04:10 ipfire dhcp[3615]: *Reloading Unbound*... Jun 17 14:04:53 ipfire dhcp[3615]: *Reloading Unbound*... [*root@ipfire*~] #
It seems like a new "Reloading Unbound…" was arriving before the "start of service (unbound 1.19.3)" actually happened:
Jun 17 13:48:35 ipfire dhcpd: DHCPDISCOVER from 20:91:df:00:01:02 via blue0 Jun 17 13:48:36 ipfire dhcpd: DHCPOFFER on 192.168.65.138 to 20:91:df:00:01:02 (JiPhoneSE) via blue0 Jun 17 13:48:37 ipfire dhcpd: DHCPREQUEST for 192.168.65.138 (192.168.65.1) from 20:91:df:00:01:02 (JiPhoneSE) via blue0 Jun 17 13:48:37 ipfire dhcpd: DHCPACK on 192.168.65.138 to 20:91:df:00:01:02 (JiPhoneSE) via blue0 Jun 17 13:48:39 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:49:13 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:49:13 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:49:36 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:49:36 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:49:36 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:49:36 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:49:36 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:49:36 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:49:42 ipfire dhcpd: DHCPREQUEST for 192.168.65.110 from a8:8f:d9:00:01:02 (iPhone-W) via blue0 Jun 17 13:49:42 ipfire dhcpd: DHCPACK on 192.168.65.110 to a8:8f:d9:00:01:02 (iPhone-W) via blue0 Jun 17 13:49:47 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:49:59 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:49:59 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:50:05 ipfire dhcpd: DHCPREQUEST for 192.168.65.103 from 00:07:a6:00:01:02 (LEVDS-Dimmer-7BCA) via blue0 Jun 17 13:50:05 ipfire dhcpd: DHCPACK on 192.168.65.103 to 00:07:a6:00:01:02 (LEVDS-Dimmer-7BCA) via blue0 Jun 17 13:50:37 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:50:37 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:50:37 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:50:37 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:50:37 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:50:37 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:50:44 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:50:44 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:50:47 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:51:30 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:51:30 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:51:37 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:51:37 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:51:37 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:51:37 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:51:37 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:51:37 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:52:00 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:52:15 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:52:15 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:52:37 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:52:37 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:52:37 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:52:37 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:52:37 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:52:37 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:52:43 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:52:44 ipfire dhcpd: DHCPREQUEST for 192.168.65.180 from 34:f3:9a:00:01:02 (deb12dell) via blue0 Jun 17 13:52:44 ipfire dhcpd: DHCPACK on 192.168.65.180 to 34:f3:9a:00:01:02 (deb12dell) via blue0 Jun 17 13:53:00 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:53:00 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:53:38 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:53:38 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:53:38 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:53:38 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:53:38 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:53:38 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:53:43 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:53:46 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:53:46 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:54:31 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:54:31 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:54:38 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:54:38 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:54:38 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:54:38 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:54:38 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:54:38 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:54:43 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:55:19 ipfire dhcpd: DHCPREQUEST for 192.168.65.10 from 74:ac:b9:00:01:02 via blue0 Jun 17 13:55:19 ipfire dhcpd: DHCPACK on 192.168.65.10 to 74:ac:b9:00:01:02 via blue0 Jun 17 13:55:22 ipfire dhcpd: DHCPDISCOVER from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:55:22 ipfire dhcpd: DHCPOFFER on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:55:22 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 (192.168.65.1) from 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:55:22 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:55:38 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:55:38 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:55:38 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:55:38 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:55:38 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:55:38 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:55:49 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:55:55 ipfire dhcpd: DHCPREQUEST for 192.168.65.105 from 44:61:32:00:01:02 (ecobee) via blue0 Jun 17 13:55:55 ipfire dhcpd: DHCPACK on 192.168.65.105 to 44:61:32:00:01:02 (ecobee) via blue0 Jun 17 13:55:58 ipfire dhcpd: DHCPREQUEST for 192.168.60.250 from 9c:93:4e:00:01:02 via green0 Jun 17 13:55:58 ipfire dhcpd: DHCPACK on 192.168.60.250 to 9c:93:4e:00:01:02 via green0 Jun 17 13:56:02 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:56:02 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:56:38 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:56:38 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:56:38 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:56:38 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:56:38 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:56:38 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:56:53 ipfire dhcpd: DHCPDISCOVER from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:56:53 ipfire dhcpd: DHCPOFFER on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:56:53 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 (192.168.65.1) from 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:56:53 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:56:55 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:57:33 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:57:33 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:57:34 ipfire dhcpd: DHCPREQUEST for 192.168.60.240 from 8c:dc:d4:00:01:02 via green0 Jun 17 13:57:34 ipfire dhcpd: DHCPACK on 192.168.60.240 to 8c:dc:d4:00:01:02 via green0 Jun 17 13:57:39 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:57:39 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:57:39 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:57:39 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:57:39 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:57:39 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:58:03 ipfire dhcpd: DHCPREQUEST for 192.168.65.160 from 14:88:e6:00:01:02 (SiPad) via blue0 Jun 17 13:58:03 ipfire dhcpd: DHCPACK on 192.168.65.160 to 14:88:e6:00:01:02 (SiPad) via blue0 Jun 17 13:58:07 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:58:08 ipfire dhcpd: DHCPREQUEST for 192.168.60.203 from dc:a6:32:00:01:02 via green0 Jun 17 13:58:08 ipfire dhcpd: DHCPACK on 192.168.60.203 to dc:a6:32:00:01:02 via green0 Jun 17 13:58:19 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:58:19 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:58:39 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:58:39 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:58:39 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:58:39 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:58:39 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:58:39 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:58:44 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:59:04 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:59:04 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:59:39 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:59:39 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:59:39 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:59:39 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:59:39 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:59:39 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:59:44 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:59:50 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:59:50 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 14:00:35 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 14:00:35 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 14:00:39 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 14:00:39 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 14:00:39 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 14:00:39 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 14:00:39 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 14:00:39 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 14:00:44 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:01:21 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 14:01:21 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 14:01:46 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 14:01:46 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 14:01:46 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 14:01:47 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 14:01:47 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 14:01:47 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 14:01:52 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:02:06 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 14:02:06 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 14:02:47 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 14:02:47 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 14:02:47 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 14:02:47 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 14:02:47 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 14:02:47 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 14:02:51 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 14:02:52 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 14:02:52 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:03:19 ipfire dhcpd: DHCPDISCOVER from 5e:24:53:00:01:02 via blue0 Jun 17 14:03:20 ipfire dhcpd: DHCPOFFER on 192.168.65.153 to 5e:24:53:00:01:02 via blue0 Jun 17 14:03:21 ipfire dhcpd: DHCPREQUEST for 192.168.65.153 (192.168.65.1) from 5e:24:53:00:01:02 via blue0 Jun 17 14:03:21 ipfire dhcpd: DHCPACK on 192.168.65.153 to 5e:24:53:00:01:02 via blue0 Jun 17 14:03:38 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 14:03:38 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 14:03:44 ipfire dhcpd: DHCPREQUEST for 192.168.65.109 from ac:c9:06:00:01:02 (MBA1) via blue0 Jun 17 14:03:44 ipfire dhcpd: DHCPACK on 192.168.65.109 to ac:c9:06:00:01:02 (MBA1) via blue0 Jun 17 14:03:47 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 14:03:47 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 14:03:47 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 14:03:47 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 14:03:47 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 14:03:47 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 14:04:02 ipfire dhcpd: DHCPREQUEST for 192.168.65.109 from ac:c9:06:00:01:02 (MBA1) via blue0 Jun 17 14:04:02 ipfire dhcpd: DHCPACK on 192.168.65.109 to ac:c9:06:00:01:02 (MBA1) via blue0 Jun 17 14:04:10 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:04:23 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 14:04:23 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 14:04:48 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 14:04:48 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 14:04:48 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 14:04:48 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 14:04:48 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 14:04:48 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 14:04:53 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:05:41 ipfire dhcpd: DHCPREQUEST for 192.168.65.106 from 40:c7:11:00:01:02 (SiPhone12) via blue0 Jun 17 14:05:41 ipfire dhcpd: DHCPACK on 192.168.65.106 to 40:c7:11:00:01:02 (SiPhone12) via blue0 Jun 17 14:05:49 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 14:05:49 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 14:05:49 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 14:05:49 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 14:05:49 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 14:05:49 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 14:06:50 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 14:06:56 ipfire dhcpd: DHCPREQUEST for 192.168.65.109 from ac:c9:06:00:01:02 (MBA1) via blue0 Jun 17 14:06:56 ipfire dhcpd: DHCPACK on 192.168.65.109 to ac:c9:06:00:01:02 (MBA1) via blue0 Jun 17 14:07:01 ipfire dhcpd: DHCPREQUEST for 192.168.65.108 from 04:c2:9b:00:01:02 (Aura-9413) via blue0 Jun 17 14:07:01 ipfire dhcpd: DHCPACK on 192.168.65.108 to 04:c2:9b:00:01:02 (Aura-9413) via blue0 Jun 17 14:10:02 ipfire dhcpd: DHCPREQUEST for 192.168.60.20 from 0c:4d:e9:00:01:02 via green0 Jun 17 14:10:02 ipfire dhcpd: DHCPACK on 192.168.60.20 to 0c:4d:e9:00:01:02 via green0 Things seem to return to normal after that…. Jon > On Apr 30, 2024, at 5:58 AM, Nick Howitt <nick@howitts.co.uk> wrote: > > > > On 27/04/2024 10:53, Nick Howitt wrote: >> >> >> On 27/04/2024 09:35, Michael Tremer wrote: >>> Hello Nick, >>> >>>> On 26 Apr 2024, at 17:48, Nick Howitt<nick@howitts.co.uk> wrote: >>>> >>>> >>>> >>>> On 26/04/2024 16:12, Michael Tremer wrote: >>>>> Hello Nick, >>>>> >>>>> >>>>>> On 25 Apr 2024, at 13:43, Nick Howitt<nick@howitts.co.uk> wrote: >>>>>> >>>>>> Please can I bump this again to request a review of my patch submitted on 28th March? Thanks, Nick. >>>>>> >>>>> It is a little bit difficult to test your proposed changes if testers have to copy and paste them out of an email. >>>> Confused. I was told I had to submit the diff in an email with [PATCH] in the title and not to attach a file. Please can you tell me how I should be operating? >>> No worries, we are going to sort this out. >>> >>> So the first email in the thread was a correct patch. It was just the patch in the format that Git uses and it was properly parsed by Patchwork. Later in the same thread, the patch was just copied and pasted which was impossible for Patchwork to pick up. There is nothing wrong with using bits of code in the conversation, but it would have been nice to have the final version submitted once again. >>> >>>>> I have just submitted a patch set to this list that breaks down all the changes that you proposed and I cleaned up the code so that we can use the filecmp function without any extra steps. We can simply jump out of the write function and return False to prevent Unbound being reloaded. >>>> I thought you didn't want me to use filecmp and pushed me to use your file compare routine. >>> Yeah, I would have preferred that too, but I thought using the module might be quicker now. >>> >>>>> We will also avoid replacing the file all the time as that causes lots of unnecessary disk I/O. I am not saying it makes a significant difference, but flash has a limited write count that we are still wasting. >>>> OK, but my changes were no worse than before, either doing a delete or rename whereas before it always did a rename. The "unnecessary I/O" was already there. >>>>> Please review and you if you are happy, please add the appropriate Git tag. >>>> Sorry but I don't know what you mean by "adding the appropriate Git tag". I am only using Git locally and have no repo access. >>> https://www.ipfire.org/docs/devel/git/tags > > I tested the patches yesterday and today and I am happy with them. I > hope I have signed them off correctly. > > FWIW I hand patched the program as it was easier than learning > something new in this case as the patch set is so small. > > One thing not mentioned in the bug report. Not only is unbound > restarting on renewals which this fixes, it also restarts on new > leases **and** expiring leases. For big work environments, I would > have thought the best workaround is to set the default DHCP lease to > 24h. It would help during the week to lessen the restarts but not on > Monday mornings. It may also need a bigger address pool to cope with > devices randomising MAC addresses. > > Roll on Jon's fix as it would get round all these restarts. I use it > modified not to update the dhcp-leases.conf. and another couple of > tweaks. Are you able to test it, but it does mean piecing it together? > >> Thanks. >>>> How can I pull your changes? Otherwise I'll need to patch my set up manually. >>> You can apply the patch to your local repository using “git am”: >>> >>> https://git-scm.com/docs/git-am >> Thanks again. >>>> I also have another problem. I have temporarily had to remove my IPF gateway for a business reason and don't know when I'll be able to add it back in. I'll have to see if I can insert it in my LAN somehow. At the same time it is also running Jon's mods which I'll have to undo. >>>> >>>> With respect to Jon's mods, I've asked a couple of times on bugzilla, but is there any point in his routine maintaining /etc/unbound/dhcp-leases.conf. As far as I see the file is only read once when unbound starts, so it only has to be valid when unbound starts and not at any other point in time. To this effect he can remove the file maintenance and I attached a modified copy unbound-dhcp-leases-bridge to his repo which generates a new dhcp-leases.conf and exits, removing all the daemonising bit and the inode watcher. This file can be run as part of the unbound init routine just before the line that actually starts or reloads unbound. It also has the advantage of cutting down a load of disk I/O, which is important as you've just pointed out. >>> I don’t think that we can fully remove the daemon part, because I don’t think that there is a way to get a script (shell or otherwise) fast enough to perform the job. I also think that we benefit a lot from keeping some state in memory, whereas the script will have to parse the old file first, generate a new file and then compare the result. That will take a long time. >> I had Jon's script down to 70ms after I removed the dhcp-leases >> file updates, used my fast method for deriving the domain name and >> removed Jon's initial setting up bit which is not needed once the >> changes are accepted. This is on a 2c VM running at 2GHz. I thought >> the goal was < 100ms which has been done comfortably. >> >> FWIW, if you no longer maintain dhcp-leases in the Jon's script you >> never read the old file or do a compare. All you do is read the >> hosts and static-leases files and nothing is written to file. >>> Do we needed the leases file generated at startup? Yes, I think so, too. DNS is quite essential for the network and for local services to be able to start. So it has to come up very early. If it comes up with proper configuration we avoid any other problems where hosts might resolve names too early and get a different result compared to a couple of seconds later when the zone is fully initialised. >> My cut-down version of unbound-dhcp-leases-bridge is at >> https://bugzilla.ipfire.org/attachment.cgi?id=1516 on >> https://bugzilla.ipfire.org/show_bug.cgi?id=13254. On my test >> system it takes 219ms to run so delays unbound starting up minimally. >> >> At the same time there is a gain in the dhcp start up as the >> unbound-dhcp-leases-bridge is no longer needed and it therefore >> avoids triggering an unbound restart which happens every time >> unbound-dhcp-leases-bridge is run. >> >> If we are generating dhcp-leases.conf at start up, then there is no >> point in maintaining it while unbound is running as the maintained >> file will never be read. >>> So all in all, I think the route I would most likely look at is changing the daemon to open a socket. We write minimal scripts which are called by dhcpd that simply pipe their data into the socket which the daemon will then parse. With all available leases in memory already, we can check if a lease has changed or if it has been renewed without any further changes. If nothing has changed, we don’t need to do anything. In case of a change, we will have to write the file again and have Unbound reload, or we even call unbound-control and make the one change without a reload. Doing that in the daemon should give us more flexibility and we avoid any performance penalties as the daemon could run multiple commands concurrently which dhcpd can’t do. >> Sockets are beyond me, I am afraid. >> >> Part of the goal of all this is to avoid reloading unbound and part >> of my goal is to avoid writing dhcp-leases.conf unless unbound is >> restarted. I am not sure why you are still mentioning writing a >> file and restarting/reloading unbound. >>>> Also on his patches, I had a significantly faster way of determining the domain name (in bash) and he and I were looking for your feedback on if we'd be allowed to use it. >>> Anything is allowed. It just has to be fast. And for that, I believe, calling any other binary will be a problem. >> OK. I was proposing to read the DOMAINNAME with: >> >> DOMAINNAME=${HOSTNAME#*.*} >> >> which is many times faster than the conventional route: >> >> eval $(/usr/local/bin/readhash /var/ipfire/main/settings) >> >> If there is any concern with bash not having the full environment >> when the program is called, the shebang could be changed to: >> >> #!/bin/bash -l >> >> Nick >>> -Michael >>> >>>> Regards, >>>> >>>> Nick >>>>> Best, >>>>> -Michael >>>>> >>>>> >>>>>> On 10/04/2024 12:04, Nick Howitt wrote: >>>>>> >>>>>>> Bump >>>>>>> >>>>>>> On 28/03/2024 17:25, Nick Howitt wrote: >>>>>>> >>>>>>>> Next version. >>>>>>>> >>>>>>>> On 28/03/2024 10:41, Michael Tremer wrote: >>>>>>>> >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> >>>>>>>>>> On 27 Mar 2024, at 15:28, Nick Howitt<nick@howitts.co.uk> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 26/03/2024 11:10, Michael Tremer wrote: >>>>>>>>>> >>>>>>>>>>>> On 25 Mar 2024, at 13:11, Nick Howitt<nick@howitts.co.uk> wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 25/03/2024 11:31, Michael Tremer wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hello, >>>>>>>>>>>>> I don’t think that the risk is very high, but I would account for this problem. >>>>>>>>>>>>> With my proposed changes from my previous email this should not be a large problem at all. When opening /etc/unbound/dhcp-leases.conf with mode “r+” you can catch the Exception and skip the comparison and open the file again with “w”. >>>>>>>>>>>>> >>>>>>>>>>>>>> On 25 Mar 2024, at 10:10, Nick Howitt<nick@howitts.co.uk> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> If there is a risk that the /etc/unbound/dhcp-leases.conf does not exist, for example on a clean install (I don't know), the filecmp.cmp line needs to be wrapped in a try...except to stop it blowing up: >>>>>>>>>>>>>> >>>>>>>>>>>>>> try: >>>>>>>>>>>>>> RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) >>>>>>>>>>>>>> except: >>>>>>>>>>>>>> RequireUnboundReload = True >>>>>>>>>>>>>> >>>>>>>>>>>>> I generally don’t like blanco exception catches. You will never know about the daemons that you are hiding. >>>>>>>>>>>>> >>>>>>>>>>>>>> This will force the file to be to be generated, even if it is only copying in a blank temp file, and unbound will restart. It will only ever happen once. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24/03/2024 15:49, Nick Howitt wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>> **** This is a repost of an email incorrectly submitted. **** >>>>>>>>>>>>>>> Please bear with me as I am new to IPFire and not at all used to this style of development, so any guidance would be appreciated. >>>>>>>>>>>>>>> There is a bug,https://bugzilla.ipfire.org/show_bug.cgi?id=13254, where unbound-dhcp-leases-bridge restarts unbound a lot when it is totally unnecessary. It appears to be because when a lease is renewed, the script gets triggered, created a new /etc/unbound/dhcp-leases.conf then restarts Unbound to read the file. Generally this seems to be unnecessary as, with a renewed lease, the old and new /etc/unbound/dhcp-leases.conf files are the same. With 5 leases in that file (one for a machine not active) I am getting 3-4 restarts an hour of Unbound when I have a min/max lease time of 60/120min. >>>>>>>>>>>>>>> Looking at the code, it is fairly easy to fix. The current code creates a temp file then copies it into place then restarts unbound. All I am doing is doing a file comparison before the copy and skipping the restart if the files are the same. >>>>>>>>>>>>>>> There were a couple of gotchas because setting the file attributes and copying the file were done inside the "with" block for generating the temporary file. This meant a file comparison always returned False as the temp file was still open and so never the same is the old file. >>>>>>>>>>>>>>> I moved those two statements outside the "with". This forced me to change the fchmod to chmod. >>>>>>>>>>>>>>> It could be argued that the file copy should not be done if the files are not different, but it would take an extra "if" and you'd have to remember to delete the temp file. If required, I can make that change. >>>>>>>>>>>>>>> Also, one small thing I noticed once is that the old and new dhcp-leases.conf files could occasionally contain the same leases but in a different order. I have been unable to reproduce, but to sidestep it, instead of stepping through the leases variable directly, I sort it and step through that. It should make the resulting file completely deterministic and make the file comparison more effective. >>>>>>>>>>>>>>> My patch is: >>>>>>>>>>>>>>> >From 73873d4944944a2f02317a73074a6894726f36f7 Mon Sep 17 00:00:00 2001 >>>>>>>>>>>>>>> From: Nick Howitt<nick@howitts.co.uk> >>>>>>>>>>>>>>> Date: Sun, 24 Mar 2024 15:17:19 +0000 >>>>>>>>>>>>>>> Subject: [PATCH] Stop unbound-dhcp-leases-bridge from restarting unbound every >>>>>>>>>>>>>>> time it write dhcp-leases.conf as it is very often unchanged and does not >>>>>>>>>>>>>>> require a restart. >>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>> config/unbound/unbound-dhcp-leases-bridge | 28 +++++++++++++++-------- >>>>>>>>>>>>>>> 1 file changed, 19 insertions(+), 9 deletions(-) >>>>>>>>>>>>>>> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>>>>>>>> index e9f022aff..d22772066 100644 >>>>>>>>>>>>>>> --- a/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>>>>>>>> +++ b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>>>>>>>> @@ -22,6 +22,7 @@ >>>>>>>>>>>>>>> import argparse >>>>>>>>>>>>>>> import datetime >>>>>>>>>>>>>>> import daemon >>>>>>>>>>>>>>> +import filecmp >>>>>>>>>>>>>>> import functools >>>>>>>>>>>>>>> import ipaddress >>>>>>>>>>>>>>> import logging >>>>>>>>>>>>>>> @@ -516,26 +517,35 @@ class UnboundConfigWriter(object): >>>>>>>>>>>>>>> def update_dhcp_leases(self, leases): >>>>>>>>>>>>>>> # Write out all leases >>>>>>>>>>>>>>> - self.write_dhcp_leases(leases) >>>>>>>>>>>>>>> + if self.write_dhcp_leases(leases): >>>>>>>>>>>>>>> - log.debug("Reloading Unbound...") >>>>>>>>>>>>>>> + log.debug("Reloading Unbound...") >>>>>>>>>>>>>>> - # Reload the configuration without dropping the cache >>>>>>>>>>>>>>> - self._control("reload_keep_cache") >>>>>>>>>>>>>>> + # Reload the configuration without dropping the cache >>>>>>>>>>>>>>> + self._control("reload_keep_cache") >>>>>>>>>>>>>>> + >>>>>>>>>>>>>>> + else: >>>>>>>>>>>>>>> + >>>>>>>>>>>>>>> + log.debug("Not reloading Unbound. Leases not changed.") >>>>>>>>>>>>>>> def write_dhcp_leases(self, leases): >>>>>>>>>>>>>>> log.debug("Writing DHCP leases...") >>>>>>>>>>>>>>> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >>>>>>>>>>>>>>> - for l in leases: >>>>>>>>>>>>>>> + for l in sorted(leases): >>>>>>>>>>>>>>> for rr in l.rrset: >>>>>>>>>>>>>>> f.write("local-data: \"%s\"\n" % " ".join(rr)) >>>>>>>>>>>>>>> - # Make file readable for everyone >>>>>>>>>>>>>>> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>>>>>>>>> + filecmp.clear_cache() >>>>>>>>>>>>>>> + RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) >>>>>>>>>>>>>>> + >>>>>>>>>>>>>>> + # Make file readable for everyone >>>>>>>>>>>>>>> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>>>>>>>>> + >>>>>>>>>>>>>>> + # Move the file to its destination >>>>>>>>>>>>>>> + os.rename(f.name, self.path) >>>>>>>>>>>>>>> - # Move the file to its destination >>>>>>>>>>>>>>> - os.rename(f.name, self.path) >>>>>>>>>>>>>>> + return RequireUnboundReload >>>>>>>>>>>>>>> def _control(self, *args): >>>>>>>>>>>>>>> command = ["unbound-control"] >>>>>>>>>>>>>>> >>>>>>>>>>>> Resubmitting the patch. >>>>>>>>>>>> >>>>>>>>>>>> Following my previous reply, I would like to stick with filecmp as I am using it with "shallow=False" to avoid doing the comparison just with stat() calls. Note that I also flush the filecmp cache before using the command so it is running with the latest file data. Talking about file descriptors, atomic file replacement etc., scares me. >>>>>>>>>>>> >>>>>>>>>>> File descriptors are great. They avoid race-conditions and all sorts of things. >>>>>>>>>>> >>>>>>>>>> But I have never used them so I don't intend to go that way. Too much for me to get my brain round and a limited ability to test. >>>>>>>>>> >>>>>>>>> Well, I pretty much gave you the entire solution… There was nothing left to do apart from copying and pasting the snippets. >>>>>>>>> >>>>>>>>> >>>>>>>> Ok I've tried to use your version but struggled to get it going. If I used it as was and called compare(file1_name, file2_name) it failed as "compare" was not defined. >>>>>>>> I could call it using self.compare but then it failed with "TypeError: UnboundConfigWriter.compare() takes 2 positional arguments but 3 were given". I changed the function to compare(self, f1, f2). >>>>>>>> Then it failed with "AttributeError: 'str' object has no attribute 'seek'", so I changed it again to accept file names then open the files before doing the seeks. >>>>>>>> I really hope it is OK as I am out of my depth when defining functions with "self" as an argument. Please can you check it? If it is not OK, can I revert to filecmp? >>>>>>>> >>>>>>>> I have made one futher change, the sorted(leases) line was not working and not sorting the object ever, and I saw lines changing places in the resulting leases file. I have changed it to "sorted(leases, key=lambda x: x.ipaddr)". It definitely sorts this time. >>>>>>>> >>>>>>>> >>>>>>>>>>> I don’t think that module is the best choice really if you have to hack around and flush the cache. Why would it even cache things in the first place? I don’t see where the cache helps. >>>>>>>>>>> >>>>>>>>>> From the filecmp.clear_cache documentation: >>>>>>>>>> >>>>>>>>>> Clear the filecmp cache. This may be useful if a file is compared so >>>>>>>>>> quickly after it is modified that it is within the mtime resolution >>>>>>>>>> of the underlying filesystem. >>>>>>>>>> >>>>>>>>>> Now this makes me believe that the clear_cache is only relevant when using shallow=True, which I'm not and I don't know how to check the underlying code. It is used to make sure the stat() info is updated, but we are not using the stat() info. I guess we could as easily add a one second sleep instead as that is way longer than the mtime resolution. >>>>>>>>>> >>>>>>>>> The function is here and it is rather simple:https://github.com/python/cpython/blob/3.12/Lib/filecmp.py#L30 >>>>>>>>> >>>>>>>>> It will always stat() both files and compares three items in case shallow is False: The type of the file, the size and mtime. If the file is not a regular file, the function will exist rather early. So in our case, that is all work the function is doing for thing. It boils down to something very similar to what I suggested to use to compare the files. >>>>>>>>> >>>>>>>>> On top of that, there is a cache which will always be used but will never return a match for us because the temporary file at least will always have another mtime. >>>>>>>>> >>>>>>>>> You could clear the cache after every call, but I would rather not have our code messy and care about implementation details of some Python module. >>>>>>>>> >>>>>>>>> >>>>>>>>>> The module is chosen because all the references I read about comparing files preferred this one-liner for comparing files, although other methods were given and there were lots of discussions about large files (GB+) because if you chomp them it may be much quicker to say they are not the same when you find the first block that does not match. Having said this, I don't know how filecmp works, but we only have a small file. At the same time, I am not a regular programmer so simple one-liners are good for me. >>>>>>>>>> >>>>>>>>> You would always want to read both files block for block and compare the blocks. There is no way to implement this significantly faster. >>>>>>>>> >>>>>>>>> >>>>>>>>>>>> I have modified the earlier try/except I proposed to catch only FileNotFoundError, and I have changed CamelCase to snake_case. >>>>>>>>>>>> >>>>>>>>>>>> --- >>>>>>>>>>>> config/unbound/unbound-dhcp-leases-bridge | 31 ++++++++++++++++------- >>>>>>>>>>>> 1 file changed, 22 insertions(+), 9 deletions(-) >>>>>>>>>>>> >>>>>>>>>>>> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>>>>> index e9f022aff..a9ec8e1e2 100644 >>>>>>>>>>>> --- a/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>>>>> +++ b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>>>>> @@ -22,6 +22,7 @@ >>>>>>>>>>>> import argparse >>>>>>>>>>>> import datetime >>>>>>>>>>>> import daemon >>>>>>>>>>>> +import filecmp >>>>>>>>>>>> import functools >>>>>>>>>>>> import ipaddress >>>>>>>>>>>> import logging >>>>>>>>>>>> @@ -516,26 +517,38 @@ class UnboundConfigWriter(object): >>>>>>>>>>>> >>>>>>>>>>>> def update_dhcp_leases(self, leases): >>>>>>>>>>>> # Write out all leases >>>>>>>>>>>> - self.write_dhcp_leases(leases) >>>>>>>>>>>> + if self.write_dhcp_leases(leases): >>>>>>>>>>>> >>>>>>>>>>>> - log.debug("Reloading Unbound...") >>>>>>>>>>>> + log.debug("Reloading Unbound...") >>>>>>>>>>>> >>>>>>>>>>>> - # Reload the configuration without dropping the cache >>>>>>>>>>>> - self._control("reload_keep_cache") >>>>>>>>>>>> + # Reload the configuration without dropping the cache >>>>>>>>>>>> + self._control("reload_keep_cache") >>>>>>>>>>>> + >>>>>>>>>>>> + else: >>>>>>>>>>>> + >>>>>>>>>>>> + log.debug("Not reloading Unbound. Leases not changed.") >>>>>>>>>>>> >>>>>>>>>>>> def write_dhcp_leases(self, leases): >>>>>>>>>>>> log.debug("Writing DHCP leases...") >>>>>>>>>>>> >>>>>>>>>>>> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >>>>>>>>>>>> - for l in leases: >>>>>>>>>>>> + for l in sorted(leases): >>>>>>>>>>>> for rr in l.rrset: >>>>>>>>>>>> f.write("local-data: \"%s\"\n" % " ".join(rr)) >>>>>>>>>>>> >>>>>>>>>>>> - # Make file readable for everyone >>>>>>>>>>>> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>>>>>> + filecmp.clear_cache() >>>>>>>>>>>> + try: >>>>>>>>>>>> + require_unbound_reload = not filecmp.cmp(f.name, self.path, shallow=False) >>>>>>>>>>>> + except FileNotFoundError: >>>>>>>>>>>> + require_unbound_reload = True >>>>>>>>>>>> >>>>>>>>>>> If you want to go this route, you will have to flush the temporary file first. Otherwise you might have some data left in the buffer that has not been written (because the file is still open). You will then compare incomplete output. This would not become a problem if you used the file descriptors. >>>>>>>>>>> >>>>>>>>>> The temp file is closed at the point of doing the filecmp, and the filecmp cache has been flushed. How do I flush the temporary file further. Reading up, is it just having the last statement in the with block as f.flush() with the same indent as "for l in sorted(leases):" >>>>>>>>>> >>>>>>>>> If you have left the with block you will already have flushed the buffers and closed the file. It looks like I misread the indentation in the patch then. >>>>>>>>> >>>>>>>>> >>>>>>>>>>> Catching FileNotFound would work for me. >>>>>>>>>>> >>>>>>>>>> Good >>>>>>>>>> >>>>>>>>>>>> + >>>>>>>>>>>> + # Make file readable for everyone >>>>>>>>>>>> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>>>>>> + >>>>>>>>>>>> + # Move the file to its destination >>>>>>>>>>>> + os.rename(f.name, self.path) >>>>>>>>>>>> >>>>>>>>>>>> - # Move the file to its destination >>>>>>>>>>>> - os.rename(f.name, self.path) >>>>>>>>>>>> + return require_unbound_reload >>>>>>>>>>>> >>>>>>>>>>>> def _control(self, *args): >>>>>>>>>>>> command = ["unbound-control"] >>>>>>>>>>>> -- >>>>>>>>>>>> 2.43.0 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Nick >>>>>>>>>>>> >>>>>>>> --- >>>>>>>> config/unbound/unbound-dhcp-leases-bridge | 50 +++++++++++++++++++---- >>>>>>>> 1 file changed, 41 insertions(+), 9 deletions(-) >>>>>>>> >>>>>>>> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>> index e9f022aff..80d9523ea 100644 >>>>>>>> --- a/config/unbound/unbound-dhcp-leases-bridge >>>>>>>> +++ b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>> @@ -516,26 +516,37 @@ class UnboundConfigWriter(object): >>>>>>>> >>>>>>>> def update_dhcp_leases(self, leases): >>>>>>>> # Write out all leases >>>>>>>> - self.write_dhcp_leases(leases) >>>>>>>> + if self.write_dhcp_leases(leases): >>>>>>>> >>>>>>>> - log.debug("Reloading Unbound...") >>>>>>>> + log.debug("Reloading Unbound...") >>>>>>>> >>>>>>>> - # Reload the configuration without dropping the cache >>>>>>>> - self._control("reload_keep_cache") >>>>>>>> + # Reload the configuration without dropping the cache >>>>>>>> + self._control("reload_keep_cache") >>>>>>>> + >>>>>>>> + else: >>>>>>>> + >>>>>>>> + log.debug("Not reloading Unbound. Leases not changed.") >>>>>>>> >>>>>>>> def write_dhcp_leases(self, leases): >>>>>>>> log.debug("Writing DHCP leases...") >>>>>>>> >>>>>>>> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >>>>>>>> - for l in leases: >>>>>>>> + for l in sorted(leases, key=lambda x: x.ipaddr): >>>>>>>> for rr in l.rrset: >>>>>>>> f.write("local-data: \"%s\"\n" % " ".join(rr)) >>>>>>>> >>>>>>>> - # Make file readable for everyone >>>>>>>> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>> + try: >>>>>>>> + require_unbound_reload = not self.compare(f.name, self.path) >>>>>>>> + except FileNotFoundError: >>>>>>>> + require_unbound_reload = True >>>>>>>> + >>>>>>>> + # Make file readable for everyone >>>>>>>> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>> + >>>>>>>> + # Move the file to its destination >>>>>>>> + os.rename(f.name, self.path) >>>>>>>> >>>>>>>> - # Move the file to its destination >>>>>>>> - os.rename(f.name, self.path) >>>>>>>> + return require_unbound_reload >>>>>>>> >>>>>>>> def _control(self, *args): >>>>>>>> command = ["unbound-control"] >>>>>>>> @@ -551,6 +562,27 @@ class UnboundConfigWriter(object): >>>>>>>> >>>>>>>> raise e >>>>>>>> >>>>>>>> + def compare(self, if1, if2): >>>>>>>> + buffersize = 16 * 1024 >>>>>>>> + >>>>>>>> + f1 = open(if1, "r") >>>>>>>> + f2 = open(if2, "r") >>>>>>>> + >>>>>>>> + # Reset file handles >>>>>>>> + f1.seek(0) >>>>>>>> + f2.seek(0) >>>>>>>> + >>>>>>>> + while True: >>>>>>>> + b1 = f1.read(buffersize) >>>>>>>> + b2 = f2.read(buffersize) >>>>>>>> + >>>>>>>> + if not b1 == b2: >>>>>>>> + return False >>>>>>>> + >>>>>>>> + elif not b1 or not b2: >>>>>>>> + break >>>>>>>> + >>>>>>>> + return True >>>>>>>> >>>>>>>> if __name__ == "__main__": >>>>>>>> parser = argparse.ArgumentParser(description="Bridge for DHCP Leases and Unbound DNS") >>>>>>>> >> > Jon Murphy jon.murphy@ipfire.org
Are you running with Michael's updates? I know that does a reload as dhcp starts.
Yes (and none of my proof-of-concept code).
On Jun 18, 2024, at 4:17 AM, Nick Howitt <nick@howitts.co.uk mailto:nick@howitts.co.uk> wrote:
Found the discussion on bugzilla from comment 150 (https://bugzilla.ipfire.org/show_bug.cgi?id=13254#c150 https://bugzilla.ipfire.org/show_bug.cgi?id=13254#c150) through to comment 153.
On 18/06/2024 09:57, Nick Howitt wrote:
Are you running with Michael's updates? I know that does a reload as dhcp starts.
We did have some correspondence on that approach and concluded it had an issue but I can't remember the details or find the correspondence. IIRC there was a significant weakness in the approach and concluded it would be better to regenerate the dhcp-leases.conf file every time unbound restarted, and at that point, the code Michael has in his unbound-dhcp-leases-script which leads up to the reload can be completely removed. This would mean using my cutdown unbound-dhcp-leases script during the unbound restart and paring back his unbound-dhcp-leases-script. This would sidestep your problem.
I had been hoping for an update along these lines but it looks like other things are taking his time. I could set up a repo locally with all the changes, but would not be able to create a multipart patch set in the way that is required, especially as new files are created. It would end up as updates to both the unbound app and the dhcp app.
Nick
On 18/06/2024 04:27, jon wrote:
Concerning CU186, I may have stumbled across an issue. It took a few days for it to pop up. See the message log below from Jun 17 13:48:34 to the end. I experienced a flurry of Reloads and DNS stalled for ~ 15 minutes.
[root@ipfire ~] # grep --color "Reloading Unbound" /var/log/messages Jun 16 10:43:48 ipfire dhcp[3615]: Reloading Unbound... Jun 16 10:58:56 ipfire dhcp[3615]: Reloading Unbound... Jun 16 11:29:26 ipfire dhcp[3615]: Reloading Unbound... Jun 16 11:43:46 ipfire dhcp[3615]: Reloading Unbound... Jun 16 15:08:10 ipfire dhcp[3615]: Reloading Unbound... Jun 16 21:48:53 ipfire dhcp[3615]: Reloading Unbound... Jun 17 06:52:30 ipfire dhcp[3615]: Reloading Unbound... Jun 17 11:38:59 ipfire dhcp[3615]: Reloading Unbound... Jun 17 11:41:16 ipfire dhcp[3615]: Reloading Unbound... Jun 17 11:41:33 ipfire dhcp[3615]: Reloading Unbound... Jun 17 11:42:23 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:39:37 ipfire dhcp[3615]: Reloading Unbound...
Jun 17 13:48:34 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:48:39 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:49:47 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:50:47 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:52:00 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:52:43 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:53:43 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:54:43 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:55:49 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:56:55 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:58:07 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:58:44 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:59:44 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:00:44 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:01:52 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:02:52 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:04:10 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:04:53 ipfire dhcp[3615]: Reloading Unbound... [root@ipfire ~] #
It seems like a new "Reloading Unbound…" was arriving before the "start of service (unbound 1.19.3)" actually happened:
Jun 17 13:48:35 ipfire dhcpd: DHCPDISCOVER from 20:91:df:00:01:02 via blue0 Jun 17 13:48:36 ipfire dhcpd: DHCPOFFER on 192.168.65.138 to 20:91:df:00:01:02 (JiPhoneSE) via blue0 Jun 17 13:48:37 ipfire dhcpd: DHCPREQUEST for 192.168.65.138 (192.168.65.1) from 20:91:df:00:01:02 (JiPhoneSE) via blue0 Jun 17 13:48:37 ipfire dhcpd: DHCPACK on 192.168.65.138 to 20:91:df:00:01:02 (JiPhoneSE) via blue0 Jun 17 13:48:39 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:49:13 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:49:13 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:49:36 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:49:36 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:49:36 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:49:36 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:49:36 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:49:36 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:49:42 ipfire dhcpd: DHCPREQUEST for 192.168.65.110 from a8:8f:d9:00:01:02 (iPhone-W) via blue0 Jun 17 13:49:42 ipfire dhcpd: DHCPACK on 192.168.65.110 to a8:8f:d9:00:01:02 (iPhone-W) via blue0 Jun 17 13:49:47 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:49:59 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:49:59 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:50:05 ipfire dhcpd: DHCPREQUEST for 192.168.65.103 from 00:07:a6:00:01:02 (LEVDS-Dimmer-7BCA) via blue0 Jun 17 13:50:05 ipfire dhcpd: DHCPACK on 192.168.65.103 to 00:07:a6:00:01:02 (LEVDS-Dimmer-7BCA) via blue0 Jun 17 13:50:37 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:50:37 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:50:37 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:50:37 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:50:37 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:50:37 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:50:44 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:50:44 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:50:47 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:51:30 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:51:30 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:51:37 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:51:37 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:51:37 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:51:37 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:51:37 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:51:37 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:52:00 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:52:15 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:52:15 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:52:37 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:52:37 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:52:37 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:52:37 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:52:37 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:52:37 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:52:43 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:52:44 ipfire dhcpd: DHCPREQUEST for 192.168.65.180 from 34:f3:9a:00:01:02 (deb12dell) via blue0 Jun 17 13:52:44 ipfire dhcpd: DHCPACK on 192.168.65.180 to 34:f3:9a:00:01:02 (deb12dell) via blue0 Jun 17 13:53:00 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:53:00 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:53:38 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:53:38 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:53:38 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:53:38 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:53:38 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:53:38 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:53:43 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:53:46 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:53:46 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:54:31 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:54:31 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:54:38 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:54:38 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:54:38 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:54:38 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:54:38 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:54:38 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:54:43 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:55:19 ipfire dhcpd: DHCPREQUEST for 192.168.65.10 from 74:ac:b9:00:01:02 via blue0 Jun 17 13:55:19 ipfire dhcpd: DHCPACK on 192.168.65.10 to 74:ac:b9:00:01:02 via blue0 Jun 17 13:55:22 ipfire dhcpd: DHCPDISCOVER from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:55:22 ipfire dhcpd: DHCPOFFER on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:55:22 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 (192.168.65.1) from 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:55:22 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:55:38 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:55:38 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:55:38 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:55:38 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:55:38 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:55:38 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:55:49 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:55:55 ipfire dhcpd: DHCPREQUEST for 192.168.65.105 from 44:61:32:00:01:02 (ecobee) via blue0 Jun 17 13:55:55 ipfire dhcpd: DHCPACK on 192.168.65.105 to 44:61:32:00:01:02 (ecobee) via blue0 Jun 17 13:55:58 ipfire dhcpd: DHCPREQUEST for 192.168.60.250 from 9c:93:4e:00:01:02 via green0 Jun 17 13:55:58 ipfire dhcpd: DHCPACK on 192.168.60.250 to 9c:93:4e:00:01:02 via green0 Jun 17 13:56:02 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:56:02 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:56:38 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:56:38 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:56:38 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:56:38 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:56:38 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:56:38 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:56:53 ipfire dhcpd: DHCPDISCOVER from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:56:53 ipfire dhcpd: DHCPOFFER on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:56:53 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 (192.168.65.1) from 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:56:53 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:56:55 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:57:33 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:57:33 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:57:34 ipfire dhcpd: DHCPREQUEST for 192.168.60.240 from 8c:dc:d4:00:01:02 via green0 Jun 17 13:57:34 ipfire dhcpd: DHCPACK on 192.168.60.240 to 8c:dc:d4:00:01:02 via green0 Jun 17 13:57:39 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:57:39 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:57:39 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:57:39 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:57:39 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:57:39 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:58:03 ipfire dhcpd: DHCPREQUEST for 192.168.65.160 from 14:88:e6:00:01:02 (SiPad) via blue0 Jun 17 13:58:03 ipfire dhcpd: DHCPACK on 192.168.65.160 to 14:88:e6:00:01:02 (SiPad) via blue0 Jun 17 13:58:07 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:58:08 ipfire dhcpd: DHCPREQUEST for 192.168.60.203 from dc:a6:32:00:01:02 via green0 Jun 17 13:58:08 ipfire dhcpd: DHCPACK on 192.168.60.203 to dc:a6:32:00:01:02 via green0 Jun 17 13:58:19 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:58:19 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:58:39 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:58:39 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:58:39 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:58:39 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:58:39 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:58:39 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:58:44 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:59:04 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:59:04 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 13:59:39 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 13:59:39 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 13:59:39 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 13:59:39 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 13:59:39 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 13:59:39 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 13:59:44 ipfire dhcp[3615]: Reloading Unbound... Jun 17 13:59:50 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 13:59:50 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 14:00:35 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 14:00:35 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 14:00:39 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 14:00:39 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 14:00:39 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 14:00:39 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 14:00:39 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 14:00:39 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 14:00:44 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:01:21 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 14:01:21 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 14:01:46 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 14:01:46 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 14:01:46 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 14:01:47 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 14:01:47 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 14:01:47 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 14:01:52 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:02:06 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 14:02:06 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 14:02:47 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 14:02:47 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 14:02:47 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 14:02:47 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 14:02:47 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 14:02:47 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 14:02:51 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 14:02:52 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 14:02:52 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:03:19 ipfire dhcpd: DHCPDISCOVER from 5e:24:53:00:01:02 via blue0 Jun 17 14:03:20 ipfire dhcpd: DHCPOFFER on 192.168.65.153 to 5e:24:53:00:01:02 via blue0 Jun 17 14:03:21 ipfire dhcpd: DHCPREQUEST for 192.168.65.153 (192.168.65.1) from 5e:24:53:00:01:02 via blue0 Jun 17 14:03:21 ipfire dhcpd: DHCPACK on 192.168.65.153 to 5e:24:53:00:01:02 via blue0 Jun 17 14:03:38 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 14:03:38 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 14:03:44 ipfire dhcpd: DHCPREQUEST for 192.168.65.109 from ac:c9:06:00:01:02 (MBA1) via blue0 Jun 17 14:03:44 ipfire dhcpd: DHCPACK on 192.168.65.109 to ac:c9:06:00:01:02 (MBA1) via blue0 Jun 17 14:03:47 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 14:03:47 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 14:03:47 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 14:03:47 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 14:03:47 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 14:03:47 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 14:04:02 ipfire dhcpd: DHCPREQUEST for 192.168.65.109 from ac:c9:06:00:01:02 (MBA1) via blue0 Jun 17 14:04:02 ipfire dhcpd: DHCPACK on 192.168.65.109 to ac:c9:06:00:01:02 (MBA1) via blue0 Jun 17 14:04:10 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:04:23 ipfire dhcpd: DHCPREQUEST for 192.168.65.192 from 4c:fc:aa:00:01:02 via blue0 Jun 17 14:04:23 ipfire dhcpd: DHCPACK on 192.168.65.192 to 4c:fc:aa:00:01:02 (Car) via blue0 Jun 17 14:04:48 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 14:04:48 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 14:04:48 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 14:04:48 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 14:04:48 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 14:04:48 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 14:04:53 ipfire dhcp[3615]: Reloading Unbound... Jun 17 14:05:41 ipfire dhcpd: DHCPREQUEST for 192.168.65.106 from 40:c7:11:00:01:02 (SiPhone12) via blue0 Jun 17 14:05:41 ipfire dhcpd: DHCPACK on 192.168.65.106 to 40:c7:11:00:01:02 (SiPhone12) via blue0 Jun 17 14:05:49 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 14:05:49 ipfire unbound: [2118:0] info: service stopped (unbound 1.19.3). Jun 17 14:05:49 ipfire unbound: [2118:0] notice: Restart of unbound 1.19.3. Jun 17 14:05:49 ipfire unbound: [2118:0] notice: init module 0: respip Jun 17 14:05:49 ipfire unbound: [2118:0] notice: init module 1: validator Jun 17 14:05:49 ipfire unbound: [2118:0] notice: init module 2: iterator Jun 17 14:06:50 ipfire unbound: [2118:0] info: start of service (unbound 1.19.3). Jun 17 14:06:56 ipfire dhcpd: DHCPREQUEST for 192.168.65.109 from ac:c9:06:00:01:02 (MBA1) via blue0 Jun 17 14:06:56 ipfire dhcpd: DHCPACK on 192.168.65.109 to ac:c9:06:00:01:02 (MBA1) via blue0 Jun 17 14:07:01 ipfire dhcpd: DHCPREQUEST for 192.168.65.108 from 04:c2:9b:00:01:02 (Aura-9413) via blue0 Jun 17 14:07:01 ipfire dhcpd: DHCPACK on 192.168.65.108 to 04:c2:9b:00:01:02 (Aura-9413) via blue0 Jun 17 14:10:02 ipfire dhcpd: DHCPREQUEST for 192.168.60.20 from 0c:4d:e9:00:01:02 via green0 Jun 17 14:10:02 ipfire dhcpd: DHCPACK on 192.168.60.20 to 0c:4d:e9:00:01:02 via green0 Things seem to return to normal after that…. Jon > On Apr 30, 2024, at 5:58 AM, Nick Howitt <nick@howitts.co.uk <mailto:nick@howitts.co.uk>> wrote: > > > > On 27/04/2024 10:53, Nick Howitt wrote: >> >> >> On 27/04/2024 09:35, Michael Tremer wrote: >>> Hello Nick, >>> >>>> On 26 Apr 2024, at 17:48, Nick Howitt <nick@howitts.co.uk> <mailto:nick@howitts.co.uk> wrote: >>>> >>>> >>>> >>>> On 26/04/2024 16:12, Michael Tremer wrote: >>>>> Hello Nick, >>>>> >>>>> >>>>>> On 25 Apr 2024, at 13:43, Nick Howitt <nick@howitts.co.uk> <mailto:nick@howitts.co.uk> wrote: >>>>>> >>>>>> Please can I bump this again to request a review of my patch submitted on 28th March? Thanks, Nick. >>>>>> >>>>> It is a little bit difficult to test your proposed changes if testers have to copy and paste them out of an email. >>>> Confused. I was told I had to submit the diff in an email with [PATCH] in the title and not to attach a file. Please can you tell me how I should be operating? >>> No worries, we are going to sort this out. >>> >>> So the first email in the thread was a correct patch. It was just the patch in the format that Git uses and it was properly parsed by Patchwork. Later in the same thread, the patch was just copied and pasted which was impossible for Patchwork to pick up. There is nothing wrong with using bits of code in the conversation, but it would have been nice to have the final version submitted once again. >>> >>>>> I have just submitted a patch set to this list that breaks down all the changes that you proposed and I cleaned up the code so that we can use the filecmp function without any extra steps. We can simply jump out of the write function and return False to prevent Unbound being reloaded. >>>> I thought you didn't want me to use filecmp and pushed me to use your file compare routine. >>> Yeah, I would have preferred that too, but I thought using the module might be quicker now. >>> >>>>> We will also avoid replacing the file all the time as that causes lots of unnecessary disk I/O. I am not saying it makes a significant difference, but flash has a limited write count that we are still wasting. >>>> OK, but my changes were no worse than before, either doing a delete or rename whereas before it always did a rename. The "unnecessary I/O" was already there. >>>>> Please review and you if you are happy, please add the appropriate Git tag. >>>> Sorry but I don't know what you mean by "adding the appropriate Git tag". I am only using Git locally and have no repo access. >>> https://www.ipfire.org/docs/devel/git/tags <https://www.ipfire.org/docs/devel/git/tags> > I tested the patches yesterday and today and I am happy with them. I hope I have signed them off correctly. > > FWIW I hand patched the program as it was easier than learning something new in this case as the patch set is so small. > > One thing not mentioned in the bug report. Not only is unbound restarting on renewals which this fixes, it also restarts on new leases **and** expiring leases. For big work environments, I would have thought the best workaround is to set the default DHCP lease to 24h. It would help during the week to lessen the restarts but not on Monday mornings. It may also need a bigger address pool to cope with devices randomising MAC addresses. > > Roll on Jon's fix as it would get round all these restarts. I use it modified not to update the dhcp-leases.conf. and another couple of tweaks. Are you able to test it, but it does mean piecing it together? > >> Thanks. >>>> How can I pull your changes? Otherwise I'll need to patch my set up manually. >>> You can apply the patch to your local repository using “git am”: >>> >>> https://git-scm.com/docs/git-am <https://git-scm.com/docs/git-am>Thanks again. >>>> I also have another problem. I have temporarily had to remove my IPF gateway for a business reason and don't know when I'll be able to add it back in. I'll have to see if I can insert it in my LAN somehow. At the same time it is also running Jon's mods which I'll have to undo. >>>> >>>> With respect to Jon's mods, I've asked a couple of times on bugzilla, but is there any point in his routine maintaining /etc/unbound/dhcp-leases.conf. As far as I see the file is only read once when unbound starts, so it only has to be valid when unbound starts and not at any other point in time. To this effect he can remove the file maintenance and I attached a modified copy unbound-dhcp-leases-bridge to his repo which generates a new dhcp-leases.conf and exits, removing all the daemonising bit and the inode watcher. This file can be run as part of the unbound init routine just before the line that actually starts or reloads unbound. It also has the advantage of cutting down a load of disk I/O, which is important as you've just pointed out. >>> I don’t think that we can fully remove the daemon part, because I don’t think that there is a way to get a script (shell or otherwise) fast enough to perform the job. I also think that we benefit a lot from keeping some state in memory, whereas the script will have to parse the old file first, generate a new file and then compare the result. That will take a long time. >> I had Jon's script down to 70ms after I removed the dhcp-leases file updates, used my fast method for deriving the domain name and removed Jon's initial setting up bit which is not needed once the changes are accepted. This is on a 2c VM running at 2GHz. I thought the goal was < 100ms which has been done comfortably. >> >> FWIW, if you no longer maintain dhcp-leases in the Jon's script you never read the old file or do a compare. All you do is read the hosts and static-leases files and nothing is written to file. >>> Do we needed the leases file generated at startup? Yes, I think so, too. DNS is quite essential for the network and for local services to be able to start. So it has to come up very early. If it comes up with proper configuration we avoid any other problems where hosts might resolve names too early and get a different result compared to a couple of seconds later when the zone is fully initialised. >> My cut-down version of unbound-dhcp-leases-bridge is at https://bugzilla.ipfire.org/attachment.cgi?id=1516 <https://bugzilla.ipfire.org/attachment.cgi?id=1516> on https://bugzilla.ipfire.org/show_bug.cgi?id=13254 <https://bugzilla.ipfire.org/show_bug.cgi?id=13254>. On my test system it takes 219ms to run so delays unbound starting up minimally. >> >> At the same time there is a gain in the dhcp start up as the unbound-dhcp-leases-bridge is no longer needed and it therefore avoids triggering an unbound restart which happens every time unbound-dhcp-leases-bridge is run. >> >> If we are generating dhcp-leases.conf at start up, then there is no point in maintaining it while unbound is running as the maintained file will never be read. >>> So all in all, I think the route I would most likely look at is changing the daemon to open a socket. We write minimal scripts which are called by dhcpd that simply pipe their data into the socket which the daemon will then parse. With all available leases in memory already, we can check if a lease has changed or if it has been renewed without any further changes. If nothing has changed, we don’t need to do anything. In case of a change, we will have to write the file again and have Unbound reload, or we even call unbound-control and make the one change without a reload. Doing that in the daemon should give us more flexibility and we avoid any performance penalties as the daemon could run multiple commands concurrently which dhcpd can’t do. >> Sockets are beyond me, I am afraid. >> >> Part of the goal of all this is to avoid reloading unbound and part of my goal is to avoid writing dhcp-leases.conf unless unbound is restarted. I am not sure why you are still mentioning writing a file and restarting/reloading unbound. >>>> Also on his patches, I had a significantly faster way of determining the domain name (in bash) and he and I were looking for your feedback on if we'd be allowed to use it. >>> Anything is allowed. It just has to be fast. And for that, I believe, calling any other binary will be a problem. >> OK. I was proposing to read the DOMAINNAME with: >> DOMAINNAME=${HOSTNAME#*.*} >> which is many times faster than the conventional route: >> eval $(/usr/local/bin/readhash /var/ipfire/main/settings) >> If there is any concern with bash not having the full environment when the program is called, the shebang could be changed to: >> #!/bin/bash -l >> Nick >>> -Michael >>> >>>> Regards, >>>> >>>> Nick >>>>> Best, >>>>> -Michael >>>>> >>>>> >>>>>> On 10/04/2024 12:04, Nick Howitt wrote: >>>>>> >>>>>>> Bump >>>>>>> >>>>>>> On 28/03/2024 17:25, Nick Howitt wrote: >>>>>>> >>>>>>>> Next version. >>>>>>>> >>>>>>>> On 28/03/2024 10:41, Michael Tremer wrote: >>>>>>>> >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> >>>>>>>>>> On 27 Mar 2024, at 15:28, Nick Howitt <nick@howitts.co.uk> <mailto:nick@howitts.co.uk> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 26/03/2024 11:10, Michael Tremer wrote: >>>>>>>>>> >>>>>>>>>>>> On 25 Mar 2024, at 13:11, Nick Howitt <nick@howitts.co.uk> <mailto:nick@howitts.co.uk> wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 25/03/2024 11:31, Michael Tremer wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hello, >>>>>>>>>>>>> I don’t think that the risk is very high, but I would account for this problem. >>>>>>>>>>>>> With my proposed changes from my previous email this should not be a large problem at all. When opening /etc/unbound/dhcp-leases.conf with mode “r+” you can catch the Exception and skip the comparison and open the file again with “w”. >>>>>>>>>>>>> >>>>>>>>>>>>>> On 25 Mar 2024, at 10:10, Nick Howitt <nick@howitts.co.uk> <mailto:nick@howitts.co.uk> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> If there is a risk that the /etc/unbound/dhcp-leases.conf does not exist, for example on a clean install (I don't know), the filecmp.cmp line needs to be wrapped in a try...except to stop it blowing up: >>>>>>>>>>>>>> >>>>>>>>>>>>>> try: >>>>>>>>>>>>>> RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) >>>>>>>>>>>>>> except: >>>>>>>>>>>>>> RequireUnboundReload = True >>>>>>>>>>>>>> >>>>>>>>>>>>> I generally don’t like blanco exception catches. You will never know about the daemons that you are hiding. >>>>>>>>>>>>> >>>>>>>>>>>>>> This will force the file to be to be generated, even if it is only copying in a blank temp file, and unbound will restart. It will only ever happen once. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24/03/2024 15:49, Nick Howitt wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>> **** This is a repost of an email incorrectly submitted. **** >>>>>>>>>>>>>>> Please bear with me as I am new to IPFire and not at all used to this style of development, so any guidance would be appreciated. >>>>>>>>>>>>>>> There is a bug, https://bugzilla.ipfire.org/show_bug.cgi?id=13254 <https://bugzilla.ipfire.org/show_bug.cgi?id=13254>, where unbound-dhcp-leases-bridge restarts unbound a lot when it is totally unnecessary. It appears to be because when a lease is renewed, the script gets triggered, created a new /etc/unbound/dhcp-leases.conf then restarts Unbound to read the file. Generally this seems to be unnecessary as, with a renewed lease, the old and new /etc/unbound/dhcp-leases.conf files are the same. With 5 leases in that file (one for a machine not active) I am getting 3-4 restarts an hour of Unbound when I have a min/max lease time of 60/120min. >>>>>>>>>>>>>>> Looking at the code, it is fairly easy to fix. The current code creates a temp file then copies it into place then restarts unbound. All I am doing is doing a file comparison before the copy and skipping the restart if the files are the same. >>>>>>>>>>>>>>> There were a couple of gotchas because setting the file attributes and copying the file were done inside the "with" block for generating the temporary file. This meant a file comparison always returned False as the temp file was still open and so never the same is the old file. >>>>>>>>>>>>>>> I moved those two statements outside the "with". This forced me to change the fchmod to chmod. >>>>>>>>>>>>>>> It could be argued that the file copy should not be done if the files are not different, but it would take an extra "if" and you'd have to remember to delete the temp file. If required, I can make that change. >>>>>>>>>>>>>>> Also, one small thing I noticed once is that the old and new dhcp-leases.conf files could occasionally contain the same leases but in a different order. I have been unable to reproduce, but to sidestep it, instead of stepping through the leases variable directly, I sort it and step through that. It should make the resulting file completely deterministic and make the file comparison more effective. >>>>>>>>>>>>>>> My patch is: >>>>>>>>>>>>>>> >From 73873d4944944a2f02317a73074a6894726f36f7 Mon Sep 17 00:00:00 2001 >>>>>>>>>>>>>>> From: Nick Howitt <nick@howitts.co.uk> <mailto:nick@howitts.co.uk> >>>>>>>>>>>>>>> Date: Sun, 24 Mar 2024 15:17:19 +0000 >>>>>>>>>>>>>>> Subject: [PATCH] Stop unbound-dhcp-leases-bridge from restarting unbound every >>>>>>>>>>>>>>> time it write dhcp-leases.conf as it is very often unchanged and does not >>>>>>>>>>>>>>> require a restart. >>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>> config/unbound/unbound-dhcp-leases-bridge | 28 +++++++++++++++-------- >>>>>>>>>>>>>>> 1 file changed, 19 insertions(+), 9 deletions(-) >>>>>>>>>>>>>>> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>>>>>>>> index e9f022aff..d22772066 100644 >>>>>>>>>>>>>>> --- a/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>>>>>>>> +++ b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>>>>>>>> @@ -22,6 +22,7 @@ >>>>>>>>>>>>>>> import argparse >>>>>>>>>>>>>>> import datetime >>>>>>>>>>>>>>> import daemon >>>>>>>>>>>>>>> +import filecmp >>>>>>>>>>>>>>> import functools >>>>>>>>>>>>>>> import ipaddress >>>>>>>>>>>>>>> import logging >>>>>>>>>>>>>>> @@ -516,26 +517,35 @@ class UnboundConfigWriter(object): >>>>>>>>>>>>>>> def update_dhcp_leases(self, leases): >>>>>>>>>>>>>>> # Write out all leases >>>>>>>>>>>>>>> - self.write_dhcp_leases(leases) >>>>>>>>>>>>>>> + if self.write_dhcp_leases(leases): >>>>>>>>>>>>>>> - log.debug("Reloading Unbound...") >>>>>>>>>>>>>>> + log.debug("Reloading Unbound...") >>>>>>>>>>>>>>> - # Reload the configuration without dropping the cache >>>>>>>>>>>>>>> - self._control("reload_keep_cache") >>>>>>>>>>>>>>> + # Reload the configuration without dropping the cache >>>>>>>>>>>>>>> + self._control("reload_keep_cache") >>>>>>>>>>>>>>> + >>>>>>>>>>>>>>> + else: >>>>>>>>>>>>>>> + >>>>>>>>>>>>>>> + log.debug("Not reloading Unbound. Leases not changed.") >>>>>>>>>>>>>>> def write_dhcp_leases(self, leases): >>>>>>>>>>>>>>> log.debug("Writing DHCP leases...") >>>>>>>>>>>>>>> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >>>>>>>>>>>>>>> - for l in leases: >>>>>>>>>>>>>>> + for l in sorted(leases): >>>>>>>>>>>>>>> for rr in l.rrset: >>>>>>>>>>>>>>> f.write("local-data: \"%s\"\n" % " ".join(rr)) >>>>>>>>>>>>>>> - # Make file readable for everyone >>>>>>>>>>>>>>> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>>>>>>>>> + filecmp.clear_cache() >>>>>>>>>>>>>>> + RequireUnboundReload = not filecmp.cmp(f.name, self.path, shallow=False) >>>>>>>>>>>>>>> + >>>>>>>>>>>>>>> + # Make file readable for everyone >>>>>>>>>>>>>>> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>>>>>>>>> + >>>>>>>>>>>>>>> + # Move the file to its destination >>>>>>>>>>>>>>> + os.rename(f.name, self.path) >>>>>>>>>>>>>>> - # Move the file to its destination >>>>>>>>>>>>>>> - os.rename(f.name, self.path) >>>>>>>>>>>>>>> + return RequireUnboundReload >>>>>>>>>>>>>>> def _control(self, *args): >>>>>>>>>>>>>>> command = ["unbound-control"] >>>>>>>>>>>>>>> >>>>>>>>>>>> Resubmitting the patch. >>>>>>>>>>>> >>>>>>>>>>>> Following my previous reply, I would like to stick with filecmp as I am using it with "shallow=False" to avoid doing the comparison just with stat() calls. Note that I also flush the filecmp cache before using the command so it is running with the latest file data. Talking about file descriptors, atomic file replacement etc., scares me. >>>>>>>>>>>> >>>>>>>>>>> File descriptors are great. They avoid race-conditions and all sorts of things. >>>>>>>>>>> >>>>>>>>>> But I have never used them so I don't intend to go that way. Too much for me to get my brain round and a limited ability to test. >>>>>>>>>> >>>>>>>>> Well, I pretty much gave you the entire solution… There was nothing left to do apart from copying and pasting the snippets. >>>>>>>>> >>>>>>>>> >>>>>>>> Ok I've tried to use your version but struggled to get it going. If I used it as was and called compare(file1_name, file2_name) it failed as "compare" was not defined. >>>>>>>> I could call it using self.compare but then it failed with "TypeError: UnboundConfigWriter.compare() takes 2 positional arguments but 3 were given". I changed the function to compare(self, f1, f2). >>>>>>>> Then it failed with "AttributeError: 'str' object has no attribute 'seek'", so I changed it again to accept file names then open the files before doing the seeks. >>>>>>>> I really hope it is OK as I am out of my depth when defining functions with "self" as an argument. Please can you check it? If it is not OK, can I revert to filecmp? >>>>>>>> >>>>>>>> I have made one futher change, the sorted(leases) line was not working and not sorting the object ever, and I saw lines changing places in the resulting leases file. I have changed it to "sorted(leases, key=lambda x: x.ipaddr)". It definitely sorts this time. >>>>>>>> >>>>>>>> >>>>>>>>>>> I don’t think that module is the best choice really if you have to hack around and flush the cache. Why would it even cache things in the first place? I don’t see where the cache helps. >>>>>>>>>>> >>>>>>>>>> From the filecmp.clear_cache documentation: >>>>>>>>>> >>>>>>>>>> Clear the filecmp cache. This may be useful if a file is compared so >>>>>>>>>> quickly after it is modified that it is within the mtime resolution >>>>>>>>>> of the underlying filesystem. >>>>>>>>>> >>>>>>>>>> Now this makes me believe that the clear_cache is only relevant when using shallow=True, which I'm not and I don't know how to check the underlying code. It is used to make sure the stat() info is updated, but we are not using the stat() info. I guess we could as easily add a one second sleep instead as that is way longer than the mtime resolution. >>>>>>>>>> >>>>>>>>> The function is here and it is rather simple: https://github.com/python/cpython/blob/3.12/Lib/filecmp.py#L30 <https://github.com/python/cpython/blob/3.12/Lib/filecmp.py#L30> >>>>>>>>> >>>>>>>>> It will always stat() both files and compares three items in case shallow is False: The type of the file, the size and mtime. If the file is not a regular file, the function will exist rather early. So in our case, that is all work the function is doing for thing. It boils down to something very similar to what I suggested to use to compare the files. >>>>>>>>> >>>>>>>>> On top of that, there is a cache which will always be used but will never return a match for us because the temporary file at least will always have another mtime. >>>>>>>>> >>>>>>>>> You could clear the cache after every call, but I would rather not have our code messy and care about implementation details of some Python module. >>>>>>>>> >>>>>>>>> >>>>>>>>>> The module is chosen because all the references I read about comparing files preferred this one-liner for comparing files, although other methods were given and there were lots of discussions about large files (GB+) because if you chomp them it may be much quicker to say they are not the same when you find the first block that does not match. Having said this, I don't know how filecmp works, but we only have a small file. At the same time, I am not a regular programmer so simple one-liners are good for me. >>>>>>>>>> >>>>>>>>> You would always want to read both files block for block and compare the blocks. There is no way to implement this significantly faster. >>>>>>>>> >>>>>>>>> >>>>>>>>>>>> I have modified the earlier try/except I proposed to catch only FileNotFoundError, and I have changed CamelCase to snake_case. >>>>>>>>>>>> >>>>>>>>>>>> --- >>>>>>>>>>>> config/unbound/unbound-dhcp-leases-bridge | 31 ++++++++++++++++------- >>>>>>>>>>>> 1 file changed, 22 insertions(+), 9 deletions(-) >>>>>>>>>>>> >>>>>>>>>>>> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>>>>> index e9f022aff..a9ec8e1e2 100644 >>>>>>>>>>>> --- a/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>>>>> +++ b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>>>>>> @@ -22,6 +22,7 @@ >>>>>>>>>>>> import argparse >>>>>>>>>>>> import datetime >>>>>>>>>>>> import daemon >>>>>>>>>>>> +import filecmp >>>>>>>>>>>> import functools >>>>>>>>>>>> import ipaddress >>>>>>>>>>>> import logging >>>>>>>>>>>> @@ -516,26 +517,38 @@ class UnboundConfigWriter(object): >>>>>>>>>>>> >>>>>>>>>>>> def update_dhcp_leases(self, leases): >>>>>>>>>>>> # Write out all leases >>>>>>>>>>>> - self.write_dhcp_leases(leases) >>>>>>>>>>>> + if self.write_dhcp_leases(leases): >>>>>>>>>>>> >>>>>>>>>>>> - log.debug("Reloading Unbound...") >>>>>>>>>>>> + log.debug("Reloading Unbound...") >>>>>>>>>>>> >>>>>>>>>>>> - # Reload the configuration without dropping the cache >>>>>>>>>>>> - self._control("reload_keep_cache") >>>>>>>>>>>> + # Reload the configuration without dropping the cache >>>>>>>>>>>> + self._control("reload_keep_cache") >>>>>>>>>>>> + >>>>>>>>>>>> + else: >>>>>>>>>>>> + >>>>>>>>>>>> + log.debug("Not reloading Unbound. Leases not changed.") >>>>>>>>>>>> >>>>>>>>>>>> def write_dhcp_leases(self, leases): >>>>>>>>>>>> log.debug("Writing DHCP leases...") >>>>>>>>>>>> >>>>>>>>>>>> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >>>>>>>>>>>> - for l in leases: >>>>>>>>>>>> + for l in sorted(leases): >>>>>>>>>>>> for rr in l.rrset: >>>>>>>>>>>> f.write("local-data: \"%s\"\n" % " ".join(rr)) >>>>>>>>>>>> >>>>>>>>>>>> - # Make file readable for everyone >>>>>>>>>>>> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>>>>>> + filecmp.clear_cache() >>>>>>>>>>>> + try: >>>>>>>>>>>> + require_unbound_reload = not filecmp.cmp(f.name, self.path, shallow=False) >>>>>>>>>>>> + except FileNotFoundError: >>>>>>>>>>>> + require_unbound_reload = True >>>>>>>>>>>> >>>>>>>>>>> If you want to go this route, you will have to flush the temporary file first. Otherwise you might have some data left in the buffer that has not been written (because the file is still open). You will then compare incomplete output. This would not become a problem if you used the file descriptors. >>>>>>>>>>> >>>>>>>>>> The temp file is closed at the point of doing the filecmp, and the filecmp cache has been flushed. How do I flush the temporary file further. Reading up, is it just having the last statement in the with block as f.flush() with the same indent as "for l in sorted(leases):" >>>>>>>>>> >>>>>>>>> If you have left the with block you will already have flushed the buffers and closed the file. It looks like I misread the indentation in the patch then. >>>>>>>>> >>>>>>>>> >>>>>>>>>>> Catching FileNotFound would work for me. >>>>>>>>>>> >>>>>>>>>> Good >>>>>>>>>> >>>>>>>>>>>> + >>>>>>>>>>>> + # Make file readable for everyone >>>>>>>>>>>> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>>>>>> + >>>>>>>>>>>> + # Move the file to its destination >>>>>>>>>>>> + os.rename(f.name, self.path) >>>>>>>>>>>> >>>>>>>>>>>> - # Move the file to its destination >>>>>>>>>>>> - os.rename(f.name, self.path) >>>>>>>>>>>> + return require_unbound_reload >>>>>>>>>>>> >>>>>>>>>>>> def _control(self, *args): >>>>>>>>>>>> command = ["unbound-control"] >>>>>>>>>>>> -- >>>>>>>>>>>> 2.43.0 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Nick >>>>>>>>>>>> >>>>>>>> --- >>>>>>>> config/unbound/unbound-dhcp-leases-bridge | 50 +++++++++++++++++++---- >>>>>>>> 1 file changed, 41 insertions(+), 9 deletions(-) >>>>>>>> >>>>>>>> diff --git a/config/unbound/unbound-dhcp-leases-bridge b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>> index e9f022aff..80d9523ea 100644 >>>>>>>> --- a/config/unbound/unbound-dhcp-leases-bridge >>>>>>>> +++ b/config/unbound/unbound-dhcp-leases-bridge >>>>>>>> @@ -516,26 +516,37 @@ class UnboundConfigWriter(object): >>>>>>>> >>>>>>>> def update_dhcp_leases(self, leases): >>>>>>>> # Write out all leases >>>>>>>> - self.write_dhcp_leases(leases) >>>>>>>> + if self.write_dhcp_leases(leases): >>>>>>>> >>>>>>>> - log.debug("Reloading Unbound...") >>>>>>>> + log.debug("Reloading Unbound...") >>>>>>>> >>>>>>>> - # Reload the configuration without dropping the cache >>>>>>>> - self._control("reload_keep_cache") >>>>>>>> + # Reload the configuration without dropping the cache >>>>>>>> + self._control("reload_keep_cache") >>>>>>>> + >>>>>>>> + else: >>>>>>>> + >>>>>>>> + log.debug("Not reloading Unbound. Leases not changed.") >>>>>>>> >>>>>>>> def write_dhcp_leases(self, leases): >>>>>>>> log.debug("Writing DHCP leases...") >>>>>>>> >>>>>>>> with tempfile.NamedTemporaryFile(mode="w", delete=False) as f: >>>>>>>> - for l in leases: >>>>>>>> + for l in sorted(leases, key=lambda x: x.ipaddr): >>>>>>>> for rr in l.rrset: >>>>>>>> f.write("local-data: \"%s\"\n" % " ".join(rr)) >>>>>>>> >>>>>>>> - # Make file readable for everyone >>>>>>>> - os.fchmod(f.fileno(), stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>> + try: >>>>>>>> + require_unbound_reload = not self.compare(f.name, self.path) >>>>>>>> + except FileNotFoundError: >>>>>>>> + require_unbound_reload = True >>>>>>>> + >>>>>>>> + # Make file readable for everyone >>>>>>>> + os.chmod(f.name, stat.S_IRUSR|stat.S_IWUSR|stat.S_IRGRP|stat.S_IROTH) >>>>>>>> + >>>>>>>> + # Move the file to its destination >>>>>>>> + os.rename(f.name, self.path) >>>>>>>> >>>>>>>> - # Move the file to its destination >>>>>>>> - os.rename(f.name, self.path) >>>>>>>> + return require_unbound_reload >>>>>>>> >>>>>>>> def _control(self, *args): >>>>>>>> command = ["unbound-control"] >>>>>>>> @@ -551,6 +562,27 @@ class UnboundConfigWriter(object): >>>>>>>> >>>>>>>> raise e >>>>>>>> >>>>>>>> + def compare(self, if1, if2): >>>>>>>> + buffersize = 16 * 1024 >>>>>>>> + >>>>>>>> + f1 = open(if1, "r") >>>>>>>> + f2 = open(if2, "r") >>>>>>>> + >>>>>>>> + # Reset file handles >>>>>>>> + f1.seek(0) >>>>>>>> + f2.seek(0) >>>>>>>> + >>>>>>>> + while True: >>>>>>>> + b1 = f1.read(buffersize) >>>>>>>> + b2 = f2.read(buffersize) >>>>>>>> + >>>>>>>> + if not b1 == b2: >>>>>>>> + return False >>>>>>>> + >>>>>>>> + elif not b1 or not b2: >>>>>>>> + break >>>>>>>> + >>>>>>>> + return True >>>>>>>> >>>>>>>> if __name__ == "__main__": >>>>>>>> parser = argparse.ArgumentParser(description="Bridge for DHCP Leases and Unbound DNS") >>>>>>>> >> > Jon Murphy jon.murphy@ipfire.org <mailto:jon.murphy@ipfire.org>
Jon Murphy jon.murphy@ipfire.org
-
jon -
Michael Tremer -
Nick Howitt