Michael & All,
I am offering a Proof Of Concept (POC) for a DHCP to DNS bridge. The POC below is written in bash and may not be fast enough for thousands of devices.
It uses the On Commit/Release/Expiry (CRE) already included into ISC DHCP. I believe these are known as EVENTS.
=== NOTE === REFERENCE: EVENTS - There are three kinds of events that can happen regarding a lease, and it is possible to declare statements that occur when any of these events happen. These events are the commit event, when the server has made a commitment of a certain lease to a client, the release event, when the client has released the server from its commitment, and the expiry event, when the commitment expires.
To declare a set of statements to execute when an event happens, you must use the on statement, followed by the name of the event, followed by a series of statements to execute when the event happens, enclosed in braces. ===
If I did this correctly is should simplify the bridge by eliminating some of the parsing though `/var/ipfire/dhcp/fixleases` and searching for new or expired leases.
If I understand CRE correctly, it should also simplify the separated Dynamic Leases and Fixed Leases. CRE seems to handle both FIXED and DYNAMIC. The downside (or upside depending on your point of view) is the elimination of the DHCP configuration "Remarks" appearing to unbound as client domain names. See DHCP configuration - Current fixed leases (at menu Network > DHCP Server)
This is not complete and it currently does not include the Hostname (static) items at `/etc/unbound/hosts.conf`
There are many extra lines included to help me test & debug (i.e., "logger"). Most to be removed.
If this helps, please offer ideas or corrections. If not, then I will be :-(
File = `/var/ipfire/dhcp/dhcpd.conf.local`
File = `dhcpEvent_v6.sh`
The bash script currently writes to `/etc/unbound/dhcp-leases2.conf` so I can compare it to `dhcp-leases.conf`. It does NOT provide any data to unbound (yet!)
Is this script helpful?
Jon
Inspiration link: https://jpmens.net/2011/07/06/execute-a-script-when-isc-dhcp-hands-out-a-new...
Reference Link: Still looking!! https://stackoverflow.com/questions/51550326/is-there-any-hook-for-finishing...
Hello Jon,
I like this idea. This might not be the most performant approach, but maybe that isn’t the biggest problem here. Correctness comes before performance at least.
On 30 Nov 2023, at 16:01, jon jon.murphy@ipfire.org wrote:
Michael & All,
I am offering a Proof Of Concept (POC) for a DHCP to DNS bridge. The POC below is written in bash and may not be fast enough for thousands of devices.
It uses the On Commit/Release/Expiry (CRE) already included into ISC DHCP. I believe these are known as EVENTS.
=== NOTE === REFERENCE: EVENTS - There are three kinds of events that can happen regarding a lease, and it is possible to declare statements that occur when any of these events happen. These events are the commit event, when the server has made a commitment of a certain lease to a client, the release event, when the client has released the server from its commitment, and the expiry event, when the commitment expires.
To declare a set of statements to execute when an event happens, you must use the on statement, followed by the name of the event, followed by a series of statements to execute when the event happens, enclosed in braces.
If I did this correctly is should simplify the bridge by eliminating some of the parsing though `/var/ipfire/dhcp/fixleases` and searching for new or expired leases.
If I understand CRE correctly, it should also simplify the separated Dynamic Leases and Fixed Leases. CRE seems to handle both FIXED and DYNAMIC. The downside (or upside depending on your point of view) is the elimination of the DHCP configuration "Remarks" appearing to unbound as client domain names. See DHCP configuration - Current fixed leases (at menu Network > DHCP Server)
This is not complete and it currently does not include the Hostname (static) items at `/etc/unbound/hosts.conf`
There are many extra lines included to help me test & debug (i.e., "logger"). Most to be removed.
If this helps, please offer ideas or corrections. If not, then I will be :-(
This is only one half the solution I would say - but probably fixes the problem of the past…
I think that we can easily use this script to dynamically load/remove entries from Unbound with removing it. Since we already have to fork unbound-control, I do not consider this too bad.
But: A new problem would be that we have a bootstrap problem: How do we load an initial set of leases into Unbound - either on boot or when Unbound gets restarted?
* Parsing the leases file in shell is painful and veeeery slow. Not really a problem since we only do this once, but it has to work. * Keeping the existing Python code feels like the wrong choice. * I played around with awk and I think that could be an option, but it is not fun to write - not the biggest problem again. * Another option could be that we don’t bother with the leases file at all. We could simply create yet another file that we write any events to and hope that we will never miss one.
I am really not sure which ones of these options I would disfavour the least. What do you think?
-Michael
File = `/var/ipfire/dhcp/dhcpd.conf.local`
<dhcpd.conf.local.txt>
File = `dhcpEvent_v6.sh`
<dhcpEvent_v6.sh.txt>
The bash script currently writes to `/etc/unbound/dhcp-leases2.conf` so I can compare it to `dhcp-leases.conf`. It does NOT provide any data to unbound (yet!)
Is this script helpful?
Jon
Inspiration link: https://jpmens.net/2011/07/06/execute-a-script-when-isc-dhcp-hands-out-a-new...
Reference Link: Still looking!! https://stackoverflow.com/questions/51550326/is-there-any-hook-for-finishing...
Jon Murphy jon.murphy@ipfire.org
On Nov 30, 2023, at 2:26 PM, Michael Tremer michael.tremer@ipfire.org wrote:
Hello Jon,
I like this idea. This might not be the most performant approach, but maybe that isn’t the biggest problem here. Correctness comes before performance at least.
On 30 Nov 2023, at 16:01, jon jon.murphy@ipfire.org wrote:
Michael & All,
I am offering a Proof Of Concept (POC) for a DHCP to DNS bridge. The POC below is written in bash and may not be fast enough for thousands of devices.
It uses the On Commit/Release/Expiry (CRE) already included into ISC DHCP. I believe these are known as EVENTS.
=== NOTE === REFERENCE: EVENTS - There are three kinds of events that can happen regarding a lease, and it is possible to declare statements that occur when any of these events happen. These events are the commit event, when the server has made a commitment of a certain lease to a client, the release event, when the client has released the server from its commitment, and the expiry event, when the commitment expires.
To declare a set of statements to execute when an event happens, you must use the on statement, followed by the name of the event, followed by a series of statements to execute when the event happens, enclosed in braces.
If I did this correctly is should simplify the bridge by eliminating some of the parsing though `/var/ipfire/dhcp/fixleases` and searching for new or expired leases.
If I understand CRE correctly, it should also simplify the separated Dynamic Leases and Fixed Leases. CRE seems to handle both FIXED and DYNAMIC. The downside (or upside depending on your point of view) is the elimination of the DHCP configuration "Remarks" appearing to unbound as client domain names. See DHCP configuration - Current fixed leases (at menu Network > DHCP Server)
This is not complete and it currently does not include the Hostname (static) items at `/etc/unbound/hosts.conf`
There are many extra lines included to help me test & debug (i.e., "logger"). Most to be removed.
If this helps, please offer ideas or corrections. If not, then I will be :-(
This is only one half the solution I would say - but probably fixes the problem of the past…
I think that we can easily use this script to dynamically load/remove entries from Unbound with removing it. Since we already have to fork unbound-control, I do not consider this too bad.
But: A new problem would be that we have a bootstrap problem: How do we load an initial set of leases into Unbound - either on boot or when Unbound gets restarted?
Bootstrap: When you say bootstrap, I think "first boot" after install. I hope that is correct! - The first time through the client would request an IP address via a DHCP request (same as today). - DHCPACK causes a "commit" EVENT entered into the `dhcpd.conf`config file at `/var/ipfire/dhcp/dhcpd.conf` - NOTE: today I have this in `/var/ipfire/dhcp/dhcpd.conf.local` for testing. - The `dhcpEvent.sh` script does its thing and drops A and PTR records into the `/etc/unbound/dhcp-leases.conf`. - It appears Unbound is constantly monitoring this file. Unbound seems to recognize the changes and accept the A and PTR records. - NOTE: I began my tests by manually entering file and they seem to be accepted fine. - NOTE: I still have testing to do to make sure this works properly! - NOTE: Unbound is the part that worries me the most since I don't understand why this works! I need to keep testing by removing/adding the unbound reload code in the existing bridge!
IPFire Reboot / Shutdown: I’ve rebooted IPFire a few times. Nothing needs to be re-populated. I did the CU 181 update with this running. The leases survive the reboot since the `/etc/unbound/dhcp-leases.conf` file is still intact and full of leases. So far it has not been an issue. More testing to do! (Especially with shutdown)
Unbound & DHCP restarts: I’ve restarted unbound and dhcp many, many times. Nothing needs to be re-populated Same as a reboot, the leases survive the restart since the `/etc/unbound/dhcp-leases.conf` file is still intact and full of leases. So far it has not been an issue. More testing to do!
Clients and DHCP: I still need to test device hostname changes. Initial tests seem OK. I still need to test static to DHCP to static changes. I still need to test Fixed to Dynamic to Fixed changes.
- Parsing the leases file in shell is painful and veeeery slow. Not really a problem since we only do this once, but it has to work.
- Keeping the existing Python code feels like the wrong choice.
- I played around with awk and I think that could be an option, but it is not fun to write - not the biggest problem again.
I tried the "davidbarnhart" awk code for the DHCP leases file. To me it seems awkward (pun certainly intended!). Awk was fun to play with, but I dont think this is needed if we use EVENTS already built into ISC dhcp.
- Another option could be that we don’t bother with the leases file at all. We could simply create yet another file that we write any events to and hope that we will never miss one.
I’m not sure I understand this option…
I am really not sure which ones of these options I would disfavour the least. What do you think?
To be honest I think I like my proof of concept (POC)! DHCP has a nice mechanism to get leases out and Unbound has a nice mechanism to get leases in. I believe we should take advantage of it (if everything works as hoped).
Like I said I am still testing so there may be a "gotcha" that causes all of the POC stop. (And we could end up being dependent of the unbound-reload)
So far the only items missing from my temp leases list are items related to "Remarks" appearing to unbound as client domain names. I am not sure if these should be added…
My question to you: What is the difference between a DHCP lease that EXPIRES and a DHCP lease that is RELEASED??
I am note sure if I answered all of your questions. Does this help? Jon
-Michael
File = `/var/ipfire/dhcp/dhcpd.conf.local`
<dhcpd.conf.local.txt>
File = `dhcpEvent_v6.sh`
<dhcpEvent_v6.sh.txt>
The bash script currently writes to `/etc/unbound/dhcp-leases2.conf` so I can compare it to `dhcp-leases.conf`. It does NOT provide any data to unbound (yet!)
Is this script helpful?
Jon
Inspiration link: https://jpmens.net/2011/07/06/execute-a-script-when-isc-dhcp-hands-out-a-new...
Reference Link: Still looking!! https://stackoverflow.com/questions/51550326/is-there-any-hook-for-finishing...
Forgot one item: doing a blended version would be good also. Replacing the DHCP parse lease code with the commit/release/expiry would be cool!
For experimenting, this code can the added `/var/ipfire/dhcp/dhcpd.conf.local`. The output will appear in the messages log with the "dhcpd" tag.
``` on commit { set noname = concat("dhcp-", binary-to-ascii(10, 8, "-", leased-address)); set ClientIP = binary-to-ascii(10, 8, ".", leased-address); set ClientDHCID = concat ( suffix (concat ("0", binary-to-ascii (16, 8, "", substring(hardware,1,1))),2), ":", suffix (concat ("0", binary-to-ascii (16, 8, "", substring(hardware,2,1))),2), ":", suffix (concat ("0", binary-to-ascii (16, 8, "", substring(hardware,3,1))),2), ":", suffix (concat ("0", binary-to-ascii (16, 8, "", substring(hardware,4,1))),2), ":", suffix (concat ("0", binary-to-ascii (16, 8, "", substring(hardware,5,1))),2), ":", suffix (concat ("0", binary-to-ascii (16, 8, "", substring(hardware,6,1))),2) );
set ClientName = pick-first-value(option host-name, config-option-host-name, client-name, noname);
set secondNibble = substring(binary-to-ascii(16, 8, "", substring(hardware, 1, 1)), 1, 1); set leftByte = substring(hardware, 1, 1);
log(concat("Commit: IP: ", ClientIP, " MAC: ", ClientDHCID, " Name: ", ClientName, " secondNibble: ", secondNibble));
#execute("/root/dhcpdconf/dhcpEvent.sh", "commit", ClientIP, ClientDHCID, ClientName); }
on release { set ClientIP = binary-to-ascii(10, 8, ".", leased-address); set ClientDHCID = concat ( suffix (concat ("0", binary-to-ascii (16, 8, "", substring(hardware,1,1))),2), ":", suffix (concat ("0", binary-to-ascii (16, 8, "", substring(hardware,2,1))),2), ":", suffix (concat ("0", binary-to-ascii (16, 8, "", substring(hardware,3,1))),2), ":", suffix (concat ("0", binary-to-ascii (16, 8, "", substring(hardware,4,1))),2), ":", suffix (concat ("0", binary-to-ascii (16, 8, "", substring(hardware,5,1))),2), ":", suffix (concat ("0", binary-to-ascii (16, 8, "", substring(hardware,6,1))),2) );
log(concat("Release: IP: ", ClientIP, " MAC: ", ClientDHCID)); #execute("/root/dhcpdconf/dhcpEvent.sh", "release", ClientIP, ClientDHCID, ClientName); }
on expiry { set ClientIP = binary-to-ascii(10, 8, ".", leased-address); # cannot get a ClientMac here, apparently this only works when actually receiving a packet
log(concat("Expired: IP: ", ClientIP)); #execute("/root/dhcpdconf/dhcpEvent.sh", "expiry", ClientIP, ClientDHCID, "");
# cannot get a ClientName here, for some reason that always fails # however the dhcp update script will obtain the short hostname. } ```
It just dawned to me some of the things you mentioned and I may have this part wrong:
- It appears Unbound is constantly monitoring this file. Unbound seems to recognize the changes and accept the > A and PTR records.
- NOTE: I began my tests by manually entering file and they seem to be accepted fine.
- NOTE: I still have testing to do to make sure this works properly!
- NOTE: Unbound is the part that worries me the most since I don't understand why this works! I need to keep testing by removing/adding the unbound reload code in the existing bridge!
Back to the drawing board!
Michael - I believe the key to getting rid of the unbound restarts is going back to the `unbound-control local_data` type commands.
But it doesn't correct the DNS leases problem you mentioned in: https://bugzilla.ipfire.org/show_bug.cgi?id=13254#c27
And I believe the DNS issue and be fixed by grabbing leases differently using the On Commit/Release/Expiry status
I am going to go back to bugzilla for now. And I'll post my updated script shortly...
The first script did not place A & PTR records into the unbound cache and this version does!
Once again I am looking for your feedback, thoughts and comments!
Jon
===============================================================================
NOTE: Do not place this into a Production environment - only for Test environment.
Description of Proof of Concept ———————————————————————————————
Transfers DHCP lease information to unbound DNS
• Enable DHCP On Commit/Release/Expire via dhcpd.conf (or dhcpd.conf.local) • A new (or changed or removed) dynamic/fixed lease causes Script to run • Script adds A & PTR records on Commit: • to unbound dhcp-leases.conf file • to unbound cache via "unbound-control load-data" • Script removes A & PTR records on Release or Expiry of lease
The Script ———————————
• Handles dynamic and fixed leases • Static hosts checked for duplicates • Re-enabled unbound-control local_data • unbound reloads not needed • A & PTR records survive IPFire reboot, unbound restart • no delays from difficult dhcp-leases parsing
Not completed (yet): • Static hosts still need to be parsed into unbound hosts.conf file • Properly handle hostnames with spaces • non-legit need to change to dashes or be removed (or completely dropped) • to be checked with large amount of clients • I’ve tested with 6 devices on test system and ~40 devices on production
NOTE: Do not place this into a Production environment. Only for Test environment.
• If placed into Test environment this current script will terminate "unbound-dhcp-leases-bridge". • Do not run the `dhcpEvent.sh` script and "unbound-dhcp-leases-bridge" in parallel.
Michael & All,
Here is another update for the `unbound-dhcp-leases-bridge` code.
I believe I have copied all the processes done in `unbound-dhcp-leases-bridge`, so if I am missing a function please speak up.
Everything seems to be working A-OK (for me!). - I’ve re-written much/most of the code, - added bash functions for unbound writes & deletes to get rid of duplicate code, - and added some error (invalid) checks.
Since the code is not parsing the dhcp-leases I believe it _may_ be fast enough for lots of dhcp clients. But I have no way of knowing without your help!
I’ve written two "compare" scripts: 1) if you want to compare the old with new. This requires some updates to the script below. 2) or if want to compare `/etc/unbound/dhcp-leases.conf` plus `/etc/unbound/hosts.conf` to the unbound cache `unbound-control list_local_data`.
Static Hosts ————————————
• Static hosts still need to be parsed into unbound hosts.conf file
I am looking for help with this since I dont fully understand the "read_static_hosts" section: https://github.com/ipfire/ipfire-2.x/blob/master/config/unbound/unbound-dhcp...
- Why are the static hosts read into the "unbound-dhcp-leases-bridge" program? - Does this section write the static hosts into the the unbound `/etc/unbound/hosts.conf` file? (I dont believe it does) - Or are the static hosts for reference only?
Currently I do a quick grep search of the `/etc/unbound/hosts.conf` file for IP address and hostname in Line 142. If an IP/hostname exists in the static hosts file, then I exit.
So I think this is what the unbound-dhcp-leases-bridge does…
Hostnames with Spaces —————————————————————
• Properly handle hostnames with spaces
I remove the spaces in the hostname - yes, I have one device with this non-legit hostname
Is this OK?
=============================================================================== NOTE: Do not place this into a Production environment - only for Test environment.
• If placed into Test environment this current script will terminate "unbound-dhcp-leases-bridge". • Do not run the `dhcpEvent.sh` script and "unbound-dhcp-leases-bridge" in parallel. ===============================================================================
Once again I am looking for help, feedback, thoughts and comments!
Jon
On Dec 8, 2023, at 10:26 AM, jon jon.murphy@ipfire.org wrote:
The first script did not place A & PTR records into the unbound cache and this version does!
Once again I am looking for your feedback, thoughts and comments!
Jon
===============================================================================
NOTE: Do not place this into a Production environment - only for Test environment.
Description of Proof of Concept ———————————————————————————————
Transfers DHCP lease information to unbound DNS
• Enable DHCP On Commit/Release/Expire via dhcpd.conf (or dhcpd.conf.local) • A new (or changed or removed) dynamic/fixed lease causes Script to run • Script adds A & PTR records on Commit: • to unbound dhcp-leases.conf file • to unbound cache via "unbound-control load-data" • Script removes A & PTR records on Release or Expiry of lease
The Script ———————————
• Handles dynamic and fixed leases • Static hosts checked for duplicates • Re-enabled unbound-control local_data • unbound reloads not needed • A & PTR records survive IPFire reboot, unbound restart • no delays from difficult dhcp-leases parsing
Not completed (yet): • Static hosts still need to be parsed into unbound hosts.conf file • Properly handle hostnames with spaces • non-legit need to change to dashes or be removed (or completely dropped) • to be checked with large amount of clients • I’ve tested with 6 devices on test system and ~40 devices on production
<dhcpEvent_v15.sh.txt><dhcpd.conf.local>
NOTE: Do not place this into a Production environment. Only for Test environment.
• If placed into Test environment this current script will terminate "unbound-dhcp-leases-bridge". • Do not run the `dhcpEvent.sh` script and "unbound-dhcp-leases-bridge" in parallel.
-
jon -
Jon Murphy
-
Michael Tremer