Re: Proof of concept: Bridge rewrite of unbound-dhcp-leases-bridge

Re: Proof of concept: Bridge rewrite of unbound-dhcp-leases-bridge

[Jon Murphy]

[-- Attachment #1: Type: text/plain, Size: 612 bytes --]

It just dawned to me some of the things you mentioned and I may have this part wrong:
> - It appears Unbound is constantly monitoring this file.  Unbound seems to recognize the changes and accept the > A and PTR records.
>    - NOTE: I began my tests by manually entering file and they seem to be accepted fine. 
>    - NOTE: I still have testing to do to make sure this works properly!
>    - NOTE: Unbound is the part that worries me the most since I don't understand why this works!  I need to keep testing by removing/adding the unbound reload code in the existing bridge!

Back to the drawing board!

Re: Proof of concept: Bridge rewrite of unbound-dhcp-leases-bridge

[Jon Murphy]

[-- Attachment #1: Type: text/plain, Size: 517 bytes --]

Michael - I believe the key to getting rid of the unbound restarts is going back to the `unbound-control local_data` type commands.

But it doesn't correct the DNS leases problem you mentioned in:  
https://bugzilla.ipfire.org/show_bug.cgi?id=13254#c27

And I believe the DNS issue and be fixed by grabbing leases differently using the On Commit/Release/Expiry status

I am going to go back to bugzilla for now.  And I'll post my updated script shortly...

https://bugzilla.ipfire.org/show_bug.cgi?id=13254

Re: Proof of concept: Bridge rewrite of unbound-dhcp-leases-bridge

[jon]

[-- Attachment #1: Type: text/plain, Size: 1809 bytes --]

The first script did not place A & PTR records into the unbound cache and this version does!

Once again I am looking for your feedback, thoughts and comments!


Jon

===============================================================================

NOTE: Do not place this into a Production environment - only for Test environment.


Description of Proof of Concept
———————————————————————————————

Transfers DHCP lease information to unbound DNS

• Enable DHCP On Commit/Release/Expire via dhcpd.conf (or dhcpd.conf.local)
• A new (or changed or removed) dynamic/fixed lease causes Script to run
• Script adds A & PTR records on Commit:
    • to unbound dhcp-leases.conf file
    • to unbound cache via "unbound-control load-data"
• Script removes A & PTR records on Release or Expiry of lease


The Script
———————————

• Handles dynamic and fixed leases
• Static hosts checked for duplicates
• Re-enabled unbound-control local_data
    • unbound reloads not needed
• A & PTR records survive IPFire reboot, unbound restart
    • no delays from difficult dhcp-leases parsing

Not completed (yet):
• Static hosts still need to be parsed into unbound hosts.conf file
• Properly handle hostnames with spaces
    • non-legit need to change to dashes or be removed (or completely dropped)
• to be checked with large amount of clients
    • I’ve tested with 6 devices on test system and ~40 devices on production









NOTE: Do not place this into a Production environment.  Only for Test environment.

• If placed into Test environment this current script will terminate "unbound-dhcp-leases-bridge".
• Do not run the `dhcpEvent.sh` script and "unbound-dhcp-leases-bridge" in parallel.



[-- Attachment #2: dhcpEvent_v15.sh.txt --]
[-- Type: text/plain, Size: 4381 bytes --]

#!/bin/bash
#set -e
#set -x

#	DHCP commit/release/expiry status drives this process

####################	following 11 lines just for testing  ####################
/usr/bin/logger --tag dhcp1 "testing --> $0 (v15)"
#	Kill the unbound-dhcp-leases-bridge process
pgrep -f unbound-dhcp-leases-bridge &&
	kill -s SIGTERM $(pgrep -d',' -f unbound-dhcp-leases-bridge) &&
	/usr/bin/logger --tag dhcp1 "terminated unbound-dhcp-leases-bridge"

#	if no dhcpLeases then create file
#[[ -f "${unboundDHCPleases}" ]] || touch "${unboundDHCPleases}"

#	chmod -v 744 /root/dhcpdconf/dhcpEvent_vN.sh
#	ln -vsf /root/dhcpdconf/dhcpEvent_vN.sh /root/dhcpdconf/dhcpEvent.sh
#-------------------------------------------------------------------------------


#	need DOMAINNAME value
eval $(/usr/local/bin/readhash /var/ipfire/main/settings)

dhcpCREstatus=$1
clientIP=$2
clientName=$3

unboundDHCPleases="/etc/unbound/dhcp-leases.conf"				#	output to unbound
unboundStaticHosts="/etc/unbound/hosts.conf"

TTL="60"														#	time to live (seconds)

#	add domain name to client name
clientFQDN=$( echo "${clientName}.${DOMAINNAME}"  )		#	 | tr '[:upper:]' '[:lower:]'

#	reverse IP
reverseIP=$( echo "${clientIP}" | awk -F. '{print $4"."$3"."$2"."$1}' )
arpaName=$( echo "${reverseIP}.in-addr.arpa" )

#	create "A" record and "PTR" record for unbound
#	made "absolute address" by adding a "." at the end of names
#	this is done for consistency with the unbound cache
aRecord="${clientFQDN}. ${TTL} IN A ${clientIP}"
ptrRecord="${arpaName}. ${TTL} IN PTR ${clientFQDN}."

#	create local data line using above Records
aRecordLD="local-data: \"${aRecord}"\"
ptrRecordLD="local-data: \"${ptrRecord}"\"

case "${dhcpCREstatus}" in
	commit)
		#	if client name is blank then exit
		[[ -z "${clientName}" ]] &&
			{ /usr/bin/logger --tag dhcp1 "${dhcpCREstatus}: no clientName - exit" ; exit ; }

		#	if client exists in static hosts then exit
		if grep --word-regexp --quiet -e "${clientIP}" -e "${clientName}" "${unboundStaticHosts}" ; then
			/usr/bin/logger --tag dhcp1 "${dhcpCREstatus}: ${clientName} or ${clientIP} found in static hosts - exit"
			exit
		fi

		#	does A record and PTR record already exist? if no, then add
		if ! grep --word-regexp --quiet -e "${clientIP}" -e "${reverseIP}" "${unboundDHCPleases}" ; then
			#	add A record and PTR record to unbound
			echo -e "${aRecordLD}\n${ptrRecordLD}" >> "${unboundDHCPleases}"
			echo -e "${aRecord}\n${ptrRecord}" | /usr/sbin/unbound-control local_datas

			/usr/bin/logger --tag dhcp1 "${dhcpCREstatus}: Record A and PTR added to unbound"
		else
			/usr/bin/logger --tag dhcp1 "${dhcpCREstatus}: Record A and PTR already exist"
		fi
		;;

	release|expiry)
		#	"expiry" and "release" are the same since I don't understand the difference    :-(

		#	does IP addr or Reverse IP appear in unbound dhcp-leases file?  if yes, delete line
		if grep --word-regexp --quiet -e "${clientIP}" -e "${reverseIP}" "${unboundDHCPleases}" ; then
			#	Since expiry does not return names, we must use the IP addr to find two FQDNs.
			#	get FQDN by searching for IP address
			uc_FQDN=$( unbound-control list_local_data | grep --word-regexp "${clientIP}" | awk '{ print $1 }' )
			[[ -z "${uc_FQDN}" ]] &&
				{ /usr/bin/logger --tag dhcp1 "${dhcpCREstatus}: Oops - no uc_FQDN - exit" ; exit ; }
				
			#	remove FQDNs from unbound dhcp-leases file (two lines in file)
			/bin/sed --in-place "/${uc_FQDN}/d" "${unboundDHCPleases}"
			/usr/bin/logger --tag dhcp1 "${dhcpCREstatus}: ${uc_FQDN} removed from ub dhcp leases"

			#	Since "local_data_remove" uses only names to delete (Client Name and arpa Name),
			#		we must get both names by searching for FQDN (returns two lines).
			#		e.g., "iMac3.localdomain."  AND  "100.6.168.192.in-addr.arpa."
			uc_Names=$( unbound-control list_local_data | awk "/${uc_FQDN}/" | awk '{ print $1 }' )

			#	for unbound cache, MUST be deleted by name for A record and arpa name for PTR record!
			echo -e "${uc_Names}" | /usr/sbin/unbound-control local_datas_remove

			/usr/bin/logger --tag dhcp1 "${dhcpCREstatus}: ${uc_Names} removed from unbound-control"
		else
			/usr/bin/logger --tag dhcp1 "${dhcpCREstatus}: ${clientIP} or ${reverseIP} not found"
		fi
		;;

	*)
		/usr/bin/logger --tag dhcp1 "CRE script: case = no status"
		echo "Usage:  dhcpEvent <status> <ip_address> <client_hostname>"
		exit
		;;
esac

exit

[-- Attachment #3: dhcpd.conf.local --]
[-- Type: application/octet-stream, Size: 1360 bytes --]

#	Do NOT delete this file!
#		it can be modified but this file cannot be deleted

#
#	commit
#
on commit {
set ClientIP = binary-to-ascii(10, 8, ".", leased-address);

#set NoName = concat("dhcp-", binary-to-ascii(10, 8, "-", leased-address));
#	if preferred, NoName can be changed to blank. and then ignored in execute script
set NoName = "";
set ClientName = pick-first-value(option host-name, config-option-host-name, client-name, NoName);

#	a MAC address can easily be added for other uses.  MAC is not needed for DNS

log(concat("CRE-Commit:  ClientIP: ", ClientIP, "  Name: ", ClientName));

execute("/root/dhcpdconf/dhcpEvent.sh", "commit", ClientIP, ClientName);
}


#
#	release
#
on release {
set ClientIP = binary-to-ascii(10, 8, ".", leased-address);

#set NoName = concat("dhcp-", binary-to-ascii(10, 8, "-", leased-address));
set NoName = "";
set ClientName = pick-first-value(option host-name, config-option-host-name, client-name, NoName);

log(concat("CRE-Release:  ClientIP: ", ClientIP, "  Name: ", ClientName));

execute("/root/dhcpdconf/dhcpEvent.sh", "release", ClientIP, ClientName);
}


#
#	expiry
#
on expiry {
set ClientIP = binary-to-ascii(10, 8, ".", leased-address);
set ClientName = "";

log(concat("CRE-Expiry:  ClientIP: ", ClientIP, "  Name: ", ClientName));

execute("/root/dhcpdconf/dhcpEvent.sh", "expiry", ClientIP, "");
}

Re: Proof of concept: Bridge rewrite of unbound-dhcp-leases-bridge

[jon]

[-- Attachment #1: Type: text/plain, Size: 4579 bytes --]

Michael & All,

Here is another update for the `unbound-dhcp-leases-bridge` code.

I believe I have copied all the processes done in `unbound-dhcp-leases-bridge`, so if I am missing a function please speak up.

Everything seems to be working A-OK (for me!).  
- I’ve re-written much/most of the code, 
- added bash functions for unbound writes & deletes to get rid of duplicate code, 
- and added some error (invalid) checks.

Since the code is not parsing the dhcp-leases I believe it _may_ be fast enough for lots of dhcp clients.
But I have no way of knowing without your help!

I’ve written two "compare" scripts:
1) if you want to compare the old with new.  This requires some updates to the script below.
2) or if want to compare `/etc/unbound/dhcp-leases.conf` plus `/etc/unbound/hosts.conf` to the unbound cache `unbound-control list_local_data`.



Static Hosts
————————————

> • Static hosts still need to be parsed into unbound hosts.conf file

I am looking for help with this since I dont fully understand the "read_static_hosts" section:
https://github.com/ipfire/ipfire-2.x/blob/master/config/unbound/unbound-dhcp-leases-bridge#L183-L223

- Why are the static hosts read into the "unbound-dhcp-leases-bridge" program?  
- Does this section write the static hosts into the the unbound `/etc/unbound/hosts.conf` file? (I dont believe it does)  
- Or are the static hosts for reference only?

Currently I do a quick grep search of the `/etc/unbound/hosts.conf` file for IP address and hostname in Line 142. 
If an IP/hostname exists in the static hosts file, then I exit.

So I think this is what the unbound-dhcp-leases-bridge does…



Hostnames with Spaces
—————————————————————

> • Properly handle hostnames with spaces

I remove the spaces in the hostname 
- yes, I have one device with this non-legit hostname

Is this OK?








===============================================================================
NOTE: Do not place this into a Production environment - only for Test environment.

• If placed into Test environment this current script will terminate "unbound-dhcp-leases-bridge".
• Do not run the `dhcpEvent.sh` script and "unbound-dhcp-leases-bridge" in parallel.
===============================================================================


Once again I am looking for help, feedback, thoughts and comments!


Jon



> On Dec 8, 2023, at 10:26 AM, jon <jon.murphy(a)ipfire.org> wrote:
> 
> The first script did not place A & PTR records into the unbound cache and this version does!
> 
> Once again I am looking for your feedback, thoughts and comments!
> 
> 
> Jon
> 
> ===============================================================================
> 
> NOTE: Do not place this into a Production environment - only for Test environment.
> 
> 
> Description of Proof of Concept
> ———————————————————————————————
> 
> Transfers DHCP lease information to unbound DNS
> 
> • Enable DHCP On Commit/Release/Expire via dhcpd.conf (or dhcpd.conf.local)
> • A new (or changed or removed) dynamic/fixed lease causes Script to run
> • Script adds A & PTR records on Commit:
>    • to unbound dhcp-leases.conf file
>    • to unbound cache via "unbound-control load-data"
> • Script removes A & PTR records on Release or Expiry of lease
> 
> 
> The Script
> ———————————
> 
> • Handles dynamic and fixed leases
> • Static hosts checked for duplicates
> • Re-enabled unbound-control local_data
>    • unbound reloads not needed
> • A & PTR records survive IPFire reboot, unbound restart
>    • no delays from difficult dhcp-leases parsing
> 
> Not completed (yet):
> • Static hosts still need to be parsed into unbound hosts.conf file
> • Properly handle hostnames with spaces
>    • non-legit need to change to dashes or be removed (or completely dropped)
> • to be checked with large amount of clients
>    • I’ve tested with 6 devices on test system and ~40 devices on production
> 
> 
> <dhcpEvent_v15.sh.txt><dhcpd.conf.local>
> 
> 
> 
> NOTE: Do not place this into a Production environment.  Only for Test environment.
> 
> • If placed into Test environment this current script will terminate "unbound-dhcp-leases-bridge".
> • Do not run the `dhcpEvent.sh` script and "unbound-dhcp-leases-bridge" in parallel.
> 
> 


[-- Attachment #2: dhcpEvent_v25.sh.txt --]
[-- Type: text/plain, Size: 5684 bytes --]

#!/bin/bash

#	DHCP commit/release/expiry status drives this process

###############     following 7 lines for testing  only    ###############
#	Kill the unbound-dhcp-leases-bridge process
pgrep -f unbound-dhcp-leases-bridge &&
	kill -s SIGTERM $(pgrep -d',' -f unbound-dhcp-leases-bridge)

#	chmod -v 744 /root/dhcpdconf/dhcpEvent_vN.sh
#	ln -vsf /root/dhcpdconf/dhcpEvent_vN.sh /root/dhcpdconf/dhcpEvent.sh
#-------------------------------------------------------------------------


dhcpCREstatus="${1:-}"
clientIP="${2:-}"
clientHostname="${3:-}"

unboundDHCPleases="/etc/unbound/dhcp-leases.conf"
unboundStaticHosts="/etc/unbound/hosts.conf"

#	time to live (seconds)
TTL="60"	#	time to live (seconds)

tagName="dhcpEv25"	#	to be "dhcpEvent"

###############     Functions     ###############

#	write A & PTR records to unbound
#
write-unbound () {
	local IPaddr
	local clientName
	local revIP
	local recA
	local recPTR

	IPaddr=$1
	clientName=$2

	#	create reverse IP for arpa name
	revIP=$( echo "${IPaddr}" | /usr/bin/awk -F. '{print $4"."$3"."$2"."$1}' )

	#	create "A" record and "PTR" record for unbound
	recA="${clientName}.${DOMAINNAME}. ${TTL} IN A ${IPaddr}"
	recPTR="${revIP}.in-addr.arpa. ${TTL} IN PTR ${clientName}.${DOMAINNAME}."

	#	send A & PTR records to unbound and dhcp-leases file
	echo -e "${recA}\n${recPTR}" | /usr/sbin/unbound-control local_datas
	echo "local-data: \"${recA}\"" >> "${unboundDHCPleases}"
	echo "local-data: \"${recPTR}\"" >> "${unboundDHCPleases}"

	msg_log "${dhcpCREstatus}: added A & PTR records to unbound"
}

#	delete A & PTR records from unbound, by IP address
#
delete_unbound () {
	local IPaddr
	local revIP
	local fqdn
	local deleteList

	local IPaddr=$1

	#	need IP address to get FQDN and arpa name
	revIP=$( echo "${IPaddr}" | /usr/bin/awk -F. '{print $4"."$3"."$2"."$1}' )
	fqdn=$( /bin/grep --word-regexp --ignore-case "${IPaddr}" <<< "${ucList}" | \
		/usr/bin/awk '{ print $1 }' )

	#	does a FQDN exist for this IP address?  If yes, then remove
	if [[ -n "${fqdn}" ]] ; then
		#	remove fqdn names from unbound AND dhcp-leases file

		#	look for multiple instances of fqdn (client hostname.domain)
		deleteList=$( /bin/grep --word-regexp --ignore-case "${fqdn}" <<< "${ucList}" | \
			/usr/bin/awk '{ print $1 }' )

		#	delete from unbound cache
		echo "${deleteList}" | /usr/sbin/unbound-control local_datas_remove

		#	delete from dhcp-leases.conf file
		/bin/sed --in-place "/\b${fqdn}/d" "${unboundDHCPleases}"
	else
		msg_log "${dhcpCREstatus}: Oops, no FQDN - exit"
		return 1
	fi
	msg_log "${dhcpCREstatus}: references to ${fqdn} removed from unbound"
}

#	send message to message log
msg_log () {
	if tty --silent ; then
		echo "${tagName}:" "$*"
	else
		/usr/bin/logger --tag "${tagName}" "$*"
	fi
}

###############     Main     ###############

#	check parameter count
if (( "$#" < 2 )) || (( $# > 3 )); then
    msg_log "Illegal number of parameters - exit"
	exit 1
fi

#	simple check for valid IP addr ( may be too simple! )
if ! /bin/grep --quiet --extended-regexp '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' <<< "${clientIP}" ; then
	msg_log "${clientIP} is an invalid IP address - exit"
	exit
fi

#	need DOMAINNAME value
eval "$(/usr/local/bin/readhash /var/ipfire/main/settings)"

#	remove invalid characters in client hostname
clientHostname=$( echo "${clientHostname}" | /bin/sed 's/[^A-Za-z0-9.-]//g' )

#	get existing records from unbound-control list_local_data
ucList=$( /usr/sbin/unbound-control list_local_data | /bin/grep -i "${DOMAINNAME}" | expand -t1 )

#	Does this Record A already exist?
UC_RecA=$( /bin/grep --ignore-case "^${clientHostname}\.${DOMAINNAME}.*${clientIP}$" <<< "${ucList}" )

#	find IP addr in unbound
UC_IP=$( /bin/grep --word-regexp --ignore-case "${clientIP}" <<< "${ucList}" )

#	find client name, at start of line, in unbound (looking for record A)
UC_FQDN=$( /bin/grep --word-regexp --ignore-case "^${clientHostname}\.${DOMAINNAME}" <<< "${ucList}" )

case "${dhcpCREstatus}" in
	commit)

		#	if client hostname is blank then exit
		if [[ -z "${clientHostname}" ]] ; then
			msg_log "${dhcpCREstatus}: received IP ${clientIP} but missing client hostname - exit"
			exit
		fi

		#	if client exists in static hosts then exit
		if /bin/grep --word-regexp --quiet --ignore-case -e "${clientIP}" -e "${clientHostname}" "${unboundStaticHosts}"
		then
			msg_log "${dhcpCREstatus}: ${clientHostname} or ${clientIP} found in static hosts - exit"
			exit
		fi
																						# (ip   fqdn)
		if [[ -n "${UC_RecA}" ]] ; then
		#	existing IP and existing FQDN												# ( 1   1 )
			msg_log "${dhcpCREstatus}: unbound hostname & IP already exist - exit"
			exit
		else

			if [[ -z "${UC_IP}" ]] ; then
				#	New IP address! - write info to unbound & dhcp-leases.conf			# ( 0   0/1 )
				write-unbound "${clientIP}" "${clientHostname}"
			else
				#	Existing IP address!
				if [[ -z "${UC_FQDN}" ]] ; then
				#	Existing IP and new FQDN
					#	hostname/fqdn changed - delete A & PTR records from unbound		# ( 1   0 )
					delete_unbound "${clientIP}"

					#	add new to unbound!
					write-unbound "${clientIP}" "${clientHostname}"
				fi
			fi
		fi
		;;

	release|expiry)
		#	NOTE: expiry & release are the same since I don't understand the difference    :-(

		if [[ -z "${clientIP}" ]] ; then
			msg_log "${dhcpCREstatus}: missing client IP address - exit"
			exit 1
		fi

		#	does IP addr appear in unbound?  if yes, delete lines
		if [[ -n "${UC_IP}" ]] ; then
			delete_unbound "${clientIP}"
		else
			msg_log "${dhcpCREstatus}: ${clientIP} not found in unbound - done"
		fi
		;;

	*)
		msg_log "CRE script: case = no status - exit"
		exit
		;;
esac

exit