Re: [PATCH] RPZ: install new add-on

Hello Jon,

I don’t think it is a good idea to send over the same patch set multiple times.

I think first of all, we need to start at the beginning which is to figure out what this actually is, and why we would want this in IPFire.

I have seen that there have been conversations on community.ipfire.org http://community.ipfire.org/ many months ago, but I did not take many things away from it. It was from my point of view mostly a prove of concept discussion. Correct me if I have missed anything here.

We have talked about RPZ many times on the monthly call since the URL filter feature is falling more and more out of fashion. I think there is also many posts about this on the forum.

Apart from generally being in favour of trying this path, we decided to postpone any further works into this because we currently have a huge backlog of things to deal with that are moving at snail speed. This is rooted in so many people being simply tied in with other things.

As you can see, the patch set that you posted did not trigger a response - a common thing on this list recently and not because of this particular patch set.

Therefore I would like to avoid opening another can of worms before we have delivered the other things that have already lots of time spent on them and code written. Otherwise we are only getting a longer pipeline and this all is going to be more frustrating for everyone involved. And this is not about the feature per se; this is about getting things done.

So what I suggest doing is the following:

You have a Git repository, push the code there and commit any fixes to it over time so that nothing is getting lost.

Explain to me from the beginning, please, how it came to this patch set. What is the motivation? What problems does it solve? What problems remain? Basically: Why would anyone want this in IPFire? Where do you see application for this? Why is this realised as an add-on and not part of the core system?

I am particularly interested in what lists are publicly available. You know from URL Filter, that lists have shut down or just been abandoned. There is now very little point running it because the lists are just not as good as they used to be. Not horrible, yet, but over time this will not be getting any better. I am not even aware of any viable paid lists. Is this the same situation for RPZ? A quick Google search did not give me anything that would be free to use over DNS. The situation is somewhat similar with the IPS and IP blocklist feature.

I would be interested in hearing your thoughts on this.

Best, -Michael

On 27 Jul 2024, at 21:31, jon jon.murphy@ipfire.org wrote:

And another question.

I found an error in the `rpz-config` script file. Simple easy one line (actually one word) fix.

Do I resubmit the entire patch? -or- Do I wait until approved and then submit the simple patch for the one line change.

Jon

...

On Jul 20, 2024, at 1:42 PM, jon jon.murphy@ipfire.org wrote:

I need help with…

1. The custom allow and block lists are currently located at `/var/ipfire/rpz`. Is this correct?

2. The three bash scripts are currently located at `/usr/sbin`. Is this correct?

3. Should this commit be separated into multiple posts? If yes, how do I decide how to split it?

Jon

...

On Jul 20, 2024, at 12:46 PM, Jon Murphy jon.murphy@ipfire.org wrote:

What is it? Response Policy Zone (RPZ) is a mechanism to define local policies in a standardised way and load those policies from external sources. Bottom line: RPZ allows admins to easily block access to websites via DNS lookup.

RPZ can block websites via categories. Examples include: fake websites, annoying pop-up ads, newly registered domains, DoH bypass sites, bad "host" services, maliscious top level domains (e.g., *.zip, *.mov), piracy, gambling, pornography, and more. RPZ lists come from various RPZ providers and their available catagories.

This RPZ add-on enables the RPZ functionality by adding a couple lines in a configuration file. This add-on simply adds a configuration file and adds three scripts (config, metrics and sleep) to make RPZ easier for the admin to use.

RPZ was release in 2010 and has been part of the IPFire build since ~2015.

Why is it needed? Some IPFire admin's utilize pihole to block unwanted websites via DNS lookup. Moving the pihole base functionality (without pretty graphs) to IPFire removes one device from the admin's local network. And hopefully this reduces the pihole questions from the Community.

A list of RPZ providers can be recommended by IPFire and coded into a set list. Or, if prefered, the local admin can choose their own RPZ providers.

This is a:

• simple replacement for pihole base functionality
• RPZ can be a nice replacement for the URL Filter

IPFire Wiki In process at: https://www.ipfire.org/docs/addons/rpz

more info:

• https://en.wikipedia.org/wiki/Response_policy_zone
• https://unbound.docs.nlnetlabs.nl/en/latest/topics/filtering/rpz.html

I need help with...

1. The custom allow and block lists are currently located at `/var/ipfire/rpz`. Is this correct?
2. The three bash scripts are currently located at `/usr/sbin`. Is this correct?

Signed-off-by: Jon Murphy jon.murphy@ipfire.org

config/backup/includes/rpz | 5 + config/rootfiles/packages/rpz | 11 ++ config/rpz/00-rpz.conf | 18 ++++ config/rpz/rpz-config | 194 ++++++++++++++++++++++++++++++++++ config/rpz/rpz-metrics | 143 +++++++++++++++++++++++++ config/rpz/rpz-sleep | 58 ++++++++++ lfs/rpz | 88 +++++++++++++++ make.sh | 2 + 8 files changed, 519 insertions(+) create mode 100644 config/backup/includes/rpz create mode 100644 config/rootfiles/packages/rpz create mode 100644 config/rpz/00-rpz.conf create mode 100644 config/rpz/rpz-config create mode 100644 config/rpz/rpz-metrics create mode 100644 config/rpz/rpz-sleep create mode 100644 lfs/rpz

diff --git a/config/backup/includes/rpz b/config/backup/includes/rpz new file mode 100644 index 000000000..4d59bb40c --- /dev/null +++ b/config/backup/includes/rpz @@ -0,0 +1,5 @@ +/var/ipfire/rpz/allowlist +/var/ipfire/rpz/blocklist +/etc/unbound/zonefiles/allow.rpz +/etc/unbound/zonefiles/block.rpz +/etc/unbound/local.d/*rpz.conf diff --git a/config/rootfiles/packages/rpz b/config/rootfiles/packages/rpz new file mode 100644 index 000000000..2ffa715dd --- /dev/null +++ b/config/rootfiles/packages/rpz @@ -0,0 +1,11 @@ +etc/unbound/local.d/00-rpz.conf +etc/unbound/zonefiles +etc/unbound/zonefiles/allow.rpz +etc/unbound/zonefiles/block.rpz +usr/sbin/rpz-config +usr/sbin/rpz-metrics +usr/sbin/rpz-sleep +var/ipfire/backup/addons/includes/rpz +var/ipfire/rpz +var/ipfire/rpz/allowlist +var/ipfire/rpz/blocklist diff --git a/config/rpz/00-rpz.conf b/config/rpz/00-rpz.conf new file mode 100644 index 000000000..72c1d12e5 --- /dev/null +++ b/config/rpz/00-rpz.conf @@ -0,0 +1,18 @@ +server:

• module-config: "respip validator iterator"

+rpz:

• name: allow.rpz
• zonefile: /etc/unbound/zonefiles/allow.rpz
• rpz-action-override: passthru
• rpz-log: yes
• rpz-log-name: allow
• rpz-signal-nxdomain-ra: yes

+rpz:

• name: block.rpz
• zonefile: /etc/unbound/zonefiles/block.rpz
• rpz-action-override: nxdomain
• rpz-log: yes
• rpz-log-name: block
• rpz-signal-nxdomain-ra: yes

diff --git a/config/rpz/rpz-config b/config/rpz/rpz-config new file mode 100644 index 000000000..98dc0a4ca --- /dev/null +++ b/config/rpz/rpz-config @@ -0,0 +1,194 @@ +#!/bin/bash +############################################################################### +# # +# IPFire.org - A linux based firewall # +# Copyright (C) 2024 IPFire Team info@ipfire.org # +# # +# This program is free software: you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation, either version 3 of the License, or # +# (at your option) any later version. # +# # +# This program is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with this program. If not, see http://www.gnu.org/licenses/. # +# # +###############################################################################

+# v22 - 2024-07-12

+############### Functions ###############

+msg_log () {

• /usr/bin/logger --tag "${tagName}" "$*"
• if tty --silent ; then
• echo "${tagName}:" "$*"
• fi

+}

+check_name () {

• local testName="${1}"
• # check for a valid name
• regex='^[a-zA-Z0-9_]+$'
• if [[ ! "${testName}" =~ $regex ]] ; then
• msg_log "error: rpz: the NAME is not valid: "${testName}". exit."
• exit 1
• fi

+}

+check_unbound_conf () {

• # check the above config files
• msg_log "info: rpz: check for errors with "unbound-checkconf""
• /usr/sbin/unbound-checkconf
• exit_code=$?
• if [[ "${exit_code}" -ne 0 ]] ; then
• msg_log "error: rpz: unbound-checkconf. exit."
• exit "${exit_code}"
• fi

+}

+make_rpz_file () {

• local theType="${1}" # allow or block
• theList="/var/ipfire/rpz/${theType}list" # input user list of domains
• theZoneFile="/etc/unbound/zonefiles/${theType}.rpz" # output file for RPZ
• theAction='.'
• if [[ "${theType}" =~ "block" ]] ; then
• theAction='rpz-passthru.'
• fi
• # does a list exist?
• if [[ -s "${theList}" ]] ; then
• # drop any extra "blanks" and add "CNAME <RPZ action>." to each line
• actionList=$( /usr/bin/awk '{$1=$1};1' "${theList}" |
• /bin/sed "/^[^;].*[[:alnum:]]/ s|$| CNAME ${theAction}|" )
• msg_log "info: rpz: create zonefile for ${theList}"
• /bin/cat <<-EOF > "${theZoneFile}"
• ; Name: ${theType} list
• ; Last modified: $(date "+%Y-%m-%d at %H.%M.%S %Z")
• ;
• ; domains with actions list
• ;
• ${actionList}
• EOF
• # reload the zone that was just updated
• zoneBase=$( basename "${theZoneFile}" )
• /usr/sbin/unbound-control auth_zone_reload -q "${zoneBase}"
• fi

+}

+############### Main ###############

+tagName="unbound"

+theAction="${1}" # input action +theName="${2}" # input RPZ name +theURL="${3}" # input RPZ URL

+check_name "${theName}" # is this a valid name?

+rpzConfig="/etc/unbound/local.d/${theName}.rpz.conf" # output zone conf file +rpzFile="/etc/unbound/zonefiles/${theName}.rpz" # output for RPZ file

+case "${theAction}" in

• # add new rpz list
• add )
• # does this config already exist? If yes, then exit
• if [[ -f "${rpzConfig}" ]] ; then
• msg_log "info: rpz: ${rpzConfig} already exists. exit"
• exit 1
• fi
• # is this a valid URL?
• regex='^https://%5B-%5B:alnum:%5D%5C+&@#/%?=~_%7C!:,.;%5D*%5B-%5B:alnum:%5D%5C+&...]'
• if ! [[ "${theURL}" =~ $regex ]] ; then
• msg_log "error: rpz: the URL is not valid: "${theURL}". exit."
• exit 1
• fi
• # create the zone config file
• msg_log "info: rpz: add config file "${theName}.rpz.conf""
• cat <<-EOF > "${rpzConfig}"
• rpz:
• name: ${theName}.rpz
• zonefile: /etc/unbound/zonefiles/${theName}.rpz
• url: ${theURL}
• rpz-action-override: nxdomain
• rpz-log: yes
• rpz-log-name: ${theName}
• rpz-signal-nxdomain-ra: yes
• EOF
• # set-up zone file
• /usr/bin/touch "${rpzFile}"
• # unbound requires these settings for rpz files
• /bin/chown --verbose nobody:nobody "${rpzFile}"
• /bin/chmod --verbose 644 "${rpzFile}"
• ;;
• # trash config file & rpz file
• remove )
• if ! [[ -f "${rpzConfig}" ]] ; then
• msg_log "info: rpz: ${rpzConfig} does not exist. exit"
• exit 1
• fi
• msg_log "info: rpz: remove config file & rpz file "${theName}""
• /bin/rm --verbose "${rpzConfig}"
• /bin/rm --verbose "${rpzFile}"
• check_unbound_conf
• ;;
• # make a new allow or block rpz file
• make )
• case "${theName}" in
• allow )
• make_rpz_file allow
• ;;
• block )
• make_rpz_file block
• ;;
• allowblock )
• make_rpz_file allow
• make_rpz_file block
• ;;
• • )
• msg_log "error: rpz: the NAME is not valid: "${theName}". exit."
• exit 1
• ;;
• esac
• check_unbound_conf
• ;;
• *)
• msg_log "error: rpz: missing or incorrect parameter"
• /usr/bin/printf "Usage: rpzConfig.sh <ACTION> <NAME> <URL>\n"
• exit 1
• ;;

+esac

+# reload due to the changes +msg_log "rpz: running "unbound-control reload"" +/usr/sbin/unbound-control reload +exit_code=$? +if [[ "${exit_code}" -ne 0 ]] ; then

• msg_log "error: rpz: unbound-control "${theName}". exit."
• exit "${exit_code}"

+fi

+exit diff --git a/config/rpz/rpz-metrics b/config/rpz/rpz-metrics new file mode 100644 index 000000000..0f97c7911 --- /dev/null +++ b/config/rpz/rpz-metrics @@ -0,0 +1,143 @@ +#!/bin/bash +############################################################################### +# # +# IPFire.org - A linux based firewall # +# Copyright (C) 2024 IPFire Team info@ipfire.org # +# # +# This program is free software: you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation, either version 3 of the License, or # +# (at your option) any later version. # +# # +# This program is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with this program. If not, see http://www.gnu.org/licenses/. # +# # +###############################################################################

+# v18 on 2024-07-05

+############### Main ###############

+weeks="${1:-2}" # default to two message logs +sortBy="${2:-name}" # by name or by hits

+# get the list of message logs for N weeks +messageLogs=$( find /var/log/messages* -type f |

• /usr/bin/sort --version-sort |
• head -"${weeks}" )

+# get the list of RPZ names & counts from the message log(s) +rpzNameCount=$( for logf in ${messageLogs} ; do

• /usr/bin/zgrep --text --fixed-strings 'info: rpz: applied' "${logf}" |
• /usr/bin/awk '$10 ~ /[\w*]/ { print $10 }' ;
• done | /usr/bin/sort | /usr/bin/uniq --count )

+# flip results and remove brackets `[` and `]` +rpzNameCount=$( /bin/echo "${rpzNameCount}" |

• /usr/bin/awk '{ print $2, $1 }' |
• /bin/sed --regexp-extended 's|^[(.*)]|\1|' )

+# grab only names +rpzNames=$( /bin/echo "${rpzNameCount}" | /usr/bin/awk '{ print $1 }' )

+# get list of RPZ files +rpzFileList=$( /bin/find /etc/unbound/zonefiles -type f -iname "*.rpz" )

+# get basename of those files +rpzBaseNames=$( /bin/echo "${rpzFileList}" |

• /bin/sed 's|/etc/unbound/zonefiles/||g ; s|.rpz||g ;' )

+# add to rpzNames +rpzNames="${rpzNames}"$'\n'"${rpzBaseNames}"

+# drop duplicate names +rpzNames=$( echo "${rpzNames}" | /usr/bin/sort --unique )

+# get line count for each RPZ +lineCount=$( /bin/echo "${rpzFileList}" | /usr/bin/xargs wc -l )

+# get comment line count and blank line count for each RPZ +commentCount=$( /bin/echo "${rpzFileList}" |

• /usr/bin/xargs /bin/grep --count -e "^$" -e "^;" )

+# get modified date each RPZ +modDateList=$( /bin/echo "${rpzFileList}" | xargs stat -c '%.10y %n' )

+ucListAuthZones=$( /usr/sbin/unbound-control list_auth_zones )

+# get width of RPZ names +pWidth=$( /bin/echo "${rpzNames}" | /usr/bin/awk '{ print $1" " }' | wc -L ) +pFormat="%-${pWidth}s %-8s %-8s %-8s %10s %12s\n"

+# print title line +printf "${pFormat}" "name" "hits" "active" "lines" "hits/line%" "last update" +printf -- "--------------"

+theResults="" +totalLines=0 +totalHits=0 +while read -r theName +do

• printf -- "-" # pretend progress bar
• # get hit count
• theHits="0"
• if output=$( /bin/grep "^${theName}\s" <<< "${rpzNameCount}" ) ; then
• theHits=$( /bin/echo "${output}" |
• /usr/bin/awk '{ print $2 }' )
• totalHits=$(( totalHits + theHits ))
• fi
• # is this RPZ list active?
• theActive="disabled"
• if /bin/grep --quiet "^${theName}.rpz" <<< "${ucListAuthZones}"
• then
• theActive="enabled"
• fi
• # get line count then subtract comment count and blank line count
• # from total line count
• theLines="n/a"
• hitsPerLine="0"
• if output=$( /bin/grep --fixed-strings "/${theName}.rpz" <<< "${lineCount}" ) ; then
• theLines=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' )
• totalLines=$(( totalLines + theLines ))
• #hitsPerLine=$( echo "scale=0 ; $theHits / $theLines" | bc )
• hitsPerLine=$(( 100 * theHits / theLines ))
• fi
• # get modification date
• theModDate="n/a"
• if output=$( /bin/grep --fixed-strings "/${theName}.rpz" <<< "${modDateList}" ) ; then
• theModDate=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' )
• fi
• # add to results list
• theResults+="${theName} ${theHits} ${theActive} ${theLines} ${hitsPerLine} ${theModDate}"$'\n'

+done <<< "${rpzNames}"

+case "${sortBy}" in

• names|name) sortArg=(-k3,3r -k1,1) ;; # sort by "active" then by "name"
• hits|hit) sortArg=(-k3,3r -k2,2nr -k1,1) ;; # sort by "active" then by "hits" then by "name"
• lines|line) sortArg=(-k3,3r -k4,4nr -k1,1) ;; # sort by "active" then by "lines" then by "name"

+esac

+printf -- "--------------\n" +# remove blank lines, sort, print as columns +/bin/echo "${theResults}" |

• /usr/bin/awk '!/^[[:space:]]*$/' |
• /usr/bin/sort "${sortArg[@]}" |
• /usr/bin/awk --assign=width="${pWidth}" \
• '{ printf "%-*s %-8s %-8s %-8s %10s %12s\n", width, $1, $2, $3, $4, $5, $6 }'

+printf "${pFormat}" "" "=======" "" "========" "" "" +printf "${pFormat}" "Totals -->" "${totalHits}" "" "${totalLines}" "" ""

+exit diff --git a/config/rpz/rpz-sleep b/config/rpz/rpz-sleep new file mode 100644 index 000000000..eeef1174a --- /dev/null +++ b/config/rpz/rpz-sleep @@ -0,0 +1,58 @@ +#!/bin/bash +############################################################################### +# # +# IPFire.org - A linux based firewall # +# Copyright (C) 2024 IPFire Team info@ipfire.org # +# # +# This program is free software: you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation, either version 3 of the License, or # +# (at your option) any later version. # +# # +# This program is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with this program. If not, see http://www.gnu.org/licenses/. # +# # +###############################################################################

+# v04 on 2024-07-05

+############### Functions ###############

+# send message to message log +msg_log () {

• /usr/bin/logger --tag "${tagName}" "$*"
• if /usr/bin/tty --silent ; then
• echo "${tagName}:" "$*"
• fi

+}

+############### Main ###############

+tagName="unbound"

+sleepTime="${1:-5m}" # default to sleep for 5m (5 minutes)

+zoneList=$( /usr/sbin/unbound-control list_auth_zones | /usr/bin/awk '{print $1}' )

+for zone in ${zoneList} ; do

• /usr/bin/printf "disable ${zone}\t"
• /usr/sbin/unbound-control rpz_disable "${zone}"

+done

+msg_log "info: rpz: disabled all zones for ${sleepTime}"

+/bin/sleep "${sleepTime}"

+for zone in ${zoneList} ; do

• /usr/bin/printf "enable ${zone}\t"
• /usr/sbin/unbound-control rpz_enable "${zone}"

+done

+msg_log "info: rpz: enabled all zones"

+exit diff --git a/lfs/rpz b/lfs/rpz new file mode 100644 index 000000000..319c10b7f --- /dev/null +++ b/lfs/rpz @@ -0,0 +1,88 @@ +############################################################################### +# # +# IPFire.org - A linux based firewall # +# Copyright (C) 2024 IPFire Team info@ipfire.org # +# # +# This program is free software: you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation, either version 3 of the License, or # +# (at your option) any later version. # +# # +# This program is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with this program. If not, see http://www.gnu.org/licenses/. # +# # +###############################################################################

+############################################################################### +# Definitions +###############################################################################

+include Config

+SUMMARY = response policy zone - RPZ reputation system for unbound DNS

+VER = 1.0.0

+THISAPP = rpz-$(VER) +DIR_APP = $(DIR_SRC)/$(THISAPP) +TARGET = $(DIR_INFO)/$(THISAPP)

+PROG = rpz +PAK_VER = 1

+DEPS =

+SERVICES =

+############################################################################### +# Top-level Rules +###############################################################################

+install : $(TARGET)

+check :

+download :

+b2 :

+dist:

• @$(PAK)

+############################################################################### +# Installation Details +###############################################################################

+$(TARGET) :

• @$(PREBUILD)
• @rm -rf $(DIR_APP)
• # install RPZ scripts
• install -v -m 755 \
• $(DIR_CONF)/rpz/{rpz-config,rpz-metrics,rpz-sleep} -t /usr/sbin
• # Install settings folder and two empty files
• mkdir -pv /var/ipfire/rpz
• touch /var/ipfire/rpz/allowlist
• touch /var/ipfire/rpz/blocklist
• # Add conf file to /etc directory
• cp -vf $(DIR_CONF)/rpz/00-rpz.conf /etc/unbound/local.d
• # create zonefiles directory for the RPZ files and add two empty RPZ
• # files to avoid a unbound config error
• mkdir -pv /etc/unbound/zonefiles
• chown -v nobody:nobody /etc/unbound/zonefiles
• touch /etc/unbound/zonefiles/allow.rpz
• touch /etc/unbound/zonefiles/block.rpz
• # Install backup definition
• cp -vf $(DIR_CONF)/backup/includes/rpz /var/ipfire/backup/addons/includes/rpz
• @rm -rf $(DIR_APP)
• @$(POSTBUILD)

diff --git a/make.sh b/make.sh index 9bbbeb0f1..886d3760a 100755 --- a/make.sh +++ b/make.sh @@ -1721,6 +1721,8 @@ buildipfire() { lfsmake2 btrfs-progs lfsmake2 inotify-tools lfsmake2 grub-btrfs

• lfsmake2 rpz

# Kernelbuild ... current we have no platform that need

# multi kernel builds so KCFG is empty

2.30.2