I need help with… 1) The custom allow and block lists are currently located at `/var/ipfire/rpz`. Is this correct? 2) The three bash scripts are currently located at `/usr/sbin`. Is this correct? 3) Should this commit be separated into multiple posts? If yes, how do I decide how to split it? Jon > On Jul 20, 2024, at 12:46 PM, Jon Murphy wrote: > > What is it? > Response Policy Zone (RPZ) is a mechanism to define local policies in a standardised > way and load those policies from external sources. > Bottom line: RPZ allows admins to easily block access to websites via DNS lookup. > > RPZ can block websites via categories. Examples include: fake websites, annoying > pop-up ads, newly registered domains, DoH bypass sites, bad "host" services, > maliscious top level domains (e.g., *.zip, *.mov), piracy, gambling, pornography, > and more. RPZ lists come from various RPZ providers and their available > catagories. > > This RPZ add-on enables the RPZ functionality by adding a couple lines in a > configuration file. This add-on simply adds a configuration file and adds > three scripts (config, metrics and sleep) to make RPZ easier for the admin to use. > > RPZ was release in 2010 and has been part of the IPFire build since ~2015. > > Why is it needed? > Some IPFire admin's utilize pihole to block unwanted websites via DNS lookup. > Moving the pihole base functionality (without pretty graphs) to IPFire removes > one device from the admin's local network. > And hopefully this reduces the pihole questions from the Community. > > A list of RPZ providers can be recommended by IPFire and coded into a set list. > Or, if prefered, the local admin can choose their own RPZ providers. > > This is a: > * simple replacement for pihole base functionality > * RPZ can be a nice replacement for the URL Filter > > IPFire Wiki > In process at: https://www.ipfire.org/docs/addons/rpz > > more info: > * https://en.wikipedia.org/wiki/Response_policy_zone > * https://unbound.docs.nlnetlabs.nl/en/latest/topics/filtering/rpz.html > > I need help with... > 1) The custom allow and block lists are currently located at `/var/ipfire/rpz`. Is this correct? > 2) The three bash scripts are currently located at `/usr/sbin`. Is this correct? > > Signed-off-by: Jon Murphy > --- > config/backup/includes/rpz | 5 + > config/rootfiles/packages/rpz | 11 ++ > config/rpz/00-rpz.conf | 18 ++++ > config/rpz/rpz-config | 194 ++++++++++++++++++++++++++++++++++ > config/rpz/rpz-metrics | 143 +++++++++++++++++++++++++ > config/rpz/rpz-sleep | 58 ++++++++++ > lfs/rpz | 88 +++++++++++++++ > make.sh | 2 + > 8 files changed, 519 insertions(+) > create mode 100644 config/backup/includes/rpz > create mode 100644 config/rootfiles/packages/rpz > create mode 100644 config/rpz/00-rpz.conf > create mode 100644 config/rpz/rpz-config > create mode 100644 config/rpz/rpz-metrics > create mode 100644 config/rpz/rpz-sleep > create mode 100644 lfs/rpz > > diff --git a/config/backup/includes/rpz b/config/backup/includes/rpz > new file mode 100644 > index 000000000..4d59bb40c > --- /dev/null > +++ b/config/backup/includes/rpz > @@ -0,0 +1,5 @@ > +/var/ipfire/rpz/allowlist > +/var/ipfire/rpz/blocklist > +/etc/unbound/zonefiles/allow.rpz > +/etc/unbound/zonefiles/block.rpz > +/etc/unbound/local.d/*rpz.conf > diff --git a/config/rootfiles/packages/rpz b/config/rootfiles/packages/rpz > new file mode 100644 > index 000000000..2ffa715dd > --- /dev/null > +++ b/config/rootfiles/packages/rpz > @@ -0,0 +1,11 @@ > +etc/unbound/local.d/00-rpz.conf > +etc/unbound/zonefiles > +etc/unbound/zonefiles/allow.rpz > +etc/unbound/zonefiles/block.rpz > +usr/sbin/rpz-config > +usr/sbin/rpz-metrics > +usr/sbin/rpz-sleep > +var/ipfire/backup/addons/includes/rpz > +var/ipfire/rpz > +var/ipfire/rpz/allowlist > +var/ipfire/rpz/blocklist > diff --git a/config/rpz/00-rpz.conf b/config/rpz/00-rpz.conf > new file mode 100644 > index 000000000..72c1d12e5 > --- /dev/null > +++ b/config/rpz/00-rpz.conf > @@ -0,0 +1,18 @@ > +server: > + module-config: "respip validator iterator" > + > +rpz: > + name: allow.rpz > + zonefile: /etc/unbound/zonefiles/allow.rpz > + rpz-action-override: passthru > + rpz-log: yes > + rpz-log-name: allow > + rpz-signal-nxdomain-ra: yes > + > +rpz: > + name: block.rpz > + zonefile: /etc/unbound/zonefiles/block.rpz > + rpz-action-override: nxdomain > + rpz-log: yes > + rpz-log-name: block > + rpz-signal-nxdomain-ra: yes > diff --git a/config/rpz/rpz-config b/config/rpz/rpz-config > new file mode 100644 > index 000000000..98dc0a4ca > --- /dev/null > +++ b/config/rpz/rpz-config > @@ -0,0 +1,194 @@ > +#!/bin/bash > +############################################################################### > +# # > +# IPFire.org - A linux based firewall # > +# Copyright (C) 2024 IPFire Team # > +# # > +# This program is free software: you can redistribute it and/or modify # > +# it under the terms of the GNU General Public License as published by # > +# the Free Software Foundation, either version 3 of the License, or # > +# (at your option) any later version. # > +# # > +# This program is distributed in the hope that it will be useful, # > +# but WITHOUT ANY WARRANTY; without even the implied warranty of # > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # > +# GNU General Public License for more details. # > +# # > +# You should have received a copy of the GNU General Public License # > +# along with this program. If not, see . # > +# # > +############################################################################### > + > +# v22 - 2024-07-12 > + > +############### Functions ############### > + > +msg_log () { > + /usr/bin/logger --tag "${tagName}" "$*" > + if tty --silent ; then > + echo "${tagName}:" "$*" > + fi > +} > + > +check_name () { > + local testName="${1}" > + # check for a valid name > + regex='^[a-zA-Z0-9_]+$' > + if [[ ! "${testName}" =~ $regex ]] ; then > + msg_log "error: rpz: the NAME is not valid: \"${testName}\". exit." > + exit 1 > + fi > +} > + > +check_unbound_conf () { > + # check the above config files > + msg_log "info: rpz: check for errors with \"unbound-checkconf\"" > + /usr/sbin/unbound-checkconf > + exit_code=$? > + if [[ "${exit_code}" -ne 0 ]] ; then > + msg_log "error: rpz: unbound-checkconf. exit." > + exit "${exit_code}" > + fi > +} > + > +make_rpz_file () { > + local theType="${1}" # allow or block > + > + theList="/var/ipfire/rpz/${theType}list" # input user list of domains > + theZoneFile="/etc/unbound/zonefiles/${theType}.rpz" # output file for RPZ > + > + theAction='.' > + if [[ "${theType}" =~ "block" ]] ; then > + theAction='rpz-passthru.' > + fi > + > + # does a list exist? > + if [[ -s "${theList}" ]] ; then > + # drop any extra "blanks" and add "CNAME ." to each line > + actionList=$( /usr/bin/awk '{$1=$1};1' "${theList}" | > + /bin/sed "/^[^;].*[[:alnum:]]/ s|$| CNAME ${theAction}|" ) > + > + msg_log "info: rpz: create zonefile for ${theList}" > + > + /bin/cat <<-EOF > "${theZoneFile}" > + ; Name: ${theType} list > + ; Last modified: $(date "+%Y-%m-%d at %H.%M.%S %Z") > + ; > + ; domains with actions list > + ; > + ${actionList} > + EOF > + > + # reload the zone that was just updated > + zoneBase=$( basename "${theZoneFile}" ) > + /usr/sbin/unbound-control auth_zone_reload -q "${zoneBase}" > + fi > +} > + > +############### Main ############### > + > +tagName="unbound" > + > +theAction="${1}" # input action > +theName="${2}" # input RPZ name > +theURL="${3}" # input RPZ URL > + > +check_name "${theName}" # is this a valid name? > + > +rpzConfig="/etc/unbound/local.d/${theName}.rpz.conf" # output zone conf file > +rpzFile="/etc/unbound/zonefiles/${theName}.rpz" # output for RPZ file > + > +case "${theAction}" in > + > + # add new rpz list > + add ) > + # does this config already exist? If yes, then exit > + if [[ -f "${rpzConfig}" ]] ; then > + msg_log "info: rpz: ${rpzConfig} already exists. exit" > + exit 1 > + fi > + > + # is this a valid URL? > + regex='^https://[-[:alnum:]\+&@#/%?=~_|!:,.;]*[-[:alnum:]\+&@#/%=~_|]' > + if ! [[ "${theURL}" =~ $regex ]] ; then > + msg_log "error: rpz: the URL is not valid: \"${theURL}\". exit." > + exit 1 > + fi > + > + # create the zone config file > + msg_log "info: rpz: add config file \"${theName}.rpz.conf\"" > + cat <<-EOF > "${rpzConfig}" > + rpz: > + name: ${theName}.rpz > + zonefile: /etc/unbound/zonefiles/${theName}.rpz > + url: ${theURL} > + rpz-action-override: nxdomain > + rpz-log: yes > + rpz-log-name: ${theName} > + rpz-signal-nxdomain-ra: yes > + EOF > + > + # set-up zone file > + /usr/bin/touch "${rpzFile}" > + # unbound requires these settings for rpz files > + /bin/chown --verbose nobody:nobody "${rpzFile}" > + /bin/chmod --verbose 644 "${rpzFile}" > + ;; > + > + # trash config file & rpz file > + remove ) > + if ! [[ -f "${rpzConfig}" ]] ; then > + msg_log "info: rpz: ${rpzConfig} does not exist. exit" > + exit 1 > + fi > + > + msg_log "info: rpz: remove config file & rpz file \"${theName}\"" > + /bin/rm --verbose "${rpzConfig}" > + /bin/rm --verbose "${rpzFile}" > + > + check_unbound_conf > + ;; > + > + # make a new allow or block rpz file > + make ) > + case "${theName}" in > + allow ) > + make_rpz_file allow > + ;; > + > + block ) > + make_rpz_file block > + ;; > + > + allowblock ) > + make_rpz_file allow > + make_rpz_file block > + ;; > + > + * ) > + msg_log "error: rpz: the NAME is not valid: \"${theName}\". exit." > + exit 1 > + ;; > + esac > + > + check_unbound_conf > + ;; > + > + *) > + msg_log "error: rpz: missing or incorrect parameter" > + /usr/bin/printf "Usage: rpzConfig.sh \n" > + exit 1 > + ;; > + > +esac > + > +# reload due to the changes > +msg_log "rpz: running \"unbound-control reload\"" > +/usr/sbin/unbound-control reload > +exit_code=$? > +if [[ "${exit_code}" -ne 0 ]] ; then > + msg_log "error: rpz: unbound-control \"${theName}\". exit." > + exit "${exit_code}" > +fi > + > +exit > diff --git a/config/rpz/rpz-metrics b/config/rpz/rpz-metrics > new file mode 100644 > index 000000000..0f97c7911 > --- /dev/null > +++ b/config/rpz/rpz-metrics > @@ -0,0 +1,143 @@ > +#!/bin/bash > +############################################################################### > +# # > +# IPFire.org - A linux based firewall # > +# Copyright (C) 2024 IPFire Team # > +# # > +# This program is free software: you can redistribute it and/or modify # > +# it under the terms of the GNU General Public License as published by # > +# the Free Software Foundation, either version 3 of the License, or # > +# (at your option) any later version. # > +# # > +# This program is distributed in the hope that it will be useful, # > +# but WITHOUT ANY WARRANTY; without even the implied warranty of # > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # > +# GNU General Public License for more details. # > +# # > +# You should have received a copy of the GNU General Public License # > +# along with this program. If not, see . # > +# # > +############################################################################### > + > +# v18 on 2024-07-05 > + > +############### Main ############### > + > +weeks="${1:-2}" # default to two message logs > +sortBy="${2:-name}" # by name or by hits > + > +# get the list of message logs for N weeks > +messageLogs=$( find /var/log/messages* -type f | > + /usr/bin/sort --version-sort | > + head -"${weeks}" ) > + > +# get the list of RPZ names & counts from the message log(s) > +rpzNameCount=$( for logf in ${messageLogs} ; do > + /usr/bin/zgrep --text --fixed-strings 'info: rpz: applied' "${logf}" | > + /usr/bin/awk '$10 ~ /\[\w*]/ { print $10 }' ; > + done | /usr/bin/sort | /usr/bin/uniq --count ) > + > +# flip results and remove brackets `[` and `]` > +rpzNameCount=$( /bin/echo "${rpzNameCount}" | > + /usr/bin/awk '{ print $2, $1 }' | > + /bin/sed --regexp-extended 's|^\[(.*)\]|\1|' ) > + > +# grab only names > +rpzNames=$( /bin/echo "${rpzNameCount}" | /usr/bin/awk '{ print $1 }' ) > + > +# get list of RPZ files > +rpzFileList=$( /bin/find /etc/unbound/zonefiles -type f -iname "*.rpz" ) > + > +# get basename of those files > +rpzBaseNames=$( /bin/echo "${rpzFileList}" | > + /bin/sed 's|/etc/unbound/zonefiles/||g ; s|\.rpz||g ;' ) > + > +# add to rpzNames > +rpzNames="${rpzNames}"$'\n'"${rpzBaseNames}" > + > +# drop duplicate names > +rpzNames=$( echo "${rpzNames}" | /usr/bin/sort --unique ) > + > +# get line count for each RPZ > +lineCount=$( /bin/echo "${rpzFileList}" | /usr/bin/xargs wc -l ) > + > +# get comment line count and blank line count for each RPZ > +commentCount=$( /bin/echo "${rpzFileList}" | > + /usr/bin/xargs /bin/grep --count -e "^$" -e "^;" ) > + > +# get modified date each RPZ > +modDateList=$( /bin/echo "${rpzFileList}" | xargs stat -c '%.10y %n' ) > + > +ucListAuthZones=$( /usr/sbin/unbound-control list_auth_zones ) > + > +# get width of RPZ names > +pWidth=$( /bin/echo "${rpzNames}" | /usr/bin/awk '{ print $1" " }' | wc -L ) > +pFormat="%-${pWidth}s %-8s %-8s %-8s %10s %12s\n" > + > +# print title line > +printf "${pFormat}" "name" "hits" "active" "lines" "hits/line%" "last update" > +printf -- "--------------" > + > +theResults="" > +totalLines=0 > +totalHits=0 > +while read -r theName > +do > + printf -- "-" # pretend progress bar > + # get hit count > + theHits="0" > + if output=$( /bin/grep "^${theName}\s" <<< "${rpzNameCount}" ) ; then > + theHits=$( /bin/echo "${output}" | > + /usr/bin/awk '{ print $2 }' ) > + totalHits=$(( totalHits + theHits )) > + fi > + > + # is this RPZ list active? > + theActive="disabled" > + if /bin/grep --quiet "^${theName}\.rpz" <<< "${ucListAuthZones}" > + then > + theActive="enabled" > + fi > + > + # get line count then subtract comment count and blank line count > + # from total line count > + theLines="n/a" > + hitsPerLine="0" > + if output=$( /bin/grep --fixed-strings "/${theName}.rpz" <<< "${lineCount}" ) ; then > + theLines=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' ) > + totalLines=$(( totalLines + theLines )) > + > + #hitsPerLine=$( echo "scale=0 ; $theHits / $theLines" | bc ) > + hitsPerLine=$(( 100 * theHits / theLines )) > + fi > + > + # get modification date > + theModDate="n/a" > + if output=$( /bin/grep --fixed-strings "/${theName}.rpz" <<< "${modDateList}" ) ; then > + theModDate=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' ) > + fi > + > + # add to results list > + theResults+="${theName} ${theHits} ${theActive} ${theLines} ${hitsPerLine} ${theModDate}"$'\n' > +done <<< "${rpzNames}" > + > +case "${sortBy}" in > + names|name) sortArg=(-k3,3r -k1,1) ;; # sort by "active" then by "name" > + > + hits|hit) sortArg=(-k3,3r -k2,2nr -k1,1) ;; # sort by "active" then by "hits" then by "name" > + > + lines|line) sortArg=(-k3,3r -k4,4nr -k1,1) ;; # sort by "active" then by "lines" then by "name" > +esac > + > +printf -- "--------------\n" > +# remove blank lines, sort, print as columns > +/bin/echo "${theResults}" | > + /usr/bin/awk '!/^[[:space:]]*$/' | > + /usr/bin/sort "${sortArg[@]}" | > + /usr/bin/awk --assign=width="${pWidth}" \ > + '{ printf "%-*s %-8s %-8s %-8s %10s %12s\n", width, $1, $2, $3, $4, $5, $6 }' > + > +printf "${pFormat}" "" "=======" "" "========" "" "" > +printf "${pFormat}" "Totals -->" "${totalHits}" "" "${totalLines}" "" "" > + > +exit > diff --git a/config/rpz/rpz-sleep b/config/rpz/rpz-sleep > new file mode 100644 > index 000000000..eeef1174a > --- /dev/null > +++ b/config/rpz/rpz-sleep > @@ -0,0 +1,58 @@ > +#!/bin/bash > +############################################################################### > +# # > +# IPFire.org - A linux based firewall # > +# Copyright (C) 2024 IPFire Team # > +# # > +# This program is free software: you can redistribute it and/or modify # > +# it under the terms of the GNU General Public License as published by # > +# the Free Software Foundation, either version 3 of the License, or # > +# (at your option) any later version. # > +# # > +# This program is distributed in the hope that it will be useful, # > +# but WITHOUT ANY WARRANTY; without even the implied warranty of # > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # > +# GNU General Public License for more details. # > +# # > +# You should have received a copy of the GNU General Public License # > +# along with this program. If not, see . # > +# # > +############################################################################### > + > +# v04 on 2024-07-05 > + > +############### Functions ############### > + > +# send message to message log > +msg_log () { > + /usr/bin/logger --tag "${tagName}" "$*" > + if /usr/bin/tty --silent ; then > + echo "${tagName}:" "$*" > + fi > +} > + > +############### Main ############### > + > +tagName="unbound" > + > +sleepTime="${1:-5m}" # default to sleep for 5m (5 minutes) > + > +zoneList=$( /usr/sbin/unbound-control list_auth_zones | /usr/bin/awk '{print $1}' ) > + > +for zone in ${zoneList} ; do > + /usr/bin/printf "disable ${zone}\t" > + /usr/sbin/unbound-control rpz_disable "${zone}" > +done > + > +msg_log "info: rpz: disabled all zones for ${sleepTime}" > + > +/bin/sleep "${sleepTime}" > + > +for zone in ${zoneList} ; do > + /usr/bin/printf "enable ${zone}\t" > + /usr/sbin/unbound-control rpz_enable "${zone}" > +done > + > +msg_log "info: rpz: enabled all zones" > + > +exit > diff --git a/lfs/rpz b/lfs/rpz > new file mode 100644 > index 000000000..319c10b7f > --- /dev/null > +++ b/lfs/rpz > @@ -0,0 +1,88 @@ > +############################################################################### > +# # > +# IPFire.org - A linux based firewall # > +# Copyright (C) 2024 IPFire Team # > +# # > +# This program is free software: you can redistribute it and/or modify # > +# it under the terms of the GNU General Public License as published by # > +# the Free Software Foundation, either version 3 of the License, or # > +# (at your option) any later version. # > +# # > +# This program is distributed in the hope that it will be useful, # > +# but WITHOUT ANY WARRANTY; without even the implied warranty of # > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # > +# GNU General Public License for more details. # > +# # > +# You should have received a copy of the GNU General Public License # > +# along with this program. If not, see . # > +# # > +############################################################################### > + > +############################################################################### > +# Definitions > +############################################################################### > + > +include Config > + > +SUMMARY = response policy zone - RPZ reputation system for unbound DNS > + > +VER = 1.0.0 > + > +THISAPP = rpz-$(VER) > +DIR_APP = $(DIR_SRC)/$(THISAPP) > +TARGET = $(DIR_INFO)/$(THISAPP) > + > +PROG = rpz > +PAK_VER = 1 > + > +DEPS = > + > +SERVICES = > + > +############################################################################### > +# Top-level Rules > +############################################################################### > + > +install : $(TARGET) > + > +check : > + > +download : > + > +b2 : > + > +dist: > + @$(PAK) > + > +############################################################################### > +# Installation Details > +############################################################################### > + > +$(TARGET) : > + @$(PREBUILD) > + @rm -rf $(DIR_APP) > + > + # install RPZ scripts > + install -v -m 755 \ > + $(DIR_CONF)/rpz/{rpz-config,rpz-metrics,rpz-sleep} -t /usr/sbin > + > + # Install settings folder and two empty files > + mkdir -pv /var/ipfire/rpz > + touch /var/ipfire/rpz/allowlist > + touch /var/ipfire/rpz/blocklist > + > + # Add conf file to /etc directory > + cp -vf $(DIR_CONF)/rpz/00-rpz.conf /etc/unbound/local.d > + > + # create zonefiles directory for the RPZ files and add two empty RPZ > + # files to avoid a unbound config error > + mkdir -pv /etc/unbound/zonefiles > + chown -v nobody:nobody /etc/unbound/zonefiles > + touch /etc/unbound/zonefiles/allow.rpz > + touch /etc/unbound/zonefiles/block.rpz > + > + # Install backup definition > + cp -vf $(DIR_CONF)/backup/includes/rpz /var/ipfire/backup/addons/includes/rpz > + > + @rm -rf $(DIR_APP) > + @$(POSTBUILD) > diff --git a/make.sh b/make.sh > index 9bbbeb0f1..886d3760a 100755 > --- a/make.sh > +++ b/make.sh > @@ -1721,6 +1721,8 @@ buildipfire() { > lfsmake2 btrfs-progs > lfsmake2 inotify-tools > lfsmake2 grub-btrfs > + lfsmake2 rpz > + > > # Kernelbuild ... current we have no platform that need > # multi kernel builds so KCFG is empty > -- > 2.30.2 >