And another question. I found an error in the `rpz-config` script file. Simple easy one line (actually one word) fix. Do I resubmit the entire patch? -or- Do I wait until approved and then submit the simple patch for the one line change. Jon > On Jul 20, 2024, at 1:42 PM, jon wrote: > > I need help with… > > 1) The custom allow and block lists are currently located at `/var/ipfire/rpz`. Is this correct? > > 2) The three bash scripts are currently located at `/usr/sbin`. Is this correct? > > 3) Should this commit be separated into multiple posts? If yes, how do I decide how to split it? > > > Jon > > >> On Jul 20, 2024, at 12:46 PM, Jon Murphy wrote: >> >> What is it? >> Response Policy Zone (RPZ) is a mechanism to define local policies in a standardised >> way and load those policies from external sources. >> Bottom line: RPZ allows admins to easily block access to websites via DNS lookup. >> >> RPZ can block websites via categories. Examples include: fake websites, annoying >> pop-up ads, newly registered domains, DoH bypass sites, bad "host" services, >> maliscious top level domains (e.g., *.zip, *.mov), piracy, gambling, pornography, >> and more. RPZ lists come from various RPZ providers and their available >> catagories. >> >> This RPZ add-on enables the RPZ functionality by adding a couple lines in a >> configuration file. This add-on simply adds a configuration file and adds >> three scripts (config, metrics and sleep) to make RPZ easier for the admin to use. >> >> RPZ was release in 2010 and has been part of the IPFire build since ~2015. >> >> Why is it needed? >> Some IPFire admin's utilize pihole to block unwanted websites via DNS lookup. >> Moving the pihole base functionality (without pretty graphs) to IPFire removes >> one device from the admin's local network. >> And hopefully this reduces the pihole questions from the Community. >> >> A list of RPZ providers can be recommended by IPFire and coded into a set list. >> Or, if prefered, the local admin can choose their own RPZ providers. >> >> This is a: >> * simple replacement for pihole base functionality >> * RPZ can be a nice replacement for the URL Filter >> >> IPFire Wiki >> In process at: https://www.ipfire.org/docs/addons/rpz >> >> more info: >> * https://en.wikipedia.org/wiki/Response_policy_zone >> * https://unbound.docs.nlnetlabs.nl/en/latest/topics/filtering/rpz.html >> >> I need help with... >> 1) The custom allow and block lists are currently located at `/var/ipfire/rpz`. Is this correct? >> 2) The three bash scripts are currently located at `/usr/sbin`. Is this correct? >> >> Signed-off-by: Jon Murphy >> --- >> config/backup/includes/rpz | 5 + >> config/rootfiles/packages/rpz | 11 ++ >> config/rpz/00-rpz.conf | 18 ++++ >> config/rpz/rpz-config | 194 ++++++++++++++++++++++++++++++++++ >> config/rpz/rpz-metrics | 143 +++++++++++++++++++++++++ >> config/rpz/rpz-sleep | 58 ++++++++++ >> lfs/rpz | 88 +++++++++++++++ >> make.sh | 2 + >> 8 files changed, 519 insertions(+) >> create mode 100644 config/backup/includes/rpz >> create mode 100644 config/rootfiles/packages/rpz >> create mode 100644 config/rpz/00-rpz.conf >> create mode 100644 config/rpz/rpz-config >> create mode 100644 config/rpz/rpz-metrics >> create mode 100644 config/rpz/rpz-sleep >> create mode 100644 lfs/rpz >> >> diff --git a/config/backup/includes/rpz b/config/backup/includes/rpz >> new file mode 100644 >> index 000000000..4d59bb40c >> --- /dev/null >> +++ b/config/backup/includes/rpz >> @@ -0,0 +1,5 @@ >> +/var/ipfire/rpz/allowlist >> +/var/ipfire/rpz/blocklist >> +/etc/unbound/zonefiles/allow.rpz >> +/etc/unbound/zonefiles/block.rpz >> +/etc/unbound/local.d/*rpz.conf >> diff --git a/config/rootfiles/packages/rpz b/config/rootfiles/packages/rpz >> new file mode 100644 >> index 000000000..2ffa715dd >> --- /dev/null >> +++ b/config/rootfiles/packages/rpz >> @@ -0,0 +1,11 @@ >> +etc/unbound/local.d/00-rpz.conf >> +etc/unbound/zonefiles >> +etc/unbound/zonefiles/allow.rpz >> +etc/unbound/zonefiles/block.rpz >> +usr/sbin/rpz-config >> +usr/sbin/rpz-metrics >> +usr/sbin/rpz-sleep >> +var/ipfire/backup/addons/includes/rpz >> +var/ipfire/rpz >> +var/ipfire/rpz/allowlist >> +var/ipfire/rpz/blocklist >> diff --git a/config/rpz/00-rpz.conf b/config/rpz/00-rpz.conf >> new file mode 100644 >> index 000000000..72c1d12e5 >> --- /dev/null >> +++ b/config/rpz/00-rpz.conf >> @@ -0,0 +1,18 @@ >> +server: >> + module-config: "respip validator iterator" >> + >> +rpz: >> + name: allow.rpz >> + zonefile: /etc/unbound/zonefiles/allow.rpz >> + rpz-action-override: passthru >> + rpz-log: yes >> + rpz-log-name: allow >> + rpz-signal-nxdomain-ra: yes >> + >> +rpz: >> + name: block.rpz >> + zonefile: /etc/unbound/zonefiles/block.rpz >> + rpz-action-override: nxdomain >> + rpz-log: yes >> + rpz-log-name: block >> + rpz-signal-nxdomain-ra: yes >> diff --git a/config/rpz/rpz-config b/config/rpz/rpz-config >> new file mode 100644 >> index 000000000..98dc0a4ca >> --- /dev/null >> +++ b/config/rpz/rpz-config >> @@ -0,0 +1,194 @@ >> +#!/bin/bash >> +############################################################################### >> +# # >> +# IPFire.org - A linux based firewall # >> +# Copyright (C) 2024 IPFire Team # >> +# # >> +# This program is free software: you can redistribute it and/or modify # >> +# it under the terms of the GNU General Public License as published by # >> +# the Free Software Foundation, either version 3 of the License, or # >> +# (at your option) any later version. # >> +# # >> +# This program is distributed in the hope that it will be useful, # >> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >> +# GNU General Public License for more details. # >> +# # >> +# You should have received a copy of the GNU General Public License # >> +# along with this program. If not, see . # >> +# # >> +############################################################################### >> + >> +# v22 - 2024-07-12 >> + >> +############### Functions ############### >> + >> +msg_log () { >> + /usr/bin/logger --tag "${tagName}" "$*" >> + if tty --silent ; then >> + echo "${tagName}:" "$*" >> + fi >> +} >> + >> +check_name () { >> + local testName="${1}" >> + # check for a valid name >> + regex='^[a-zA-Z0-9_]+$' >> + if [[ ! "${testName}" =~ $regex ]] ; then >> + msg_log "error: rpz: the NAME is not valid: \"${testName}\". exit." >> + exit 1 >> + fi >> +} >> + >> +check_unbound_conf () { >> + # check the above config files >> + msg_log "info: rpz: check for errors with \"unbound-checkconf\"" >> + /usr/sbin/unbound-checkconf >> + exit_code=$? >> + if [[ "${exit_code}" -ne 0 ]] ; then >> + msg_log "error: rpz: unbound-checkconf. exit." >> + exit "${exit_code}" >> + fi >> +} >> + >> +make_rpz_file () { >> + local theType="${1}" # allow or block >> + >> + theList="/var/ipfire/rpz/${theType}list" # input user list of domains >> + theZoneFile="/etc/unbound/zonefiles/${theType}.rpz" # output file for RPZ >> + >> + theAction='.' >> + if [[ "${theType}" =~ "block" ]] ; then >> + theAction='rpz-passthru.' >> + fi >> + >> + # does a list exist? >> + if [[ -s "${theList}" ]] ; then >> + # drop any extra "blanks" and add "CNAME ." to each line >> + actionList=$( /usr/bin/awk '{$1=$1};1' "${theList}" | >> + /bin/sed "/^[^;].*[[:alnum:]]/ s|$| CNAME ${theAction}|" ) >> + >> + msg_log "info: rpz: create zonefile for ${theList}" >> + >> + /bin/cat <<-EOF > "${theZoneFile}" >> + ; Name: ${theType} list >> + ; Last modified: $(date "+%Y-%m-%d at %H.%M.%S %Z") >> + ; >> + ; domains with actions list >> + ; >> + ${actionList} >> + EOF >> + >> + # reload the zone that was just updated >> + zoneBase=$( basename "${theZoneFile}" ) >> + /usr/sbin/unbound-control auth_zone_reload -q "${zoneBase}" >> + fi >> +} >> + >> +############### Main ############### >> + >> +tagName="unbound" >> + >> +theAction="${1}" # input action >> +theName="${2}" # input RPZ name >> +theURL="${3}" # input RPZ URL >> + >> +check_name "${theName}" # is this a valid name? >> + >> +rpzConfig="/etc/unbound/local.d/${theName}.rpz.conf" # output zone conf file >> +rpzFile="/etc/unbound/zonefiles/${theName}.rpz" # output for RPZ file >> + >> +case "${theAction}" in >> + >> + # add new rpz list >> + add ) >> + # does this config already exist? If yes, then exit >> + if [[ -f "${rpzConfig}" ]] ; then >> + msg_log "info: rpz: ${rpzConfig} already exists. exit" >> + exit 1 >> + fi >> + >> + # is this a valid URL? >> + regex='^https://[-[:alnum:]\+&@#/%?=~_|!:,.;]*[-[:alnum:]\+&@#/%=~_|]' >> + if ! [[ "${theURL}" =~ $regex ]] ; then >> + msg_log "error: rpz: the URL is not valid: \"${theURL}\". exit." >> + exit 1 >> + fi >> + >> + # create the zone config file >> + msg_log "info: rpz: add config file \"${theName}.rpz.conf\"" >> + cat <<-EOF > "${rpzConfig}" >> + rpz: >> + name: ${theName}.rpz >> + zonefile: /etc/unbound/zonefiles/${theName}.rpz >> + url: ${theURL} >> + rpz-action-override: nxdomain >> + rpz-log: yes >> + rpz-log-name: ${theName} >> + rpz-signal-nxdomain-ra: yes >> + EOF >> + >> + # set-up zone file >> + /usr/bin/touch "${rpzFile}" >> + # unbound requires these settings for rpz files >> + /bin/chown --verbose nobody:nobody "${rpzFile}" >> + /bin/chmod --verbose 644 "${rpzFile}" >> + ;; >> + >> + # trash config file & rpz file >> + remove ) >> + if ! [[ -f "${rpzConfig}" ]] ; then >> + msg_log "info: rpz: ${rpzConfig} does not exist. exit" >> + exit 1 >> + fi >> + >> + msg_log "info: rpz: remove config file & rpz file \"${theName}\"" >> + /bin/rm --verbose "${rpzConfig}" >> + /bin/rm --verbose "${rpzFile}" >> + >> + check_unbound_conf >> + ;; >> + >> + # make a new allow or block rpz file >> + make ) >> + case "${theName}" in >> + allow ) >> + make_rpz_file allow >> + ;; >> + >> + block ) >> + make_rpz_file block >> + ;; >> + >> + allowblock ) >> + make_rpz_file allow >> + make_rpz_file block >> + ;; >> + >> + * ) >> + msg_log "error: rpz: the NAME is not valid: \"${theName}\". exit." >> + exit 1 >> + ;; >> + esac >> + >> + check_unbound_conf >> + ;; >> + >> + *) >> + msg_log "error: rpz: missing or incorrect parameter" >> + /usr/bin/printf "Usage: rpzConfig.sh \n" >> + exit 1 >> + ;; >> + >> +esac >> + >> +# reload due to the changes >> +msg_log "rpz: running \"unbound-control reload\"" >> +/usr/sbin/unbound-control reload >> +exit_code=$? >> +if [[ "${exit_code}" -ne 0 ]] ; then >> + msg_log "error: rpz: unbound-control \"${theName}\". exit." >> + exit "${exit_code}" >> +fi >> + >> +exit >> diff --git a/config/rpz/rpz-metrics b/config/rpz/rpz-metrics >> new file mode 100644 >> index 000000000..0f97c7911 >> --- /dev/null >> +++ b/config/rpz/rpz-metrics >> @@ -0,0 +1,143 @@ >> +#!/bin/bash >> +############################################################################### >> +# # >> +# IPFire.org - A linux based firewall # >> +# Copyright (C) 2024 IPFire Team # >> +# # >> +# This program is free software: you can redistribute it and/or modify # >> +# it under the terms of the GNU General Public License as published by # >> +# the Free Software Foundation, either version 3 of the License, or # >> +# (at your option) any later version. # >> +# # >> +# This program is distributed in the hope that it will be useful, # >> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >> +# GNU General Public License for more details. # >> +# # >> +# You should have received a copy of the GNU General Public License # >> +# along with this program. If not, see . # >> +# # >> +############################################################################### >> + >> +# v18 on 2024-07-05 >> + >> +############### Main ############### >> + >> +weeks="${1:-2}" # default to two message logs >> +sortBy="${2:-name}" # by name or by hits >> + >> +# get the list of message logs for N weeks >> +messageLogs=$( find /var/log/messages* -type f | >> + /usr/bin/sort --version-sort | >> + head -"${weeks}" ) >> + >> +# get the list of RPZ names & counts from the message log(s) >> +rpzNameCount=$( for logf in ${messageLogs} ; do >> + /usr/bin/zgrep --text --fixed-strings 'info: rpz: applied' "${logf}" | >> + /usr/bin/awk '$10 ~ /\[\w*]/ { print $10 }' ; >> + done | /usr/bin/sort | /usr/bin/uniq --count ) >> + >> +# flip results and remove brackets `[` and `]` >> +rpzNameCount=$( /bin/echo "${rpzNameCount}" | >> + /usr/bin/awk '{ print $2, $1 }' | >> + /bin/sed --regexp-extended 's|^\[(.*)\]|\1|' ) >> + >> +# grab only names >> +rpzNames=$( /bin/echo "${rpzNameCount}" | /usr/bin/awk '{ print $1 }' ) >> + >> +# get list of RPZ files >> +rpzFileList=$( /bin/find /etc/unbound/zonefiles -type f -iname "*.rpz" ) >> + >> +# get basename of those files >> +rpzBaseNames=$( /bin/echo "${rpzFileList}" | >> + /bin/sed 's|/etc/unbound/zonefiles/||g ; s|\.rpz||g ;' ) >> + >> +# add to rpzNames >> +rpzNames="${rpzNames}"$'\n'"${rpzBaseNames}" >> + >> +# drop duplicate names >> +rpzNames=$( echo "${rpzNames}" | /usr/bin/sort --unique ) >> + >> +# get line count for each RPZ >> +lineCount=$( /bin/echo "${rpzFileList}" | /usr/bin/xargs wc -l ) >> + >> +# get comment line count and blank line count for each RPZ >> +commentCount=$( /bin/echo "${rpzFileList}" | >> + /usr/bin/xargs /bin/grep --count -e "^$" -e "^;" ) >> + >> +# get modified date each RPZ >> +modDateList=$( /bin/echo "${rpzFileList}" | xargs stat -c '%.10y %n' ) >> + >> +ucListAuthZones=$( /usr/sbin/unbound-control list_auth_zones ) >> + >> +# get width of RPZ names >> +pWidth=$( /bin/echo "${rpzNames}" | /usr/bin/awk '{ print $1" " }' | wc -L ) >> +pFormat="%-${pWidth}s %-8s %-8s %-8s %10s %12s\n" >> + >> +# print title line >> +printf "${pFormat}" "name" "hits" "active" "lines" "hits/line%" "last update" >> +printf -- "--------------" >> + >> +theResults="" >> +totalLines=0 >> +totalHits=0 >> +while read -r theName >> +do >> + printf -- "-" # pretend progress bar >> + # get hit count >> + theHits="0" >> + if output=$( /bin/grep "^${theName}\s" <<< "${rpzNameCount}" ) ; then >> + theHits=$( /bin/echo "${output}" | >> + /usr/bin/awk '{ print $2 }' ) >> + totalHits=$(( totalHits + theHits )) >> + fi >> + >> + # is this RPZ list active? >> + theActive="disabled" >> + if /bin/grep --quiet "^${theName}\.rpz" <<< "${ucListAuthZones}" >> + then >> + theActive="enabled" >> + fi >> + >> + # get line count then subtract comment count and blank line count >> + # from total line count >> + theLines="n/a" >> + hitsPerLine="0" >> + if output=$( /bin/grep --fixed-strings "/${theName}.rpz" <<< "${lineCount}" ) ; then >> + theLines=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' ) >> + totalLines=$(( totalLines + theLines )) >> + >> + #hitsPerLine=$( echo "scale=0 ; $theHits / $theLines" | bc ) >> + hitsPerLine=$(( 100 * theHits / theLines )) >> + fi >> + >> + # get modification date >> + theModDate="n/a" >> + if output=$( /bin/grep --fixed-strings "/${theName}.rpz" <<< "${modDateList}" ) ; then >> + theModDate=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' ) >> + fi >> + >> + # add to results list >> + theResults+="${theName} ${theHits} ${theActive} ${theLines} ${hitsPerLine} ${theModDate}"$'\n' >> +done <<< "${rpzNames}" >> + >> +case "${sortBy}" in >> + names|name) sortArg=(-k3,3r -k1,1) ;; # sort by "active" then by "name" >> + >> + hits|hit) sortArg=(-k3,3r -k2,2nr -k1,1) ;; # sort by "active" then by "hits" then by "name" >> + >> + lines|line) sortArg=(-k3,3r -k4,4nr -k1,1) ;; # sort by "active" then by "lines" then by "name" >> +esac >> + >> +printf -- "--------------\n" >> +# remove blank lines, sort, print as columns >> +/bin/echo "${theResults}" | >> + /usr/bin/awk '!/^[[:space:]]*$/' | >> + /usr/bin/sort "${sortArg[@]}" | >> + /usr/bin/awk --assign=width="${pWidth}" \ >> + '{ printf "%-*s %-8s %-8s %-8s %10s %12s\n", width, $1, $2, $3, $4, $5, $6 }' >> + >> +printf "${pFormat}" "" "=======" "" "========" "" "" >> +printf "${pFormat}" "Totals -->" "${totalHits}" "" "${totalLines}" "" "" >> + >> +exit >> diff --git a/config/rpz/rpz-sleep b/config/rpz/rpz-sleep >> new file mode 100644 >> index 000000000..eeef1174a >> --- /dev/null >> +++ b/config/rpz/rpz-sleep >> @@ -0,0 +1,58 @@ >> +#!/bin/bash >> +############################################################################### >> +# # >> +# IPFire.org - A linux based firewall # >> +# Copyright (C) 2024 IPFire Team # >> +# # >> +# This program is free software: you can redistribute it and/or modify # >> +# it under the terms of the GNU General Public License as published by # >> +# the Free Software Foundation, either version 3 of the License, or # >> +# (at your option) any later version. # >> +# # >> +# This program is distributed in the hope that it will be useful, # >> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >> +# GNU General Public License for more details. # >> +# # >> +# You should have received a copy of the GNU General Public License # >> +# along with this program. If not, see . # >> +# # >> +############################################################################### >> + >> +# v04 on 2024-07-05 >> + >> +############### Functions ############### >> + >> +# send message to message log >> +msg_log () { >> + /usr/bin/logger --tag "${tagName}" "$*" >> + if /usr/bin/tty --silent ; then >> + echo "${tagName}:" "$*" >> + fi >> +} >> + >> +############### Main ############### >> + >> +tagName="unbound" >> + >> +sleepTime="${1:-5m}" # default to sleep for 5m (5 minutes) >> + >> +zoneList=$( /usr/sbin/unbound-control list_auth_zones | /usr/bin/awk '{print $1}' ) >> + >> +for zone in ${zoneList} ; do >> + /usr/bin/printf "disable ${zone}\t" >> + /usr/sbin/unbound-control rpz_disable "${zone}" >> +done >> + >> +msg_log "info: rpz: disabled all zones for ${sleepTime}" >> + >> +/bin/sleep "${sleepTime}" >> + >> +for zone in ${zoneList} ; do >> + /usr/bin/printf "enable ${zone}\t" >> + /usr/sbin/unbound-control rpz_enable "${zone}" >> +done >> + >> +msg_log "info: rpz: enabled all zones" >> + >> +exit >> diff --git a/lfs/rpz b/lfs/rpz >> new file mode 100644 >> index 000000000..319c10b7f >> --- /dev/null >> +++ b/lfs/rpz >> @@ -0,0 +1,88 @@ >> +############################################################################### >> +# # >> +# IPFire.org - A linux based firewall # >> +# Copyright (C) 2024 IPFire Team # >> +# # >> +# This program is free software: you can redistribute it and/or modify # >> +# it under the terms of the GNU General Public License as published by # >> +# the Free Software Foundation, either version 3 of the License, or # >> +# (at your option) any later version. # >> +# # >> +# This program is distributed in the hope that it will be useful, # >> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >> +# GNU General Public License for more details. # >> +# # >> +# You should have received a copy of the GNU General Public License # >> +# along with this program. If not, see . # >> +# # >> +############################################################################### >> + >> +############################################################################### >> +# Definitions >> +############################################################################### >> + >> +include Config >> + >> +SUMMARY = response policy zone - RPZ reputation system for unbound DNS >> + >> +VER = 1.0.0 >> + >> +THISAPP = rpz-$(VER) >> +DIR_APP = $(DIR_SRC)/$(THISAPP) >> +TARGET = $(DIR_INFO)/$(THISAPP) >> + >> +PROG = rpz >> +PAK_VER = 1 >> + >> +DEPS = >> + >> +SERVICES = >> + >> +############################################################################### >> +# Top-level Rules >> +############################################################################### >> + >> +install : $(TARGET) >> + >> +check : >> + >> +download : >> + >> +b2 : >> + >> +dist: >> + @$(PAK) >> + >> +############################################################################### >> +# Installation Details >> +############################################################################### >> + >> +$(TARGET) : >> + @$(PREBUILD) >> + @rm -rf $(DIR_APP) >> + >> + # install RPZ scripts >> + install -v -m 755 \ >> + $(DIR_CONF)/rpz/{rpz-config,rpz-metrics,rpz-sleep} -t /usr/sbin >> + >> + # Install settings folder and two empty files >> + mkdir -pv /var/ipfire/rpz >> + touch /var/ipfire/rpz/allowlist >> + touch /var/ipfire/rpz/blocklist >> + >> + # Add conf file to /etc directory >> + cp -vf $(DIR_CONF)/rpz/00-rpz.conf /etc/unbound/local.d >> + >> + # create zonefiles directory for the RPZ files and add two empty RPZ >> + # files to avoid a unbound config error >> + mkdir -pv /etc/unbound/zonefiles >> + chown -v nobody:nobody /etc/unbound/zonefiles >> + touch /etc/unbound/zonefiles/allow.rpz >> + touch /etc/unbound/zonefiles/block.rpz >> + >> + # Install backup definition >> + cp -vf $(DIR_CONF)/backup/includes/rpz /var/ipfire/backup/addons/includes/rpz >> + >> + @rm -rf $(DIR_APP) >> + @$(POSTBUILD) >> diff --git a/make.sh b/make.sh >> index 9bbbeb0f1..886d3760a 100755 >> --- a/make.sh >> +++ b/make.sh >> @@ -1721,6 +1721,8 @@ buildipfire() { >> lfsmake2 btrfs-progs >> lfsmake2 inotify-tools >> lfsmake2 grub-btrfs >> + lfsmake2 rpz >> + >> >> # Kernelbuild ... current we have no platform that need >> # multi kernel builds so KCFG is empty >> -- >> 2.30.2 >> >