From mboxrd@z Thu Jan 1 00:00:00 1970 From: jon To: development@lists.ipfire.org Subject: Re: [PATCH] RPZ: install new add-on Date: Sat, 20 Jul 2024 13:42:53 -0500 Message-ID: <42099D9F-1D6E-4C2A-B1BF-F21D036369AD@ipfire.org> In-Reply-To: <20240720174639.1419870-1-jon.murphy@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============7928514985191039943==" List-Id: --===============7928514985191039943== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable I need help with=E2=80=A6 1) The custom allow and block lists are currently located at `/var/ipfire/rpz= `. Is this correct? 2) The three bash scripts are currently located at `/usr/sbin`. Is this corr= ect? 3) Should this commit be separated into multiple posts? If yes, how do I dec= ide how to split it? Jon=20 > On Jul 20, 2024, at 12:46=E2=80=AFPM, Jon Murphy = wrote: >=20 > What is it? > Response Policy Zone (RPZ) is a mechanism to define local policies in a sta= ndardised > way and load those policies from external sources. > Bottom line: RPZ allows admins to easily block access to websites via DNS l= ookup. >=20 > RPZ can block websites via categories. Examples include: fake websites, an= noying > pop-up ads, newly registered domains, DoH bypass sites, bad "host" services, > maliscious top level domains (e.g., *.zip, *.mov), piracy, gambling, pornog= raphy, > and more. RPZ lists come from various RPZ providers and their available > catagories. >=20 > This RPZ add-on enables the RPZ functionality by adding a couple lines in a > configuration file. This add-on simply adds a configuration file and adds > three scripts (config, metrics and sleep) to make RPZ easier for the admin = to use. >=20 > RPZ was release in 2010 and has been part of the IPFire build since ~2015. >=20 > Why is it needed? > Some IPFire admin's utilize pihole to block unwanted websites via DNS looku= p. > Moving the pihole base functionality (without pretty graphs) to IPFire remo= ves > one device from the admin's local network. > And hopefully this reduces the pihole questions from the Community. >=20 > A list of RPZ providers can be recommended by IPFire and coded into a set l= ist. > Or, if prefered, the local admin can choose their own RPZ providers. >=20 > This is a: > * simple replacement for pihole base functionality > * RPZ can be a nice replacement for the URL Filter >=20 > IPFire Wiki > In process at: https://www.ipfire.org/docs/addons/rpz >=20 > more info: > * https://en.wikipedia.org/wiki/Response_policy_zone > * https://unbound.docs.nlnetlabs.nl/en/latest/topics/filtering/rpz.html >=20 > I need help with... > 1) The custom allow and block lists are currently located at `/var/ipfire/r= pz`. Is this correct? > 2) The three bash scripts are currently located at `/usr/sbin`. Is this co= rrect? >=20 > Signed-off-by: Jon Murphy > --- > config/backup/includes/rpz | 5 + > config/rootfiles/packages/rpz | 11 ++ > config/rpz/00-rpz.conf | 18 ++++ > config/rpz/rpz-config | 194 ++++++++++++++++++++++++++++++++++ > config/rpz/rpz-metrics | 143 +++++++++++++++++++++++++ > config/rpz/rpz-sleep | 58 ++++++++++ > lfs/rpz | 88 +++++++++++++++ > make.sh | 2 + > 8 files changed, 519 insertions(+) > create mode 100644 config/backup/includes/rpz > create mode 100644 config/rootfiles/packages/rpz > create mode 100644 config/rpz/00-rpz.conf > create mode 100644 config/rpz/rpz-config > create mode 100644 config/rpz/rpz-metrics > create mode 100644 config/rpz/rpz-sleep > create mode 100644 lfs/rpz >=20 > diff --git a/config/backup/includes/rpz b/config/backup/includes/rpz > new file mode 100644 > index 000000000..4d59bb40c > --- /dev/null > +++ b/config/backup/includes/rpz > @@ -0,0 +1,5 @@ > +/var/ipfire/rpz/allowlist > +/var/ipfire/rpz/blocklist > +/etc/unbound/zonefiles/allow.rpz > +/etc/unbound/zonefiles/block.rpz > +/etc/unbound/local.d/*rpz.conf > diff --git a/config/rootfiles/packages/rpz b/config/rootfiles/packages/rpz > new file mode 100644 > index 000000000..2ffa715dd > --- /dev/null > +++ b/config/rootfiles/packages/rpz > @@ -0,0 +1,11 @@ > +etc/unbound/local.d/00-rpz.conf > +etc/unbound/zonefiles > +etc/unbound/zonefiles/allow.rpz > +etc/unbound/zonefiles/block.rpz > +usr/sbin/rpz-config > +usr/sbin/rpz-metrics > +usr/sbin/rpz-sleep > +var/ipfire/backup/addons/includes/rpz > +var/ipfire/rpz > +var/ipfire/rpz/allowlist > +var/ipfire/rpz/blocklist > diff --git a/config/rpz/00-rpz.conf b/config/rpz/00-rpz.conf > new file mode 100644 > index 000000000..72c1d12e5 > --- /dev/null > +++ b/config/rpz/00-rpz.conf > @@ -0,0 +1,18 @@ > +server: > + module-config: "respip validator iterator" > + > +rpz: > + name: allow.rpz > + zonefile: /etc/unbound/zonefiles/allow.rpz > + rpz-action-override: passthru > + rpz-log: yes > + rpz-log-name: allow > + rpz-signal-nxdomain-ra: yes > + > +rpz: > + name: block.rpz > + zonefile: /etc/unbound/zonefiles/block.rpz > + rpz-action-override: nxdomain > + rpz-log: yes > + rpz-log-name: block > + rpz-signal-nxdomain-ra: yes > diff --git a/config/rpz/rpz-config b/config/rpz/rpz-config > new file mode 100644 > index 000000000..98dc0a4ca > --- /dev/null > +++ b/config/rpz/rpz-config > @@ -0,0 +1,194 @@ > +#!/bin/bash > +##########################################################################= ##### > +# = # > +# IPFire.org - A linux based firewall = # > +# Copyright (C) 2024 IPFire Team = # > +# = # > +# This program is free software: you can redistribute it and/or modify = # > +# it under the terms of the GNU General Public License as published by = # > +# the Free Software Foundation, either version 3 of the License, or = # > +# (at your option) any later version. = # > +# = # > +# This program is distributed in the hope that it will be useful, = # > +# but WITHOUT ANY WARRANTY; without even the implied warranty of = # > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the = # > +# GNU General Public License for more details. = # > +# = # > +# You should have received a copy of the GNU General Public License = # > +# along with this program. If not, see . = # > +# = # > +##########################################################################= ##### > + > +# v22 - 2024-07-12 > + > +############### Functions ############### > + > +msg_log () { > + /usr/bin/logger --tag "${tagName}" "$*" > + if tty --silent ; then > + echo "${tagName}:" "$*" > + fi > +} > + > +check_name () { > + local testName=3D"${1}" > + # check for a valid name > + regex=3D'^[a-zA-Z0-9_]+$' > + if [[ ! "${testName}" =3D~ $regex ]] ; then > + msg_log "error: rpz: the NAME is not valid: \"${testName}\". exit." > + exit 1 > + fi > +} > + > +check_unbound_conf () { > + # check the above config files > + msg_log "info: rpz: check for errors with \"unbound-checkconf\"" > + /usr/sbin/unbound-checkconf > + exit_code=3D$? > + if [[ "${exit_code}" -ne 0 ]] ; then > + msg_log "error: rpz: unbound-checkconf. exit." > + exit "${exit_code}" > + fi > +} > + > +make_rpz_file () { > + local theType=3D"${1}" # allow or block > + > + theList=3D"/var/ipfire/rpz/${theType}list" # input user list of domains > + theZoneFile=3D"/etc/unbound/zonefiles/${theType}.rpz" # output file for R= PZ > + > + theAction=3D'.' > + if [[ "${theType}" =3D~ "block" ]] ; then > + theAction=3D'rpz-passthru.' > + fi > + > + # does a list exist? > + if [[ -s "${theList}" ]] ; then > + # drop any extra "blanks" and add "CNAME ." to each line > + actionList=3D$( /usr/bin/awk '{$1=3D$1};1' "${theList}" | > + /bin/sed "/^[^;].*[[:alnum:]]/ s|$| CNAME ${theAction}|" ) > + > + msg_log "info: rpz: create zonefile for ${theList}" > + > + /bin/cat <<-EOF > "${theZoneFile}" > + ; Name: ${theType} list > + ; Last modified: $(date "+%Y-%m-%d at %H.%M.%S %Z") > + ; > + ; domains with actions list > + ; > + ${actionList} > + EOF > + > + # reload the zone that was just updated > + zoneBase=3D$( basename "${theZoneFile}" ) > + /usr/sbin/unbound-control auth_zone_reload -q "${zoneBase}" > + fi > +} > + > +############### Main ############### > + > +tagName=3D"unbound" > + > +theAction=3D"${1}" # input action > +theName=3D"${2}" # input RPZ name > +theURL=3D"${3}" # input RPZ URL > + > +check_name "${theName}" # is this a valid name? > + > +rpzConfig=3D"/etc/unbound/local.d/${theName}.rpz.conf" # output zone conf = file > +rpzFile=3D"/etc/unbound/zonefiles/${theName}.rpz" # output for RPZ file > + > +case "${theAction}" in > + > + # add new rpz list > + add ) > + # does this config already exist? If yes, then exit > + if [[ -f "${rpzConfig}" ]] ; then > + msg_log "info: rpz: ${rpzConfig} already exists. exit" > + exit 1 > + fi > + > + # is this a valid URL? > + regex=3D'^https://[-[:alnum:]\+&@#/%?=3D~_|!:,.;]*[-[:alnum:]\+&@#/%=3D~_= |]' > + if ! [[ "${theURL}" =3D~ $regex ]] ; then > + msg_log "error: rpz: the URL is not valid: \"${theURL}\". exit." > + exit 1 > + fi > + > + # create the zone config file > + msg_log "info: rpz: add config file \"${theName}.rpz.conf\"" > + cat <<-EOF > "${rpzConfig}" > + rpz: > + name: ${theName}.rpz > + zonefile: /etc/unbound/zonefiles/${theName}.rpz > + url: ${theURL} > + rpz-action-override: nxdomain > + rpz-log: yes > + rpz-log-name: ${theName} > + rpz-signal-nxdomain-ra: yes > + EOF > + > + # set-up zone file > + /usr/bin/touch "${rpzFile}" > + # unbound requires these settings for rpz files > + /bin/chown --verbose nobody:nobody "${rpzFile}" > + /bin/chmod --verbose 644 "${rpzFile}" > + ;; > + > + # trash config file & rpz file > + remove ) > + if ! [[ -f "${rpzConfig}" ]] ; then > + msg_log "info: rpz: ${rpzConfig} does not exist. exit" > + exit 1 > + fi > + > + msg_log "info: rpz: remove config file & rpz file \"${theName}\"" > + /bin/rm --verbose "${rpzConfig}" > + /bin/rm --verbose "${rpzFile}" > + > + check_unbound_conf > + ;; > + > + # make a new allow or block rpz file > + make ) > + case "${theName}" in > + allow ) > + make_rpz_file allow > + ;; > + > + block ) > + make_rpz_file block > + ;; > + > + allowblock ) > + make_rpz_file allow > + make_rpz_file block > + ;; > + > + * ) > + msg_log "error: rpz: the NAME is not valid: \"${theName}\". exit." > + exit 1 > + ;; > + esac > + > + check_unbound_conf > + ;; > + > + *) > + msg_log "error: rpz: missing or incorrect parameter" > + /usr/bin/printf "Usage: rpzConfig.sh \n" > + exit 1 > + ;; > + > +esac > + > +# reload due to the changes > +msg_log "rpz: running \"unbound-control reload\"" > +/usr/sbin/unbound-control reload > +exit_code=3D$? > +if [[ "${exit_code}" -ne 0 ]] ; then > + msg_log "error: rpz: unbound-control \"${theName}\". exit." > + exit "${exit_code}" > +fi > + > +exit > diff --git a/config/rpz/rpz-metrics b/config/rpz/rpz-metrics > new file mode 100644 > index 000000000..0f97c7911 > --- /dev/null > +++ b/config/rpz/rpz-metrics > @@ -0,0 +1,143 @@ > +#!/bin/bash > +##########################################################################= ##### > +# = # > +# IPFire.org - A linux based firewall = # > +# Copyright (C) 2024 IPFire Team = # > +# = # > +# This program is free software: you can redistribute it and/or modify = # > +# it under the terms of the GNU General Public License as published by = # > +# the Free Software Foundation, either version 3 of the License, or = # > +# (at your option) any later version. = # > +# = # > +# This program is distributed in the hope that it will be useful, = # > +# but WITHOUT ANY WARRANTY; without even the implied warranty of = # > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the = # > +# GNU General Public License for more details. = # > +# = # > +# You should have received a copy of the GNU General Public License = # > +# along with this program. If not, see . = # > +# = # > +##########################################################################= ##### > + > +# v18 on 2024-07-05 > + > +############### Main ############### > + > +weeks=3D"${1:-2}" # default to two message logs > +sortBy=3D"${2:-name}" # by name or by hits > + > +# get the list of message logs for N weeks > +messageLogs=3D$( find /var/log/messages* -type f | > + /usr/bin/sort --version-sort | > + head -"${weeks}" ) > + > +# get the list of RPZ names & counts from the message log(s) > +rpzNameCount=3D$( for logf in ${messageLogs} ; do > + /usr/bin/zgrep --text --fixed-strings 'info: rpz: applied' "${logf}" | > + /usr/bin/awk '$10 ~ /\[\w*]/ { print $10 }' ; > + done | /usr/bin/sort | /usr/bin/uniq --count ) > + > +# flip results and remove brackets `[` and `]` > +rpzNameCount=3D$( /bin/echo "${rpzNameCount}" | > + /usr/bin/awk '{ print $2, $1 }' | > + /bin/sed --regexp-extended 's|^\[(.*)\]|\1|' ) > + > +# grab only names > +rpzNames=3D$( /bin/echo "${rpzNameCount}" | /usr/bin/awk '{ print $1 }' ) > + > +# get list of RPZ files > +rpzFileList=3D$( /bin/find /etc/unbound/zonefiles -type f -iname "*.rpz" ) > + > +# get basename of those files > +rpzBaseNames=3D$( /bin/echo "${rpzFileList}" | > + /bin/sed 's|/etc/unbound/zonefiles/||g ; s|\.rpz||g ;' ) > + > +# add to rpzNames > +rpzNames=3D"${rpzNames}"$'\n'"${rpzBaseNames}" > + > +# drop duplicate names > +rpzNames=3D$( echo "${rpzNames}" | /usr/bin/sort --unique ) > + > +# get line count for each RPZ > +lineCount=3D$( /bin/echo "${rpzFileList}" | /usr/bin/xargs wc -l ) > + > +# get comment line count and blank line count for each RPZ > +commentCount=3D$( /bin/echo "${rpzFileList}" | > + /usr/bin/xargs /bin/grep --count -e "^$" -e "^;" ) > + > +# get modified date each RPZ > +modDateList=3D$( /bin/echo "${rpzFileList}" | xargs stat -c '%.10y %n' ) > + > +ucListAuthZones=3D$( /usr/sbin/unbound-control list_auth_zones ) > + > +# get width of RPZ names > +pWidth=3D$( /bin/echo "${rpzNames}" | /usr/bin/awk '{ print $1" " }' | w= c -L ) > +pFormat=3D"%-${pWidth}s %-8s %-8s %-8s %10s %12s\n" > + > +# print title line > +printf "${pFormat}" "name" "hits" "active" "lines" "hits/line%" "last upda= te" > +printf -- "--------------" > + > +theResults=3D"" > +totalLines=3D0 > +totalHits=3D0 > +while read -r theName > +do > + printf -- "-" # pretend progress bar > + # get hit count > + theHits=3D"0" > + if output=3D$( /bin/grep "^${theName}\s" <<< "${rpzNameCount}" ) ; then > + theHits=3D$( /bin/echo "${output}" | > + /usr/bin/awk '{ print $2 }' ) > + totalHits=3D$(( totalHits + theHits )) > + fi > + > + # is this RPZ list active? > + theActive=3D"disabled" > + if /bin/grep --quiet "^${theName}\.rpz" <<< "${ucListAuthZones}" > + then > + theActive=3D"enabled" > + fi > + > + # get line count then subtract comment count and blank line count > + # from total line count > + theLines=3D"n/a" > + hitsPerLine=3D"0" > + if output=3D$( /bin/grep --fixed-strings "/${theName}.rpz" <<< "${lineCou= nt}" ) ; then > + theLines=3D$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' ) > + totalLines=3D$(( totalLines + theLines )) > + > + #hitsPerLine=3D$( echo "scale=3D0 ; $theHits / $theLines" | bc ) > + hitsPerLine=3D$(( 100 * theHits / theLines )) > + fi > + > + # get modification date > + theModDate=3D"n/a" > + if output=3D$( /bin/grep --fixed-strings "/${theName}.rpz" <<< "${modDate= List}" ) ; then > + theModDate=3D$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' ) > + fi > + > + # add to results list > + theResults+=3D"${theName} ${theHits} ${theActive} ${theLines} ${hitsPerLi= ne} ${theModDate}"$'\n' > +done <<< "${rpzNames}" > + > +case "${sortBy}" in > + names|name) sortArg=3D(-k3,3r -k1,1) ;; # sort by "active" then by "name" > + > + hits|hit) sortArg=3D(-k3,3r -k2,2nr -k1,1) ;; # sort by "active" then by = "hits" then by "name" > + > + lines|line) sortArg=3D(-k3,3r -k4,4nr -k1,1) ;; # sort by "active" then b= y "lines" then by "name" > +esac > + > +printf -- "--------------\n" > +# remove blank lines, sort, print as columns > +/bin/echo "${theResults}" | > + /usr/bin/awk '!/^[[:space:]]*$/' | > + /usr/bin/sort "${sortArg[@]}" | > + /usr/bin/awk --assign=3Dwidth=3D"${pWidth}" \ > + '{ printf "%-*s %-8s %-8s %-8s %10s %12s\n", width, $1, $2, $3, $4, $5, $= 6 }' > + > +printf "${pFormat}" "" "=3D=3D=3D=3D=3D=3D=3D" "" "=3D=3D=3D=3D=3D=3D=3D= =3D" "" "" > +printf "${pFormat}" "Totals -->" "${totalHits}" "" "${totalLines}" "" "" > + > +exit > diff --git a/config/rpz/rpz-sleep b/config/rpz/rpz-sleep > new file mode 100644 > index 000000000..eeef1174a > --- /dev/null > +++ b/config/rpz/rpz-sleep > @@ -0,0 +1,58 @@ > +#!/bin/bash > +##########################################################################= ##### > +# = # > +# IPFire.org - A linux based firewall = # > +# Copyright (C) 2024 IPFire Team = # > +# = # > +# This program is free software: you can redistribute it and/or modify = # > +# it under the terms of the GNU General Public License as published by = # > +# the Free Software Foundation, either version 3 of the License, or = # > +# (at your option) any later version. = # > +# = # > +# This program is distributed in the hope that it will be useful, = # > +# but WITHOUT ANY WARRANTY; without even the implied warranty of = # > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the = # > +# GNU General Public License for more details. = # > +# = # > +# You should have received a copy of the GNU General Public License = # > +# along with this program. If not, see . = # > +# = # > +##########################################################################= ##### > + > +# v04 on 2024-07-05 > + > +############### Functions ############### > + > +# send message to message log > +msg_log () { > + /usr/bin/logger --tag "${tagName}" "$*" > + if /usr/bin/tty --silent ; then > + echo "${tagName}:" "$*" > + fi > +} > + > +############### Main ############### > + > +tagName=3D"unbound" > + > +sleepTime=3D"${1:-5m}" # default to sleep for 5m (5 minutes) > + > +zoneList=3D$( /usr/sbin/unbound-control list_auth_zones | /usr/bin/awk '{p= rint $1}' ) > + > +for zone in ${zoneList} ; do > + /usr/bin/printf "disable ${zone}\t" > + /usr/sbin/unbound-control rpz_disable "${zone}" > +done > + > +msg_log "info: rpz: disabled all zones for ${sleepTime}" > + > +/bin/sleep "${sleepTime}" > + > +for zone in ${zoneList} ; do > + /usr/bin/printf "enable ${zone}\t" > + /usr/sbin/unbound-control rpz_enable "${zone}" > +done > + > +msg_log "info: rpz: enabled all zones" > + > +exit > diff --git a/lfs/rpz b/lfs/rpz > new file mode 100644 > index 000000000..319c10b7f > --- /dev/null > +++ b/lfs/rpz > @@ -0,0 +1,88 @@ > +##########################################################################= ##### > +# = # > +# IPFire.org - A linux based firewall = # > +# Copyright (C) 2024 IPFire Team = # > +# = # > +# This program is free software: you can redistribute it and/or modify = # > +# it under the terms of the GNU General Public License as published by = # > +# the Free Software Foundation, either version 3 of the License, or = # > +# (at your option) any later version. = # > +# = # > +# This program is distributed in the hope that it will be useful, = # > +# but WITHOUT ANY WARRANTY; without even the implied warranty of = # > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the = # > +# GNU General Public License for more details. = # > +# = # > +# You should have received a copy of the GNU General Public License = # > +# along with this program. If not, see . = # > +# = # > +##########################################################################= ##### > + > +##########################################################################= ##### > +# Definitions > +##########################################################################= ##### > + > +include Config > + > +SUMMARY =3D response policy zone - RPZ reputation system for unbound DNS > + > +VER =3D 1.0.0 > + > +THISAPP =3D rpz-$(VER) > +DIR_APP =3D $(DIR_SRC)/$(THISAPP) > +TARGET =3D $(DIR_INFO)/$(THISAPP) > + > +PROG =3D rpz > +PAK_VER =3D 1 > + > +DEPS =3D > + > +SERVICES =3D > + > +##########################################################################= ##### > +# Top-level Rules > +##########################################################################= ##### > + > +install : $(TARGET) > + > +check : > + > +download : > + > +b2 : > + > +dist: > + @$(PAK) > + > +##########################################################################= ##### > +# Installation Details > +##########################################################################= ##### > + > +$(TARGET) : > + @$(PREBUILD) > + @rm -rf $(DIR_APP) > + > + # install RPZ scripts > + install -v -m 755 \ > + $(DIR_CONF)/rpz/{rpz-config,rpz-metrics,rpz-sleep} -t /usr/sbin > + > + # Install settings folder and two empty files > + mkdir -pv /var/ipfire/rpz > + touch /var/ipfire/rpz/allowlist > + touch /var/ipfire/rpz/blocklist > + > + # Add conf file to /etc directory > + cp -vf $(DIR_CONF)/rpz/00-rpz.conf /etc/unbound/local.d > + > + # create zonefiles directory for the RPZ files and add two empty RPZ > + # files to avoid a unbound config error > + mkdir -pv /etc/unbound/zonefiles > + chown -v nobody:nobody /etc/unbound/zonefiles > + touch /etc/unbound/zonefiles/allow.rpz > + touch /etc/unbound/zonefiles/block.rpz > + > + # Install backup definition > + cp -vf $(DIR_CONF)/backup/includes/rpz /var/ipfire/backup/addons/includes= /rpz > + > + @rm -rf $(DIR_APP) > + @$(POSTBUILD) > diff --git a/make.sh b/make.sh > index 9bbbeb0f1..886d3760a 100755 > --- a/make.sh > +++ b/make.sh > @@ -1721,6 +1721,8 @@ buildipfire() { > lfsmake2 btrfs-progs > lfsmake2 inotify-tools > lfsmake2 grub-btrfs > + lfsmake2 rpz > + >=20 > # Kernelbuild ... current we have no platform that need > # multi kernel builds so KCFG is empty > --=20 > 2.30.2 >=20 --===============7928514985191039943==--