From mboxrd@z Thu Jan 1 00:00:00 1970 From: Adolf Belka To: development@lists.ipfire.org Subject: Re: [PATCH] RPZ: install new add-on Date: Tue, 30 Jul 2024 12:05:44 +0200 Message-ID: <2e0bceb3-3d40-41c5-b96e-d4f06329638b@ipfire.org> In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2794457153386085151==" List-Id: --===============2794457153386085151== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi Jon, I have given some feedback in your patches below. On 29/07/2024 04:28, jon wrote: > See comments below... > >> On Jul 28, 2024, at 6:22=E2=80=AFAM, Michael Tremer wrote: >> >> Hello Jon, >> >> I don=E2=80=99t think it is a good idea to send over the same patch set mu= ltiple times. >> >> I think first of all, we need to start at the beginning which is to figure= out what this actually is, and why we would want this in IPFire. >> >> I have seen that there have been conversations on community.ipfire.org many months ago, but I did not take many things a= way from it. It was from my point of view mostly a prove of concept discussio= n. Correct me if I have missed anything here. >> >> We have talked about RPZ many times on the monthly call since the URL filt= er feature is falling more and more out of fashion. I think there is also man= y posts about this on the forum. > Yes, I have seen the posts about the url filter feature falling out of fash= ion. I did not see notes about RPZ from the monthly calls. I will go back a= nd search telco call notes. > > >> Apart from generally being in favour of trying this path, we decided to po= stpone any further works into this because we currently have a huge backlog o= f things to deal with that are moving at snail speed. This is rooted in so m= any people being simply tied in with other things. >> >> As you can see, the patch set that you posted did not trigger a response -= a common thing on this list recently and not because of this particular patc= h set. >> >> Therefore I would like to avoid opening another can of worms before we hav= e delivered the other things that have already lots of time spent on them and= code written. Otherwise we are only getting a longer pipeline and this all i= s going to be more frustrating for everyone involved. And this is not about t= he feature per se; this is about getting things done. > This is why I wrote the RPZ patch instead of asking for the Developers to w= rite it. I realize there is a very long pipeline. > > >> So what I suggest doing is the following: >> >> You have a Git repository, push the code there and commit any fixes to it = over time so that nothing is getting lost. > This was done a few days ago. But I am not sure if I did it correctly. If= not, please let me know. > > >> Explain to me from the beginning, please, how it came to this patch set. W= hat is the motivation? > In the beginning (Fall of 2023) the main motivation was blocking DoH. I ha= d noticed apps using their own method of DNS lookup. Some were friendly DoH = lookups (Apple iCloud?) and some not (mozilla.cloudflare-dns.com but not from= Firefox). Examples I saw are: > mozilla.cloudflare-dns.com > www.switch.ch > use-application-dns.net (a canary for Fir= efox?) > And a few others... > > This worked better than expected and was easier to configure than expected.= All of the main code was part of unbound and I just need to activate it wit= h a config file. Updates to the RPZ files are already part of the unbound RP= Z code. > > As I experimented with RPZ I quickly realized it could easily be used to bl= ock other websites. I have been using pihole for a number of years to limit = the number of ads and trackers. And RPZ helped with those areas also. > > >> What problems does it solve? What problems remain? Basically: Why would an= yone want this in IPFire? > It solved the issue of blocking unwanted DoH from apps. And it solved the = issue of unwanted ads and tracking. For me those are huge benefits and the b= iggest reason for using RPZ. > > To test other possibilities, I increased the "type" of lists to include oth= er categories other than ads & trackers. Like blocking bad TLDs, gambling, p= iracy, bad host sites, etc. This is a nice bonus of using RPZ for me. > >> Where do you see application for this? > As a replacement for the URL Filter. And as a base functional replacement = for pihole (but no pretty graphics). > > >> Why is this realised as an add-on and not part of the core system? > In my opinion it should be part of the core. But I wanted to "test the wat= ers" first to determine interest. If only four people use it, then it isn=E2= =80=99t worth adding to core and maybe not as an add-on. > > Either way I will support it because it helps me! > > Here are my metrics for the past eight days (searching `messages` and `mess= ages.1.gz`): > > [root(a)ipfire ~] # rpz-metrics > name hits > ---------------------------- > allow 5560 This is a custom list > AmazonTrkrHZ 5726 > AppleTrkrHZ 2956 > block 128 This is a custom list > DOHblockHZ 25077 > dohJPG 3448 > HosterHZ 19 > MxProPlusHZ 17287 > NotSafeSearchHZ 0 > ProxyBypassHZ 11 > tldHZ 15 > WinTrkrHZ 507 > =3D=3D=3D=3D=3D=3D=3D > Totals --> 60734 > > For the DoH blocking, the above "hits" including blocking Apple DoH. I=E2= =80=99ve been testing this for the past few weeks. > >> I am particularly interested in what lists are publicly available. You kno= w from URL Filter, that lists have shut down or just been abandoned. There is= now very little point running it because the lists are just not as good as t= hey used to be. Not horrible, yet, but over time this will not be getting any= better. I am not even aware of any viable paid lists. Is this the same situa= tion for RPZ? A quick Google search did not give me anything that would be f= ree to use over DNS. The situation is somewhat similar with the IPS and IP bl= ocklist feature. > These are free/open lists: > > Hagezi - DNS Blocklists > =E2=80=A2 at https://github.com/hagezi/dns-blocklists?tab=3Dreadme-ov-file#= zap-dns-blocklists---for-a-better-internet > =E2=80=A2 This list was recommended within the IPFire Community and this is= the list I strongly recommend. Gerd (Hagezi) is very responsive to support = requests. The Hagezi RPZ list is well supported and the RPZ lists are updated= twice per day (if changes are needed). > =E2=80=A2 Example: I came across a domain name that was blocked `f005.backb= lazeb2.com` and it was fixed (removed from the block list) within a few days. > > ThreatFox - DNS Response Policy Zone (RPZ) > =E2=80=A2 at https://threatfox.abuse.ch/export/#rpz > > URLHaus - DNS Response Policy Zone (RPZ) > =E2=80=A2 at https://urlhaus.abuse.ch/api/#rpz > > jpgpi250 - DNS block list for DoH > =E2=80=A2 at https://github.com/jpgpi250/piholemanual/blob/master/DOH.rpz > =E2=80=A2 this is the original DoH block list that I started with. Peter (= jpgpi250) has a nice PDF document explaining RPZ and was very helpful getting= me started with RPZ. See https://jpgpi250.github.io/piholemanual/doc/Unboun= d%20response%20policy%20zones.pdf > > I have come across other RPZ lists but these are the best supported lists. > > I did not look into paid lists. > >> >> I would be interested in hearing your thoughts on this. > Does the above help? > >> Best, >> -Michael >> >>> On 27 Jul 2024, at 21:31, jon wrote: >>> >>> And another question. >>> >>> I found an error in the `rpz-config` script file. Simple easy one line (= actually one word) fix. >>> >>> Do I resubmit the entire patch? >>> -or- >>> Do I wait until approved and then submit the simple patch for the one lin= e change. >>> >>> >>> Jon >>> >>> >>>> On Jul 20, 2024, at 1:42=E2=80=AFPM, jon wrote: >>>> >>>> I need help with=E2=80=A6 >>>> >>>> 1) The custom allow and block lists are currently located at `/var/ipfir= e/rpz`. Is this correct? >>>> >>>> 2) The three bash scripts are currently located at `/usr/sbin`. Is this= correct? >>>> >>>> 3) Should this commit be separated into multiple posts? If yes, how do = I decide how to split it? >>>> >>>> >>>> Jon >>>> >>>> >>>>> On Jul 20, 2024, at 12:46=E2=80=AFPM, Jon Murphy wrote: >>>>> >>>>> What is it? >>>>> Response Policy Zone (RPZ) is a mechanism to define local policies in a= standardised >>>>> way and load those policies from external sources. >>>>> Bottom line: RPZ allows admins to easily block access to websites via D= NS lookup. >>>>> >>>>> RPZ can block websites via categories. Examples include: fake websites= , annoying >>>>> pop-up ads, newly registered domains, DoH bypass sites, bad "host" serv= ices, >>>>> maliscious top level domains (e.g., *.zip, *.mov), piracy, gambling, po= rnography, >>>>> and more. RPZ lists come from various RPZ providers and their available >>>>> catagories. >>>>> >>>>> This RPZ add-on enables the RPZ functionality by adding a couple lines = in a >>>>> configuration file. This add-on simply adds a configuration file and a= dds >>>>> three scripts (config, metrics and sleep) to make RPZ easier for the ad= min to use. >>>>> >>>>> RPZ was release in 2010 and has been part of the IPFire build since ~20= 15. >>>>> >>>>> Why is it needed? >>>>> Some IPFire admin's utilize pihole to block unwanted websites via DNS l= ookup. >>>>> Moving the pihole base functionality (without pretty graphs) to IPFire = removes >>>>> one device from the admin's local network. >>>>> And hopefully this reduces the pihole questions from the Community. >>>>> >>>>> A list of RPZ providers can be recommended by IPFire and coded into a s= et list. >>>>> Or, if prefered, the local admin can choose their own RPZ providers. >>>>> >>>>> This is a: >>>>> * simple replacement for pihole base functionality >>>>> * RPZ can be a nice replacement for the URL Filter >>>>> >>>>> IPFire Wiki >>>>> In process at: https://www.ipfire.org/docs/addons/rpz >>>>> >>>>> more info: >>>>> * https://en.wikipedia.org/wiki/Response_policy_zone >>>>> * https://unbound.docs.nlnetlabs.nl/en/latest/topics/filtering/rpz.html >>>>> >>>>> I need help with... >>>>> 1) The custom allow and block lists are currently located at `/var/ipfi= re/rpz`. Is this correct? rpz is supposed to be part of the dns so I would either put all the=20 files in /var/ipfire/dns/ or create a directory for rpz under the dns=20 directory. >>>>> 2) The three bash scripts are currently located at `/usr/sbin`. Is thi= s correct? >>>>> >>>>> Signed-off-by: Jon Murphy >>>>> --- >>>>> config/backup/includes/rpz | 5 + >>>>> config/rootfiles/packages/rpz | 11 ++ >>>>> config/rpz/00-rpz.conf | 18 ++++ >>>>> config/rpz/rpz-config | 194 ++++++++++++++++++++++++++++++++++ >>>>> config/rpz/rpz-metrics | 143 +++++++++++++++++++++++++ >>>>> config/rpz/rpz-sleep | 58 ++++++++++ >>>>> lfs/rpz | 88 +++++++++++++++ >>>>> make.sh | 2 + >>>>> 8 files changed, 519 insertions(+) >>>>> create mode 100644 config/backup/includes/rpz >>>>> create mode 100644 config/rootfiles/packages/rpz >>>>> create mode 100644 config/rpz/00-rpz.conf >>>>> create mode 100644 config/rpz/rpz-config >>>>> create mode 100644 config/rpz/rpz-metrics >>>>> create mode 100644 config/rpz/rpz-sleep >>>>> create mode 100644 lfs/rpz >>>>> >>>>> diff --git a/config/backup/includes/rpz b/config/backup/includes/rpz >>>>> new file mode 100644 >>>>> index 000000000..4d59bb40c >>>>> --- /dev/null >>>>> +++ b/config/backup/includes/rpz >>>>> @@ -0,0 +1,5 @@ >>>>> +/var/ipfire/rpz/allowlist >>>>> +/var/ipfire/rpz/blocklist >>>>> +/etc/unbound/zonefiles/allow.rpz >>>>> +/etc/unbound/zonefiles/block.rpz >>>>> +/etc/unbound/local.d/*rpz.conf >>>>> diff --git a/config/rootfiles/packages/rpz b/config/rootfiles/packages/= rpz >>>>> new file mode 100644 >>>>> index 000000000..2ffa715dd >>>>> --- /dev/null >>>>> +++ b/config/rootfiles/packages/rpz >>>>> @@ -0,0 +1,11 @@ >>>>> +etc/unbound/local.d/00-rpz.conf >>>>> +etc/unbound/zonefiles >>>>> +etc/unbound/zonefiles/allow.rpz >>>>> +etc/unbound/zonefiles/block.rpz >>>>> +usr/sbin/rpz-config >>>>> +usr/sbin/rpz-metrics >>>>> +usr/sbin/rpz-sleep >>>>> +var/ipfire/backup/addons/includes/rpz >>>>> +var/ipfire/rpz >>>>> +var/ipfire/rpz/allowlist >>>>> +var/ipfire/rpz/blocklist >>>>> diff --git a/config/rpz/00-rpz.conf b/config/rpz/00-rpz.conf >>>>> new file mode 100644 >>>>> index 000000000..72c1d12e5 >>>>> --- /dev/null >>>>> +++ b/config/rpz/00-rpz.conf >>>>> @@ -0,0 +1,18 @@ >>>>> +server: >>>>> + module-config: "respip validator iterator" >>>>> + >>>>> +rpz: >>>>> + name: allow.rpz >>>>> + zonefile: /etc/unbound/zonefiles/allow.rpz >>>>> + rpz-action-override: passthru >>>>> + rpz-log: yes >>>>> + rpz-log-name: allow >>>>> + rpz-signal-nxdomain-ra: yes >>>>> + >>>>> +rpz: >>>>> + name: block.rpz >>>>> + zonefile: /etc/unbound/zonefiles/block.rpz >>>>> + rpz-action-override: nxdomain >>>>> + rpz-log: yes >>>>> + rpz-log-name: block >>>>> + rpz-signal-nxdomain-ra: yes >>>>> diff --git a/config/rpz/rpz-config b/config/rpz/rpz-config >>>>> new file mode 100644 >>>>> index 000000000..98dc0a4ca >>>>> --- /dev/null >>>>> +++ b/config/rpz/rpz-config >>>>> @@ -0,0 +1,194 @@ >>>>> +#!/bin/bash >>>>> +######################################################################= ######### >>>>> +# = # >>>>> +# IPFire.org - A linux based firewall = # >>>>> +# Copyright (C) 2024 IPFire Team = # >>>>> +# = # >>>>> +# This program is free software: you can redistribute it and/or modif= y # >>>>> +# it under the terms of the GNU General Public License as published b= y # >>>>> +# the Free Software Foundation, either version 3 of the License, or = # >>>>> +# (at your option) any later version. = # >>>>> +# = # >>>>> +# This program is distributed in the hope that it will be useful, = # >>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of = # >>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the = # >>>>> +# GNU General Public License for more details. = # >>>>> +# = # >>>>> +# You should have received a copy of the GNU General Public License = # >>>>> +# along with this program. If not, see . # >>>>> +# = # >>>>> +######################################################################= ######### >>>>> + >>>>> +# v22 - 2024-07-12 >>>>> + >>>>> +############### Functions ############### >>>>> + >>>>> +msg_log () { >>>>> + /usr/bin/logger --tag "${tagName}" "$*" >>>>> + if tty --silent ; then >>>>> + echo "${tagName}:" "$*" >>>>> + fi >>>>> +} >>>>> + >>>>> +check_name () { >>>>> + local testName=3D"${1}" >>>>> + # check for a valid name >>>>> + regex=3D'^[a-zA-Z0-9_]+$' Is there a specific reason why you have chosen only letters, numbers and=20 underscore and not also the dash. If you allowed that then you could use=20 the validdomainname subroutine in the general-functions.pl file rather=20 than creating another valid name function. Does unbound not accept dash's in the rpz config filename? >>>>> + if [[ ! "${testName}" =3D~ $regex ]] ; then >>>>> + msg_log "error: rpz: the NAME is not valid: \"${testName}\". exit." >>>>> + exit 1 >>>>> + fi >>>>> +} >>>>> + >>>>> +check_unbound_conf () { >>>>> + # check the above config files >>>>> + msg_log "info: rpz: check for errors with \"unbound-checkconf\"" >>>>> + /usr/sbin/unbound-checkconf I think checkconf should be added into the unboundctrl.c file https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dblob;f=3Dsrc/misc-progs/unboun= dctrl.c;h=3D86c6ac42b9010f9d77fa49754bf217875aed3cc0;hb=3Drefs/heads/next so that the command can be ensured to be running in safe mode. >>>>> + exit_code=3D$? >>>>> + if [[ "${exit_code}" -ne 0 ]] ; then >>>>> + msg_log "error: rpz: unbound-checkconf. exit." >>>>> + exit "${exit_code}" >>>>> + fi >>>>> +} >>>>> + >>>>> +make_rpz_file () { >>>>> + local theType=3D"${1}" # allow or block >>>>> + >>>>> + theList=3D"/var/ipfire/rpz/${theType}list" # input user list of domai= ns >>>>> + theZoneFile=3D"/etc/unbound/zonefiles/${theType}.rpz" # output file f= or RPZ >>>>> + >>>>> + theAction=3D'.' >>>>> + if [[ "${theType}" =3D~ "block" ]] ; then >>>>> + theAction=3D'rpz-passthru.' >>>>> + fi >>>>> + >>>>> + # does a list exist? >>>>> + if [[ -s "${theList}" ]] ; then >>>>> + # drop any extra "blanks" and add "CNAME ." to each line >>>>> + actionList=3D$( /usr/bin/awk '{$1=3D$1};1' "${theList}" | >>>>> + /bin/sed "/^[^;].*[[:alnum:]]/ s|$| CNAME ${theAction}|" ) >>>>> + >>>>> + msg_log "info: rpz: create zonefile for ${theList}" >>>>> + >>>>> + /bin/cat <<-EOF > "${theZoneFile}" >>>>> + ; Name: ${theType} list >>>>> + ; Last modified: $(date "+%Y-%m-%d at %H.%M.%S %Z") >>>>> + ; >>>>> + ; domains with actions list >>>>> + ; >>>>> + ${actionList} >>>>> + EOF >>>>> + >>>>> + # reload the zone that was just updated >>>>> + zoneBase=3D$( basename "${theZoneFile}" ) >>>>> + /usr/sbin/unbound-control auth_zone_reload -q "${zoneBase}" This should also be run via the unboundctrl.c file especially as the=20 unbound-control command is accessing remote systems. >>>>> + fi >>>>> +} >>>>> + >>>>> +############### Main ############### >>>>> + >>>>> +tagName=3D"unbound" >>>>> + >>>>> +theAction=3D"${1}" # input action >>>>> +theName=3D"${2}" # input RPZ name >>>>> +theURL=3D"${3}" # input RPZ URL >>>>> + >>>>> +check_name "${theName}" # is this a valid name? >>>>> + >>>>> +rpzConfig=3D"/etc/unbound/local.d/${theName}.rpz.conf" # output zone c= onf file >>>>> +rpzFile=3D"/etc/unbound/zonefiles/${theName}.rpz" # output for RPZ file >>>>> + >>>>> +case "${theAction}" in >>>>> + >>>>> + # add new rpz list >>>>> + add ) >>>>> + # does this config already exist? If yes, then exit >>>>> + if [[ -f "${rpzConfig}" ]] ; then >>>>> + msg_log "info: rpz: ${rpzConfig} already exists. exit" >>>>> + exit 1 >>>>> + fi >>>>> + >>>>> + # is this a valid URL? >>>>> + regex=3D'^https://[-[:alnum:]\+&@#/%?=3D~_|!:,.;]*[-[:alnum:]\+&@#/%= =3D~_|]' >>>>> + if ! [[ "${theURL}" =3D~ $regex ]] ; then >>>>> + msg_log "error: rpz: the URL is not valid: \"${theURL}\". exit." >>>>> + exit 1 >>>>> + fi >>>>> + >>>>> + # create the zone config file >>>>> + msg_log "info: rpz: add config file \"${theName}.rpz.conf\"" >>>>> + cat <<-EOF > "${rpzConfig}" >>>>> + rpz: >>>>> + name: ${theName}.rpz >>>>> + zonefile: /etc/unbound/zonefiles/${theName}.rpz >>>>> + url: ${theURL} >>>>> + rpz-action-override: nxdomain >>>>> + rpz-log: yes >>>>> + rpz-log-name: ${theName} >>>>> + rpz-signal-nxdomain-ra: yes >>>>> + EOF >>>>> + >>>>> + # set-up zone file >>>>> + /usr/bin/touch "${rpzFile}" >>>>> + # unbound requires these settings for rpz files >>>>> + /bin/chown --verbose nobody:nobody "${rpzFile}" >>>>> + /bin/chmod --verbose 644 "${rpzFile}" These should only be using --verbose while doing development. In=20 production they should not be used as the development should have=20 ensured that everything is working as required. >>>>> + ;; >>>>> + >>>>> + # trash config file & rpz file >>>>> + remove ) >>>>> + if ! [[ -f "${rpzConfig}" ]] ; then >>>>> + msg_log "info: rpz: ${rpzConfig} does not exist. exit" >>>>> + exit 1 >>>>> + fi >>>>> + >>>>> + msg_log "info: rpz: remove config file & rpz file \"${theName}\"" >>>>> + /bin/rm --verbose "${rpzConfig}" >>>>> + /bin/rm --verbose "${rpzFile}" Same thing with the --verbose. >>>>> + >>>>> + check_unbound_conf >>>>> + ;; >>>>> + >>>>> + # make a new allow or block rpz file >>>>> + make ) >>>>> + case "${theName}" in >>>>> + allow ) >>>>> + make_rpz_file allow >>>>> + ;; >>>>> + >>>>> + block ) >>>>> + make_rpz_file block >>>>> + ;; >>>>> + >>>>> + allowblock ) >>>>> + make_rpz_file allow >>>>> + make_rpz_file block >>>>> + ;; >>>>> + >>>>> + * ) >>>>> + msg_log "error: rpz: the NAME is not valid: \"${theName}\". exit." >>>>> + exit 1 >>>>> + ;; >>>>> + esac >>>>> + >>>>> + check_unbound_conf >>>>> + ;; >>>>> + >>>>> + *) >>>>> + msg_log "error: rpz: missing or incorrect parameter" >>>>> + /usr/bin/printf "Usage: rpzConfig.sh \n" >>>>> + exit 1 >>>>> + ;; >>>>> + >>>>> +esac >>>>> + >>>>> +# reload due to the changes >>>>> +msg_log "rpz: running \"unbound-control reload\"" >>>>> +/usr/sbin/unbound-control reload Use unboundctrl >>>>> +exit_code=3D$? >>>>> +if [[ "${exit_code}" -ne 0 ]] ; then >>>>> + msg_log "error: rpz: unbound-control \"${theName}\". exit." >>>>> + exit "${exit_code}" >>>>> +fi >>>>> + >>>>> +exit This is a hell of a long bash script. Did you consider doing it in Perl=20 or python instead of bash. Maybe it would be better in c even as then it=20 would be compiled and could not be modified on the IPFire system if some=20 malware got in. >>>>> diff --git a/config/rpz/rpz-metrics b/config/rpz/rpz-metrics >>>>> new file mode 100644 >>>>> index 000000000..0f97c7911 >>>>> --- /dev/null >>>>> +++ b/config/rpz/rpz-metrics >>>>> @@ -0,0 +1,143 @@ >>>>> +#!/bin/bash >>>>> +######################################################################= ######### >>>>> +# = # >>>>> +# IPFire.org - A linux based firewall = # >>>>> +# Copyright (C) 2024 IPFire Team = # >>>>> +# = # >>>>> +# This program is free software: you can redistribute it and/or modif= y # >>>>> +# it under the terms of the GNU General Public License as published b= y # >>>>> +# the Free Software Foundation, either version 3 of the License, or = # >>>>> +# (at your option) any later version. = # >>>>> +# = # >>>>> +# This program is distributed in the hope that it will be useful, = # >>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of = # >>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the = # >>>>> +# GNU General Public License for more details. = # >>>>> +# = # >>>>> +# You should have received a copy of the GNU General Public License = # >>>>> +# along with this program. If not, see . # >>>>> +# = # >>>>> +######################################################################= ######### >>>>> + >>>>> +# v18 on 2024-07-05 >>>>> + >>>>> +############### Main ############### >>>>> + >>>>> +weeks=3D"${1:-2}" # default to two message logs >>>>> +sortBy=3D"${2:-name}" # by name or by hits >>>>> + >>>>> +# get the list of message logs for N weeks >>>>> +messageLogs=3D$( find /var/log/messages* -type f | >>>>> + /usr/bin/sort --version-sort | >>>>> + head -"${weeks}" ) >>>>> + >>>>> +# get the list of RPZ names & counts from the message log(s) >>>>> +rpzNameCount=3D$( for logf in ${messageLogs} ; do >>>>> + /usr/bin/zgrep --text --fixed-strings 'info: rpz: applied' "${logf}" | >>>>> + /usr/bin/awk '$10 ~ /\[\w*]/ { print $10 }' ; >>>>> + done | /usr/bin/sort | /usr/bin/uniq --count ) >>>>> + >>>>> +# flip results and remove brackets `[` and `]` >>>>> +rpzNameCount=3D$( /bin/echo "${rpzNameCount}" | >>>>> + /usr/bin/awk '{ print $2, $1 }' | >>>>> + /bin/sed --regexp-extended 's|^\[(.*)\]|\1|' ) >>>>> + >>>>> +# grab only names >>>>> +rpzNames=3D$( /bin/echo "${rpzNameCount}" | /usr/bin/awk '{ print $1 }= ' ) >>>>> + >>>>> +# get list of RPZ files >>>>> +rpzFileList=3D$( /bin/find /etc/unbound/zonefiles -type f -iname "*.rp= z" ) >>>>> + >>>>> +# get basename of those files >>>>> +rpzBaseNames=3D$( /bin/echo "${rpzFileList}" | >>>>> + /bin/sed 's|/etc/unbound/zonefiles/||g ; s|\.rpz||g ;' ) >>>>> + >>>>> +# add to rpzNames >>>>> +rpzNames=3D"${rpzNames}"$'\n'"${rpzBaseNames}" >>>>> + >>>>> +# drop duplicate names >>>>> +rpzNames=3D$( echo "${rpzNames}" | /usr/bin/sort --unique ) >>>>> + >>>>> +# get line count for each RPZ >>>>> +lineCount=3D$( /bin/echo "${rpzFileList}" | /usr/bin/xargs wc -l ) >>>>> + >>>>> +# get comment line count and blank line count for each RPZ >>>>> +commentCount=3D$( /bin/echo "${rpzFileList}" | >>>>> + /usr/bin/xargs /bin/grep --count -e "^$" -e "^;" ) >>>>> + >>>>> +# get modified date each RPZ >>>>> +modDateList=3D$( /bin/echo "${rpzFileList}" | xargs stat -c '%.10y %n= ' ) >>>>> + >>>>> +ucListAuthZones=3D$( /usr/sbin/unbound-control list_auth_zones ) >>>>> + >>>>> +# get width of RPZ names >>>>> +pWidth=3D$( /bin/echo "${rpzNames}" | /usr/bin/awk '{ print $1" " }'= | wc -L ) >>>>> +pFormat=3D"%-${pWidth}s %-8s %-8s %-8s %10s %12s\n" >>>>> + >>>>> +# print title line >>>>> +printf "${pFormat}" "name" "hits" "active" "lines" "hits/line%" "last = update" >>>>> +printf -- "--------------" >>>>> + >>>>> +theResults=3D"" >>>>> +totalLines=3D0 >>>>> +totalHits=3D0 >>>>> +while read -r theName >>>>> +do >>>>> + printf -- "-" # pretend progress bar >>>>> + # get hit count >>>>> + theHits=3D"0" >>>>> + if output=3D$( /bin/grep "^${theName}\s" <<< "${rpzNameCount}" ) ; th= en >>>>> + theHits=3D$( /bin/echo "${output}" | >>>>> + /usr/bin/awk '{ print $2 }' ) >>>>> + totalHits=3D$(( totalHits + theHits )) >>>>> + fi >>>>> + >>>>> + # is this RPZ list active? >>>>> + theActive=3D"disabled" >>>>> + if /bin/grep --quiet "^${theName}\.rpz" <<< "${ucListAuthZones}" >>>>> + then >>>>> + theActive=3D"enabled" >>>>> + fi >>>>> + >>>>> + # get line count then subtract comment count and blank line count >>>>> + # from total line count >>>>> + theLines=3D"n/a" >>>>> + hitsPerLine=3D"0" >>>>> + if output=3D$( /bin/grep --fixed-strings "/${theName}.rpz" <<< "${lin= eCount}" ) ; then >>>>> + theLines=3D$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' ) >>>>> + totalLines=3D$(( totalLines + theLines )) >>>>> + >>>>> + #hitsPerLine=3D$( echo "scale=3D0 ; $theHits / $theLines" | bc ) >>>>> + hitsPerLine=3D$(( 100 * theHits / theLines )) >>>>> + fi >>>>> + >>>>> + # get modification date >>>>> + theModDate=3D"n/a" >>>>> + if output=3D$( /bin/grep --fixed-strings "/${theName}.rpz" <<< "${mod= DateList}" ) ; then >>>>> + theModDate=3D$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' ) >>>>> + fi >>>>> + >>>>> + # add to results list >>>>> + theResults+=3D"${theName} ${theHits} ${theActive} ${theLines} ${hitsP= erLine} ${theModDate}"$'\n' >>>>> +done <<< "${rpzNames}" >>>>> + >>>>> +case "${sortBy}" in >>>>> + names|name) sortArg=3D(-k3,3r -k1,1) ;; # sort by "active" then by "n= ame" >>>>> + >>>>> + hits|hit) sortArg=3D(-k3,3r -k2,2nr -k1,1) ;; # sort by "active" then= by "hits" then by "name" >>>>> + >>>>> + lines|line) sortArg=3D(-k3,3r -k4,4nr -k1,1) ;; # sort by "active" th= en by "lines" then by "name" >>>>> +esac >>>>> + >>>>> +printf -- "--------------\n" >>>>> +# remove blank lines, sort, print as columns >>>>> +/bin/echo "${theResults}" | >>>>> + /usr/bin/awk '!/^[[:space:]]*$/' | >>>>> + /usr/bin/sort "${sortArg[@]}" | >>>>> + /usr/bin/awk --assign=3Dwidth=3D"${pWidth}" \ >>>>> + '{ printf "%-*s %-8s %-8s %-8s %10s %12s\n", width, $1, $2, $3, $4, $= 5, $6 }' >>>>> + >>>>> +printf "${pFormat}" "" "=3D=3D=3D=3D=3D=3D=3D" "" "=3D=3D=3D=3D=3D=3D= =3D=3D" "" "" >>>>> +printf "${pFormat}" "Totals -->" "${totalHits}" "" "${totalLines}" "" = "" >>>>> + >>>>> +exit If I understand right the rpz-metrics info is being shown on a weekly=20 basis. Wouldn't this make more sense to create a log page for=20 unbound-rpz that shows the results in a similar way to the Firewall=20 Logs, IP Address Blocklist Logs etc. So you can see for a whole month or=20 a specific day and you can also export the data into a text format on a=20 page of the browser. Basically that would just be copying one of the=20 existing cgi scripts and modifying the content elements. >>>>> diff --git a/config/rpz/rpz-sleep b/config/rpz/rpz-sleep >>>>> new file mode 100644 >>>>> index 000000000..eeef1174a >>>>> --- /dev/null >>>>> +++ b/config/rpz/rpz-sleep >>>>> @@ -0,0 +1,58 @@ >>>>> +#!/bin/bash >>>>> +######################################################################= ######### >>>>> +# = # >>>>> +# IPFire.org - A linux based firewall = # >>>>> +# Copyright (C) 2024 IPFire Team = # >>>>> +# = # >>>>> +# This program is free software: you can redistribute it and/or modif= y # >>>>> +# it under the terms of the GNU General Public License as published b= y # >>>>> +# the Free Software Foundation, either version 3 of the License, or = # >>>>> +# (at your option) any later version. = # >>>>> +# = # >>>>> +# This program is distributed in the hope that it will be useful, = # >>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of = # >>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the = # >>>>> +# GNU General Public License for more details. = # >>>>> +# = # >>>>> +# You should have received a copy of the GNU General Public License = # >>>>> +# along with this program. If not, see . # >>>>> +# = # >>>>> +######################################################################= ######### >>>>> + >>>>> +# v04 on 2024-07-05 >>>>> + >>>>> +############### Functions ############### >>>>> + >>>>> +# send message to message log >>>>> +msg_log () { >>>>> + /usr/bin/logger --tag "${tagName}" "$*" >>>>> + if /usr/bin/tty --silent ; then >>>>> + echo "${tagName}:" "$*" >>>>> + fi >>>>> +} >>>>> + >>>>> +############### Main ############### >>>>> + >>>>> +tagName=3D"unbound" >>>>> + >>>>> +sleepTime=3D"${1:-5m}" # default to sleep for 5m (5 minutes) >>>>> + >>>>> +zoneList=3D$( /usr/sbin/unbound-control list_auth_zones | /usr/bin/awk= '{print $1}' ) >>>>> + >>>>> +for zone in ${zoneList} ; do >>>>> + /usr/bin/printf "disable ${zone}\t" >>>>> + /usr/sbin/unbound-control rpz_disable "${zone}" unboundctrl >>>>> +done >>>>> + >>>>> +msg_log "info: rpz: disabled all zones for ${sleepTime}" >>>>> + >>>>> +/bin/sleep "${sleepTime}" >>>>> + >>>>> +for zone in ${zoneList} ; do >>>>> + /usr/bin/printf "enable ${zone}\t" >>>>> + /usr/sbin/unbound-control rpz_enable "${zone}" unboundctrl >>>>> +done >>>>> + >>>>> +msg_log "info: rpz: enabled all zones" >>>>> + >>>>> +exit >>>>> diff --git a/lfs/rpz b/lfs/rpz >>>>> new file mode 100644 >>>>> index 000000000..319c10b7f >>>>> --- /dev/null >>>>> +++ b/lfs/rpz >>>>> @@ -0,0 +1,88 @@ >>>>> +######################################################################= ######### >>>>> +# = # >>>>> +# IPFire.org - A linux based firewall = # >>>>> +# Copyright (C) 2024 IPFire Team = # >>>>> +# = # >>>>> +# This program is free software: you can redistribute it and/or modify= # >>>>> +# it under the terms of the GNU General Public License as published by= # >>>>> +# the Free Software Foundation, either version 3 of the License, or = # >>>>> +# (at your option) any later version. = # >>>>> +# = # >>>>> +# This program is distributed in the hope that it will be useful, = # >>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of = # >>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the = # >>>>> +# GNU General Public License for more details. = # >>>>> +# = # >>>>> +# You should have received a copy of the GNU General Public License = # >>>>> +# along with this program. If not, see = . # >>>>> +# = # >>>>> +######################################################################= ######### >>>>> + >>>>> +######################################################################= ######### >>>>> +# Definitions >>>>> +######################################################################= ######### >>>>> + >>>>> +include Config >>>>> + >>>>> +SUMMARY =3D response policy zone - RPZ reputation system for unboun= d DNS >>>>> + >>>>> +VER =3D 1.0.0 >>>>> + >>>>> +THISAPP =3D rpz-$(VER) >>>>> +DIR_APP =3D $(DIR_SRC)/$(THISAPP) >>>>> +TARGET =3D $(DIR_INFO)/$(THISAPP) >>>>> + >>>>> +PROG =3D rpz >>>>> +PAK_VER =3D 1 >>>>> + >>>>> +DEPS =3D >>>>> + >>>>> +SERVICES =3D >>>>> + >>>>> +######################################################################= ######### >>>>> +# Top-level Rules >>>>> +######################################################################= ######### >>>>> + >>>>> +install : $(TARGET) >>>>> + >>>>> +check : >>>>> + >>>>> +download : >>>>> + >>>>> +b2 : >>>>> + >>>>> +dist: >>>>> + @$(PAK) >>>>> + >>>>> +######################################################################= ######### >>>>> +# Installation Details >>>>> +######################################################################= ######### >>>>> + >>>>> +$(TARGET) : >>>>> + @$(PREBUILD) >>>>> + @rm -rf $(DIR_APP) >>>>> + >>>>> + # install RPZ scripts >>>>> + install -v -m 755 \ >>>>> + $(DIR_CONF)/rpz/{rpz-config,rpz-metrics,rpz-sleep} -t /usr/sbin >>>>> + >>>>> + # Install settings folder and two empty files >>>>> + mkdir -pv /var/ipfire/rpz >>>>> + touch /var/ipfire/rpz/allowlist >>>>> + touch /var/ipfire/rpz/blocklist >>>>> + >>>>> + # Add conf file to /etc directory >>>>> + cp -vf $(DIR_CONF)/rpz/00-rpz.conf /etc/unbound/local.d >>>>> + >>>>> + # create zonefiles directory for the RPZ files and add two empty RPZ >>>>> + # files to avoid a unbound config error >>>>> + mkdir -pv /etc/unbound/zonefiles >>>>> + chown -v nobody:nobody /etc/unbound/zonefiles >>>>> + touch /etc/unbound/zonefiles/allow.rpz >>>>> + touch /etc/unbound/zonefiles/block.rpz >>>>> + >>>>> + # Install backup definition >>>>> + cp -vf $(DIR_CONF)/backup/includes/rpz /var/ipfire/backup/addons/incl= udes/rpz >>>>> + >>>>> + @rm -rf $(DIR_APP) >>>>> + @$(POSTBUILD) >>>>> diff --git a/make.sh b/make.sh >>>>> index 9bbbeb0f1..886d3760a 100755 >>>>> --- a/make.sh >>>>> +++ b/make.sh >>>>> @@ -1721,6 +1721,8 @@ buildipfire() { >>>>> lfsmake2 btrfs-progs >>>>> lfsmake2 inotify-tools >>>>> lfsmake2 grub-btrfs >>>>> + lfsmake2 rpz >>>>> + >>>>> >>>>> # Kernelbuild ... current we have no platform that need >>>>> # multi kernel builds so KCFG is empty >>>>> --=20 >>>>> 2.30.2 >>>>> --=20 Sent from my laptop --===============2794457153386085151==--