From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: [PATCH] RPZ: bug fix and code update Date: Thu, 15 Aug 2024 19:33:51 +0100 Message-ID: <4DBD7D82-268D-4D9B-8C39-E97751570E7D@ipfire.org> In-Reply-To: <0CC66649-F5ED-4E87-80F8-A979BB877DA7@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2221061858985836045==" List-Id: --===============2221061858985836045== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello, > On 15 Aug 2024, at 17:57, jon wrote: >=20 >=20 > Comments below. >=20 > Jon >=20 >=20 >> On Aug 15, 2024, at 10:33=E2=80=AFAM, Michael Tremer wrote: >>=20 >> Hello, >>=20 >>> On 14 Aug 2024, at 18:14, jon wrote: >>>=20 >>> Michael, >>>=20 >>> Sorry for putting you on the spot, but what do you want to do with this R= PZ add-on? >>=20 >> I am not sure this is a question for only me. >>=20 >> I personally just don=E2=80=99t have any capacity to take on another rathe= r large project as I have a huge backlog of so many things and I feel like a = lonely fighter getting these all over the line. For my own sanity I need to h= ave a couple of those closed before thinking about the next ones. >>=20 >> A couple of months ago we have talked on the monthly call about RPZ and th= e decision that was made by the people was to not look at this now, just beca= use there were other more pressing things. We could also not find answers to = the questions that led us into RPZ: >>=20 >> * Are there any good lists out there that would allow us to replace the UR= L Filter? That thing is basically on its last leg because of the lack of list= s. We need to be able to block ads and pr0n and that very reliably. >=20 > I believe this exists with the Hagezi lists at https://github.com/hagezi/dn= s-blocklists and/or the addition of a chosen "oisd NSFW" list at https://oisd= .nl/setup >=20 > I understand there is no time to review to see if this meet the needs. So = we can leave this as "I believe these exist" for now. So, I am not sure what to think of all these things on GitHub. It seems that = there are many repositories that pop up and presumably disappear in the same = kind of way. We see this with the IP blocklists. We want quality blocklists. I know that we don=E2=80=99t have too much on the= URL Filter front, but I would prefer to avoid putting so much work in if we = in the end stand where we started. >>=20 >> * We need to look into privacy when RPZs are being realised over DNS - I d= on=E2=80=99t even understand why we are suddenly starting to pull text files = over HTTP again. The IPS seems to have most of these lists already. >=20 > All of the lists that I am pull are HTTPS. And HTTPS only is currently che= cked (validated) in the existing `rpz-config` code. What a lost case. DNS is a globally distributed, cacheable protocol. We could= simply put all those blacklists into the DNS caches all around the world. Th= ey would respond fast, be local to so many people, but instead we download ma= ssive text files. That means that whenever something changes the entire file = needs to be transferred again. They are huge. We would want many fast updates= which are not possible that way=E2=80=A6 but I digress. How much traffic did you observe? How is Unbound refreshing these? Is there a= local cache? >> Regarding your code, there are some issues with the coding style, but I he= avily appreciate the pioneering to bring this feature to life. >=20 > My style is camelCase, but if you prefer snake_case that is an easy change = (please speak up). >=20 > As for as other style changes, that can wait until time is available by the= core developers. It is mainly that you make things very complicated for yourself. Call =E2=80= =9Ccat=E2=80=9D, not =E2=80=9C/bin/cat=E2=80=9D. There is no guarantee that t= hings will stay where they are now and the shell has lots of builtins that ex= ecute a lot faster. And this is all designed to work on the shell. We need something for the web = UI. >> What it would need to be finally merged would be a web UI though. So I thi= nk we have a long way ahead of us. >=20 > The attachment is an early mock-up I made a few months ago (made with Pixel= mator Pro). The WebGUI is being worked on now. >=20 > I think we might look for something that is very similar to the IPS ruleset e= ditor or the IP blocklists. I am not even sure if we would require users to manage this or whether we wou= ld curate a list like with the IPS and IP blocklists. > Does this help? > Jon >=20 >=20 >>=20 >> -Michael >>=20 >>> I saw your comments in the Dev Mailing List of "generally being in favor = of trying this path" (bad paraphrasing on my part) >>>=20 >>> I saw your comments in bugzilla at https://bugzilla.ipfire.org/show_bug.c= gi?id=3D13254#c171 >>>> I am not interested in anything regarding the RPZs right now. They have = not been properly put on the agenda and looking at how much time we have on o= ur hands, this won't make it on the agenda for years. >>>>=20 >>>> I don't want to build blockers, but this ticket is about a different pro= blem which I want to solve first. >>>=20 >>>=20 >>> How do you want to go forward? >>>=20 >>>=20 >>> Jon >>>=20 >>>=20 >>>> On Aug 12, 2024, at 2:11=E2=80=AFPM, jon wrote: >>>>=20 >>>> More questions! >>>>=20 >>>> Currently RPZ config files are at `/etc/unbound/local.d` but this direct= ory seems like it is for user (admin) customizations. =20 >>>>=20 >>>> ``` >>>> [root(a)ipfire ~] # ls -al /etc/unbound/local.d >>>> total 68 >>>> drwxr-xr-x 2 nobody nobody 4096 Aug 12 13:41 . >>>> drwxr-xr-x 4 root root 4096 Aug 12 00:52 .. >>>> -rw-r--r-- 1 nobody nobody 436 Jul 12 15:45 00-rpz.conf >>>> -rw-r--r-- 1 nobody nobody 285 Mar 1 22:12 AmazonTrkrHZ.rpz.conf >>>> -rw-r--r-- 1 nobody nobody 281 Mar 1 22:02 AppleTrkrHZ.rpz.conf >>>> -rw-r--r-- 1 nobody nobody 269 Mar 1 21:40 DOHblockHZ.rpz.conf >>>> ... >>>> -rw-r--r-- 1 nobody nobody 299 Aug 1 19:42 WinTrkrHZ.rpz.conf >>>> [root(a)ipfire ~] #=20 >>>> ``` >>>>=20 >>>>=20 >>>> Each file is a config file per category (or one per RPZ file). This mak= es it easy to add or remove a category (or RPZ file). >>>>=20 >>>> Should I create a new unbound directory for RPZ config files? Maybe `/e= tc/unbound/rpz.d`? Or `/etc/unbound/rpz`? >>>>=20 >>>>=20 >>>> Jon >>>>=20 >>>>=20 >>>>> On Aug 1, 2024, at 1:45=E2=80=AFPM, Jon Murphy wrote: >>>>>=20 >>>>> changed all paths from `/var/ipfire/rpz/` to `/var/ipfire/dns/rpz/` >>>>> (thank you to Adolf!) >>>>>=20 >>>>> rpz-config: >>>>> - bug: corrected "Type" test from block to allow >>>>> - removed verbose parameter from various commands >>>>>=20 >>>>> rpz-metrics: >>>>> - bug: corrected grep for rpz name count >>>>> - bug: fixed divide by zero error (thank you Peppe!) >>>>>=20 >>>>> install/uninstall: >>>>> - bug: corrected scripts (thank you Bernhard!) >>>>>=20 >>>>> Signed-off-by: Jon Murphy >>>>> --- >>>>> config/backup/includes/rpz | 4 ++-- >>>>> config/rootfiles/packages/rpz | 6 +++--- >>>>> config/rpz/rpz-config | 14 +++++++------- >>>>> config/rpz/rpz-metrics | 9 +++++---- >>>>> lfs/rpz | 6 +++--- >>>>> src/paks/rpz/install.sh | 27 +++++++++++++++++++++++++++ >>>>> src/paks/rpz/uninstall.sh | 31 +++++++++++++++++++++++++++++++ >>>>> src/paks/rpz/update.sh | 25 +++++++++++++++++++++++++ >>>>> 8 files changed, 103 insertions(+), 19 deletions(-) >>>>> create mode 100644 src/paks/rpz/install.sh >>>>> create mode 100644 src/paks/rpz/uninstall.sh >>>>> create mode 100644 src/paks/rpz/update.sh >>>>>=20 >>>>> diff --git a/config/backup/includes/rpz b/config/backup/includes/rpz >>>>> index 4d59bb40c..8c7410ebd 100644 >>>>> --- a/config/backup/includes/rpz >>>>> +++ b/config/backup/includes/rpz >>>>> @@ -1,5 +1,5 @@ >>>>> -/var/ipfire/rpz/allowlist >>>>> -/var/ipfire/rpz/blocklist >>>>> +/var/ipfire/dns/rpz/allowlist >>>>> +/var/ipfire/dns/rpz/blocklist >>>>> /etc/unbound/zonefiles/allow.rpz >>>>> /etc/unbound/zonefiles/block.rpz >>>>> /etc/unbound/local.d/*rpz.conf >>>>> diff --git a/config/rootfiles/packages/rpz b/config/rootfiles/packages/= rpz >>>>> index 2ffa715dd..183825362 100644 >>>>> --- a/config/rootfiles/packages/rpz >>>>> +++ b/config/rootfiles/packages/rpz >>>>> @@ -6,6 +6,6 @@ usr/sbin/rpz-config >>>>> usr/sbin/rpz-metrics >>>>> usr/sbin/rpz-sleep >>>>> var/ipfire/backup/addons/includes/rpz >>>>> -var/ipfire/rpz >>>>> -var/ipfire/rpz/allowlist >>>>> -var/ipfire/rpz/blocklist >>>>> +var/ipfire/dns/rpz >>>>> +var/ipfire/dns/rpz/allowlist >>>>> +var/ipfire/dns/rpz/blocklist >>>>> diff --git a/config/rpz/rpz-config b/config/rpz/rpz-config >>>>> index 98dc0a4ca..a24a5c132 100644 >>>>> --- a/config/rpz/rpz-config >>>>> +++ b/config/rpz/rpz-config >>>>> @@ -19,7 +19,7 @@ >>>>> # = # >>>>> #######################################################################= ######## >>>>>=20 >>>>> -# v22 - 2024-07-12 >>>>> +# v23 - 2024-07-30 >>>>>=20 >>>>> ############### Functions ############### >>>>>=20 >>>>> @@ -54,11 +54,11 @@ check_unbound_conf () { >>>>> make_rpz_file () { >>>>> local theType=3D"${1}" # allow or block >>>>>=20 >>>>> - theList=3D"/var/ipfire/rpz/${theType}list" # input user list of domai= ns >>>>> + theList=3D"/var/ipfire/dns/rpz/${theType}list" # input custom list of= domains >>>>> theZoneFile=3D"/etc/unbound/zonefiles/${theType}.rpz" # output file for= RPZ >>>>>=20 >>>>> theAction=3D'.' >>>>> - if [[ "${theType}" =3D~ "block" ]] ; then >>>>> + if [[ "${theType}" =3D~ "allow" ]] ; then >>>>> theAction=3D'rpz-passthru.' >>>>> fi >>>>>=20 >>>>> @@ -131,8 +131,8 @@ case "${theAction}" in >>>>> # set-up zone file >>>>> /usr/bin/touch "${rpzFile}" >>>>> # unbound requires these settings for rpz files >>>>> - /bin/chown --verbose nobody:nobody "${rpzFile}" >>>>> - /bin/chmod --verbose 644 "${rpzFile}" >>>>> + /bin/chown nobody:nobody "${rpzFile}" >>>>> + /bin/chmod 644 "${rpzFile}" >>>>> ;; >>>>>=20 >>>>> # trash config file & rpz file >>>>> @@ -143,8 +143,8 @@ case "${theAction}" in >>>>> fi >>>>>=20 >>>>> msg_log "info: rpz: remove config file & rpz file \"${theName}\"" >>>>> - /bin/rm --verbose "${rpzConfig}" >>>>> - /bin/rm --verbose "${rpzFile}" >>>>> + /bin/rm "${rpzConfig}" >>>>> + /bin/rm "${rpzFile}" >>>>>=20 >>>>> check_unbound_conf >>>>> ;; >>>>> diff --git a/config/rpz/rpz-metrics b/config/rpz/rpz-metrics >>>>> index 0f97c7911..4d932726e 100644 >>>>> --- a/config/rpz/rpz-metrics >>>>> +++ b/config/rpz/rpz-metrics >>>>> @@ -19,7 +19,7 @@ >>>>> # = # >>>>> #######################################################################= ######## >>>>>=20 >>>>> -# v18 on 2024-07-05 >>>>> +# v19 on 2024-07-30 >>>>>=20 >>>>> ############### Main ############### >>>>>=20 >>>>> @@ -33,7 +33,7 @@ messageLogs=3D$( find /var/log/messages* -type f | >>>>>=20 >>>>> # get the list of RPZ names & counts from the message log(s) >>>>> rpzNameCount=3D$( for logf in ${messageLogs} ; do >>>>> - /usr/bin/zgrep --text --fixed-strings 'info: rpz: applied' "${logf}" | >>>>> + /usr/bin/zgrep --text --extended-regexp 'info: rpz: applied.* A IN$' = "${logf}" | >>>>> /usr/bin/awk '$10 ~ /\[\w*]/ { print $10 }' ; >>>>> done | /usr/bin/sort | /usr/bin/uniq --count ) >>>>>=20 >>>>> @@ -107,8 +107,9 @@ do >>>>> theLines=3D$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' ) >>>>> totalLines=3D$(( totalLines + theLines )) >>>>>=20 >>>>> - #hitsPerLine=3D$( echo "scale=3D0 ; $theHits / $theLines" | bc ) >>>>> - hitsPerLine=3D$(( 100 * theHits / theLines )) >>>>> + if [[ "${theLines}" -gt 2 ]] ; then >>>>> + hitsPerLine=3D$(( 100 * theHits / theLines )) >>>>> + fi >>>>> fi >>>>>=20 >>>>> # get modification date >>>>> diff --git a/lfs/rpz b/lfs/rpz >>>>> index 319c10b7f..73f6f2b1b 100644 >>>>> --- a/lfs/rpz >>>>> +++ b/lfs/rpz >>>>> @@ -67,9 +67,9 @@ $(TARGET) : >>>>> $(DIR_CONF)/rpz/{rpz-config,rpz-metrics,rpz-sleep} -t /usr/sbin >>>>>=20 >>>>> # Install settings folder and two empty files >>>>> - mkdir -pv /var/ipfire/rpz >>>>> - touch /var/ipfire/rpz/allowlist >>>>> - touch /var/ipfire/rpz/blocklist >>>>> + mkdir -pv /var/ipfire/dns/rpz >>>>> + touch /var/ipfire/dns/rpz/allowlist >>>>> + touch /var/ipfire/dns/rpz/blocklist >>>>>=20 >>>>> # Add conf file to /etc directory >>>>> cp -vf $(DIR_CONF)/rpz/00-rpz.conf /etc/unbound/local.d >>>>> diff --git a/src/paks/rpz/install.sh b/src/paks/rpz/install.sh >>>>> new file mode 100644 >>>>> index 000000000..0a797e158 >>>>> --- /dev/null >>>>> +++ b/src/paks/rpz/install.sh >>>>> @@ -0,0 +1,27 @@ >>>>> +#!/bin/bash >>>>> +######################################################################= ######### >>>>> +# = # >>>>> +# IPFire.org - A linux based firewall = # >>>>> +# Copyright (C) 2024 IPFire Team = # >>>>> +# = # >>>>> +# This program is free software: you can redistribute it and/or modif= y # >>>>> +# it under the terms of the GNU General Public License as published b= y # >>>>> +# the Free Software Foundation, either version 3 of the License, or = # >>>>> +# (at your option) any later version. = # >>>>> +# = # >>>>> +# This program is distributed in the hope that it will be useful, = # >>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of = # >>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the = # >>>>> +# GNU General Public License for more details. = # >>>>> +# = # >>>>> +# You should have received a copy of the GNU General Public License = # >>>>> +# along with this program. If not, see . # >>>>> +# = # >>>>> +######################################################################= ######### >>>>> +# >>>>> +. /opt/pakfire/lib/functions.sh >>>>> +extract_files >>>>> +restore_backup ${NAME} >>>>> + >>>>> +# restart unbound to load config file >>>>> +/etc/init.d/unbound restart >>>>> diff --git a/src/paks/rpz/uninstall.sh b/src/paks/rpz/uninstall.sh >>>>> new file mode 100644 >>>>> index 000000000..4fb20e127 >>>>> --- /dev/null >>>>> +++ b/src/paks/rpz/uninstall.sh >>>>> @@ -0,0 +1,31 @@ >>>>> +#!/bin/bash >>>>> +######################################################################= ######### >>>>> +# = # >>>>> +# IPFire.org - A linux based firewall = # >>>>> +# Copyright (C) 2024 IPFire Team = # >>>>> +# = # >>>>> +# This program is free software: you can redistribute it and/or modif= y # >>>>> +# it under the terms of the GNU General Public License as published b= y # >>>>> +# the Free Software Foundation, either version 3 of the License, or = # >>>>> +# (at your option) any later version. = # >>>>> +# = # >>>>> +# This program is distributed in the hope that it will be useful, = # >>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of = # >>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the = # >>>>> +# GNU General Public License for more details. = # >>>>> +# = # >>>>> +# You should have received a copy of the GNU General Public License = # >>>>> +# along with this program. If not, see . # >>>>> +# = # >>>>> +######################################################################= ######### >>>>> +# >>>>> +. /opt/pakfire/lib/functions.sh >>>>> + >>>>> +# stop unbound to delete RPZ conf file >>>>> +/etc/init.d/unbound stop >>>>> + >>>>> +make_backup ${NAME} >>>>> +remove_files >>>>> + >>>>> +# start unbound to load unbound config file >>>>> +/etc/init.d/unbound start >>>>> diff --git a/src/paks/rpz/update.sh b/src/paks/rpz/update.sh >>>>> new file mode 100644 >>>>> index 000000000..938a93a40 >>>>> --- /dev/null >>>>> +++ b/src/paks/rpz/update.sh >>>>> @@ -0,0 +1,25 @@ >>>>> +#!/bin/bash >>>>> +######################################################################= ######### >>>>> +# = # >>>>> +# IPFire.org - A linux based firewall = # >>>>> +# Copyright (C) 2024 IPFire Team = # >>>>> +# = # >>>>> +# This program is free software: you can redistribute it and/or modif= y # >>>>> +# it under the terms of the GNU General Public License as published b= y # >>>>> +# the Free Software Foundation, either version 3 of the License, or = # >>>>> +# (at your option) any later version. = # >>>>> +# = # >>>>> +# This program is distributed in the hope that it will be useful, = # >>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of = # >>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the = # >>>>> +# GNU General Public License for more details. = # >>>>> +# = # >>>>> +# You should have received a copy of the GNU General Public License = # >>>>> +# along with this program. If not, see . # >>>>> +# = # >>>>> +######################################################################= ######### >>>>> +# >>>>> +. /opt/pakfire/lib/functions.sh >>>>> +extract_backup_includes >>>>> +./uninstall.sh >>>>> +./install.sh >>>>> --=20 >>>>> 2.30.2 --===============2221061858985836045==--