Hello Jon,
On 15 Aug 2024, at 23:22, jon jon.murphy@ipfire.org wrote:
Comments below…
On Aug 15, 2024, at 1:33 PM, Michael Tremer michael.tremer@ipfire.org wrote:
Hello,
On 15 Aug 2024, at 17:57, jon jon.murphy@ipfire.org wrote:
Comments below.
Jon
On Aug 15, 2024, at 10:33 AM, Michael Tremer michael.tremer@ipfire.org wrote:
Hello,
On 14 Aug 2024, at 18:14, jon jon.murphy@ipfire.org wrote:
Michael,
Sorry for putting you on the spot, but what do you want to do with this RPZ add-on?
I am not sure this is a question for only me.
I personally just don’t have any capacity to take on another rather large project as I have a huge backlog of so many things and I feel like a lonely fighter getting these all over the line. For my own sanity I need to have a couple of those closed before thinking about the next ones.
A couple of months ago we have talked on the monthly call about RPZ and the decision that was made by the people was to not look at this now, just because there were other more pressing things. We could also not find answers to the questions that led us into RPZ:
- Are there any good lists out there that would allow us to replace the URL Filter? That thing is basically on its last leg because of the lack of lists. We need to be able to block ads and pr0n and that very reliably.
I believe this exists with the Hagezi lists at https://github.com/hagezi/dns-blocklists and/or the addition of a chosen "oisd NSFW" list at https://oisd.nl/setup
I understand there is no time to review to see if this meet the needs. So we can leave this as "I believe these exist" for now.
So, I am not sure what to think of all these things on GitHub. It seems that there are many repositories that pop up and presumably disappear in the same kind of way. We see this with the IP blocklists.
We want quality blocklists. I know that we don’t have too much on the URL Filter front, but I would prefer to avoid putting so much work in if we in the end stand where we started.
This makes perfect sense. Based on the comments I’ve seen it seems like Hagezi is growing in popularity and in offerings.
Gerd (the owner) is very responsive, involved and I keep running into his posts when searching for RPZ "things". Gerd quickly made changes per my request: see https://github.com/hagezi/dns-blocklists/issues/2301. Now there are two TLD RPZ lists.
Like everything else I cannot predict how long it might last.
If you have such a good connection, could you do me the favour and ask why the prefer distributing those lists using HTTPS and not DNS? Is it simply that GitHub is free and does not offer DNS? :)
- We need to look into privacy when RPZs are being realised over DNS - I don’t even understand why we are suddenly starting to pull text files over HTTP again. The IPS seems to have most of these lists already.
All of the lists that I am pull are HTTPS. And HTTPS only is currently checked (validated) in the existing `rpz-config` code.
What a lost case. DNS is a globally distributed, cacheable protocol. We could simply put all those blacklists into the DNS caches all around the world. They would respond fast, be local to so many people, but instead we download massive text files. That means that whenever something changes the entire file needs to be transferred again. They are huge. We would want many fast updates which are not possible that way… but I digress.
I came across one DNS service that offers the Hagezi lists as part of their available DNS services. If interested I can find and post the details.
I run a global DNS system where we could easily add this all to. This would scale like hell. It would be a fun project.
Just for fun, I have imported "jpgpi250.github.io http://jpgpi250.github.io/” which you can pull from either dfw.lwldns.net http://dfw.lwldns.net/ or haj.lwldns.net http://haj.lwldns.net/. This would be so much better since DNS zone transfer checks the serial of the zone and does not retransfer it if it has not changed (and I suppose none of these lists change that frequently).
Sadly our DNS system does not support this, but there is also IXFR for incremental zone updates where you would only load the changes.
How much traffic did you observe?
I have 11 RPZ list (honestly too many but I am experimenting) and that eat up about 230M bytes per day.
Oh wow that is huuuuuge. I assume that is repeat downloads? How large are the zone files just downloaded once?
How is Unbound refreshing these?
Unbound RPZ takes care of the updates automagically. There is no cron entry.
Each RPZ list includes an SOA line similar to this: `@ SOA localhost. root.localhost. 1723093380 43200 3600 259200 300`
The 43200 is the refresh period in seconds. So this RPZ list auto updates every 12 hours.
Is there a local cache?
Yes, it is part of unbound. (Nothing we need to setup)
Would it use the cache for zones transferred over DNS, too?
Regarding your code, there are some issues with the coding style, but I heavily appreciate the pioneering to bring this feature to life.
My style is camelCase, but if you prefer snake_case that is an easy change (please speak up).
As for as other style changes, that can wait until time is available by the core developers.
It is mainly that you make things very complicated for yourself. Call “cat”, not “/bin/cat”. There is no guarantee that things will stay where they are now and the shell has lots of builtins that execute a lot faster.
I will remove the executable paths. What builtin did you see that I missed? Anything else?
A builtin is just a command that really isn’t one. So there is a /bin/echo somewhere, but when you just write “echo” in the shell, it will simply output the string itself. That is a lot faster than forking a new process. That is all.
And this is all designed to work on the shell. We need something for the web UI.
For commands within the WegGUI there are `safe_system` commands on the `.cgi` page.
But I am guess you are referring to something else.
That was mainly for the metrics… We would want those to be shown on the web UI and the tool that you currently have outputs it on the shell. That is what that meant.
What it would need to be finally merged would be a web UI though. So I think we have a long way ahead of us.
The attachment is an early mock-up I made a few months ago (made with Pixelmator Pro). The WebGUI is being worked on now.
<mock-up.png>
I think we might look for something that is very similar to the IPS ruleset editor or the IP blocklists.
I am not even sure if we would require users to manage this or whether we would curate a list like with the IPS and IP blocklists.
The first "DRAFT" of this was for the user/admin to manage the list. For the RPZ add-on, I’d prefer to keep it this way until we figure out if RPZ is popular and used by the user/admins.
And yes, it could be changed to a set list like IPS. FYI - there are 34 different Hagezi list. So it will take some experiments to pick a "the" list.
Well it doesn’t have to be just one. People might want to block different things…
It just seems that they come as a package “Light”, “Normal”, “Pro”, “Pro++” and it is either that or nothing. In URL Filter you would choose the categories more freely I would say.
-Michael
Also, here is current DRAFT (first draft) of user/admin managed WebGUI. See attachment.
<Screenshot 2024-08-11 at 2.07.19 PM copy.png>
Jon
Does this help? Jon
-Michael
I saw your comments in the Dev Mailing List of "generally being in favor of trying this path" (bad paraphrasing on my part)
I saw your comments in bugzilla at https://bugzilla.ipfire.org/show_bug.cgi?id=13254#c171
I am not interested in anything regarding the RPZs right now. They have not been properly put on the agenda and looking at how much time we have on our hands, this won't make it on the agenda for years.
I don't want to build blockers, but this ticket is about a different problem which I want to solve first.
How do you want to go forward?
Jon
On Aug 12, 2024, at 2:11 PM, jon jon.murphy@ipfire.org wrote:
More questions!
Currently RPZ config files are at `/etc/unbound/local.d` but this directory seems like it is for user (admin) customizations.
[root@ipfire ~] # ls -al /etc/unbound/local.d total 68 drwxr-xr-x 2 nobody nobody 4096 Aug 12 13:41 . drwxr-xr-x 4 root root 4096 Aug 12 00:52 .. -rw-r--r-- 1 nobody nobody 436 Jul 12 15:45 00-rpz.conf -rw-r--r-- 1 nobody nobody 285 Mar 1 22:12 AmazonTrkrHZ.rpz.conf -rw-r--r-- 1 nobody nobody 281 Mar 1 22:02 AppleTrkrHZ.rpz.conf -rw-r--r-- 1 nobody nobody 269 Mar 1 21:40 DOHblockHZ.rpz.conf ... -rw-r--r-- 1 nobody nobody 299 Aug 1 19:42 WinTrkrHZ.rpz.conf [root@ipfire ~] #Each file is a config file per category (or one per RPZ file). This makes it easy to add or remove a category (or RPZ file).
Should I create a new unbound directory for RPZ config files? Maybe `/etc/unbound/rpz.d`? Or `/etc/unbound/rpz`?
Jon
> On Aug 1, 2024, at 1:45 PM, Jon Murphy jon.murphy@ipfire.org wrote: > > changed all paths from `/var/ipfire/rpz/` to `/var/ipfire/dns/rpz/` > (thank you to Adolf!) > > rpz-config: > - bug: corrected "Type" test from block to allow > - removed verbose parameter from various commands > > rpz-metrics: > - bug: corrected grep for rpz name count > - bug: fixed divide by zero error (thank you Peppe!) > > install/uninstall: > - bug: corrected scripts (thank you Bernhard!) > > Signed-off-by: Jon Murphy jon.murphy@ipfire.org > --- > config/backup/includes/rpz | 4 ++-- > config/rootfiles/packages/rpz | 6 +++--- > config/rpz/rpz-config | 14 +++++++------- > config/rpz/rpz-metrics | 9 +++++---- > lfs/rpz | 6 +++--- > src/paks/rpz/install.sh | 27 +++++++++++++++++++++++++++ > src/paks/rpz/uninstall.sh | 31 +++++++++++++++++++++++++++++++ > src/paks/rpz/update.sh | 25 +++++++++++++++++++++++++ > 8 files changed, 103 insertions(+), 19 deletions(-) > create mode 100644 src/paks/rpz/install.sh > create mode 100644 src/paks/rpz/uninstall.sh > create mode 100644 src/paks/rpz/update.sh > > diff --git a/config/backup/includes/rpz b/config/backup/includes/rpz > index 4d59bb40c..8c7410ebd 100644 > --- a/config/backup/includes/rpz > +++ b/config/backup/includes/rpz > @@ -1,5 +1,5 @@ > -/var/ipfire/rpz/allowlist > -/var/ipfire/rpz/blocklist > +/var/ipfire/dns/rpz/allowlist > +/var/ipfire/dns/rpz/blocklist > /etc/unbound/zonefiles/allow.rpz > /etc/unbound/zonefiles/block.rpz > /etc/unbound/local.d/*rpz.conf > diff --git a/config/rootfiles/packages/rpz b/config/rootfiles/packages/rpz > index 2ffa715dd..183825362 100644 > --- a/config/rootfiles/packages/rpz > +++ b/config/rootfiles/packages/rpz > @@ -6,6 +6,6 @@ usr/sbin/rpz-config > usr/sbin/rpz-metrics > usr/sbin/rpz-sleep > var/ipfire/backup/addons/includes/rpz > -var/ipfire/rpz > -var/ipfire/rpz/allowlist > -var/ipfire/rpz/blocklist > +var/ipfire/dns/rpz > +var/ipfire/dns/rpz/allowlist > +var/ipfire/dns/rpz/blocklist > diff --git a/config/rpz/rpz-config b/config/rpz/rpz-config > index 98dc0a4ca..a24a5c132 100644 > --- a/config/rpz/rpz-config > +++ b/config/rpz/rpz-config > @@ -19,7 +19,7 @@ > # # > ############################################################################### > > -# v22 - 2024-07-12 > +# v23 - 2024-07-30 > > ############### Functions ############### > > @@ -54,11 +54,11 @@ check_unbound_conf () { > make_rpz_file () { > local theType="${1}" # allow or block > > - theList="/var/ipfire/rpz/${theType}list" # input user list of domains > + theList="/var/ipfire/dns/rpz/${theType}list" # input custom list of domains > theZoneFile="/etc/unbound/zonefiles/${theType}.rpz" # output file for RPZ > > theAction='.' > - if [[ "${theType}" =~ "block" ]] ; then > + if [[ "${theType}" =~ "allow" ]] ; then > theAction='rpz-passthru.' > fi > > @@ -131,8 +131,8 @@ case "${theAction}" in > # set-up zone file > /usr/bin/touch "${rpzFile}" > # unbound requires these settings for rpz files > - /bin/chown --verbose nobody:nobody "${rpzFile}" > - /bin/chmod --verbose 644 "${rpzFile}" > + /bin/chown nobody:nobody "${rpzFile}" > + /bin/chmod 644 "${rpzFile}" > ;; > > # trash config file & rpz file > @@ -143,8 +143,8 @@ case "${theAction}" in > fi > > msg_log "info: rpz: remove config file & rpz file "${theName}"" > - /bin/rm --verbose "${rpzConfig}" > - /bin/rm --verbose "${rpzFile}" > + /bin/rm "${rpzConfig}" > + /bin/rm "${rpzFile}" > > check_unbound_conf > ;; > diff --git a/config/rpz/rpz-metrics b/config/rpz/rpz-metrics > index 0f97c7911..4d932726e 100644 > --- a/config/rpz/rpz-metrics > +++ b/config/rpz/rpz-metrics > @@ -19,7 +19,7 @@ > # # > ############################################################################### > > -# v18 on 2024-07-05 > +# v19 on 2024-07-30 > > ############### Main ############### > > @@ -33,7 +33,7 @@ messageLogs=$( find /var/log/messages* -type f | > > # get the list of RPZ names & counts from the message log(s) > rpzNameCount=$( for logf in ${messageLogs} ; do > - /usr/bin/zgrep --text --fixed-strings 'info: rpz: applied' "${logf}" | > + /usr/bin/zgrep --text --extended-regexp 'info: rpz: applied.* A IN$' "${logf}" | > /usr/bin/awk '$10 ~ /[\w*]/ { print $10 }' ; > done | /usr/bin/sort | /usr/bin/uniq --count ) > > @@ -107,8 +107,9 @@ do > theLines=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' ) > totalLines=$(( totalLines + theLines )) > > - #hitsPerLine=$( echo "scale=0 ; $theHits / $theLines" | bc ) > - hitsPerLine=$(( 100 * theHits / theLines )) > + if [[ "${theLines}" -gt 2 ]] ; then > + hitsPerLine=$(( 100 * theHits / theLines )) > + fi > fi > > # get modification date > diff --git a/lfs/rpz b/lfs/rpz > index 319c10b7f..73f6f2b1b 100644 > --- a/lfs/rpz > +++ b/lfs/rpz > @@ -67,9 +67,9 @@ $(TARGET) : > $(DIR_CONF)/rpz/{rpz-config,rpz-metrics,rpz-sleep} -t /usr/sbin > > # Install settings folder and two empty files > - mkdir -pv /var/ipfire/rpz > - touch /var/ipfire/rpz/allowlist > - touch /var/ipfire/rpz/blocklist > + mkdir -pv /var/ipfire/dns/rpz > + touch /var/ipfire/dns/rpz/allowlist > + touch /var/ipfire/dns/rpz/blocklist > > # Add conf file to /etc directory > cp -vf $(DIR_CONF)/rpz/00-rpz.conf /etc/unbound/local.d > diff --git a/src/paks/rpz/install.sh b/src/paks/rpz/install.sh > new file mode 100644 > index 000000000..0a797e158 > --- /dev/null > +++ b/src/paks/rpz/install.sh > @@ -0,0 +1,27 @@ > +#!/bin/bash > +############################################################################### > +# # > +# IPFire.org - A linux based firewall # > +# Copyright (C) 2024 IPFire Team info@ipfire.org # > +# # > +# This program is free software: you can redistribute it and/or modify # > +# it under the terms of the GNU General Public License as published by # > +# the Free Software Foundation, either version 3 of the License, or # > +# (at your option) any later version. # > +# # > +# This program is distributed in the hope that it will be useful, # > +# but WITHOUT ANY WARRANTY; without even the implied warranty of # > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # > +# GNU General Public License for more details. # > +# # > +# You should have received a copy of the GNU General Public License # > +# along with this program. If not, see http://www.gnu.org/licenses/. # > +# # > +############################################################################### > +# > +. /opt/pakfire/lib/functions.sh > +extract_files > +restore_backup ${NAME} > + > +# restart unbound to load config file > +/etc/init.d/unbound restart > diff --git a/src/paks/rpz/uninstall.sh b/src/paks/rpz/uninstall.sh > new file mode 100644 > index 000000000..4fb20e127 > --- /dev/null > +++ b/src/paks/rpz/uninstall.sh > @@ -0,0 +1,31 @@ > +#!/bin/bash > +############################################################################### > +# # > +# IPFire.org - A linux based firewall # > +# Copyright (C) 2024 IPFire Team info@ipfire.org # > +# # > +# This program is free software: you can redistribute it and/or modify # > +# it under the terms of the GNU General Public License as published by # > +# the Free Software Foundation, either version 3 of the License, or # > +# (at your option) any later version. # > +# # > +# This program is distributed in the hope that it will be useful, # > +# but WITHOUT ANY WARRANTY; without even the implied warranty of # > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # > +# GNU General Public License for more details. # > +# # > +# You should have received a copy of the GNU General Public License # > +# along with this program. If not, see http://www.gnu.org/licenses/. # > +# # > +############################################################################### > +# > +. /opt/pakfire/lib/functions.sh > + > +# stop unbound to delete RPZ conf file > +/etc/init.d/unbound stop > + > +make_backup ${NAME} > +remove_files > + > +# start unbound to load unbound config file > +/etc/init.d/unbound start > diff --git a/src/paks/rpz/update.sh b/src/paks/rpz/update.sh > new file mode 100644 > index 000000000..938a93a40 > --- /dev/null > +++ b/src/paks/rpz/update.sh > @@ -0,0 +1,25 @@ > +#!/bin/bash > +############################################################################### > +# # > +# IPFire.org - A linux based firewall # > +# Copyright (C) 2024 IPFire Team info@ipfire.org # > +# # > +# This program is free software: you can redistribute it and/or modify # > +# it under the terms of the GNU General Public License as published by # > +# the Free Software Foundation, either version 3 of the License, or # > +# (at your option) any later version. # > +# # > +# This program is distributed in the hope that it will be useful, # > +# but WITHOUT ANY WARRANTY; without even the implied warranty of # > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # > +# GNU General Public License for more details. # > +# # > +# You should have received a copy of the GNU General Public License # > +# along with this program. If not, see http://www.gnu.org/licenses/. # > +# # > +############################################################################### > +# > +. /opt/pakfire/lib/functions.sh > +extract_backup_includes > +./uninstall.sh > +./install.sh > -- > 2.30.2