Morning,
On 22 Aug 2024, at 16:37, jon jon.murphy@ipfire.org wrote:
Hi Michael,
On Aug 21, 2024, at 5:03 AM, Michael Tremer michael.tremer@ipfire.org wrote:
Hello Jon,
On 15 Aug 2024, at 23:22, jon jon.murphy@ipfire.org wrote:
Comments below…
On Aug 15, 2024, at 1:33 PM, Michael Tremer michael.tremer@ipfire.org wrote:
Hello,
On 15 Aug 2024, at 17:57, jon jon.murphy@ipfire.org wrote:
Comments below.
Jon
On Aug 15, 2024, at 10:33 AM, Michael Tremer michael.tremer@ipfire.org wrote:
Hello,
> On 14 Aug 2024, at 18:14, jon jon.murphy@ipfire.org wrote: > > Michael, > > Sorry for putting you on the spot, but what do you want to do with this RPZ add-on?
I am not sure this is a question for only me.
I personally just don’t have any capacity to take on another rather large project as I have a huge backlog of so many things and I feel like a lonely fighter getting these all over the line. For my own sanity I need to have a couple of those closed before thinking about the next ones.
A couple of months ago we have talked on the monthly call about RPZ and the decision that was made by the people was to not look at this now, just because there were other more pressing things. We could also not find answers to the questions that led us into RPZ:
- Are there any good lists out there that would allow us to replace the URL Filter? That thing is basically on its last leg because of the lack of lists. We need to be able to block ads and pr0n and that very reliably.
I believe this exists with the Hagezi lists at https://github.com/hagezi/dns-blocklists and/or the addition of a chosen "oisd NSFW" list at https://oisd.nl/setup
I understand there is no time to review to see if this meet the needs. So we can leave this as "I believe these exist" for now.
So, I am not sure what to think of all these things on GitHub. It seems that there are many repositories that pop up and presumably disappear in the same kind of way. We see this with the IP blocklists.
We want quality blocklists. I know that we don’t have too much on the URL Filter front, but I would prefer to avoid putting so much work in if we in the end stand where we started.
This makes perfect sense. Based on the comments I’ve seen it seems like Hagezi is growing in popularity and in offerings.
Gerd (the owner) is very responsive, involved and I keep running into his posts when searching for RPZ "things". Gerd quickly made changes per my request: see https://github.com/hagezi/dns-blocklists/issues/2301. Now there are two TLD RPZ lists.
Like everything else I cannot predict how long it might last.
If you have such a good connection, could you do me the favour and ask why the prefer distributing those lists using HTTPS and not DNS?
I’d be happy to ask Gerd, but I do not understand what you want me to ask.
It sounded like you have a personal connection. That is all.
I guessed it is related to this: https://www.ipfire.org/blog/ipfire-location-a-decentralised-signed-database-... https://www.ipfire.org/location/how-to-use/dns
I understand the APIs you had added, but I do not understand how this related to the DNS blocked lists.
Yes, this is slightly related to this. I am just running a network of authoritative DNS servers scattered a little bit across the world. It is an essential part of the IPFire infrastructure but also hosts a couple of other things. And since it is there, it is a nice thing to bolt on more things.
What might the query do? Look up a requested domain?
The relationship to RPZs is that we could use that service to host them. It is a DNS-based technology after all.
Is it simply that GitHub is free and does not offer DNS? :)
I did not find GitHub DNS server(s) listed anywhere. Just lots of references on setting up DNS on other servers...
No, GitHub does not offer any DNS services.
Let me know what you want asked, specifically, and I will pass it on to Gerd!
The question is only whether it was a conscious decision to distribute the lists over HTTP instead of using DNS. Maybe it was just chosen because that is what was available to them.
- We need to look into privacy when RPZs are being realised over DNS - I don’t even understand why we are suddenly starting to pull text files over HTTP again. The IPS seems to have most of these lists already.
All of the lists that I am pull are HTTPS. And HTTPS only is currently checked (validated) in the existing `rpz-config` code.
What a lost case. DNS is a globally distributed, cacheable protocol. We could simply put all those blacklists into the DNS caches all around the world. They would respond fast, be local to so many people, but instead we download massive text files. That means that whenever something changes the entire file needs to be transferred again. They are huge. We would want many fast updates which are not possible that way… but I digress.
I came across one DNS service that offers the Hagezi lists as part of their available DNS services. If interested I can find and post the details.
I run a global DNS system where we could easily add this all to. This would scale like hell. It would be a fun project.
Just for fun, I have imported "jpgpi250.github.io http://jpgpi250.github.io/” which you can pull from either dfw.lwldns.nethttp://dfw.lwldns.net/ or haj.lwldns.net http://haj.lwldns.net/. This would be so much better since DNS zone transfer checks the serial of the zone and does not retransfer it if it has not changed (and I suppose none of these lists change that frequently).
Peter’s jpgpi250 DoH rpz list updates one per day near 04:00 CEST (02:00 UTC). A handful of adds or deletes every night.
Gerd’s block DoH rpz list updates twice per day as needed. Sometime once per day, sometimes twice per day.
Yeah, that is not a lot, so DNS would qualify to distribute the lists.
Sadly our DNS system does not support this,
I do not know what "global DNS system" is support. Is it not unbound based?
No those servers are running PowerDNS behind a load-balancer. Unbound is just a recursor and not an authoritative DNS server.
There are many different formats to feed to DNS system: https://github.com/hagezi/dns-blocklists?tab=readme-ov-file#outbox_tray-encr...
Maybe one of these might help?
but there is also IXFR for incremental zone updates where you would only load the changes.
How much traffic did you observe?
I have 11 RPZ list (honestly too many but I am experimenting) and that eat up about 230M bytes per day.
Oh wow that is huuuuuge. I assume that is repeat downloads? How large are the zone files just downloaded once?
Here is the size of each file (grapped via `ls -lS /etc/unbound/zonefiles` and snipped):
10,655,321 MxProPlusHZ.rpz 90,909 dohJPG.rpz 76,591 HosterHZ.rpz 72,628 DOHblockHZ.rpz 37,411 urlhaus.rpz 27,863 WinTrkrHZ.rpz 12,333 NotSafeSearchHZ.rpz 7,573 AppleTrkrHZ1.rpz 7,308 tldAggHZ.rpz 2,631 tldHZ.rpz 966 allow.rpz 238 block.rpz
How is Unbound refreshing these?
Unbound RPZ takes care of the updates automagically. There is no cron entry.
Each RPZ list includes an SOA line similar to this: `@ SOA localhost. root.localhost. 1723093380 43200 3600 259200 300`
The 43200 is the refresh period in seconds. So this RPZ list auto updates every 12 hours.
Is there a local cache?
Yes, it is part of unbound. (Nothing we need to setup)
Would it use the cache for zones transferred over DNS, too?
I do not know.
Regarding your code, there are some issues with the coding style, but I heavily appreciate the pioneering to bring this feature to life.
My style is camelCase, but if you prefer snake_case that is an easy change (please speak up).
As for as other style changes, that can wait until time is available by the core developers.
It is mainly that you make things very complicated for yourself. Call “cat”, not “/bin/cat”. There is no guarantee that things will stay where they are now and the shell has lots of builtins that execute a lot faster.
I will remove the executable paths. What builtin did you see that I missed? Anything else?
A builtin is just a command that really isn’t one. So there is a /bin/echo somewhere, but when you just write “echo” in the shell, it will simply output the string itself. That is a lot faster than forking a new process. That is all.
Got it! All changed to:
echo "multi line string" > "${rpzConfig}"
And this is all designed to work on the shell. We need something for the web UI.
For commands within the WegGUI there are `safe_system` commands on the `.cgi` page.
But I am guess you are referring to something else.
That was mainly for the metrics… We would want those to be shown on the web UI and the tool that you currently have outputs it on the shell. That is what that meant.
For short term the metrics will be via shell only.
Long term, agree, metrics via the WebGUI!
What it would need to be finally merged would be a web UI though. So I think we have a long way ahead of us.
The attachment is an early mock-up I made a few months ago (made with Pixelmator Pro). The WebGUI is being worked on now.
<mock-up.png>
I think we might look for something that is very similar to the IPS ruleset editor or the IP blocklists.
I am not even sure if we would require users to manage this or whether we would curate a list like with the IPS and IP blocklists.
The first "DRAFT" of this was for the user/admin to manage the list. For the RPZ add-on, I’d prefer to keep it this way until we figure out if RPZ is popular and used by the user/admins.
And yes, it could be changed to a set list like IPS. FYI - there are 34 different Hagezi list. So it will take some experiments to pick a "the" list.
Well it doesn’t have to be just one. People might want to block different things…
Agree!
It just seems that they come as a package “Light”, “Normal”, “Pro”, “Pro++” and it is either that or nothing.
More than nothing - Here are the individual (non-package) lists for Hagezi:
• Fake - Protects against internet scams, traps & fakes! • Pop-Up Ads - Protects against annoying and malicious pop-up ads! • Threat Intelligence Feeds - Increases security significantly! (Recommended) : Full - Medium - Mini - IPs • Newly Registered Domains - Favoured by threat actors to launch malicious campaigns! : 14 days - 30 days • DoH/VPN/TOR/Proxy Bypass - Prevent methods to bypass your DNS! : Full - DoH only - DoH IPs • Safesearch not supported - Prevent the use of search engines that do not support Safesearch! • Dynamic DNS - Protects against the malicious use of dynamic DNS services! • Badware Hoster - Protects against the malicious use of free host services! • Most Abused TLDs - Protects against known malicious Top Level Domains! • Anti Piracy - Protects against piracy! • Gambling - Protects against gambling content! : Full - Medium - Mini • NSFW (external) - oisd NSFW - Protects against adult content! • Native Tracker - Broadband tracker of devices, services and operating systems
And there are others out there also.
Others might exist, but as it seems they all mix and match their lists together. So there will a lot of overlap which becomes a waste of memory and data transfer.
Looking at that list, I am not sure what I would want to turn on in my home/if I was a school/a generic office company.
Do I have your blessing to keep moving forward?
What do you want to move forward with? What are the next steps?
-Michael
Jon
PS - Here is the current WebGUI for RPZ.
<Screenshot 2024-08-22 at 10.28.54 AM.png>
And the EDIT pencil.
<Screenshot 2024-08-22 at 10.29.25 AM.png>
-Michael
Also, here is current DRAFT (first draft) of user/admin managed WebGUI. See attachment.
<Screenshot 2024-08-11 at 2.07.19 PM copy.png>
Jon
Does this help? Jon
-Michael
> I saw your comments in the Dev Mailing List of "generally being in favor of trying this path" (bad paraphrasing on my part) > > I saw your comments in bugzilla at https://bugzilla.ipfire.org/show_bug.cgi?id=13254#c171 >> I am not interested in anything regarding the RPZs right now. They have not been properly put on the agenda and looking at how much time we have on our hands, this won't make it on the agenda for years. >> >> I don't want to build blockers, but this ticket is about a different problem which I want to solve first. > > > How do you want to go forward? > > > Jon > > >> On Aug 12, 2024, at 2:11 PM, jon jon.murphy@ipfire.org wrote: >> >> More questions! >> >> Currently RPZ config files are at `/etc/unbound/local.d` but this directory seems like it is for user (admin) customizations. >> >> ``` >> [root@ipfire ~] # ls -al /etc/unbound/local.d >> total 68 >> drwxr-xr-x 2 nobody nobody 4096 Aug 12 13:41 . >> drwxr-xr-x 4 root root 4096 Aug 12 00:52 .. >> -rw-r--r-- 1 nobody nobody 436 Jul 12 15:45 00-rpz.conf >> -rw-r--r-- 1 nobody nobody 285 Mar 1 22:12 AmazonTrkrHZ.rpz.conf >> -rw-r--r-- 1 nobody nobody 281 Mar 1 22:02 AppleTrkrHZ.rpz.conf >> -rw-r--r-- 1 nobody nobody 269 Mar 1 21:40 DOHblockHZ.rpz.conf >> ... >> -rw-r--r-- 1 nobody nobody 299 Aug 1 19:42 WinTrkrHZ.rpz.conf >> [root@ipfire ~] # >> ``` >> >> >> Each file is a config file per category (or one per RPZ file). This makes it easy to add or remove a category (or RPZ file). >> >> Should I create a new unbound directory for RPZ config files? Maybe `/etc/unbound/rpz.d`? Or `/etc/unbound/rpz`? >> >> >> Jon >> >> >>> On Aug 1, 2024, at 1:45 PM, Jon Murphy jon.murphy@ipfire.org wrote: >>> >>> changed all paths from `/var/ipfire/rpz/` to `/var/ipfire/dns/rpz/` >>> (thank you to Adolf!) >>> >>> rpz-config: >>> - bug: corrected "Type" test from block to allow >>> - removed verbose parameter from various commands >>> >>> rpz-metrics: >>> - bug: corrected grep for rpz name count >>> - bug: fixed divide by zero error (thank you Peppe!) >>> >>> install/uninstall: >>> - bug: corrected scripts (thank you Bernhard!) >>> >>> Signed-off-by: Jon Murphy jon.murphy@ipfire.org >>> --- >>> config/backup/includes/rpz | 4 ++-- >>> config/rootfiles/packages/rpz | 6 +++--- >>> config/rpz/rpz-config | 14 +++++++------- >>> config/rpz/rpz-metrics | 9 +++++---- >>> lfs/rpz | 6 +++--- >>> src/paks/rpz/install.sh | 27 +++++++++++++++++++++++++++ >>> src/paks/rpz/uninstall.sh | 31 +++++++++++++++++++++++++++++++ >>> src/paks/rpz/update.sh | 25 +++++++++++++++++++++++++ >>> 8 files changed, 103 insertions(+), 19 deletions(-) >>> create mode 100644 src/paks/rpz/install.sh >>> create mode 100644 src/paks/rpz/uninstall.sh >>> create mode 100644 src/paks/rpz/update.sh >>> >>> diff --git a/config/backup/includes/rpz b/config/backup/includes/rpz >>> index 4d59bb40c..8c7410ebd 100644 >>> --- a/config/backup/includes/rpz >>> +++ b/config/backup/includes/rpz >>> @@ -1,5 +1,5 @@ >>> -/var/ipfire/rpz/allowlist >>> -/var/ipfire/rpz/blocklist >>> +/var/ipfire/dns/rpz/allowlist >>> +/var/ipfire/dns/rpz/blocklist >>> /etc/unbound/zonefiles/allow.rpz >>> /etc/unbound/zonefiles/block.rpz >>> /etc/unbound/local.d/*rpz.conf >>> diff --git a/config/rootfiles/packages/rpz b/config/rootfiles/packages/rpz >>> index 2ffa715dd..183825362 100644 >>> --- a/config/rootfiles/packages/rpz >>> +++ b/config/rootfiles/packages/rpz >>> @@ -6,6 +6,6 @@ usr/sbin/rpz-config >>> usr/sbin/rpz-metrics >>> usr/sbin/rpz-sleep >>> var/ipfire/backup/addons/includes/rpz >>> -var/ipfire/rpz >>> -var/ipfire/rpz/allowlist >>> -var/ipfire/rpz/blocklist >>> +var/ipfire/dns/rpz >>> +var/ipfire/dns/rpz/allowlist >>> +var/ipfire/dns/rpz/blocklist >>> diff --git a/config/rpz/rpz-config b/config/rpz/rpz-config >>> index 98dc0a4ca..a24a5c132 100644 >>> --- a/config/rpz/rpz-config >>> +++ b/config/rpz/rpz-config >>> @@ -19,7 +19,7 @@ >>> # # >>> ############################################################################### >>> >>> -# v22 - 2024-07-12 >>> +# v23 - 2024-07-30 >>> >>> ############### Functions ############### >>> >>> @@ -54,11 +54,11 @@ check_unbound_conf () { >>> make_rpz_file () { >>> local theType="${1}" # allow or block >>> >>> - theList="/var/ipfire/rpz/${theType}list" # input user list of domains >>> + theList="/var/ipfire/dns/rpz/${theType}list" # input custom list of domains >>> theZoneFile="/etc/unbound/zonefiles/${theType}.rpz" # output file for RPZ >>> >>> theAction='.' >>> - if [[ "${theType}" =~ "block" ]] ; then >>> + if [[ "${theType}" =~ "allow" ]] ; then >>> theAction='rpz-passthru.' >>> fi >>> >>> @@ -131,8 +131,8 @@ case "${theAction}" in >>> # set-up zone file >>> /usr/bin/touch "${rpzFile}" >>> # unbound requires these settings for rpz files >>> - /bin/chown --verbose nobody:nobody "${rpzFile}" >>> - /bin/chmod --verbose 644 "${rpzFile}" >>> + /bin/chown nobody:nobody "${rpzFile}" >>> + /bin/chmod 644 "${rpzFile}" >>> ;; >>> >>> # trash config file & rpz file >>> @@ -143,8 +143,8 @@ case "${theAction}" in >>> fi >>> >>> msg_log "info: rpz: remove config file & rpz file "${theName}"" >>> - /bin/rm --verbose "${rpzConfig}" >>> - /bin/rm --verbose "${rpzFile}" >>> + /bin/rm "${rpzConfig}" >>> + /bin/rm "${rpzFile}" >>> >>> check_unbound_conf >>> ;; >>> diff --git a/config/rpz/rpz-metrics b/config/rpz/rpz-metrics >>> index 0f97c7911..4d932726e 100644 >>> --- a/config/rpz/rpz-metrics >>> +++ b/config/rpz/rpz-metrics >>> @@ -19,7 +19,7 @@ >>> # # >>> ############################################################################### >>> >>> -# v18 on 2024-07-05 >>> +# v19 on 2024-07-30 >>> >>> ############### Main ############### >>> >>> @@ -33,7 +33,7 @@ messageLogs=$( find /var/log/messages* -type f | >>> >>> # get the list of RPZ names & counts from the message log(s) >>> rpzNameCount=$( for logf in ${messageLogs} ; do >>> - /usr/bin/zgrep --text --fixed-strings 'info: rpz: applied' "${logf}" | >>> + /usr/bin/zgrep --text --extended-regexp 'info: rpz: applied.* A IN$' "${logf}" | >>> /usr/bin/awk '$10 ~ /[\w*]/ { print $10 }' ; >>> done | /usr/bin/sort | /usr/bin/uniq --count ) >>> >>> @@ -107,8 +107,9 @@ do >>> theLines=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' ) >>> totalLines=$(( totalLines + theLines )) >>> >>> - #hitsPerLine=$( echo "scale=0 ; $theHits / $theLines" | bc ) >>> - hitsPerLine=$(( 100 * theHits / theLines )) >>> + if [[ "${theLines}" -gt 2 ]] ; then >>> + hitsPerLine=$(( 100 * theHits / theLines )) >>> + fi >>> fi >>> >>> # get modification date >>> diff --git a/lfs/rpz b/lfs/rpz >>> index 319c10b7f..73f6f2b1b 100644 >>> --- a/lfs/rpz >>> +++ b/lfs/rpz >>> @@ -67,9 +67,9 @@ $(TARGET) : >>> $(DIR_CONF)/rpz/{rpz-config,rpz-metrics,rpz-sleep} -t /usr/sbin >>> >>> # Install settings folder and two empty files >>> - mkdir -pv /var/ipfire/rpz >>> - touch /var/ipfire/rpz/allowlist >>> - touch /var/ipfire/rpz/blocklist >>> + mkdir -pv /var/ipfire/dns/rpz >>> + touch /var/ipfire/dns/rpz/allowlist >>> + touch /var/ipfire/dns/rpz/blocklist >>> >>> # Add conf file to /etc directory >>> cp -vf $(DIR_CONF)/rpz/00-rpz.conf /etc/unbound/local.d >>> diff --git a/src/paks/rpz/install.sh b/src/paks/rpz/install.sh >>> new file mode 100644 >>> index 000000000..0a797e158 >>> --- /dev/null >>> +++ b/src/paks/rpz/install.sh >>> @@ -0,0 +1,27 @@ >>> +#!/bin/bash >>> +############################################################################### >>> +# # >>> +# IPFire.org - A linux based firewall # >>> +# Copyright (C) 2024 IPFire Team info@ipfire.org # >>> +# # >>> +# This program is free software: you can redistribute it and/or modify # >>> +# it under the terms of the GNU General Public License as published by # >>> +# the Free Software Foundation, either version 3 of the License, or # >>> +# (at your option) any later version. # >>> +# # >>> +# This program is distributed in the hope that it will be useful, # >>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >>> +# GNU General Public License for more details. # >>> +# # >>> +# You should have received a copy of the GNU General Public License # >>> +# along with this program. If not, see http://www.gnu.org/licenses/. # >>> +# # >>> +############################################################################### >>> +# >>> +. /opt/pakfire/lib/functions.sh >>> +extract_files >>> +restore_backup ${NAME} >>> + >>> +# restart unbound to load config file >>> +/etc/init.d/unbound restart >>> diff --git a/src/paks/rpz/uninstall.sh b/src/paks/rpz/uninstall.sh >>> new file mode 100644 >>> index 000000000..4fb20e127 >>> --- /dev/null >>> +++ b/src/paks/rpz/uninstall.sh >>> @@ -0,0 +1,31 @@ >>> +#!/bin/bash >>> +############################################################################### >>> +# # >>> +# IPFire.org - A linux based firewall # >>> +# Copyright (C) 2024 IPFire Team info@ipfire.org # >>> +# # >>> +# This program is free software: you can redistribute it and/or modify # >>> +# it under the terms of the GNU General Public License as published by # >>> +# the Free Software Foundation, either version 3 of the License, or # >>> +# (at your option) any later version. # >>> +# # >>> +# This program is distributed in the hope that it will be useful, # >>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >>> +# GNU General Public License for more details. # >>> +# # >>> +# You should have received a copy of the GNU General Public License # >>> +# along with this program. If not, see http://www.gnu.org/licenses/. # >>> +# # >>> +############################################################################### >>> +# >>> +. /opt/pakfire/lib/functions.sh >>> + >>> +# stop unbound to delete RPZ conf file >>> +/etc/init.d/unbound stop >>> + >>> +make_backup ${NAME} >>> +remove_files >>> + >>> +# start unbound to load unbound config file >>> +/etc/init.d/unbound start >>> diff --git a/src/paks/rpz/update.sh b/src/paks/rpz/update.sh >>> new file mode 100644 >>> index 000000000..938a93a40 >>> --- /dev/null >>> +++ b/src/paks/rpz/update.sh >>> @@ -0,0 +1,25 @@ >>> +#!/bin/bash >>> +############################################################################### >>> +# # >>> +# IPFire.org - A linux based firewall # >>> +# Copyright (C) 2024 IPFire Team info@ipfire.org # >>> +# # >>> +# This program is free software: you can redistribute it and/or modify # >>> +# it under the terms of the GNU General Public License as published by # >>> +# the Free Software Foundation, either version 3 of the License, or # >>> +# (at your option) any later version. # >>> +# # >>> +# This program is distributed in the hope that it will be useful, # >>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >>> +# GNU General Public License for more details. # >>> +# # >>> +# You should have received a copy of the GNU General Public License # >>> +# along with this program. If not, see http://www.gnu.org/licenses/. # >>> +# # >>> +############################################################################### >>> +# >>> +. /opt/pakfire/lib/functions.sh >>> +extract_backup_includes >>> +./uninstall.sh >>> +./install.sh >>> -- >>> 2.30.2
Jon
-- Jon Murphy jon.murphy@ipfire.org