From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail02.haj.ipfire.org (localhost [IPv6:::1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4dgcwp5Jc3z333F for ; Tue, 30 Dec 2025 15:50:02 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [IPv6:2001:678:b28::25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (Client CN "mail01.haj.ipfire.org", Issuer "R12" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4dgcwl1D9cz2xRj for ; Tue, 30 Dec 2025 15:49:59 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4dgcwh6tG8z1N3; Tue, 30 Dec 2025 15:49:56 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1767109797; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2OaVh/ixVVT50gALZVJNl9/gHbyU7FpftvALY4RmGe8=; b=PBl8Pka/5YkF6/RDEsCaWALAZXD9t0YZfpWQ9ffLQDZDKbQiwLaQXFf2sJ6S7vHczNjQis tVBZalQkAv9wObBA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1767109797; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2OaVh/ixVVT50gALZVJNl9/gHbyU7FpftvALY4RmGe8=; b=HT0MjrDKziLyP+LFfUV0jsBeWFHJn6sfQACH+aDrh1Kz5M3nOEaAX9bkrcv97h00+JxOFM qyu+lrfBpsPtmUdV0CLqflrBy3FOFtcVl7MoRhn0Oj4EyozY/FyoITmxtlTiRU6KIfEeIL TRDokkO7QA0uZ3lA2YrXW0HgI4f0y6Ur2tORsgxyBXNU27xogtFKXMubacfmDCCCoOgIH8 uq24T/sVtkgoRXyowzZdd5aSeAg+oAPm/SqYiUyZuknk/YUAxEoj/zarsVIn5u6twlU7N1 AI6mCzTtxO2E8opMUa+lL8uayXpEavT75eMf+l7IuLSc4rdMzsudfOdrITHUTw== From: "Jon Murphy" To: "Adolf Belka" , "Michael Tremer" Subject: Re[2]: Let's launch our own blocklists... Cc: "IPFire: Development-List" Date: Tue, 30 Dec 2025 15:49:53 +0000 Message-Id: In-Reply-To: References: <7EF00B55-81C0-493F-A70F-B1DDD45363E2@ipfire.org> Reply-To: "Jon Murphy" Precedence: list List-Id: List-Subscribe: , List-Unsubscribe: , List-Post: List-Help: Sender: Mail-Followup-To: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable ------ Original Message ------ >From "Adolf Belka" To "Michael Tremer" Cc "IPFire: Development-List" Date 12/30/2025 8:05:20=E2=80=AFAM Subject Re: Let's launch our own blocklists... >Hi Michael, > >On 29/12/2025 13:05, Michael Tremer wrote: >>Hello everyone, >> >>I hope everyone had a great Christmas and a couple of quiet days to relax = from all the stress that was the year 2025. >Still relaxing. >>Having a couple of quieter days, I have been working on a new, little (ho= pefully) side project that has probably been high up on our radar since the = Shalla list has shut down in 2020, or maybe even earlier. The goal of the= project is to provide good lists with categories of domain names which are= usually used to block access to these domains. >> >>I simply call this IPFire DNSBL which is short for IPFire DNS Blocklists. >> >>How did we get here? >> >>As stated before, the URL filter feature in IPFire has the problem that t= here are not many good blocklists available any more. There used to be a co= uple more - most famously the Shalla list - but we are now down to a single = list from the University of Toulouse. It is a great list, but it is not al= ways the best fit for all users. >> >>Then there has been talk about whether we could implement more blocking f= eatures into IPFire that don=E2=80=99t involve the proxy. Most famously blo= cking over DNS. The problem here remains a the blocking feature is only as= good as the data that is fed into it. Some people have been putting forward = a number of lists that were suitable for them, but they would not have rep= laced the blocking functionality as we know it. Their aim is to provide =E2= =80=9Cone list for everything=E2=80=9D but that is not what people usually= want. It is targeted at a classic home user and the only separation that is = being made is any adult/porn/NSFW content which usually is put into a sepa= rate list. >> >>It would have been technically possible to include these lists and let th= e users decide, but that is not the aim of IPFire. We want to do the job fo= r the user so that their job is getting easier. Including obscure lists tha= t don=E2=80=99t have a clear outline of what they actually want to block (= =E2=80=9Cbad content=E2=80=9D is not a category) and passing the burden of= figuring out whether they need the =E2=80=9CLight=E2=80=9D, =E2=80=9CNormal= =E2=80=9D, =E2=80=9CPro=E2=80=9D, =E2=80=9CPro++=E2=80=9D, =E2=80=9CUltimat= e=E2=80=9D or even a =E2=80=9CVenti=E2=80=9D list with cream on top is real= ly not going to work. It is all confusing and will lead to a bad user exper= ience. >> >>An even bigger problem that is however completely impossible to solve is= bad licensing of these lists. A user has asked the publisher of the HaGeZi= list whether they could be included in IPFire and under what terms. The res= ponse was that the list is available under the terms of the GNU General Pub= lic License v3, but that does not seem to be true. The list contains data f= rom various sources. Many of them are licensed under incompatible licenses= (CC BY-SA 4.0, MPL, Apache2, =E2=80=A6) and unless there is a non-public ag= reement that this data may be redistributed, there is a huge legal issue he= re. We would expose our users to potential copyright infringement which we= cannot do under any circumstances. Furthermore many lists are available und= er a non-commercial license which excludes them from being used in any kind = of business. Plenty of IPFire systems are running in businesses, if not ev= en the vast majority. >> >>In short, these lists are completely unusable for us. Apart from HaGeZi,= I consider OISD to have the same problem. >> >>Enough about all the things that are bad. Let=E2=80=99s talk about the ne= w, good things: >> >>Many blacklists on the internet are an amalgamation of other lists. These = lists vary in quality with some of them being not that good and without a= clear focus and others being excellent data. Since we don=E2=80=99t have th= e man power to start from scratch, I felt that we can copy the concept that = HaGeZi and OISD have started and simply create a new list that is based on = other lists at the beginning to have a good starting point. That way, we h= ave much better control over what is going on these lists and we can shape= and mould them as we need them. Most importantly, we don=E2=80=99t create a = single lists, but many lists that have a clear focus and allow users to ch= oose what they want to block and what not. >> >>So the current experimental stage that I am in has these lists: >> >> * Ads >> * Dating >> * DoH >> * Gambling >> * Malware >> * Porn >> * Social >> * Violence >> >>The categories have been determined by what source lists we have availabl= e with good data and are compatible with our chosen license CC BY-SA 4.0. T= his is the same license that we are using for the IPFire Location database, = too. >> >>The main use-cases for any kind of blocking are to comply with legal requ= irements in networks with children (i.e. schools) to remove any kind of por= nographic content, sometimes block social media as well. Gambling and viole= nce are commonly blocked, too. Even more common would be filtering advertis= ing and any malicious content. >> >>The latter is especially difficult because so many source lists throw phi= shing, spyware, malvertising, tracking and other things into the same bucke= t. Here this is currently all in the malware list which has therefore becom= e quite large. I am not sure whether this will stay like this in the future = or if we will have to make some adjustments, but that is exactly why this= is now entering some larger testing. >> >>What has been built so far? In order to put these lists together properly= , track any data about where it is coming from, I have built a tool in Pyth= on available here: >> >> https://git.ipfire.org/?p=3Ddnsbl.git;a=3Dsummary >> >>This tool will automatically update all lists once an hour if there have= been any changes and export them in various formats. The exported lists are = available for download here: >> >> https://dnsbl.ipfire.org/lists/ >The download using dnsbl.ipfire.org/lists/squidguard.tar.gz as the custom= url works fine. > >However you need to remember not to put the https:// at the front of the u= rl otherwise the WUI page completes without any error messages but leaves a= n error message in the system logs saying > >URL filter blacklist - ERROR: Not a valid URL filter blacklist > >I found this out the hard way. > >The other thing I noticed is that if you already have the Toulouse Univers= ity list downloaded and you then change to the ipfire custom url then all t= he existing Toulouse blocklists stay in the directory on IPFire and so you= end up with a huge number of category tick boxes, most of which are the old = Toulouse ones, which are still available to select and it is not clear whi= ch ones are from Toulouse and which ones from IPFire. > >I think if the blocklist URL source is changed or a custom url is provided = the first step should be to remove the old ones already existing. >That might be a problem because users can also create their own blocklists = and I believe those go into the same directory. > >Without clearing out the old blocklists you end up with a huge number of c= heckboxes for lists but it is not clear what happens if there is a category = that has the same name for the Toulouse list and the IPFire list such as g= ambling. I will have a look at that and see what happens. > >Not sure what the best approach to this is. To find the older files I used this: find /var/ipfire/urlfilter/blacklists -mtime +365 -type f -ls To delete the older files and folders I did this: find /var/ipfire/urlfilter/blacklists -mtime +365 -type f -delete -o= =20 -type d -empty -delete > > >Manually deleting all contents of the urlfilter/blacklists/ directory and= then selecting the IPFire blocklist url for the custom url I end up with on= ly the 8 categories from the IPFire list. > >I have tested some gambling sites from the IPFire list and the block worke= d on some. On others the site no longer exists so there is nothing to block = or has been changed to an https site and in that case it went straight thr= ough. Also if I chose the http version of the link, it was automatically ch= anged to https and went through without being blocked. > > >>If you download and open any of the files, you will see a large header th= at includes copyright information and lists all sources that have been used = to create the individual lists. This way we ensure maximum transparency, c= omply with the terms of the individual licenses of the source lists and giv= e credit to the people who help us to put together the most perfect list fo= r our users. >> >>I would like this to become a project that is not only being used in IPFi= re. We can and will be compatible with other solutions like AdGuard, PiHole = so that people can use our lists if they would like to even though they ar= e not using IPFire. Hopefully, these users will also feed back to us so tha= t we can improve our lists over time and make them one of the best options= out there. >> >>All lists are available as a simple text file that lists the domains. The= n there is a hosts file available as well as a DNS zone file and an RPZ fil= e. Each list is individually available to be used in squidGuard and there i= s a larger tarball available with all lists that can be used in IPFire=E2= =80=99s URL Filter. > I tested the URL Filter with the change to the autoupdate.urls. For me=20 it only picked up one URL but I think my Web Proxy is configured wrong. =E2=80=A2 Is the non-Transparent needed to utilize the IPFire DNSBL (a.k.= a.=20 IPFire DNS Blocklists)?? =E2=80=A2 Does Web Proxy Auto-Discovery Protocol (WPAD) need to be setup? I ask because I disabled the Web Proxy once Shalla Services stopped a=20 few years ago. I think the answer is yes to get HTTPS sites recognized. > >>I am planning to add Suricata/Snort signatures whenever I have time to do = so. Even though it is not a good idea to filter pornographic content this= way, I suppose that catching malware and blocking DoH are good use-cases fo= r an IPS. Time will tell=E2=80=A6 >> >>As a start, we will make these lists available in IPFire=E2=80=99s URL Fi= lter and collect some feedback about how we are doing. Afterwards, we can s= ee where else we can take this project. >> >>If you want to enable this on your system, simply add the URL to your aut= oupdate.urls file like here: >> >> https://git.ipfire.org/?p=3Dpeople/ms/ipfire-2.x.git;a=3Dcommitdiff;h= =3Dbf675bb937faa7617474b3cc84435af3b1f7f45f >I also tested out adding the IPFire url to autoupdate.urls and that also w= orked fine for me. > >Regards, >Adolf. >>This email is just a brain dump from me to this list. I would be happy to = answer any questions about implementation details, etc. if people are inte= rested. Right now, this email is long enough already=E2=80=A6 >> >>All the best, >>-Michael > >-- Sent from my laptop > > >